Running Unit Tests in Parallel using VSTS

A lot of effort has been put into running UI/integration tests in parallel across multiple machines. This is a standard practice these days. You can achieve this with some paid services such as BrowserStack. Microsoft recently added inbuilt support for this in VSTS using the Visual Studio Test task (version 2).

I wanted to see if I could use the Visual Studio Test task to distribute unit test execution. In short, it is possible, but there are some caveats that I will explain in this post.

Why?

The first question worth asking is why would you want to do this? It makes perfect sense to distribute UI tests across agents. The main reason is that UI tests are generally slower to execute than Unit Test. Some UI test suites can take many hours to execute, so it makes a lot of sense to want to distribute that.

In my case, our test suite contains approximately 5000 tests. These tests execute on a standard VSTS Hosted Agent in between 10-15 mins. Whilst this isn't a huge issue, I want my CI builds to finish as fast as possible and 10-15 mins just for the testing phase is very significant. I would like to reduce the test execution to less than 5 mins.

The Approach

I'm going to take an iterative approach to this issue, showing each step I make and providing commentary on the decisions made. If you wish to jump to the final solution, just scroll to the bottom.

Baseline

The baseline is our starting point. There are a few points to outline at the start:

  1. We are using the YAML build system. This is required for one of the features we are going to leverage later on. It also allows us to make changes to the build on a branch without affecting the main build process.
  2. There are some other quirks to this build definition, such as the artifact compression and SonarCloud, but that is not relevant to this blog post.

Baseline Build Log

Ignore the errors/warnings - they are not relevant

Baseline Build Log

Baseline YAML File

queue:  
  name: Hosted VS2017
  demands: 
  - msbuild
  - visualstudio
  - vstest
  - java

variables:  
  Parameters.Solution: iSAMS.New/iSAMS.New.sln
  BuildConfiguration: 'release'
  BuildPlatform: 'any cpu'

name: $(BuildDefinitionName).$(Year:yy).$(Month).$(DayOfMonth)$(Rev:.r)-$(Build.SourceBranchName)

steps:  
- task: [email protected]
  displayName: Use Latest NuGet 4.X
  inputs:
    versionSpec: 4.x
    checkLatest: true

- task: [email protected]
  displayName: NuGet restore
  inputs:
    restoreSolution: '$(Parameters.solution)'
    feedsToUse: config
    nugetConfigPath: 'iSAMS.New/NuGet.Config'

- task: richardfennellBM.BM[email protected]1
  displayName: Version Assemblies

- task: SonarSource.so[email protected]1
  displayName: Prepare analysis on SonarCloud
  inputs:
    SonarCloud: SonarCloud
    organization: isams
    projectKey: isams.newframework
    projectName: iSAMS.NewFramework
    projectVersion: '$(Build.BuildNumber)'
    extraProperties: |
     # Additional properties that will be passed to the scanner, 
     # Put one key=value per line, example:
     sonar.exclusions=**/bower_components/**/*,**/node_modules/**/*,**/Scripts/lib/**/*
  condition: and(succeeded(), eq(variables['ExecuteSonarSource'], true))

- task: [email protected]
  displayName: Build solution
  inputs:
    solution: '$(Parameters.solution)'
    msbuildArgs: '/p:DeployOnBuild=true /p:PublishProfile=UpdateSystem /p:SkipInvalidConfigurations=true /p:publishBaseUrl="$(build.artifactstagingdirectory)\\"'
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'

- task: [email protected]
  displayName: VsTest Platform Installer
  enabled: false

- task: [email protected]
  displayName: Test Assemblies
  inputs:
    testAssemblyVer2: |
     **\$(BuildConfiguration)\*test*.dll
     !**\$(BuildConfiguration)\*IntegrationTests*.dll
     !**\obj\**
     !**\Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll
    runOnlyImpactedTests: false
    runInParallel: true
    runTestsInIsolation: false
    codeCoverageEnabled: true
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'

- task: SonarSource.so[email protected]1
  displayName: Run Code Analysis
  condition: and(succeeded(), eq(variables['ExecuteSonarSource'], true))

- task: SonarSource.so[email protected]1
  displayName: Publish Quality Gate Result
  condition: and(succeeded(), eq(variables['ExecuteSonarSource'], true))

- task: [email protected]
  displayName: Publish symbols path
  inputs:
    SearchPattern: '**\bin\**\*.pdb'
    PublishSymbols: false
  enabled: false
  continueOnError: true

- task: roshkovski.2B9619D5-7BE9-4ED7-BF10-707CB[email protected]1
  condition: and(succeeded(), eq(variables['PullRequest'], false))
  displayName: 'Compress iSAMS.Root'
  inputs:
    source: '$(Build.ArtifactStagingDirectory)\iSAMS.Root'
    dest: '$(Build.ArtifactStagingDirectory)\iSAMS.Root.zip'
    compressionLevel: NoCompression
    multiplePackages: false

- task: [email protected]
  condition: and(succeeded(), eq(variables['PullRequest'], false))
  displayName: 'Publish iSAMS.Root'
  inputs:
    PathtoPublish: '$(build.artifactstagingdirectory)\\iSAMS.Root.zip'
    ArtifactName: 'iSAMS.Root'
    ArtifactType: 'Container'

- task: roshkovski.2B9619D5-7BE9-4ED7-BF10-707CB[email protected]1
  condition: and(succeeded(), eq(variables['PullRequest'], false))
  displayName: 'Compress iSAMS.Web'
  inputs:
    source: '$(Build.ArtifactStagingDirectory)\iSAMS.Web'
    dest: '$(Build.ArtifactStagingDirectory)\iSAMS.Web.zip'
    compressionLevel: NoCompression
    multiplePackages: false

- task: [email protected]
  condition: and(succeeded(), eq(variables['PullRequest'], false))
  displayName: 'Publish iSAMS.Web'
  inputs:
    PathtoPublish: '$(build.artifactstagingdirectory)\\iSAMS.Web.zip'
    ArtifactName: 'iSAMS.Web'
    ArtifactType: 'Container'

The baseline timings are as follows:

  • Overall Time: 29 mins 11 seconds
  • Testing Time: 11 mins 26 seconds

Another important point is that our Git source is over 1.7 GB so the Get Sources task takes 2.5 minutes to run. This is actually extremely fast and is one of the big benefits we have using the Hosted Agents in Azure. This will become important later on.

Attempt 1: Adding Parallelism

The first thing we are going to try is to leverage the parallelism in the VSTest (v2) task. We do this by simply changing our YAML file as follows:

queue:  
  name: Hosted VS2017
  parallel: 2    <-------- New Line
  demands: 
  - msbuild
  - visualstudio

The idea behind this is that version 2 of the VSTest task actually detects the parallelism and will automatically share the tests between the two build agents. This should be exactly what we want. You can read more about it here. It is referred to as batching. We haven't changed anything in the VSTest task configuration because the defaults are what we want for the time being. Visually, these are the defaults:

Batching Defaults

Build Log

Build Log

The first thing to notice is that VSTS has parallelised our build across two agents. This is great. The total run time has dropped to 22 mins 54 seconds. That is a 6 minute improvement!

Digging further into this, we can see that the speed gains were in the Test task. Exactly where we expected it to be:

Attempt 1 Build Log

We can also see that it has split the tests across the two agents causing the testing task to be much quicker, reducing the overall time to by xxxx minutes. We can see this happening here:

2018-08-16T01:20:18.5054098Z ==============================================================================  
2018-08-16T01:20:18.5054980Z Task         : Visual Studio Test  
2018-08-16T01:20:18.5055438Z Description  : Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test (VsTest) runner. Test frameworks that have a Visual Studio test adapter such as MsTest, xUnit, NUnit, Chutzpah (for JavaScript tests using QUnit, Mocha and Jasmine), etc. can be run. Tests can be distributed on multiple agents using this task (version 2).  
2018-08-16T01:20:18.5055843Z Version      : 2.138.14  
2018-08-16T01:20:18.5056583Z Author       : Microsoft Corporation  
2018-08-16T01:20:18.5056798Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=835764)  
2018-08-16T01:20:18.5057043Z ==============================================================================  
2018-08-16T01:20:20.2096944Z SystemVssConnection exists true  
2018-08-16T01:20:20.3791584Z In distributed testing flow  
2018-08-16T01:20:20.3792011Z ======================================================  
2018-08-16T01:20:20.3793268Z Test selector : Test assemblies  
2018-08-16T01:20:20.3793813Z Test filter criteria : null  
2018-08-16T01:20:20.3795167Z Search folder : D:\a\1\s  
2018-08-16T01:20:20.3799887Z VisualStudio version selected for test execution : latest  
2018-08-16T01:20:20.8452438Z Distributed test execution, number of agents in phase : 2  

The obvious negative issue is that the entire build was duplicated in each agent, meaning that we are now using up 2 agents for 23 minutes, rather than a single agent for 29 mins. Given that hosted agents are not free, we need to use as few agents as possible and in this scenario, there is a lot of wasted time.

Next steps

I only want the testing phase to run in parallel. Everything else, I want to run in a single agent.

Attempt 2: Multi-Phase

Here we are going to try and leverage the multi-phase builds available in VSTS. We are going to have a first build phase which runs on one agent, and then a second test phase which runs in parallel.

Here is the new build file:

name: $(BuildDefinitionName).$(Year:yy).$(Month).$(DayOfMonth)$(Rev:.r)-$(Build.SourceBranchName)  
phases:  
- phase: Build
  queue:
    name: Hosted VS2017
    demands: 
    - msbuild
    - visualstudio
    - vstest
    - java

  variables:
    Parameters.Solution: iSAMS.New/iSAMS.New.sln
    BuildConfiguration: 'release'
    BuildPlatform: 'any cpu'

  steps:
  - task: [email protected]
    displayName: Use Latest NuGet 4.X
    inputs:
      versionSpec: 4.x
      checkLatest: true

  - task: [email protected]
    displayName: NuGet restore
    inputs:
      restoreSolution: '$(Parameters.solution)'
      feedsToUse: config
      nugetConfigPath: 'iSAMS.New/NuGet.Config'

  - task: richardfennellBM.BM[email protected]1
    displayName: Version Assemblies

  - task: SonarSource.so[email protected]1
    displayName: Prepare analysis on SonarCloud
    inputs:
      SonarCloud: SonarCloud
      organization: isams
      projectKey: isams.newframework
      projectName: iSAMS.NewFramework
      projectVersion: '$(Build.BuildNumber)'
      extraProperties: |
       # Additional properties that will be passed to the scanner, 
       # Put one key=value per line, example:
       sonar.exclusions=**/bower_components/**/*,**/node_modules/**/*,**/Scripts/lib/**/*
    condition: and(succeeded(), eq(variables['ExecuteSonarSource'], true))

  - task: [email protected]
    displayName: Build solution
    inputs:
      solution: '$(Parameters.solution)'
      msbuildArgs: '/p:DeployOnBuild=true /p:PublishProfile=UpdateSystem /p:SkipInvalidConfigurations=true /p:publishBaseUrl="$(build.artifactstagingdirectory)\\"'
      platform: '$(BuildPlatform)'
      configuration: '$(BuildConfiguration)'

  - task: [email protected]
    displayName: VsTest Platform Installer
    enabled: false

  - task: [email protected]
    displayName: Test Assemblies
    enabled: false
    inputs:
      testAssemblyVer2: |
       **\$(BuildConfiguration)\*test*.dll
       !**\$(BuildConfiguration)\*IntegrationTests*.dll
       !**\obj\**
       !**\Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll
      runOnlyImpactedTests: false
      runInParallel: true
      runTestsInIsolation: false
      codeCoverageEnabled: true
      platform: '$(BuildPlatform)'
      configuration: '$(BuildConfiguration)'

  - task: SonarSource.so[email protected]1
    displayName: Run Code Analysis
    condition: and(succeeded(), eq(variables['ExecuteSonarSource'], true))

  - task: SonarSource.so[email protected]1
    displayName: Publish Quality Gate Result
    condition: and(succeeded(), eq(variables['ExecuteSonarSource'], true))

  - task: [email protected]
    displayName: Publish symbols path
    inputs:
      SearchPattern: '**\bin\**\*.pdb'
      PublishSymbols: false
    enabled: false
    continueOnError: true

  - task: roshkovski.2B9619D5-7BE9-4ED7-BF10-707CB[email protected]1
    condition: and(succeeded(), eq(variables['PullRequest'], false))
    displayName: 'Compress iSAMS.Root'
    inputs:
      source: '$(Build.ArtifactStagingDirectory)\iSAMS.Root'
      dest: '$(Build.ArtifactStagingDirectory)\iSAMS.Root.zip'
      compressionLevel: NoCompression
      multiplePackages: false

  - task: [email protected]
    condition: and(succeeded(), eq(variables['PullRequest'], false))
    displayName: 'Publish iSAMS.Root'
    inputs:
      PathtoPublish: '$(build.artifactstagingdirectory)\\iSAMS.Root.zip'
      ArtifactName: 'iSAMS.Root'
      ArtifactType: 'Container'

  - task: roshkovski.2B9619D5-7BE9-4ED7-BF10-707CB[email protected]1
    condition: and(succeeded(), eq(variables['PullRequest'], false))
    displayName: 'Compress iSAMS.Web'
    inputs:
      source: '$(Build.ArtifactStagingDirectory)\iSAMS.Web'
      dest: '$(Build.ArtifactStagingDirectory)\iSAMS.Web.zip'
      compressionLevel: NoCompression
      multiplePackages: false

  - task: [email protected]
    condition: and(succeeded(), eq(variables['PullRequest'], false))
    displayName: 'Publish iSAMS.Web'
    inputs:
      PathtoPublish: '$(build.artifactstagingdirectory)\\iSAMS.Web.zip'
      ArtifactName: 'iSAMS.Web'
      ArtifactType: 'Container'

- phase: Test
  condition: succeeded()
  dependsOn: Build
  queue:
    name: Hosted VS2017
    parallel: 2
    demands: vstest

  steps:
  - task: [email protected]
    displayName: Download Build Artifacts
    inputs:
      downloadType: specific

  - powershell: 'Get-ChildItem "$(System.ArtifactsDirectory)\*.zip" | Expand-Archive' 
    displayName: PowerShell Script

  - task: [email protected]
    displayName: Unit Test
    inputs:
      testAssemblyVer2: |
       **\*test*.dll
       !**\*TestAdapter.dll
       !**\*IntegrationTests*.dll
       !**\Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll
       !**\obj\**
      searchFolder: '$(System.ArtifactsDirectory)'
      runInParallel: true
      codeCoverageEnabled: true
      distributionBatchType: basedOnExecutionTime
      platform: '$(BuildPlatform)'
      configuration: '$(BuildConfiguration)'

Build Log

Multi Phase Build Log

As we can see here, the build phase is correctly run in just a single agent and the testing phase is run over 2 agents.

The glaring issue here is that the build failed. This is because we have taken the build artifacts from the first build and downloaded them in the second build. However, the build artifacts generally do not contain any testing assemblies.

Next steps

We need to somehow transfer the test assemblies from the first phase into the second phase. As the phases run on completely independent VMs, we are going to have to do this manually.

Attempt 3: Passing the test assets between phases

In order to pass the test assemblies from the build phase into the test phase, we are going to have to collect them up and publish them as a new artifact which can we downloaded.

I am going to add the following tasks to the build phase:

- powershell: |
      Mkdir "$(Build.ArtifactStagingDirectory)\Testing" -Force; 
      (Get-ChildItem -Path "$(System.DefaultWorkingDirectory)\isams.new\" -Recurse | Select -ExpandProperty FullName) -like "*test*\bin\*" -notlike "*\node_modules\*" -notlike "*.pdb" | 
      Copy-Item -Destination "$(Build.ArtifactStagingDirectory)\Testing" -Recurse -Force
    displayName: Copy all dlls to "Testing" folder
    workingDirectory: $(Build.ArtifactStagingDirectory)

  - task: roshkovski.2B9619D5-7BE9-4ED7-BF10-707CB[email protected]1
    displayName: 'Compress Testing folder'
    inputs:
      source: '$(Build.ArtifactStagingDirectory)\Testing'
      dest: '$(Build.ArtifactStagingDirectory)\Testing.zip'
      compressionLevel: NoCompression
      multiplePackages: false

  - task: [email protected]
    displayName: 'Publish Testing'
    inputs:
      PathtoPublish: '$(build.artifactstagingdirectory)\\Testing.zip'
      ArtifactName: 'Testing'
      ArtifactType: 'Container'

These tasks perform the work of scooping up all the dlls, putting them in a single folder, compress them and finally publish the archive as an artifact.

Important: Due to the fact that we are scooping up all dlls from all projects in the solution and putting them in a single folder, this can introduce some errors. For example, if you have different versions of a dll referenced in your solution, then this method will not work as only the last dll will end up in the folder. This did expose a number of incorrect references in our project that we were not aware of.

The additions to the testing phase look like this:

  - powershell: 'Get-ChildItem "Testing.zip" -recurse | %{ Write-Host "Expanding [$_] to $((Join-Path "$(System.ArtifactsDirectory)" ([System.IO.Path]::GetFileNameWithoutExtension($_))))"; Expand-Archive -Path $_ -DestinationPath (Join-Path "$(System.ArtifactsDirectory)" ([System.IO.Path]::GetFileNameWithoutExtension($_))) }'
    displayName: PowerShell Script
    workingDirectory: $(System.ArtifactsDirectory)

The entire build file now looks like this:

name: $(BuildDefinitionName).$(Year:yy).$(Month).$(DayOfMonth)$(Rev:.r)-$(Build.SourceBranchName)  
phases:  
- phase: Build
  queue:
    name: Hosted VS2017
    demands: 
    - msbuild
    - visualstudio
    - vstest
    - java

  variables:
    Parameters.Solution: iSAMS.New/iSAMS.New.sln
    BuildConfiguration: 'release'
    BuildPlatform: 'any cpu'

  steps:
  - task: [email protected]
    displayName: Use Latest NuGet 4.X
    inputs:
      versionSpec: 4.x
      checkLatest: true

  - task: [email protected]
    displayName: NuGet restore
    inputs:
      restoreSolution: '$(Parameters.solution)'
      feedsToUse: config
      nugetConfigPath: 'iSAMS.New/NuGet.Config'

  - task: richardfennellBM.BM[email protected]1
    displayName: Version Assemblies

  - task: SonarSource.so[email protected]1
    displayName: Prepare analysis on SonarCloud
    inputs:
      SonarCloud: SonarCloud
      organization: isams
      projectKey: isams.newframework
      projectName: iSAMS.NewFramework
      projectVersion: '$(Build.BuildNumber)'
      extraProperties: |
       # Additional properties that will be passed to the scanner, 
       # Put one key=value per line, example:
       sonar.exclusions=**/bower_components/**/*,**/node_modules/**/*,**/Scripts/lib/**/*
    condition: and(succeeded(), eq(variables['ExecuteSonarSource'], true))

  - task: [email protected]
    displayName: Build solution
    inputs:
      solution: '$(Parameters.solution)'
      msbuildArgs: '/p:DeployOnBuild=true /p:PublishProfile=UpdateSystem /p:SkipInvalidConfigurations=true /p:publishBaseUrl="$(build.artifactstagingdirectory)\\"'
      platform: '$(BuildPlatform)'
      configuration: '$(BuildConfiguration)'

  - task: [email protected]
    displayName: VsTest Platform Installer
    enabled: false

  - task: [email protected]
    displayName: Test Assemblies
    enabled: false
    inputs:
      testAssemblyVer2: |
       **\$(BuildConfiguration)\*test*.dll
       !**\$(BuildConfiguration)\*IntegrationTests*.dll
       !**\obj\**
       !**\Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll
      runOnlyImpactedTests: false
      runInParallel: true
      runTestsInIsolation: false
      codeCoverageEnabled: true
      platform: '$(BuildPlatform)'
      configuration: '$(BuildConfiguration)'

  - task: SonarSource.so[email protected]1
    displayName: Run Code Analysis
    condition: and(succeeded(), eq(variables['ExecuteSonarSource'], true))

  - task: SonarSource.so[email protected]1
    displayName: Publish Quality Gate Result
    condition: and(succeeded(), eq(variables['ExecuteSonarSource'], true))

  - task: [email protected]
    displayName: Publish symbols path
    inputs:
      SearchPattern: '**\bin\**\*.pdb'
      PublishSymbols: false
    enabled: false
    continueOnError: true

  - powershell: |
      Mkdir "$(Build.ArtifactStagingDirectory)\Testing" -Force; 
      (Get-ChildItem -Path "$(System.DefaultWorkingDirectory)\isams.new\" -Recurse | Select -ExpandProperty FullName) -like "*test*\bin\*" -notlike "*\node_modules\*" -notlike "*.pdb" | 
      Copy-Item -Destination "$(Build.ArtifactStagingDirectory)\Testing" -Recurse -Force
    displayName: Copy all dlls to "Testing" folder
    workingDirectory: $(Build.ArtifactStagingDirectory)

  - task: roshkovski.2B9619D5-7BE9-4ED7-BF10-707CB[email protected]1
    displayName: 'Compress Testing folder'
    inputs:
      source: '$(Build.ArtifactStagingDirectory)\Testing'
      dest: '$(Build.ArtifactStagingDirectory)\Testing.zip'
      compressionLevel: NoCompression
      multiplePackages: false

  - task: [email protected]
    displayName: 'Publish Testing'
    inputs:
      PathtoPublish: '$(build.artifactstagingdirectory)\\Testing.zip'
      ArtifactName: 'Testing'
      ArtifactType: 'Container'

  - task: roshkovski.2B9619D5-7BE9-4ED7-BF10-707CB[email protected]1
    condition: and(succeeded(), eq(variables['PullRequest'], false))
    displayName: 'Compress iSAMS.Root'
    inputs:
      source: '$(Build.ArtifactStagingDirectory)\iSAMS.Root'
      dest: '$(Build.ArtifactStagingDirectory)\iSAMS.Root.zip'
      compressionLevel: NoCompression
      multiplePackages: false

  - task: [email protected]
    condition: and(succeeded(), eq(variables['PullRequest'], false))
    displayName: 'Publish iSAMS.Root'
    inputs:
      PathtoPublish: '$(build.artifactstagingdirectory)\\iSAMS.Root.zip'
      ArtifactName: 'iSAMS.Root'
      ArtifactType: 'Container'

  - task: roshkovski.2B9619D5-7BE9-4ED7-BF10-707CB[email protected]1
    condition: and(succeeded(), eq(variables['PullRequest'], false))
    displayName: 'Compress iSAMS.Web'
    inputs:
      source: '$(Build.ArtifactStagingDirectory)\iSAMS.Web'
      dest: '$(Build.ArtifactStagingDirectory)\iSAMS.Web.zip'
      compressionLevel: NoCompression
      multiplePackages: false

  - task: [email protected]
    condition: and(succeeded(), eq(variables['PullRequest'], false))
    displayName: 'Publish iSAMS.Web'
    inputs:
      PathtoPublish: '$(build.artifactstagingdirectory)\\iSAMS.Web.zip'
      ArtifactName: 'iSAMS.Web'
      ArtifactType: 'Container'

- phase: Test
  condition: succeeded()
  dependsOn: Build
  queue:
    name: Hosted VS2017
    parallel: 2
    demands: vstest

  variables:
    BuildConfiguration: 'release'
    BuildPlatform: 'any cpu'

  steps:
  - task: [email protected]
    displayName: Download Build Artifacts
    inputs:
      downloadType: specific

  - powershell: 'Get-ChildItem "Testing.zip" -recurse | %{ Write-Host "Expanding [$_] to $((Join-Path "$(System.ArtifactsDirectory)" ([System.IO.Path]::GetFileNameWithoutExtension($_))))"; Expand-Archive -Path $_ -DestinationPath (Join-Path "$(System.ArtifactsDirectory)" ([System.IO.Path]::GetFileNameWithoutExtension($_))) }'
    displayName: PowerShell Script
    workingDirectory: $(System.ArtifactsDirectory)

  - task: [email protected]
    displayName: Unit Test
    inputs:
      testAssemblyVer2: |
       **\*test*.dll
       !**\*TestAdapter.dll
       !**\*IntegrationTests*.dll
       !**\Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll
       !**\obj\**
      searchFolder: '$(System.ArtifactsDirectory)'
      runInParallel: true
      codeCoverageEnabled: true
      distributionBatchType: basedOnExecutionTime
      platform: '$(BuildPlatform)'
      configuration: '$(BuildConfiguration)'

Build Log

3 Build Log

So we now have a successful build with the tests split across two agents. The timings are as follows:

  • Overall Time: 29 mins 44 seconds
  • Testing Time: 7 mins 07 seconds

So although we have decreased the testing time significantly, it looks like the other parts we have brought into our build to achieve this have actually cost more time than we have saved.

Next Steps

We need to see if we can stop the source download in the testing phase. We don't use the source and it is taking up 2.5 valuable minutes.

Attempt 4: Stopping source download

The aim here is to stop the testing phase from downloading the sources. I had to ask on twitter how to do this. You can only do it from a YAML build. This does not work using the graphical interface as far as I know.

All we need to do is add agent.source.skip: true to the variables of the agent phase.

The new YAML file looks like this:

name: $(BuildDefinitionName).$(Year:yy).$(Month).$(DayOfMonth)$(Rev:.r)-$(Build.SourceBranchName)  
phases:  
- phase: Build
  queue:
    name: Hosted VS2017
    demands: 
    - msbuild
    - visualstudio
    - vstest
    - java

  variables:
    Parameters.Solution: iSAMS.New/iSAMS.New.sln
    BuildConfiguration: 'release'
    BuildPlatform: 'any cpu'

  steps:
  - task: [email protected]
    displayName: Use Latest NuGet 4.X
    inputs:
      versionSpec: 4.x
      checkLatest: true

  - task: [email protected]
    displayName: NuGet restore
    inputs:
      restoreSolution: '$(Parameters.solution)'
      feedsToUse: config
      nugetConfigPath: 'iSAMS.New/NuGet.Config'

  - task: richardfennellBM.BM[email protected]1
    displayName: Version Assemblies

  - task: SonarSource.so[email protected]1
    displayName: Prepare analysis on SonarCloud
    inputs:
      SonarCloud: SonarCloud
      organization: isams
      projectKey: isams.newframework
      projectName: iSAMS.NewFramework
      projectVersion: '$(Build.BuildNumber)'
      extraProperties: |
       # Additional properties that will be passed to the scanner, 
       # Put one key=value per line, example:
       sonar.exclusions=**/bower_components/**/*,**/node_modules/**/*,**/Scripts/lib/**/*
    condition: and(succeeded(), eq(variables['ExecuteSonarSource'], true))

  - task: [email protected]
    displayName: Build solution
    inputs:
      solution: '$(Parameters.solution)'
      msbuildArgs: '/p:DeployOnBuild=true /p:PublishProfile=UpdateSystem /p:SkipInvalidConfigurations=true /p:publishBaseUrl="$(build.artifactstagingdirectory)\\"'
      platform: '$(BuildPlatform)'
      configuration: '$(BuildConfiguration)'

  - task: [email protected]
    displayName: VsTest Platform Installer
    enabled: false

  - task: [email protected]
    displayName: Test Assemblies
    enabled: false
    inputs:
      testAssemblyVer2: |
       **\$(BuildConfiguration)\*test*.dll
       !**\$(BuildConfiguration)\*IntegrationTests*.dll
       !**\obj\**
       !**\Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll
      runOnlyImpactedTests: false
      runInParallel: true
      runTestsInIsolation: false
      codeCoverageEnabled: true
      platform: '$(BuildPlatform)'
      configuration: '$(BuildConfiguration)'

  - task: SonarSource.so[email protected]1
    displayName: Run Code Analysis
    condition: and(succeeded(), eq(variables['ExecuteSonarSource'], true))

  - task: SonarSource.so[email protected]1
    displayName: Publish Quality Gate Result
    condition: and(succeeded(), eq(variables['ExecuteSonarSource'], true))

  - task: [email protected]
    displayName: Publish symbols path
    inputs:
      SearchPattern: '**\bin\**\*.pdb'
      PublishSymbols: false
    enabled: false
    continueOnError: true

  - powershell: |
      Mkdir "$(Build.ArtifactStagingDirectory)\Testing" -Force; 
      (Get-ChildItem -Path "$(System.DefaultWorkingDirectory)\isams.new\" -Recurse | Select -ExpandProperty FullName) -like "*test*\bin\*" -notlike "*\node_modules\*" -notlike "*.pdb" | 
      Copy-Item -Destination "$(Build.ArtifactStagingDirectory)\Testing" -Recurse -Force
    displayName: Copy all dlls to "Testing" folder
    workingDirectory: $(Build.ArtifactStagingDirectory)

  - task: roshkovski.2B9619D5-7BE9-4ED7-BF10-707CB[email protected]1
    displayName: 'Compress Testing folder'
    inputs:
      source: '$(Build.ArtifactStagingDirectory)\Testing'
      dest: '$(Build.ArtifactStagingDirectory)\Testing.zip'
      compressionLevel: NoCompression
      multiplePackages: false

  - task: [email protected]
    displayName: 'Publish Testing'
    inputs:
      PathtoPublish: '$(build.artifactstagingdirectory)\\Testing.zip'
      ArtifactName: 'Testing'
      ArtifactType: 'Container'

  - task: roshkovski.2B9619D5-7BE9-4ED7-BF10-707CB[email protected]1
    condition: and(succeeded(), eq(variables['PullRequest'], false))
    displayName: 'Compress iSAMS.Root'
    inputs:
      source: '$(Build.ArtifactStagingDirectory)\iSAMS.Root'
      dest: '$(Build.ArtifactStagingDirectory)\iSAMS.Root.zip'
      compressionLevel: NoCompression
      multiplePackages: false

  - task: [email protected]
    condition: and(succeeded(), eq(variables['PullRequest'], false))
    displayName: 'Publish iSAMS.Root'
    inputs:
      PathtoPublish: '$(build.artifactstagingdirectory)\\iSAMS.Root.zip'
      ArtifactName: 'iSAMS.Root'
      ArtifactType: 'Container'

  - task: roshkovski.2B9619D5-7BE9-4ED7-BF10-707CB[email protected]1
    condition: and(succeeded(), eq(variables['PullRequest'], false))
    displayName: 'Compress iSAMS.Web'
    inputs:
      source: '$(Build.ArtifactStagingDirectory)\iSAMS.Web'
      dest: '$(Build.ArtifactStagingDirectory)\iSAMS.Web.zip'
      compressionLevel: NoCompression
      multiplePackages: false

  - task: [email protected]
    condition: and(succeeded(), eq(variables['PullRequest'], false))
    displayName: 'Publish iSAMS.Web'
    inputs:
      PathtoPublish: '$(build.artifactstagingdirectory)\\iSAMS.Web.zip'
      ArtifactName: 'iSAMS.Web'
      ArtifactType: 'Container'

- phase: Test
  condition: succeeded()
  dependsOn: Build
  queue:
    name: Hosted VS2017
    parallel: 2
    demands: vstest

  variables:
    agent.source.skip: true
    BuildConfiguration: 'release'
    BuildPlatform: 'any cpu'

  steps:
  - task: [email protected]
    displayName: Download Build Artifacts
    inputs:
      downloadType: specific

  - powershell: 'Get-ChildItem "Testing.zip" -recurse | %{ Write-Host "Expanding [$_] to $((Join-Path "$(System.ArtifactsDirectory)" ([System.IO.Path]::GetFileNameWithoutExtension($_))))"; Expand-Archive -Path $_ -DestinationPath (Join-Path "$(System.ArtifactsDirectory)" ([System.IO.Path]::GetFileNameWithoutExtension($_))) }'
    displayName: PowerShell Script
    workingDirectory: $(System.ArtifactsDirectory)

  - task: [email protected]
    displayName: Unit Test
    inputs:
      testAssemblyVer2: |
       **\*test*.dll
       !**\*TestAdapter.dll
       !**\*IntegrationTests*.dll
       !**\Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll
       !**\obj\**
      searchFolder: '$(System.ArtifactsDirectory)'
      runInParallel: true
      codeCoverageEnabled: true
      distributionBatchType: basedOnExecutionTime
      platform: '$(BuildPlatform)'
      configuration: '$(BuildConfiguration)'

Build Log

4 Build Log

That was a success. As you can see the Get Sources task finished in 1.677 seconds, saving us approximately 2 minutes.

  • Overall Time: 26 mins 51 seconds
  • Testing Time: 6 mins 11 seconds

So this gives us a saving of 3 minutes against our baseline build. To be honest, I was hoping for more.

Lets see what else we can do.

Next Steps

Lets see what happens if we distribute this over 3 agents. In theory, this should deliver another couple of minutes improvement.

Attempt 5: More Agents

I am going to increase the number of agents by changing the Parallel variable to 3 in the testing phase.

The results are not really what we would expect:

3 Agents

Overall Time: 25 mins 32 seconds

Lets increase the number of agents to 4:

4 Agents

Overall Time: 26 mins 2 seconds

So it seems now that we are not getting any improvement by increasing the number of agents that we spread this across.

Next Steps

I think that if I had more tests, I might see more of an improvement by adding more agents, but in my specific scenario, I am not seeing much of an improvement.

I am not going to see what other batching options I have.

Attempt 6: Batching Options

I am going to revert back to two agents and take a look at the batching options. Here are the batching options available to us in the VSTest task.

  • Based on number of tests and agents: Simple batching based on the number of tests and agents participating in the test run.
  • Based on past running time of tests: This batching considers past running time to create batches of tests such that each batch has approximately equal running time.
  • Based on test assemblies: Tests from an assembly are batched together.

All of the tests so far have been done using the Based on past running time of tests setting.

Here are the results from the other options:

Based on number of tests and agents

6 Number Of Agents

Overall Time: 25 mins 0 seconds

Based on test assemblies

6 test assemblies

Overall Time: 31 mins 0 seconds

So there is quite a significant difference here. In all honesty, I don't know why there is such a difference, but you can see that both Test jobs ran for 11.5 mins, which is much longer than the other settings.

Conclusion

Although I managed to achieve what I wanted to, I don't think it was very successful in my scenario.

I have managed to reduce the build time by approximately 4 minutes. Your mileage may vary. If you have a much larger set of unit tests, then I can see that you might get some more gains than I was able to.

I don't think I will be using this system in production for a few reasons:

  1. We are now using an extra agent for 5 minutes. Agents are very valuable in our environment and I would need to see a larger gain in time to justify using an extra agent.
  2. Our testing is now happening in a different phase. This disrupts the order of the build, which affect the following:
    • In the final setup, we are publishing artifacts regardless of whether the unit tests succeed or fail. This might not be an issue, but it is different from the standard behaviour.
    • You may have noticed that we use SonarCloud in our build. Ordinarily, SonarCloud uploads the Unit Test results file and provides all kinds of exciting metrics on it. This no longer works. This is because we upload the SonarCloud results in the build phase and the testing has not been conducted yet. I have tried to get this to work, but so far have been unable to.

The Best Performing YAML File

name: $(BuildDefinitionName).$(Year:yy).$(Month).$(DayOfMonth)$(Rev:.r)-$(Build.SourceBranchName)  
phases:  
- phase: Build
  queue:
    name: Hosted VS2017
    demands: 
    - msbuild
    - visualstudio
    - vstest
    - java

  variables:
    Parameters.Solution: iSAMS.New/iSAMS.New.sln
    BuildConfiguration: 'release'
    BuildPlatform: 'any cpu'

  steps:
  - task: [email protected]
    displayName: Use Latest NuGet 4.X
    inputs:
      versionSpec: 4.x
      checkLatest: true

  - task: [email protected]
    displayName: NuGet restore
    inputs:
      restoreSolution: '$(Parameters.solution)'
      feedsToUse: config
      nugetConfigPath: 'iSAMS.New/NuGet.Config'

  - task: richardfennellBM.BM[email protected]1
    displayName: Version Assemblies

  - task: SonarSource.so[email protected]1
    displayName: Prepare analysis on SonarCloud
    inputs:
      SonarCloud: SonarCloud
      organization: isams
      projectKey: isams.newframework
      projectName: iSAMS.NewFramework
      projectVersion: '$(Build.BuildNumber)'
      extraProperties: |
       # Additional properties that will be passed to the scanner, 
       # Put one key=value per line, example:
       sonar.exclusions=**/bower_components/**/*,**/node_modules/**/*,**/Scripts/lib/**/*
    condition: and(succeeded(), eq(variables['ExecuteSonarSource'], true))

  - task: [email protected]
    displayName: Build solution
    inputs:
      solution: '$(Parameters.solution)'
      msbuildArgs: '/p:DeployOnBuild=true /p:PublishProfile=UpdateSystem /p:SkipInvalidConfigurations=true /p:publishBaseUrl="$(build.artifactstagingdirectory)\\"'
      platform: '$(BuildPlatform)'
      configuration: '$(BuildConfiguration)'

  - task: [email protected]
    displayName: VsTest Platform Installer
    enabled: false

  - task: [email protected]
    displayName: Test Assemblies
    enabled: false
    inputs:
      testAssemblyVer2: |
       **\$(BuildConfiguration)\*test*.dll
       !**\$(BuildConfiguration)\*IntegrationTests*.dll
       !**\obj\**
       !**\Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll
      runOnlyImpactedTests: false
      runInParallel: true
      runTestsInIsolation: false
      codeCoverageEnabled: true
      platform: '$(BuildPlatform)'
      configuration: '$(BuildConfiguration)'

  - task: SonarSource.so[email protected]1
    displayName: Run Code Analysis
    condition: and(succeeded(), eq(variables['ExecuteSonarSource'], true))

  - task: SonarSource.so[email protected]1
    displayName: Publish Quality Gate Result
    condition: and(succeeded(), eq(variables['ExecuteSonarSource'], true))

  - task: [email protected]
    displayName: Publish symbols path
    inputs:
      SearchPattern: '**\bin\**\*.pdb'
      PublishSymbols: false
    enabled: false
    continueOnError: true

  - powershell: |
      Mkdir "$(Build.ArtifactStagingDirectory)\Testing" -Force; 
      (Get-ChildItem -Path "$(System.DefaultWorkingDirectory)\isams.new\" -Recurse | Select -ExpandProperty FullName) -like "*test*\bin\*" -notlike "*\node_modules\*" -notlike "*.pdb" | 
      Copy-Item -Destination "$(Build.ArtifactStagingDirectory)\Testing" -Recurse -Force
    displayName: Copy all dlls to "Testing" folder
    workingDirectory: $(Build.ArtifactStagingDirectory)

  - task: roshkovski.2B9619D5-7BE9-4ED7-BF10-707CB[email protected]1
    displayName: 'Compress Testing folder'
    inputs:
      source: '$(Build.ArtifactStagingDirectory)\Testing'
      dest: '$(Build.ArtifactStagingDirectory)\Testing.zip'
      compressionLevel: NoCompression
      multiplePackages: false

  - task: [email protected]
    displayName: 'Publish Testing'
    inputs:
      PathtoPublish: '$(build.artifactstagingdirectory)\\Testing.zip'
      ArtifactName: 'Testing'
      ArtifactType: 'Container'

  - task: roshkovski.2B9619D5-7BE9-4ED7-BF10-707CB[email protected]1
    condition: and(succeeded(), eq(variables['PullRequest'], false))
    displayName: 'Compress iSAMS.Root'
    inputs:
      source: '$(Build.ArtifactStagingDirectory)\iSAMS.Root'
      dest: '$(Build.ArtifactStagingDirectory)\iSAMS.Root.zip'
      compressionLevel: NoCompression
      multiplePackages: false

  - task: [email protected]
    condition: and(succeeded(), eq(variables['PullRequest'], false))
    displayName: 'Publish iSAMS.Root'
    inputs:
      PathtoPublish: '$(build.artifactstagingdirectory)\\iSAMS.Root.zip'
      ArtifactName: 'iSAMS.Root'
      ArtifactType: 'Container'

  - task: roshkovski.2B9619D5-7BE9-4ED7-BF10-707CB[email protected]1
    condition: and(succeeded(), eq(variables['PullRequest'], false))
    displayName: 'Compress iSAMS.Web'
    inputs:
      source: '$(Build.ArtifactStagingDirectory)\iSAMS.Web'
      dest: '$(Build.ArtifactStagingDirectory)\iSAMS.Web.zip'
      compressionLevel: NoCompression
      multiplePackages: false

  - task: [email protected]
    condition: and(succeeded(), eq(variables['PullRequest'], false))
    displayName: 'Publish iSAMS.Web'
    inputs:
      PathtoPublish: '$(build.artifactstagingdirectory)\\iSAMS.Web.zip'
      ArtifactName: 'iSAMS.Web'
      ArtifactType: 'Container'

- phase: Test
  condition: succeeded()
  dependsOn: Build
  queue:
    name: Hosted VS2017
    parallel: 2
    demands: vstest

  variables:
    agent.source.skip: true
    BuildConfiguration: 'release'
    BuildPlatform: 'any cpu'

  steps:
  - task: [email protected]
    displayName: Download Build Artifacts
    inputs:
      downloadType: specific

  - powershell: 'Get-ChildItem "Testing.zip" -recurse | %{ Write-Host "Expanding [$_] to $((Join-Path "$(System.ArtifactsDirectory)" ([System.IO.Path]::GetFileNameWithoutExtension($_))))"; Expand-Archive -Path $_ -DestinationPath (Join-Path "$(System.ArtifactsDirectory)" ([System.IO.Path]::GetFileNameWithoutExtension($_))) }'
    displayName: PowerShell Script
    workingDirectory: $(System.ArtifactsDirectory)

  - task: [email protected]
    displayName: Unit Test
    inputs:
      testAssemblyVer2: |
       **\*test*.dll
       !**\*TestAdapter.dll
       !**\*IntegrationTests*.dll
       !**\Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll
       !**\obj\**
      searchFolder: '$(System.ArtifactsDirectory)'
      runInParallel: true
      codeCoverageEnabled: true
      platform: '$(BuildPlatform)'
      configuration: '$(BuildConfiguration)'