At my current client, Nordic Leisure Travel Group , we have a large number of web tests that automates some of the QA effort. The tests are written in Selenium and Playwright. They run on a schedule in Jenkins and we get reports on email. They take quite some time to run. Unfortunately some of the tests are a bit flaky and it is hard to make them stable.

Therefore I set out trying to mitigate failed test runs by rerunning the failing tests.

A question/answer on Stack Overflow inspired me to write my own test script.

My requirements were:

  • Use PowerShell
  • Use the dotnet test CLI command
  • Keep track of failing tests and retry them
  • Notify the test framework of the current retry iteration
  • Make it possible to accept a certain percentage of failing tests
  • Output the test result in a coherent way

This resulted in the following script:

The script runs dotnet test and uses the trx logger result file to collect failed tests. Then reruns the failed tests and reports the final result.

Parameters

  • Path - Path to the: project | solution | directory | dll | exe
  • Configuration - Build configuration for environment specific appsettings.json file.
    • Default value: Debug
    • Valid values: Debug | Release | Development | Production
  • Filter - Filter to run selected tests based on: TestCategory | Priority | Name | FullyQualifiedName
  • Settings - Path to the .runsettings file.
  • Retries - Number of retries for each failed test.
    • Default value: 2
    • Valid range: 1-9
  • Percentage - Required percentage of passed tests.
    • Default value: 100
    • Valid range: 0-100

Examples

Run regression tests:

.\test.ps1 -filter "TestCategory=RegressionTest"

Run FooBar smoke tests in Development:

.\test.ps1 .\FooBar.Tests\FooBar.Tests.csproj -filter "TestCategory=SmokeTest" -configuration "Development"

Retry failed tests once and reports the run as green if 95% of the tests passed:

.\test.ps1 -retries 1 -percentage 95

Run tests configured with a .runsettings file:

.\test.ps1 -settings .\test.runsettings

Output

If the tests passed, or the required percentage of tests passed, the script returns the exitcode 0. Otherwise the number of failed tests is returned as exitcode.

Output when successful:

Test Rerun Successful.

Output when required percentage of tests passed:

Test Rerun Successful.

Output when failed:

Test Rerun Failed.

Showcase:

.\test.ps1 -filter FullyQualifiedName=ConductOfCode.NUnitTests

Tests

To showcase the test script I have written some tests in three popular frameworks.

The script and tests are available in this repo:

NUnit

  • The Category attribute corresponds to the TestCategory filter in the script.
  • The Retry parameter in the TestContext contains the current retry iteration.

MSTest

  • The TestCategory attribute corresponds to the TestCategory filter in the script.
  • The Retry property in the TestContext contains the current retry iteration.

XUnit

Playwright

Playwright enables reliable end-to-end testing for modern web apps.

Web testing might be hard to get right and flaky tests can benefit from being retried.

Here is an example with Playwright and NUnit:

  • The SetUp method starts a trace recording if the test is in retry mode.
  • The TearDown method exports the trace into a zip archive if the test failed.

The trace zip archives of failing tests can be examined at:

A .runsettings file can be used to customizing Playwright options:

Run the Playwright tests with the .runsettings file:

.\test.ps1 -filter FullyQualifiedName=ConductOfCode.PlaywrightTests -settings .\test.runsettings

Resources

If you use Azure Pipelines consider using the Visual Studio Test v2 task with the rerunFailedTests option: