Smart Orchestration
Cypress Cloud provides four distinct Smart Orchestration features for use in CI to speed up test runs, accelerate debugging workflows, and reduce costs:
- Parallelization: Reduce test duration by running multiple test spec files simultaneously across available CI machines.
- Load Balancing: Optimize CI resources and minimize test duration by intelligently prioritizing running of test spec files during parallelization.
- Spec Prioritization: Quickly verify that your latest changes fixed a build by prioritizing the specs that failed in the previous Cypress run.
- Auto Cancellation: Save on CI resources by automatically canceling a Cypress run upon the first test failure.
Spec Prioritization
It is often helpful to be aware of test failures earlier within a CI test run so that debugging and iterations can resume and progress much faster. Being able to catch issues sooner within the CI process can save valuable time in troubleshooting failures and deploying fixes.
For example, you have a test suite consisting of several spec files that take 30 minutes to execute, but failures start to occur within one spec at the 20 minute marker. A subsequent code change aimed at resolving the issue will have to be validated after running the CI build and waiting 20 minutes. This process and waiting continues until the issue is fixed.
By running failed specs first, the issues can be surfaced earlier and reduce that 20 minute wait to potentially a few minutes. Considering that a test suite has multiple spec files, this orchestration strategy will always save time.
Activate Spec Prioritization in Cypress Cloud
Spec Prioritization is a Smart Orchestration feature available to users with a Business or Enterprise tier Cypress Cloud plan. Wondering if upgrading your Team tier subscription is worth it? Check out the recommendations on the run overview page to see how much time we predict these features would save you.
Spec Prioritization is a Smart Orchestration feature that is managed within a project's settings.
To enable or disable this feature at the Project level (must be an admin user):
- Select the desired project within Projects view to visit Cypress Cloud.
- Click "Project Settings" with the right-hand sidebar.
- Scroll to the Smart Orchestration section within Project Settings page.
- Toggle "Spec Prioritization".
Note that Spec Prioritization alone will not save time as it purely alters the order in which specs are run. However, when used in tandem with Auto Cancellation it can result in shorter test runs due to cancellation being triggered earlier, leading to significantly reduced CI costs.
Auto Cancellation
Continuous Integration (CI) pipelines are typically costly processes that can demand significant compute time. When a test failure occurs in CI, it often does not make sense to continue running the remainder of a test suite since the process has to start again upon merging of subsequent fixes and other code changes. When Auto Cancellation is enabled, once the number of failed tests goes over a preset threshold, the entire test run is canceled. Note that any in-progress specs will continue to run to completion.
Canceling an entire test run, even if parallelized, upon the first test failure will:
- Save time. Resolve test outcomes faster.
- Reduce CI costs. These cost savings can be significant for large test suites.
- Free-up CI resources for validating fixes, and helping other users.
Activate Auto Cancellation in Cypress Cloud
Auto Cancellation is a Smart Orchestration feature available to users with a Business or Enterprise tier Cypress Cloud plan. Wondering if upgrading your Team tier subscription is worth it? Check out the recommendations on the run overview page to see how much time we predict these features would save you.
Auto Cancellation is a Smart Orchestration feature that is managed within a project's settings.
If your development, testing, or QA teams operate in a highly collaborative workflow where multiple people are working on various test failures at the same time, it may be helpful to increase the failures threshold or even disable Auto Cancellation altogether, so multiple failing tests can be surfaced for each test run.
To enable or disable this feature at the Project level (must be an admin user):
- Select the desired project within Projects view to visit Cypress Cloud.
- Click "Project Settings" with the right-hand sidebar.
- Scroll to the Smart Orchestration section within Project Settings page.
- Toggle "Auto Cancellation".
- Set the "failures before auto canceling" threshold (or just leave it at the default value of 1).
In addition to Auto Cancellation, consider reducing CI costs further by enabling Spec Prioritization to surface test failures earlier in the run, and reduce the time it takes to fix issues and deploy changes.
Specifying failures before auto canceling in the CLI
The auto-cancel-after-failures flag in the CLI is only available in Cypress
12.6.0 and later. Including the flag in an older version will throw an error.
With the Cypress CLI, you can specify the number of test failures to execute before canceling the run via a command flag. This allows more granular control of Auto Cancellation on a per-run basis.
Some common use cases for doing so might be:
- Setting a lower-than-normal value on dev environments to fail a test suite early for faster feedback
- Disabling Auto Cancellation on important branches that need the full test suite to run every time
- Overriding the project value on a case-by-case basis
Specifying the flag will override the project setting in Cypress Cloud, and not specifying it will default to the value in the project settings.
To set it, pass the --auto-cancel-after-failures flag to the cypress run
command, and give it a numeric value:
npx cypress run --record --key <<your_record_key>> --auto-cancel-after-failures 1
You can also specify false for the value to disable Auto Cancellation for the
run:
npx cypress run --record --key <<your_record_key>> --auto-cancel-after-failures false
For more info on passing CLI flags, see our guide on the Cypress Command Line.