How to Run a Test Multiple Times in Cypress.io: Automatically Restart and Stop at Threshold to Fix Intermittent Bugs

Table of Contents#

1. Understanding Intermittent Bugs in Cypress
   • What Are Intermittent Bugs?
   • Common Causes of Flakiness
2. Why Run Tests Multiple Times?
3. Cypress Built-in Retry Logic: What You Need to Know
   • How Retries Work in Cypress
   • Limitations of Built-in Retries
4. Custom Solutions: Running a Test Multiple Times
   • CLI Scripts for Multiple Spec Runs
   • In-Test Loop vs. CLI Restart
5. Stopping at a Threshold: When to Stop Testing
   • Setting a Maximum Run Count
   • Stopping on First Failure
6. Tracking Results: Logging and Reporting
7. Best Practices for Testing Intermittent Bugs
8. Example Workflow: Fixing a Flaky Test
9. Tools and Plugins to Simplify the Process
10. Conclusion
11. References

Understanding Intermittent Bugs in Cypress#

What Are Intermittent Bugs?#

Intermittent bugs (or "flaky tests") are tests that pass or fail unpredictably without changes to the code or test itself. For example, a login test might pass 9/10 times but fail the 10th due to a slow API response or race condition. These bugs are notoriously hard to fix because they're not consistently reproducible.

Common Causes of Flakiness#

• Timing Issues: Tests that rely on hardcoded cy.wait() times or assume elements load instantly.
• API/Backend Flakiness: Dependencies on external services with variable response times.
• State Leakage: Tests that don't clean up state (e.g., local storage, cookies) between runs.
• Asynchronous Race Conditions: JavaScript execution order mismatches (e.g., a test asserting before data loads).
• Environmental Factors: Network latency, CPU usage, or browser inconsistencies.

Why Run Tests Multiple Times?#

Intermittent bugs are stochastic—they depend on random variables like timing or network speed. Running a test once might miss the bug entirely, but running it 10, 50, or 100 times increases the odds of triggering the failure. This makes it easier to:

• Reproduce the bug consistently.
• Validate fixes (e.g., "After the fix, the test passes 100/100 runs").
• Gather data on failure patterns (e.g., "It fails only when the API takes >2s to respond").

Cypress Built-in Retry Logic: What You Need to Know#

Before building custom solutions, it's critical to understand Cypress' built-in retry logic, as it may already solve simple flakiness.

How Retries Work in Cypress#

Cypress automatically retries failed commands and assertions to handle transient issues. By default:

• In cypress run (headless mode): Commands/assertions retry up to 4 times.
• In cypress open (interactive mode): Retries are disabled by default (to speed up development).

You can customize retries globally in cypress.config.js or per-test with the retries() method:

// cypress.config.js
module.exports = {
  retries: {
    runMode: 5, // Retry 5 times in headless mode
    openMode: 2, // Retry 2 times in interactive mode
  },
};
 
// Per-test override
it('logs in successfully', { retries: { runMode: 3 } }, () => {
  cy.visit('/login');
  cy.get('[data-testid="username"]').type('user');
  cy.get('[data-testid="password"]').type('pass');
  cy.get('[data-testid="submit"]').click();
  cy.url().should('include', '/dashboard'); // Retried up to 3 times if it fails
});

Limitations of Built-in Retries#

Built-in retries only apply to individual commands/assertions, not the entire test. If the flakiness stems from the test as a whole (e.g., the login flow works 90% of the time), retries won't help. For example:

• If the API fails to return data once during the test, the entire test fails—even if retries would fix a later assertion.

To catch whole-test flakiness, we need to run the entire test multiple times.

Custom Solutions: Running a Test Multiple Times#

When built-in retries aren't enough, we need to run the entire test repeatedly. Because Cypress commands are queued and executed in a chain, traditional loops and recursive functions inside a test will not work as expected. Instead, use external scripts to run the same spec multiple times in separate Cypress processes.

CLI Scripts for Multiple Spec Runs#

Use a shell script or npm script to run the same spec file repeatedly. For example, in package.json:

{
  "scripts": {
    "test:flaky": "for i in {1..5}; do cypress run --spec cypress/e2e/login.cy.js; done"
  }
}

Run it with npm run test:flaky to execute the login test 5 separate times (each as a fresh Cypress run).

Pros: Full isolation (no state leakage between runs).
Cons: Slower (starts Cypress from scratch each time).

In-Test Loop vs. CLI Restart#

Stopping at a Threshold: When to Stop Testing#

To avoid infinite loops or unnecessary runs, define thresholds for stopping:

Setting a Maximum Run Count#

In your shell script, set a maximum number of iterations using a loop counter. This prevents the test from running indefinitely.

Stopping on First Failure#

If your goal is to reproduce the bug quickly, stop as soon as it fails. Use the || break pattern in shell scripts:

# Run until failure (useful for reproducing bugs quickly)
for i in {1..100}; do cypress run --spec cypress/e2e/login.cy.js || break; done

Tracking Results: Logging and Reporting#

To debug flakiness, log detailed results. Parse the Cypress JSON reporter output or use a wrapper script to capture pass/fail status:

# Run test 10 times and log results
for i in {1..10}; do
  cypress run --spec cypress/e2e/login.cy.js --reporter json > results.json 2>&1
  if [ $? -eq 0 ]; then
    echo "Run $i: PASSED" >> test-runs.log
  else
    echo "Run $i: FAILED" >> test-runs.log
  fi
done

Best Practices for Testing Intermittent Bugs#

1. Isolate the Test: Remove dependencies on external services (use mocks/fixtures with cy.intercept()).

   cy.intercept('GET', '/api/user', { fixture: 'user.json' }).as('getUser');

2. Clean Up State: Always reset cookies, local storage, and backend state (e.g., via API calls to /reset endpoints) between runs.

3. Optimize Speed: Run tests in headless mode (cypress run) and reduce unnecessary waits to iterate faster.

4. Run in Parallel: Use Cypress Cloud or cypress-parallel to run multiple test copies simultaneously (e.g., 10 runs at once).

Example Workflow: Fixing a Flaky Test#

1. Identify the Flaky Test: Notice a test passes 9/10 runs in CI.
2. Reproduce Locally: Use a CLI script to run it 50 times. It fails on run 12.
3. Debug the Failure: Check the logs/video (Cypress records videos by default) to see the API took 3s to respond.
4. Fix the Bug: Add a dynamic wait for the API with cy.wait('@getUser') instead of a hardcoded cy.wait(1000).
5. Validate the Fix: Run the test 100 times. If it passes all, the bug is fixed!

Tools and Plugins to Simplify the Process#

Instead of dedicated plugins, you can use npm scripts or shell scripts to run tests multiple times. The approach shown earlier in the CLI Scripts for Multiple Spec Runs section is the recommended way:

# Run a spec file 10 times
for i in {1..10}; do cypress run --spec cypress/e2e/login.cy.js; done

You can also use npm scripts in package.json:

{
  "scripts": {
    "test:repeat": "for i in {1..10}; do cypress run --spec cypress/e2e/login.cy.js; done",
    "test:until-fail": "for i in {1..100}; do cypress run --spec cypress/e2e/login.cy.js || break; done"
  }
}

• test:repeat: Runs the test 10 times regardless of pass/fail
• test:until-fail: Runs until the test fails (useful for reproducing bugs quickly)

Conclusion#

Intermittent bugs are frustrating, but running tests multiple times in Cypress can turn "unreproducible" into "fixed." Use built-in retries for step-level flakiness and CLI scripts for whole-test repetition with full isolation. By combining these strategies with careful logging and state management, you'll squash flaky tests for good.

References#

• Cypress Retry Documentation
• Cypress Best Practices for Flaky Tests
• Debugging Flaky Tests in Cypress