How to Reduce Flaky Cypress Tests

In this article you'll learn:

• What are flaky tests?
• How to identify tests that are flaky.
• How to reduce the flakiness of your tests.
• How to use SIFERS

What Are Flaky Tests?

Flaky tests are defined as those tests that exhibit the following:

• Will fail less than 100% of the time
• With no changes to the configuration
• With no changes to the code under test

Flaky tests are, well, just part of life. While we intend to write automated tests for our applications that are concise, fast, and accurately reflect the stability of the system under test, flaky tests arise.

You can read more about flaky tests at Google and how they work to mitigate this issue.

Identify Flaky Tests

There are 4 primary sources of flaky tests:

1. Concurrency
2. Non-deterministic code under test
3. Third-party code
4. Infrastructure

Concurrency is a result of the parallelization of executing tests, most commonly in your CI/CD pipeline. While executing tests in parallel decreases the time to execute the test suite, it is the most common source of flaky tests. The reason for this is that the run order is not guaranteed with running tests in parallel.

Non-deterministic code under test can further cause flaky tests. This means that when executing the code under test, given the same input, the code results in different outputs. The most common cause of this is poor state management. Another common cause of non-deterministic code is inconsistent server/backend state.

Let's look at an example of two tests.

describe('List Books Page', () => {
  let book: BookModel;

  beforeEach(() => {
    book = {
      id: uuid.v4(),
      name: 'The Lord of the Rings',
      earnings: '100',
      description:
        'The Lord of the Rings is an epic high fantasy novel written by English author and scholar J. R. R. Tolkien.',
    };
    BooksApi.createBook(book);
    AuthApi.login('Admin', 'password');
  });

  afterEach(() => {
    BooksApi.deleteAllBooks();
  });

  it('should show a list all of the books', () => {
    cy.visit('/');
    cy.wait(1000);

    cy.getByTestId(`book-${book.id}`).should('contain', book.name);
  });
});

describe('Delete Book Page', () => {
  let book: BookModel;

  beforeEach(() => {
    book = {
      id: uuid.v4(),
      name: 'The Lord of the Rings',
      earnings: '100',
      description:
        'The Lord of the Rings is an epic high fantasy novel written by English author and scholar J. R. R. Tolkien.',
    };
    BooksApi.createBook(book);
    AuthApi.login('Admin', 'password');
  });

  afterEach(() => {
    BooksApi.deleteAllBooks();
  });

  it('should delete a book', () => {
    cy.visit(`/book/${book.id}`);
    cy.getByTestId(`book-${book.id}`).find('[data-test-id="bookDeleteButton"]').click();
    cy.wait(1000);

    cy.visit('/');
    cy.wait('@getBooks');
    cy.getByTestId(`book-${book.id}`).should('not.exist');
  });
});

Let's quickly review the code above:

• In the first test, we are asserting that the books page shows the list of books in our API.
• Ideally, we want to use our real API to assert that our application is without regressions from end (client) to end (server).
• In the first test, we assert that the list of books displays as we expect.
• In the second test, we assert that a book is successfully deleted.
• In both tests, we are using the beforeEach() and afterEach() lifecycle methods to create a test book, login, and then remove all books in the API.

For those of us that use Cypress a lot, this test code should look pretty familiar.

Identify Flaky Cypress Tests

First, we highly recommend using the Flaky Tests Management suite as part of the Cypress Dashboard. This tool in incredibly valuable to identify and isolate tests that are flaky.

Second, running tests in parallel can enable us to easily identify tests that are flaky:

cypress run --record --parallel

Third, we want to identify parts of our tests that setup and tear down the state of the application. This code is most prone to causing flake.

Fourth, we want to look for timing issues in our tests. The most common culprit is using cy.wait() with a specified number of milliseconds.

Finally, well, 🤷, you probably know the tests in your test suite that are already flaky because they occasionally fail you likely just force push or merge and hope for the best.

SIFERS

Before we refactor our tests to reduce the flakiness let's learn about SIFERS. SIFERS is an acronym for:

• Simple
• Injectable
• Functions
• Explicitly
• Returning
• State

SIFERS is a replacement for the Cypress test hooks:

• beforeAll and beforeEach
• afterAll and afterEach

Rather, our tests will contain a single setup() function that is responsible for explicitly returning the state of our system under test. The setup() function is invoked within each test.

Resolving Flaky Tests

Let's refactor our tests above to reduce the flakiness.

function setup(options: { throwErrorWhenLoadingBooks?: boolean } = {}) {
  const book = {
    id: uuid.v4(),
    name: 'The Lord of the Rings',
    earnings: '100',
    description:
      'The Lord of the Rings is an epic high fantasy novel written by English author and scholar J. R. R. Tolkien.',
  };

  BooksApi.deleteAllBooks();
  BooksApi.createBook(book);

  if (options.throwErrorWhenLoadingBooks) {
    cy.intercept('GET', 'http://localhost:3000/books', {
      statusCode: 500,
      body: {
        error: 'Internal Server Error',
      },
    }).as('getBooks');
  } else {
    cy.intercept('GET', 'http://localhost:3000/books').as('getBooks');
  }

  cy.intercept('POST', 'http://localhost:3000/books').as('createBook');
  cy.intercept('PATCH', 'http://localhost:3000/books/*').as('updateBook');
  cy.intercept('DELETE', 'http://localhost:3000/books/*').as('deleteBook');

  AuthApi.login('Admin', 'password');

  return {
    book,
    teardown: () => {}
  };
}

describe('List Books Page', () => {
  it('should show a list all of the books', () => {
    const { book } = setup();

    cy.visit('/');
    cy.wait('@getBooks');

    BookListComponent.getBook(book.id).should('contain', book.name);
  });
});

describe('Delete Book Page', () => {
  it('should delete a book', () => {
    const { book } = setup();

    BookListComponent.clickDeleteButtonOnBook(book.id);
    cy.wait('@deleteBook');

    BooksApi.getBooks()
      .its('body')
      .should((books) => {
        const bookExists = books.some((b) => b.id === book.id);
        expect(bookExists).to.be.false;
      });
  });
});

Let's review our refactored tests:

• First, we defined a new setup() function that will perform all of the necessary setup.
• The setup() function returns an object containing the newly created book, as well as a (placeholder) teardown() function.
• We refactored the test to not use the afterEach() lifecycle method. It's a best practice to avoid using this method, and, rather, explicitly clean up the state before each test. As such, we're removing all books in the API when invoking the setup() function.
• We removed all instances of cy.wait(ms) that we had hard-coded previously. Now, we're referencing aliases that we can use to explicitly wait for our API.
• We refactored the assertion for deleting a book to not rely on the UI. Rather, we'll go directly to the API to assert that the book was indeed removed.

Page Objects and Component Fixtures

Here at LiveLoveApp, we're also fans of using page objects and component fixtures for our Cypress tests.

In the previous example, you likely noticed that we referenced the BookListComponent harness. Let's look at the code:

export const getBook = (bookId: string) => cy.getByTestId(`book-${bookId}`);
export const getEditBookButton = (bookId: string) =>
  getBook(bookId).find('[data-test-id="bookEditButton"]');
export const getDeleteBookButton = (bookId: string) =>
  getBook(bookId).find('[data-test-id="bookDeleteButton"]');

export const clickEditButtonOnBook = (bookId: string) => {
  getEditBookButton(bookId).click();
};
export const clickDeleteButtonOnBook = (bookId: string) => {
  getDeleteBookButton(bookId).click();
};

The component harness above enables us to isolate interactions with the DOM into a single module. We can then use this module and the exported functions to query the DOM as necessary.

Key Takeaways

Flaky tests are part of life. Yeah, I know that sounds like something your grandmother said to you.

To reduce the flakiness of your Cypress tests we recommend:

1. First, identify those tests that are flaky. Using the Cypress Dashboard makes this simple.
2. Second, identify parts of your tests that introduce flakiness through poor state management and timing issues.
3. Third, refactor your tests to use SIFERS.
4. Fourth, consider using component harnesses to isolate DOM interaction.