Overview

We’ve been using and recommending our High ROI Testing approach (which is an E2E style of testing in React and Redux) for a good while now and I just fixed a production bug using this approach that I feel really codify’s it’s value and wanted to share.

The Bug’s Challenge

I was assigned a bug on a project that requires a large number of steps to reproduce which is shown below.

As you can see we have to navigate to 6 different pages to get this bug to reproduce as it deals with our infrastructure components that control our busy spinner while allowing us to do things like

• resolve request from our client side cache
• silently load request in the background

This infrastructure code is getting out of wack after we navigate around the site a bit and there are a number of challenges to being able to quickly fix this bug including:

1. We are working in a dev build which doesn’t have API caching so each page load can take 30 seconds up to a few minutes
2. We have to do an awful lot of clicking around in the app to reproduce this scenario
3. We can pretty easily limit the number of rows being fetched in the dev server to reduce the page load speed but we would have to do that across a lot of a API calls which would require a lot of setup time

Calculating the Overhead Time

TLDR: It will cost us around 1.5 hours of overhead to manually test and debug our code not to mention the mental strain of having to click around and wait so much.

Let’s assume we spend 20 minutes tracking down all the APIs to limit the rows being fetched and then another 20 minutes cleaning up the code after. And then assume that we get the overhead that it takes to run the tests down to 10 seconds per page on average for the async loading time and time it takes for us to click the menus.  Let’s also assume that it takes us 30 debug runs to isolate and then fix the bug. We have to go through 6 page transitions for each test and so that means we have 30 x 6 x 10 =  1800 seconds or 30 minutes of overhead where we are just navigating around the app. Then we add in the 20 minutes upfront for limiting the API server calls to reduce our wait time and another 20 minutes to clean up. So this gives up a total overhead of 30 + 20 + 20 = 1h 20 mins. And then just image how hard it would be to maintain your train of thought and flow as you are clicking endless through this series of steps.

How High ROI E2E Testing Helped

Because we have been using the High ROI E2E Testing approach for a good bit we already have reusable steps that we can string together to quickly create this scenario in our tests. It took me about 5 minutes to write this test and it reproduce the issue first try.

describe('Given we are on the items page ', () => {
  afterEach(() => {
    fetch.mockReset();
    resetRedux();
    resetHttpRequestCache();
  });
  describe('and we navigate around a good bit ', () => {
    it('Then the busy spinner is turned off', done => {
      expectNoErrors(done);

      return mountApp()
        .then(navigateToRetailers)
        .then(navigateToPromotions)
        .then(navigateToRetailers)
        .then(navigateToGroups)
        .then(navigateToRetailers)
        .then(({store}) => {
          expect(store.getState().busyIndicator.busyCount).toBe(0)
        })
        .then(() => done())
        .catch(done.fail);
    });
  });
});

The test took just under 5 seconds to run and here’s the test output.

Error: expect(received).toBe(expected)

Expected value to be (using ===):
  0
Received:
  1
    at then.then.then.then.then.then( ...removed for brevity

Recalculating the Overhead Time

It took a few hours to fix this bug but I was able to move quick and stay productive as each thing I tried only required 5 seconds of overhead to run and see the results. If we recalculate our over head then it took about 5 mins to write the test and then 5 seconds per test so thats 30 attempts at 5 seconds each or 150 seconds. So with our setup we are at about 7.5 minutes of overhead using the same math that got us to 1.5 hours of overhead for manual testing.