Embracing E2E testing with Cypress

Photo by Sigmund on Unsplash

I’ve worked extensively with end-to-end testing in the past, enough to deter me from using it owing to its fragility.

I recently decided that OnCare’s at the point of having a sufficiently advanced set of features that we can’t continue developing with confidence without an E2E testing suite. We of course have unit tests but these don’t quite cut it for testing system integration.

A previous employee started exploring implementing Cypress but canned the project for various reasons. This prior effort was dusted off (some projects are worth hoarding) and served as a substrate to work off. What I did love about this original project was how good the Cypress developer experience was relative to what I’d used in the past with Cucumber and Selenium.

Some prescient issues were apparent;

1. How to point Cypress to a running web server allowing for per-branch test assurance. It’s too late to get feedback that you’ve broken something when you’ve merged it into a staging/integration environment.
2. How to avoid exposing security loopholes by adding backdoors for the test suite to mutate system and DB state. E.g you need to programmatically create DB entities to speed up the process or fill gaps that are usually run manually.

Isolated E2E testing environments

Background; we’re a Django & MySQL shop. Prior to working on this project, we had 3 environments to work with: experimental, staging and production. There’s real overhead and cost associated with each environment so I wasn’t willing to multiply this complexity on each branch or commit by spinning up a new infrastructure complete with DB, Redis, managed secrets, S3 buckets etc.

I had a brainwave; keep it simple — treat CI as another dev environment with some minor adjustments. We use CircleCI which has been already configured to set up a MySQL instance for unit testing so I could make use of this for the DB backend. Another feature of CircleCI is that you can run a background task whilst executing a test/operation so I took advantage of that:

Background task definition:

python -m venv ve
source ve/bin/activate
pip install -r requirements.txt
# ... Some Webpack stuff for static assets
python manage.py collectstatic --noinput
./scripts/import_seed_sql.sh # snapshot of good local DB
python manage.py migrate
APP_ENV=CI python manage.py runserver

I’ve essentially listed the steps required to run a dev environment locally (although we mostly use docker-compose) which serves as being good enough. We can now introspect localhost:8000 on CI 🎊.

Adding a security backdoor

Cypress knows nothing about the inner workings of your app, it just loads web pages and interacts with them.

I wanted a way to easily say: “Generate me a care agency with these clients and care workers” as quickly as possible. I could have followed the E2E philosophy of emulating user behaviour but this would have been very taxing and repetitive. This led me to create some Django views that allowed for creating new agencies and other entities such as users. E.g:

POST /cypress/create-agency
{
  "agency_name": "Florence & the E2E machine",
  "care_workers": [{"email":"florence@machine.com", "name": "Florence"}],
  "clients": [{"name":"Well being"}],
  "is_magical_feature_enabled": true
}

Now obviously this isn’t something I want on live web servers as it serves as an obvious backdoor to exploit.

After lots of brainstorming and cunning custom auth hacks, I had another brainwave to keep it simple; conditionally load the Cypress-related views, URLs etc only in a testing environment:

# settings.py

INSTALLED_APPS = [
  ...
]

if settings.APP_ENV in ["DEV", "CI"]:
  INSTALLED_APPS += ['cypress']


# urls.py
urlpatterns = [
  ...
]

if settings.APP_ENV in ["DEV", "CI"]:
  urlpatterns += [path("cypress/", include("cypress.urls")),]

This conditional loading of the Cypress Django app means we can be confident in adding risky functionality to satisfy smooth testing

I was chuffed with these two points coming together as it made writing and running tests a doddle.

It does still pose the risk that there’s bundled code that’s not accessible but a risk I’m willing to take.

When things go wrong

Once this was set up and running, encountering failed tests started becoming tedious to understand why it was failing. Useful as it was knowing which test failed, it required opening Cypress up on the correct branch locally to do the investigation work.

I then found a feature of Cypress itself that would output videos of each test run and screenshots at the point of failed tests. This is fine on your dev machine but more cumbersome on a CI environment as you need somewhere persistent to host the files. Thankfully I found Circle’s artifacts feature which allows you to dump files for post-analysis, perfect for this use-case.

This was as simple as tweaking my Circle manifest file as Cypress is aware of the CIRCLE_ARTIFACTS env setting:

- environment:
    ...
    CIRCLE_ARTIFACTS: /tmp/artifacts
- store_artifacts:
    path: /tmp/artifacts

Wishlist for the future

Short of the tests being written for me based on replaying common behaviours (something something AI), I’d love:

• Failed test notifications to include the screenshot of the failure and direct links to the hosted videos.
• Indications of common flaky tests, by looking at a sample of > 5 runs
• Tips on how tests could be written better (Cypress specific). I guess a linter of some description.

Disclosure; I have no affiliation with Cypress or Circle.