In this post, I’m covering the task of integrating an end-to-end testing framework into a React project and its GitHub pull requests. When comparing testing frameworks, the two key points I look for are: a) the technology solves our problems, and b) it is simple enough to attract others into writing tests. Sometimes abstraction can lead to over-engineering, and before you know it, your testing framework needs a 300-page user’s manual to operate. In my mind, a project strikes that balance when it’s actively (and happily) contributed to.
Of the frameworks, Playwright sticks out the most to me for being cross-domain friendly. It started as a fork of another crowd pleaser, Puppeteer. The library playwright-cli is a utility tool with a code generator for recording tests. And as Jest is already in our project, I’ll use that for running the tests.
Let’s Get Started!
Note: This guide was written on a Mac, so there’ll be minor differences if you’re on a Windows machine (for example: replace “Terminal” references with “Command Prompt”).
1. Install Dependencies
Let’s install all of the plugins at once and I’ll reference what the devDependencies are for in later steps. Both npm and yarn commands are supplied, but only one needs to be executed.
npm i --save-dev jest-html-reporters jest playwright playwright-cli jest-playwright-preset eslint-plugin-jest-playwrightyarn add --D jest-html-reporters jest playwright playwright-cli jest-playwright-preset eslint-plugin-jest-playwright2. Running Configurations
jest.e2e.config.js
You can get away with a slimmer configuration, but in case you want to run with this configuration, the Jest documentation gives a good description on what each line does.
As you determine what an average time for your tests are, you can play around with slowTestThreshold to detect when scripts are running too long. This can be a sign that they’re becoming too complex or having too much overhead.
It’s worth pointing out the addition of jest-html-reporters at the bottom of this file. This generates the report for you after the run completes.
module.exports = {
verbose: true,
preset: 'jest-playwright-preset',
testMatch: ['**/?(*.)+(spec).js'],
testPathIgnorePatterns: ['/node_modules/'],
setupFilesAfterEnv: ['./jest.e2e.setup.js'],
reporters: [
'default',
[
'jest-html-reporters',
{
publicPath: './test-results/',
filename: 'report.html',
expand: true
}
]
]
}jest.e2e.setup.js
This script is called before each test is run. As I continue to add tests to this project, I’ll be looking at using slowTestThreshold and this timeout as knobs for catching slow tests and killing hanging ones.
jest.setTimeout(120000)jest-playwright.config.js
In jest.e2e.config.js, you may have noticed the preset of “jest-playwright-preset”. This file is the configuration for that plugin, which takes on a heavy lift. It starts the local server, launches the necessary browsers, and handles the creation of browser, context, and page objects for tests. You can read more about its capabilities on the Jest Playwright project page.
module.exports = {
browsers: ['chromium'], // other available options ['firefox', 'webkit']
exitOnPageError: false,
launchOptions: {
headless: true
},
serverOptions: {
command: '',
port: 3000,
protocol: 'http',
usedPortAction: 'kill',
launchTimeout: 60000
}
}.env.test
I used a test environment configuration for it to run completely headless locally. It’s fine if you don’t want this, but a browser will launch when the server starts.
BROWSER=none.eslintrc (or wherever you’re configuring eslint)
Since jest-playright-preset handles the creation of the browser, context, and page, eslint gets confused over the no_def rule. This plugin defines those variables, so eslint doesn’t trip over them.
extends: [
... other plugins,
'plugin:jest-playwright/recommend'
]<testDirectory>/.eslintrc
Import commands can’t be used outside of modules. There are ways to adjust the configuration for babel to properly transpile imports to require statements. I felt there was less value when the project is a create-react-app project and the babel configuration is managed. There are still ways to inject it, but I chose this as a path forward for now. Just be sure this file is in the testing directory, so it doesn’t affect other parts of your project.
{
"rules": {
"@typescript-eslint/no-var-requires": 0
}
}3. Setting up custom scripts
For this project, I set up two scripts in my package.json file:
"test:e2e": "DEBUG='pw:api,pw:browser*,pw:protocol*' NODE_ENV=test jest --config=jest.e2e.config.js"test:record": "npx playwright-cli codegen http://localhost:3000"test:e2e – Runs my tests using the custom configuration. The DEBUG section is optional, but helps dearly with troubleshooting inside GitHub Actions.
test:record – This command kicks off the test recorder and starts the browser at http://localhost:3000. The server must already be running.
4. Writing your first test
In this test, addAttach is used from the Jest reporter plugin for attaching a screenshot to the test report. Since it’s within a catch statement, it’s only called when an error occurs. The error is then thrown again, so the test can carry on as it normally would. This would be good functionality to pull into an abstract class as it’d most likely apply to all tests in the project.
<testDirectory>/**/test.spec.js
const { addAttach } = require('jest-html-reporters/helper')
describe('Application Launch', () => {
it('page loads and navigates to dashboard', async () => {
try {
const response = await page.goto('http://localhost:3000/login')
await page.fill('input[id="username"]', 'demoUser')
await page.fill('input[id="password"]', 'badPassword')
await page.click('text="Login"')
expect(page.url()).toMatch('/dashboard')
} catch (ex) {
const screenshot = await page.screenshot()
await addAttach(screenshot, 'Screenshot at time of failure')
throw ex
}
})
})5. Running the test recorder
The test recorder is great for the arranging steps in the test. The test:record command in the package.json file is for triggering this. You call it with one of the following:
yarn test:recordor
npm run test:recordThe browser launches and begins listening immediately. As you interact with the website, your actions are converted to javascript and displayed in the terminal.
The plugin is very ambitious with capturing a lot more than you’re probably requiring. I suggest cleaning it up a bit when you’re copy+pasting into your test file.
6. Running the tests locally
The tests can now be run with
yarn test:e2eor
npm run test:e2eIf it fails to find your file, make sure its name ends with .spec.js (for example, test.spec.js) or modify the testMatch in jest.e2e.config.js.
Typescript is supported with the help of ts-jest. I opted out of it for tests for a couple of reasons:
– Our project depends on [email protected] and it requires [email protected] while ts-jest requires [email protected] It’s a rabbit hole of updates that didn’t seem worth it at this stage. The upgrades will come, but I’d rather separate that from this work.
– The test recorder outputs javascript. It seems like unnecessary work to convert to Typescript without much gain in the e2e tests.
7. GitHub Action
This is the part where things got fun. GitHub Actions (GHA) are written in .yml format and placed in your <rootDir>/.github/workflows directory. You can search for any of the actions listed below and get detailed information about them.
This action below does the following:
- Loads a docker image of ubuntu-latest
- Grabs the latest code
- Sets up node
- Installs browser dependencies (microsoft/[email protected])
- Runs yarn to get project dependencies (There were still missing dependencies upon running for me. I discuss this later in “Troubleshooting GHA”)
- Tests are run, which outputs the test report
- If the test has any failures, the test report is uploaded to GHA artifacts. To find where the artifacts end up, check out the Upload-Artifact project page
.github/workflows/e2e.yml
name: E2E Tests
on:
pull_request:
env:
NODE_VERSION: 14.15.0
jobs:
e2e:
name: e2e
runs-on: ubuntu-latest
steps:
- name: Check out code
uses: actions/[email protected]
- name: Setup node
uses: actions/[email protected]
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup dependencies for playwright/browsers
uses: microsoft/[email protected]
- name: Install dependencies
run: yarn
- name: Run additional dependencies for GAH
run: yarn add playwright-chromium @babel/plugin-transform-typescript nyc
- name: Start local service and run tests
run: yarn test:e2e
- name: Push test report to artifacts
uses: actions/[email protected]
if: failure()
with:
name: Test Results
path: test-results/8. Running GHA locally
The project Act was designed for running GitHub Actions locally. One gotcha is the Docker image they use by default is a slimmed down version of what GitHub Actions use in production. There is an alternative you can use by appending a parameter, however the image is 18gb in size. There’s also a high chance though if you also use the microsoft-playwright-github-action, you’ll need it after encountering the following issue:
“::error::Unable to locate executable file: sudo. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also check the file mode to verify the file is executable.”
To use this image, pass this parameter when executing Act:
-P ubuntu-latest=nektos/act-environments-ubuntu:18.04Hopefully this requirement changes with plugin versions later than v1.44 or in the slim Docker image.
Here is an example call for Act if you want to trigger pull_request actions with the large Docker image:
act -P ubuntu-latest=nektos/act-environments-ubuntu:18.04 pull_requestOr if you want to run only the e2e action:
act -P ubuntu-latest=nektos/act-environments-ubuntu:18.04 -j e2e9. Troubleshooting GHA
In the test:e2e script above, there’s a prefix of “DEBUG=’pw:api,pw:browser*,pw:protocol*'”. This more or less spams the terminal with information as your tests are running. The pw:api gives beginning and ending action information like “navigating to…” and “navigated to…”. If that’s all you’re looking for, then you can remove browser and protocol. Those come in handy when working through an issue and you’ll most likely be asked to post that information if you’re reaching out to forums for help.
Our tests had encountered issues with launching our site in the Docker image. It gave an error of missing libraries, but the tests didn’t fail fast. All of a sudden, it’d just stop abruptly. This behavior is what triggered the step of “yarn add playwright-chromium @babel/plugin-transform-typescript nyc.” There are certainly better ways to handle this, so just make sure your Docker image is getting the dependencies it needs to launch your server.
10. Summary
I’m curious to know others’ opinions on the Playwright framework, as well as any other struggles they had with getting GHA set up. So far I like it. I also hope this brain dump has helped accelerate this process for others!
Interested in learning more about the world of Armory? Check out our blog or contact us!
