One of the key features of Playwright is the ability to run tests in parallel using a technique called sharding. This allows you to divide your test suite into smaller pieces and run them on different machines or browser instances, which can significantly speed up your test execution. However, the main limitation is that when you run tests in parallel like this, you end up with multiple test reports that need to be combined into a single report in order to get a complete overview of the test results.
In this post, we’ll show you how to merge Playwright reports within a GitHub Actions workflow.
Sharding is a powerful technique for parallelizing test execution, and it can be especially useful when running tests on GitHub Actions. GitHub Actions is a continuous integration and delivery platform that allows you to automate your software development workflow, including running tests and deploying code. One of the key benefits of GitHub Actions is that it provides access to a wide range of computing resources, including multiple machines and browser instances, which makes it well-suited for running tests in parallel.
To use sharding with GitHub Actions, you can set up a matrix job in your workflow file. A matrix job allows you to define a set of variables that will be used to create multiple jobs, each with a different combination of values for those variables. For example, you might want to run your tests on different combinations of operating systems, browser versions, and screen sizes. To do this, you can define a matrix job in your workflow file like this:
This workflow will create a matrix of 5 jobs. Each job will run on a different machine or browser instance, and you can use Playwright's sharding feature to divide your test suite into smaller pieces and run them in parallel on these different instances.
To use sharding with Playwright, you can specify the "--shard-count" and "--shard-index" command-line arguments when running your tests. The "--shard-count" argument specifies the total number of shards you want to create, and the "--shard-index" argument specifies the index of the current shard (starting from 0). For example, to divide your test suite into five shards and run the third shard on a particular machine or browser instance, you could use the following command:
When you run your tests like this, Playwright will create five shards and run the tests in each shard in parallel on different machines or browser instances. Once all of the tests have finished running, you will end up with multiple test reports, one for each shard.
One additional piece of GitHub Actions functionality that is critical is "needs", which allows you to specify dependencies between jobs in a workflow. This can be useful when you want to ensure that certain jobs are completed before others are started. For example, when merging Playwright reports, it is important to make sure that all of the jobs that generate the reports have already completed before you start the merge job.
To do this, you can use the "needs" keyword in your workflow file to specify that the merge job depends on the test jobs. This will ensure that the merge job is not started until all of the test jobs have finished running. For example, you might have a workflow that looks like this:
In this example, the "example1" and "example2" jobs are responsible for generating the Playwright reports, and the "merge" job depends on them. This means that the "merge" job will not be started until both of the "test" jobs have completed. This is important because it ensures that all of the reports have been generated before the merge job starts, which is necessary for the merge to be successful.
Unfortunately, Playwright doesn't currently have native support for merging reports after sharding. However, the Playwright team is actively working on this feature, and there is an open thread on GitHub discussing the various approaches that have been proposed. In the meantime, it is possible to use a third-party library called "merge-playwright-reports" to achieve this.
To install "merge-playwright-reports", you can run the following command in your terminal:
npm install playwright-merge-html-reports --dev
This will add a few files to your project, including a file called "merge-reports.ts" which contains the function that does the actual merging. To use this function, you just need to run it in your terminal, passing in the file paths of the reports you want to merge as arguments. For example, if you have a Playwright test suite that has been sharded into five different shards, and each shard has produced a test report with a name like "playwright-1.html", "playwright-2.html", etc., you can merge all of these reports into a single report using the following command:
This command will run the "merge-reports" function and pass in the paths to all of the individual test reports as arguments. The function will then combine all of the reports into a single report, which you can view in your browser.
One thing to note is that the "merge-reports" function only works if it can find all of the reports you want to merge. If there are any reports missing, the function will fail. This can be a bit of a headache, as it means you have to make sure to adjust the list of reports whenever you adjust the number of shards or the infrastructure you are using. However, the outcome is worth the hassle, as it allows you to combine all of your test reports into a single, easy-to-read report.
In conclusion, while Playwright doesn't currently have native support for merging reports after sharding, it is possible to use a third-party library called "merge-playwright-reports" to achieve this. This library is easy to use and can save you a lot of time and effort when dealing with large test suites that have been sharded into multiple reports. We hope that the Playwright team will eventually add native support for merging reports, which would make this process even easier and more reliable. In the meantime, using "merge-playwright-reports" is a good way to combine multiple test reports into a single report when running tests with Playwright and GitHub Actions.
%20(1).svg)