How to Select and Maintain CI Runners for Playwright

If you’re running end-to-end tests with Playwright, you’ll know that runners are a crucial part of your CI/CD pipeline. After all, even well written tests can turn flakey due to poorly configured runners.

Whether it’s GitHub Actions, GitLab, Jenkins or another tool, there’s challenges to selecting and maintaining your runners which we’ll cover in this guide.

Goldilocks runners - Not too big, not too small

Selecting the right runners is a matter of balancing cost and performance.

Over-provisioning the size or number of runners can result in excessive costs without any added performance benefits. On the other hand, insufficient resources can slow down test execution and lead to flaky testing, especially with large test suites or memory leak issues.

Poorly configured runners can also lead to flaky or failed tests, making it difficult to pinpoint issues with the actual application and ultimately slowing down the overall development cycle.

How to select the right instances

There are four key factors for selecting the right instances.

Assessing RAM requirements

Assessing the resource requirements of your Playwright test suite is a complex task. Determining the exact amount of RAM needed per parallel test process isn't straightforward, especially when dealing with varying test scenarios and workloads. 

While allocating 4-8 GB of RAM per process is a general guideline, identifying the precise requirements often involves trial and error, extensive monitoring, and a deep understanding of your test suite's behavior. 

Failing to correctly assess these requirements can lead to frequent resource exhaustion errors, like out-of-memory (OOM) crashes, which are not only disruptive but difficult to diagnose and resolve.

Optimizing CPU performance

Playwright tests can be CPU-intensive, particularly when running in headless mode, making it important to ensure that your runners have adequate processing power. 

Multi-core CPUs are beneficial for running tests in parallel, but achieving optimal performance requires attention to detail. For example, you’ll need to configure the correct number of parallel test processes to match the available cores to avoid overloading the CPU. 

While compute-optimized instances can offer the enhanced CPU performance needed, balancing this against cost efficiency is crucial. Careful monitoring and occasional adjustments are required to strike the right balance between performance and resource utilization, ensuring that your tests run smoothly without unnecessary overhead.

Storage and auto-scaling considerations

Storage capacity and speed are crucial for ensuring efficient test execution, especially when dealing with file uploads, downloads, or large log generation. 

Ensuring that your CI runners have sufficient disk space with fast I/O, such as SSDs, helps avoid bottlenecks during these operations. 

Implementing auto-scaling adds flexibility in managing varying workloads by dynamically adjusting runner capacity based on demand. This approach allows you to efficiently handle peak loads while maintaining resource efficiency during lower demand periods. With thoughtful configuration and regular monitoring, you can create a robust setup that keeps your CI pipeline running smoothly under all conditions.

Monitoring and adjusting runners

Monitoring and adjusting runners for optimal performance is an ongoing, resource-intensive task. Tools like Grafana and Prometheus are essential, but effectively utilizing them to track and interpret CPU, memory, and disk usage requires significant expertise. 

Making informed decisions about when to scale up or down, and which instance types to use, involves sifting through a vast amount of data, identifying trends, and acting on insights that are often subtle and not immediately obvious. The process of keeping your runners adequately provisioned while avoiding wasteful over-provisioning is a continual challenge that requires a proactive and informed approach.

Signs it’s time to upgrade your runners

Over time, you may notice your Playwright tests taking longer to complete, or worse, encountering intermittent failures. These are often early indicators that your current CI runner setup is no longer sufficient. Here are some specific signs to look out for:

1. Increased test execution time: If your test suite starts to take significantly longer to execute, this could be due to resource bottlenecks. An under-provisioned runner may struggle to keep up with the demands of a growing test suite, leading to longer wait times for test results. This is especially true if your tests are running in parallel and the available CPU or memory is insufficient to handle the load.
2. Frequent test failures: Test failures that are not reproducible locally could indicate that your CI runners are experiencing resource contention. This can lead to flaky tests, where tests fail sporadically due to factors unrelated to the code being tested. If you notice an uptick in such failures, it might be time to reconsider your runner configurations.
3. Out-of-memory errors: As your test suite grows or as the complexity of individual tests increases, the memory footprint of your tests may exceed the capacity of your runners. If you start encountering out-of-memory (OOM) errors, this is a clear signal that your runners need more memory or that you need to move to larger instance types.
4. Slow feedback loops: The purpose of CI is to provide fast feedback to developers, allowing them to address issues quickly. If your runners are underperforming, this feedback loop can become sluggish, slowing down the entire development process. This can lead to frustration among developers and delayed releases.
5. High utilization metrics: Regular monitoring should include tracking the utilization of CPU, memory, and disk I/O on your runners. If these metrics consistently show high utilization (e.g., above 80-90%), it’s likely time to upgrade to more powerful instances or increase the number of runners to distribute the load more effectively.

Planning the upgrade

When you recognize these signs, it's important to plan the upgrade process carefully. Rushing into changes without fully understanding the implications can lead to overspending on infrastructure or failing to resolve the underlying issues. Here’s a step-by-step approach:

1. Analyze test suite growth: Assess how your test suite has grown over time in terms of the number of tests, the complexity of those tests, and their resource demands. This will give you a clearer picture of what kind of upgrade is necessary.
2. Review resource usage data: Use monitoring tools to review historical data on CPU, memory, and disk usage. Look for patterns that indicate when and where resources are being maxed out.
3. Evaluate instance types: Based on your findings, evaluate different instance types that offer the right balance of CPU, memory, and storage for your needs. Consider whether a shift to more powerful instances, additional runners, or a combination of both is required.
4. Test the new setup: Before fully committing to the upgrade, it’s wise to test the new setup with a subset of your test suite. This allows you to identify any issues and make adjustments before rolling out changes across your entire CI environment.
5. ‍Implement gradually: Implementing the upgrade gradually, such as by switching over to new runners for specific jobs or pipelines first, can help minimize disruptions. Monitor the impact closely and be ready to make further adjustments as needed.

Skip the hassle with our pool of managed browsers

Maintaining CI runners is an ongoing challenge. As your test suite continues to evolve and your application's complexity grows, the need for regular adjustments and scaling becomes a constant demand. 

To avoid the time drain of maintaining runners, try Browserless.io.

Our managed pool of Chrome, Firefox and Webkit browsers are ready for connecting to with Playwright or using our REST APIs. You don’t need to mess around with autoscaling workers or sharding tests, just use our pool of concurrent browsers. Run your existing scripts, or use our APIs for tasks such as taking screenshots or running Lighthouse tests.

Try it yourself today with our free trial.