Conduct load testing that measures end-to-end journey completion, not request-by-request success
Ren Iris
Introduction
From the glitchy rollout of HealthCare.gov in 2013 to unemployment sites crashing during COVID-19, users tend to have a long memory for failures to launch. Some of these failures happened because load testing occurred too late or wasn’t optimized. For those who don’t (yet) know, load testing “is a performance-testing technique where you simulate expected real-world user activity on an application or system to measure how it performs under normal and peak load conditions.”
Rob Bayliss has supported the performance and stability of resilient, surge-ready infrastructure for several high-profile digital services: Mass.gov, Massachusetts Paid Family and Medical Leave, MyMassGov, and New Jersey Unemployment Insurance. Across these platforms, his work centers on a simple promise for agencies and constituents alike: simulate real traffic, stress real user journeys end to end, and use those insights to keep critical services resilient when demand suddenly surges.
We revisited Rob’s recent Open Source Summit talk on load testing, which aims to prevent dramatically memorable crashes.
Making resilience a core requirement
Rob starts with the notion that the web can be a force for good. Broadening information access gives more people the knowledge they need to make the most informed, context-based decisions. As Rob highlighted in his Summit talk, sharing open standards and methodologies behind performant systems can materially improve people’s lives. Bringing that mindset to government technology means we need to view resilience as a core requirement, not an optional feature.
The digital chaos amid COVID-19 resurfaced public discussions on the stability of government digital services at scale. When unemployment websites were crashing across the US, the constituents who most needed help—and the staff supporting them—were locked out of the systems supposed to support them. In site-reliability engineering circles, those kinds of sudden spikes are considered “black swan” events. “It’s the nightmare scenario,” Rob said. “You get relatively consistent traffic over time, and then suddenly there’s a massive spike. You can’t really predict that; you just have to be ready to scale when it happens.”
Mighty Acorn has collaborated with NJ’s unemployment modernization effort, which offers a helpful case study. The NJ Office of Innovation has described how a flood of claims collided with a legacy mainframe and COBOL coding, complex application wording, mobile incompatibilities, and shifting federal guidelines. This resulted in delayed benefits, confused constituents, and frustrated staff. Modernization under these circumstances isn’t abstract: when someone needs to file a claim in crisis, the system needs to be available, understandable, and trustworthy.
Planning for the complexity of unemployment insurance systems and how they span several different platforms
Many interconnected applications comprise NJ’s unemployment system. Each application has its own data store and communication patterns, which ultimately all flow back to a long-running COBOL mainframe. The complexity and pace of iterative development add more pressure to testing processes.
Mighty Acorn joined the NJ unemployment insurance modernization project in 2023 to help stand up and lead a quality operations practice, including a series of automated testing capabilities. When Mighty Acorn joined, NJ tasked us with identifying, reducing, and mitigating risk, especially risks related to high traffic and ongoing change. The Commissioner at the time, Robert Asaro-Angelo, advocated for an agile, modular unemployment system—one that makes changes easy to roll out, but also recognizes that modernization is never “a one-and-done deal”—it must keep adapting to claimant needs and policy shifts.
Rob discussed work that occurred across several layers:
- End-to-end system tests that examine critical journeys across systems
- Practices to spot risks near where code changes are introduced
- Load testing for any changes that could affect stability or performance
As we modernize pieces, they shift and change. That makes testing very complicated—so much is changing, and so many systems are involved. There isn’t a single test strategy that covers everything; you need multiple approaches.
Thinking in this multi-pronged way, Rob noted, moves load testing into a broader agile system. Teams scrutinize each release, and if a change identifies a performance issue, agencies can pause, investigate further with DevOps and application teams, and make adjustments before constituents are affected.
Without a consistent, predictable load testing practice, teams have no reliable way to know when a seemingly small change has quietly degraded stability or performance. Configuration tweaks, new maintenance features, or integration changes may look safe at code review, but only show their impact when traffic spikes—exactly when constituents most need the service to work.
By making journey-focused load testing a regular part of the release process, agencies create a safety net. Changes are exercised under realistic loads before they reach production, performance regressions are caught early instead of leaking out to users, and engineers can move faster without trading off safety. This discipline is what allows Mighty Acorn to ship iterative improvements with confidence: we increase the pace of change while keeping a steady, reliable experience for people filing claims, applying for benefits, or trying to get information during a crisis.
Turning load testing insights into better experiences, not just better graphs
As Rob urged in his talk, the best way to get ahead of crashes is to plan for them. Position for long-term platform / service success by designing systems that can scale, stay running, and remain usable under stress.
For anyone new to load testing: imagine you’ve built a bridge and want to know if it supports the number of people who will cross it. You don’t test by sending people across until it collapses. You gradually add weight and see if it supports what you expect.
In load testing, software engineers do something similar, said Rob: they “simulate traffic to the website, expect it to handle normal traffic and handle peak traffic (a really busy day), and test to find where or at what level it starts to break.”
Breakpoint-level tests, dynamic flows, and graph tracking
For the NJ unemployment insurance project, the joint team tested breakpoint levels to locate where response times lagged and user journeys started to deteriorate. The team’s first tool was K6, an open-source tool from Grafana written in Go. They viewed the unemployment homepage, where at the bottom were the “waterfall” of requests in the browser’s network tab. Visiting the homepage generated a couple dozen requests.
Rob detailed the team’s first approach: “we visited each page of the site, found what requests were made, then simulated those requests 1 by 1. The application is complicated—20 or 30 pages in the full claim form. Multiply the number of requests on the homepage by the number of pages, and that’s the number of HTTP requests we needed to simulate. It was daunting, but we tried it.”
They implemented code with the following dynamic flow, with each line as a single HTTP request:
- Access the homepage
- Access the claim start page
- Send a request to start an application
- Return an application ID
- Apply that ID in further requests
The team entered all of this into a single script, said Rob, “written in a JavaScript-like language that compiles down to Go. Each function call is 1 iteration through the claim-submission process. We set a concurrency level (how much traffic), put it into an AWS CodeBuild job, and ran it. It worked.”
CloudWatch graphs for the project displayed response times that were mostly flat (under half a second), with requests ramping up, then steadying for 20 minutes, and then ramping down, with a small number of errors. Overall, Rob said, “the system looked reasonably healthy under that test.”
A new maintenance request and a gap in protocol-level testing
A few weeks later, while manually testing another feature, the team noticed a new request in the waterfall: an API endpoint for maintenance. This maintenance request was being made on every page view—potentially adding several requests to each iteration. And when the application was in maintenance mode, it stopped new applications. While the feature was developed, the testing team hadn’t updated its load test scripts. Rob learned that they were testing the HTTP requests they “thought should be there, not the requests the browser was actually making.”
This exposed a larger gap. Rob explained that “traditional load testing focuses on protocol-level testing—make one request, get one response—but [it] doesn’t account for modern web applications,” which issue dozens of requests per page to different endpoints. Additionally, said Rob, traditional load testing “doesn’t account for apps rendered differently on the server or client side, or dynamic behaviors based on those choices.”
Browser-based load testing: Playwright + Artillery
To address this gap, the team decided to use the browser itself for load testing. They reused code and page-object models from Playwright, a browser automation tool with an end-to-end test framework. As Rob noted, Playwright gave the team “coverage aligned with real user flows”; because they’d be “driving the browser,” they wouldn’t miss new requests. “If someone introduced a new HTTP request, [they’d] still be testing the webpage, not a stale list of expected requests.” Around this time, added Rob, both K6 and Artillery (another open-source load-testing tool) “released support for browser-based load testing using Playwright. Artillery had a clear advantage: it’s JavaScript-based and uses the same Playwright APIs we were already using, while K6’s Go-based APIs were different. So we chose Artillery.”
The team also stopped “focusing on individual HTTP requests.” They began focusing on and measuring the user-journey success rate of “whether the interaction we care about—the full claim submission—can be completed.”
This also lets the team “handle application behavior like retry logic.” Rob said that with “browser-based load tests, we don’t care if a specific request fails temporarily, as long as users aren’t affected; retries can succeed behind the scenes.” Conversely, the failure of a single, critical request could stop an entire user journey.
“We care about the journey completion,” emphasized Rob, “not every request.”
Conclusion
Load testing for complex government systems is complicated, but it’s among the most effective ways to protect trust when demand suddenly surges. By testing an application in the ways constituents use it—driving browsers through end-to-end journeys, and making those tests a predictable part of every change—teams can catch hidden regressions before they reach production. This keeps critical services available and understandable under stress, ensuring that people can reliably complete the tasks that matter most, even on the busiest day of the year.