On Accuracy of Load Tests

An honest conversation that never happened (but should, more often than you’d think)

Manager: So this is the test report. What are those numbers?

Tester: It’s response times, in milliseconds

Manager: So this represents what our customers will see?

Tester: … No

Manager: Hold on, so what’s the response times our customers will see?

Tester: We don’t know…

Manager: Alright, so what do we need to find out?

Tester: We need to run a browser-based test. JMeter doesn’t do that

Manager: Let’s use this Plugin I’ve found on the web. So will we have the full client-side response times?

Tester: … also no

Manager: So what else do we need?

Tester: Most of our users use mobile data, we need to emulate their network conditions and latency

Manager: Great, do that too. Is that it?

Tester: …

Manager: Alright, what else?

Tester: Our test data is so uniform, the database won’t even use the indexes we,ve defined for heavy traffic. We can’t predict the database latency until it hits production. Then we’ll have to fix stuff it in prod.


If you’re at least aware of the problems listed in here, you know more than most of your peers.

If you address them actively, you’re a rockstar.