The Observer Effect in Observability

by Perfluencer

Observer effect is most commonly attributed to physics experiments. It’s a phenomenon of reporting different results simply because someone was watching. Does it also occur in software engineering?

Modern osbservability plaftorms let you measure your code down to the smallest pieces of code – but is it free? And is it completely accurate?

Here are a few factors to consider before you start profiling every single piece of code and surrounding systems „just in case”:

– Every instrumentation has a cost, although you may not see it in your spans, traces. It is still a piece of code that cost you CPU cycles, memory allocations, conditional resolution, system clock access etc.

– In most cases, traces are batched, transformed and dispatched asunchronously. Asynchronous does not mean free. Your CPU thread will still be affected by context switching. Spans will have to be forwarded at some point anyway. The bigger the batch size, the more memory your system will use to store them before they can be flushed.

– For critical services, chose manual over automated instrumentation. Automated instrumentation in Java is implemented via a javaagent. This agent smartly modifies the bytecode of loaded classes using a process called Transformation. Transformation is not free, happens every time your class loader reloads the class and it adds the same overhead as the manual instrumentation. With Automated instrumentation you have a limited control over your span attributes and what exactly is measured with your spans.

– Instrumentation can be your blind spot.
If you only measure the code inside your spans, then who’s measuring your instrumentation?…

…Your customers.
Relying solely on APM/OTel data can lead to many discrepancies between the real vs measured outcomes. That’s why it’s important to look at the difference between your load test results in both load testing tool and your APM because on production system, you’ll only have the latter.