Meta News

Performance engineering has spent years relying on familiar metrics such as averages, P95 and P99 latency. They are convenient, easy to place on dashboards and useful for quickly understanding whether a service is meeting its objectives. But Adrian Cockcroft argues that modern distributed systems have become too complicated to understand through a handful of percentile measurements.

The veteran engineer, whose career has included Sun Microsystems, eBay, Netflix and Amazon, believes engineers should pay much more attention to the full distribution of latency. His argument comes at an interesting moment, as AI-assisted coding is also making it easier to build the specialized analysis tools required to investigate performance in greater detail.

Percentiles improved performance monitoring because averages frequently hide poor user experiences. A service can have an acceptable average response time while a small but important percentage of requests take several seconds.

But P99 still represents only one point in a much larger distribution.

Modern applications contain many different execution paths. Some requests hit caches while others reach databases, external APIs or additional services. Those different paths can produce distinct groups of response times, creating multiple peaks inside the latency distribution.

If engineers monitor only P99, those differences disappear behind a single number. A change in P99 may indicate that something became slower, but it could also mean that the proportion of requests following a slower execution path increased.

Understanding that distinction can dramatically change where engineers investigate.

Cockcroft has been experimenting with techniques that identify peaks in latency distributions and track how they evolve.

Consider an application where cached requests normally complete in 20 milliseconds while cache misses require 180 milliseconds. If the percentage of cache misses suddenly increases, P99 may deteriorate even though neither execution path actually became slower.

Looking at the full distribution makes that behavior much easier to recognize.

Instead of asking only whether P99 increased, engineers can examine whether an existing latency peak moved, whether a new population appeared or whether traffic simply shifted between existing populations.

That provides a more useful picture of what the system is actually doing.

Cockcroft describes performance engineering as a process of progressively increasing magnification.

Engineers can begin with high-level metrics to identify unusual behavior, examine latency distributions to isolate the affected workload and then move into distributed traces and profiling when deeper analysis is required.

The objective is not to collect every possible metric. It is to use one level of observability to determine where the next investigation should happen.

This becomes particularly valuable in modern cloud architectures, where a single user request may cross numerous services, databases, queues and external APIs before completing.

The other important part of Cockcroft’s argument involves generative AI.

Performance engineers frequently encounter questions that existing observability products were never designed to answer. Historically, creating a custom analysis tool might require enough development effort that engineers simply worked around the limitation.

LLMs change that calculation.

Cockcroft has been using AI to rapidly create specialized tools for analyzing performance data, including software capable of identifying and tracking peaks across latency distributions.

The significance is not simply that AI can write code. It is that engineers can now afford to create temporary, highly specialized software for individual investigations.

A tool might exist only to answer one performance question. Once the analysis is finished, maintaining that software may not even matter.

This represents an interesting use of AI-assisted development.

Much of the discussion around AI coding focuses on whether agents can build applications or replace parts of traditional software development. Cockcroft’s approach is different: AI becomes an engineering instrument.

An engineer identifies an unusual pattern, generates a small tool to explore it, examines the results and then goes deeper where necessary.

That could make performance engineering considerably more exploratory. Instead of being limited by whatever visualizations an observability platform happens to provide, engineers can increasingly create the analysis they need for the problem in front of them.

The broader lesson is that AI may influence software performance in two directions at once. AI workloads themselves create new infrastructure challenges, but AI-assisted coding also gives engineers new ways to investigate those systems.

And as Cockcroft’s argument about P99 suggests, those tools may be particularly useful for revealing something dashboards often hide:the interesting behavior is not always in the percentile. Sometimes it is in the shape of the data itself.