Trufflow: Why an Anomaly Needs Its Neighbors
Follow a latency spike through Trufflow's design to understand service-specific baselines, graph context, and the gap between an unusual event and a useful alert.
Two services slow down. Only one change looks dramatic.
Imagine a small service whose usual latency is 20 milliseconds. Today it jumps to 80. A busier service moves from 150 to 170 milliseconds. A single threshold of 200 milliseconds would flag neither, even though the small service now takes four times its usual time.
These are illustrative values, not project measurements. They reveal the question behind Trufflow's anomaly-detection design: unusual compared with what? An event becomes more informative when you know the service's history, the current load, and what its neighbors are doing.
This is the first of three connected walkthroughs. It explains the feature pipeline represented in the project's archived writeups. The companions cover inference measurement and evaluation and early warnings. The figures remain historical artifacts; this series does not present them as newly reproduced experiments.
Follow one suspicious event
From a raw metric to a decision
Choose a step to see what happens and why it matters.
Start with the measured value, service identity, and time. These define which observation the detector is trying to interpret.
The pipeline has three views of the same event. Indigo provides observations, teal builds context, amber makes a decision, and slate holds the result. The model sits inside the pipeline; it doesn't replace the work before or after it.
Give each service a meaningful point of comparison
A raw value and a deviation answer different questions. “Latency is 80ms” describes the observation. “Latency is four times this service's usual value” describes a relationship to its history.
| Illustrative service | Usual latency | Current latency |
|---|---|---|
| Small service | 20ms | 80ms |
| Busy service | 150ms | 170ms |
Neither ratio is a diagnosis. Perhaps the small service is processing a different workload. Perhaps the busy service has a strict latency budget. Context improves the question; it does not settle the answer by itself.
The archived design combines statistical summaries with temporal features. A rolling baseline can describe recent behavior, while a change feature can describe movement relative to it. A robust baseline can reduce the influence of extreme observations, but its window and update rules still need to match the workload.
Pause and predict: could a baseline learn the incident as “normal”?
Yes. If a baseline updates continuously during a prolonged incident, the elevated values can become part of its reference history. The design needs a policy for updating or freezing baselines and a way to evaluate recovery, rather than assuming that a rolling statistic always represents healthy behavior.
Add relationships, not just more columns
Suppose the small service calls a dependency whose response time rose first. Neighboring callers then begin to slow down too. That pattern offers a different explanation from an isolated spike in one service.
Graph-aware features represent relationships such as who calls whom or which nodes share a dependency. In this design, they provide context to a tabular predictor such as XGBoost. “Graph-aware” therefore does not necessarily mean that the model itself is a graph neural network.
Dependency information can make a score easier to interpret. It also introduces questions about missing edges, stale topology, and when each relationship became known. A feature available only after an incident cannot serve as an honest early-warning input.
The same event can also affect multiple downstream services. A dependency map helps explain a possible blast radius, but a reachable path is not a measured probability that every downstream service will fail. The graph is context for investigation.
Why separate preparation from prediction?
The archived pipeline uses Polars for data preparation and XGBoost for one prediction path. That separation lets you ask which stage actually dominates runtime: reading data, constructing features, running the predictor, or preparing an explanation.
Polars' lazy API can optimize a query plan before execution. That makes it useful to think about which columns and rows a feature job really needs. It doesn't imply that every feature calculation or every workload will become faster by the same factor. The Polars lazy API guide explains the distinction between constructing and executing a query plan.
The inference companion follows this boundary further. It explains why a model's events-per-second number cannot describe the time a user waits for a complete alert.
A score becomes an alert only after a decision
The predictor produces a score. An alert policy decides when that score should interrupt someone. Lowering the threshold may catch more incidents while also creating more false alarms. Raising it can reduce interruptions while missing more useful signals.
A good alert carries the evidence that makes it worth investigating: which service changed, which time window was used, which nearby observations matter, and what the score means under the chosen threshold.
An explanation layer can summarize that context. It should not turn correlations into a proven root cause or treat a rule-based check as a guarantee that every explanation is correct.
What would show that the context helps?
Compare a baseline without graph features against a version with them, keeping the data split and evaluation policy fixed. Check whether gains remain for small services and for later time periods. Test what happens when dependency information is absent or stale.
Those are proposed comparisons, not results claimed by this article. They separate the useful hypothesis—relationships can improve detection—from the evidence required to establish it for a particular dataset.
The design lesson is to choose the comparison before choosing the alarm. An event's value, its deviation from history, and its position in a system each tell you something different.
Continue the series
- Part 2: What does a fast inference number actually measure?
- Part 3: How would you evaluate an early warning?
- Archived pipeline figure, retained with its original labels; those labels are not a substitute for a reproducible benchmark record.