Scaling Anomaly Detection to 1.06M events/sec

Saksham Kapoor | University of Maryland — College Park

Key Systems Benchmarks

Verified Throughput:1,061,348 samples/sec (XGBoost 3.1)
Inference Latency:0.94µs per sample
The "Lift":73x PR-AUC improvement over biased global baselines
Lead Time:3-hour proactive warning window at 87.5% recall

Introduction: When the Lab Fails the Real World

I remember the moment I realized our "perfect" model was failing. During the Break Through Tech AI Studio, I was staring at a K-Nearest Neighbors (KNN) baseline that boasted a 0.99 F1-score. In the lab, we were celebrating. But when I looked at how it handled our smaller microservices, I felt a sense of dread.

The model was suffering from what I now call Statistical Shadowing.

Imagine a smoke alarm designed only to trigger for skyscrapers. It is incredibly accurate at detecting massive infernos, but it stays completely silent when a small home is burning down. In a multi-tenant system, the high-traffic enterprise giants are the skyscrapers. Their massive data volume creates a "shadow" that hides critical failures in smaller apps. This isn't just a math problem; it's a production liability. This is the story of how I moved from global bias to a system that "sees" every tenant, regardless of scale.

The Infrastructure Pivot: Ditching Lazy Learners

To handle a firehose of 1.06M events/sec, we couldn't rely on "lazy" algorithms like KNN that get slower as data grows. We needed a decoupled, high-performance pipeline.

Trade-off: XGBoost 3.1 + Polars vs. KNN/Pandas

Decision: XGBoost 3.1 + Polars for constant-time inference and sub-microsecond overhead

KNN (Lazy Learner): Inference time grows with dataset size ( $O(N)$ ), failing under high load.
Pandas: Single-threaded and memory-inefficient, creating bottlenecks in feature engineering.
XGBoost 3.1: Optimized for speed with constant-time inference ( $O(d \cdot \text{trees})$ ).
Polars: Rust-based, lazy evaluation, and Arrow-based memory keep feature engineering overhead under $1\mu s$ .

We pivoted to XGBoost 3.1 for its constant-time inference and paired it with Polars for feature engineering. If you are still using Pandas for real-time ML features, you are leaving an order of magnitude of performance on the table. By leveraging Polars' arrow-based memory, we kept the total overhead under $1\mu s$ .

Figure 1: The Architecture: Moving from raw metrics to insights in under a microsecond.

The Math of Robustness: Neighborhood-Aware Alerts

The "Average" is the enemy of the anomaly detector. To solve the Skyscraper Problem, I replaced standard Z-scores with Robust Relative Deviation Scaling.

Standard averages get skewed by massive spikes. A Robust Z-score, however, uses the Median and Interquartile Range (IQR). It acts like a neighborhood-aware smoke alarm: it doesn't care if the building next door is fine; it only cares if this specific service is deviating from its own unique history.

\text{Robust Z} = \frac{x - \text{Median}_{\text{app}}}{\text{IQR}_{\text{app}} + \epsilon}

Context > Model Tuning: The Blast Radius

Most ML engineers spend weeks grid-searching hyperparameters. I spent that time building transaction topology features. In a microservice environment, no app is an island.

Constraint: The Skyscraper Problem (Statistical Shadowing)

Impact: Hiding critical failures in smaller microservices

High-traffic services dominate global metrics, creating a "shadow".
Smaller services can fail completely without triggering global thresholds.
Solution: Localized, topology-aware baselines that respect individual service scale.

I implemented a Blast Radius feature. Think of it like a domino effect in a power grid. If a lightbulb in one house burns out, it's a local issue. If the substation fails, every house goes dark. By quantifying the "downstream impact" across 3-hop cascades, we gave the model the context it needed to prioritize critical infrastructure failures over background noise.

Figure 2: Service Topology & Cascading Impact.

Figure 3: Audit: The Top Drivers of Accuracy.

Neuro-Symbolic Intelligence: Trusting the Machine

In high-stakes FinTech, "trust me" isn't an answer. Standard LLMs (like T5) hallucinate explanations 40% of the time. They might see a cost spike and blame "latency" just because it sounds plausible.

To fix this, I designed a Neuro-Symbolic Analyst. We forced the LLM to follow rigid, symbolic business rules. If the XGBoost model (the detective) finds a clue, the Symbolic layer (the judge) verifies it against the facts. The result? A 0.0% hallucination rate. We provided SREs with reports they could actually bet their infrastructure on.

The 2026 Horizon: Proactive Prevention

The ultimate goal of observability is to prevent the fire, not just describe it. Our proactive engine now identifies system degradation 3 hours before it manifests in billing spikes. By optimizing a Logistic Regression layer to process 9.76M samples/sec, we've moved from reactive detection to proactive early warning.

Figure 4: Early Warning: Predicting failure 180 minutes before system degradation.

Conclusion: Lessons from the 1-to-100 Scaling Journey

If there is one thing this implementation proved, it is that Context beats Model Tuning every time. By focusing on robust math, graph topology, and sub-microsecond latency, we've built more than a detector; we've built a security net for modern cloud infrastructure. The scaling journey from 1 to 100 isn't about the newest algorithm—it's about building systems that understand the reality they operate in.