Trufflow: A Million Events per Second Can Still Mean a Waiting User
An illustrated guide to batching, preparation cost, and model choice—and why throughput, amortized inference time, and alert latency need different measurements.
One fast number, two very different experiences
Imagine a predictor that processes a batch of 10,000 events in 10 milliseconds. Dividing 10,000 by 0.01 seconds gives one million events per second. Dividing 10 milliseconds by 10,000 gives one microsecond of amortized processing time per event.
Did the first event get its answer in one microsecond? No. It may have waited for the batch to fill, then for preparation, prediction, and result delivery.
This illustrative calculation explains the question behind the second Trufflow walkthrough: what exactly did the stopwatch include? The archived project writeups describe XGBoost and lighter logistic-regression inference paths, batching, and data-layout experiments. Here, we'll use those design choices to understand a measurement, rather than combine their headline rates into a single performance ladder.
Start with the feature-pipeline walkthrough if you want the context before the speed discussion.
Follow the event's waiting time
Four stages between arrival and an answer
Choose a step to see what happens and why it matters.
An event may wait for available capacity or enough neighbors to form a batch. That wait matters even if the later model call is fast.
Every stage contributes to the experience. Amber represents waiting, indigo preparation, teal prediction, and slate delivery. A model-only benchmark times the predictor; an end-to-end measurement follows the entire path.
Batching changes the unit of work
Calling a predictor once per event repeatedly pays per-call overhead. Passing a matrix of events can amortize some of that cost across the batch. But collecting the matrix also takes time and memory.
Pause and calculate: what if that same batch spends 90ms waiting before its 10ms prediction?
The predictor still completes 10,000 events in 10ms. An event that waited the full 90ms has now spent at least 100ms in the workflow, before any additional delivery cost. The predictor's amortized microsecond remains a valid calculation for its timed stage; it is not that event's response latency.
This is why an online service often needs both a maximum batch size and a maximum wait policy. A full batch may help throughput, while a small arriving workload needs a way to proceed without waiting indefinitely.
Measure preparation before rewriting the model
Prediction may be only part of the work. Parsing records, selecting columns, converting types, allocating arrays, and copying features can all sit outside a narrowly timed model call.
Before changing algorithms, measure those stages separately. Reusing a prepared matrix answers a useful predictor question, but it removes preparation from the experiment. Allocating it inside the timed region answers a different question.
Memory layout belongs in this discussion. How features are arranged can affect a particular implementation's access patterns. A column-oriented representation that helps one operation is not automatically the best layout for every predictor. Compare layouts with the actual workload rather than treating “column-major” as a universal optimization.
A lighter model is a new quality tradeoff
The archived writeups discuss both XGBoost and logistic regression. Moving between them changes the predictor, not merely a tuning parameter. It can change the decision boundary, calibration, error distribution, and suitability for a chosen threshold.
A simpler model may be sufficient when the features carry useful structure. That is a hypothesis to test on the same evaluation data, with an explicit operating point. A faster result is useful only if its errors are acceptable for the task.
Don't multiply speedups from different models, hardware settings, batch sizes, and measurement scopes as though each were an independent improvement to one unchanged system.
A benchmark record a reader can interpret
| Record | Why it matters |
|---|---|
| Model and input shape | Defines the operation performed |
| Hardware and thread settings | Defines the available resources |
| Batch size and wait policy | Separates processing from waiting |
| Timed stages and repetitions | Makes the boundary and variability visible |
| Quality metric and split | Keeps speed attached to a useful predictor |
The archived pipeline figure labels an approximately 1.06M-sample/sec XGBoost stage. Earlier prose also reports 1.2M and roughly 9.8M rates for different paths. Without the corresponding benchmark records, those labels should not be treated as one controlled experiment or as end-to-end service latency. This rewrite does not claim a fresh reproduction of them.
What to optimize next
Take one workload and write down the timing boundary before changing it. Compare repeated calls with batching, then measure preparation separately. If you replace the predictor, repeat the quality evaluation. Finally, test the whole path under realistic arrivals so waiting time becomes visible.
The goal is a useful answer delivered within a known budget. Throughput helps describe capacity; latency describes how the work feels to the person waiting for it.
Continue with evaluating alerts and early warnings.