Testing MedInsight: What Happens When a Model Worker Fails?

The happy path is only the beginning

A client sends observations. A Go service accepts the job. A Python worker runs Gemma 2 and publishes an analysis. That sequence is easy to draw. The interesting test starts when the worker disappears halfway through.

This is the testing companion to Following a Request from Go to Gemma 2. That article explains why the services are separated. Here, let's use the same design to ask what each boundary must guarantee when things go wrong.

The scenarios below are a test plan for the architecture. They are not a report of completed failure-injection experiments.

Case 1: The same job arrives twice

Imagine a worker stores its result, then crashes before acknowledging the message. The message can be delivered again even though the expensive part already happened.

If the second attempt creates a new unrelated result, one click can produce two answers. A stable job identifier lets the result layer recognize that both deliveries refer to the same logical operation. That's idempotency: repeating the operation leaves the intended final state unchanged.

Test it: deliver two messages with the same job ID. Check that there is one logical completed result and that a retry cannot overwrite it with a conflicting answer. Because model output may vary between runs, deduplicating the final effect matters even if duplicate computation is allowed.

Case 2: Go and Python disagree about the payload

A producer changes a field name. Its unit tests pass. The consumer still expects the old name, so every new message fails after deployment.

This is why a polyglot system needs contract tests in addition to language-specific unit tests. Keep a shared example payload and validate it from both sides. Include a schema version and define how a consumer handles an unsupported version.

Check	Producer's responsibility	Consumer's responsibility
Required job ID	Include a stable identifier	Reject a missing identifier
Payload version	Publish a documented version	Accept supported versions explicitly
Invalid input	Reject errors known at intake	Validate again at the trust boundary
Error result	Define a consistent shape	Expose a terminal failure the client understands

Test it: remove a required field, send an unknown version, and send an empty payload. Each should produce a deliberate outcome rather than an unbounded retry loop.

Case 3: The worker is healthy, but the user keeps waiting

A running container tells you that a process exists. It doesn't tell you that a particular job is making progress. Slow inference, a growing queue, or a stuck dependency can leave the service technically alive while the user waits indefinitely.

Separate three questions: can the process respond, can it accept work, and are jobs actually finishing? These correspond to different health and progress signals.

Test it: delay inference while continuing to accept jobs. Watch queue age and completion time, not only successful API responses. Check that the interface keeps pending jobs distinct from failed jobs.

Case 4: A job can never succeed

Some failures are temporary: a dependency may recover. Others are permanent: a malformed payload won't repair itself on the next attempt.

A retry policy needs a stopping rule. After bounded attempts, preserve the failed job in a controlled recovery path and mark its user-visible state appropriately. A dead-letter queue is a place to investigate work that exhausted its processing attempts; it is not an automatic repair mechanism.

Test it: force the same failure repeatedly. Verify that retry attempts stop according to the chosen policy, that the job can be investigated, and that the client isn't left waiting forever.

Case 5: Debugging exposes the original input

The normal model path may use carefully selected fields while an exception handler logs the entire original request. Reviewing only the successful path misses that second route.

Test it: use synthetic records with recognizable marker strings. Trigger validation errors, model errors, and retries, then inspect the resulting logs and failure records for those markers. This is a useful targeted check; it does not establish that all sensitive information is absent under every condition.

What a coverage number cannot answer

Line coverage tells you which code ran during a test suite. It doesn't tell you whether a duplicate message can corrupt state, whether both languages interpret a payload the same way, or whether the client sees the correct final status.

A stronger reliability story pairs each promise with an observable test: one job produces one logical result; unsupported messages fail clearly; repeated failures eventually stop; and pending work remains visible.

Those promises are useful beyond MedInsight. Any application that hands work from an API to a background model worker inherits the same questions.