Failure handling
What happens when each part fails, and how the platform recovers. The live-path drills are automated in tests/Kaleido.Gre.Tests (KafkaPipelineTests.Redis_outage_loses_nothing_and_commits_nothing_until_it_is_back, StreamStateTests.Replaying_events_changes_nothing, StreamStateTests.Resuming_from_an_earlier_offset_after_a_restore_gives_the_same_state).
Failure matrix
| Failure | Effect | Recovery |
|---|---|---|
| Redis down (ingest) | Batch apply fails; ingest retries the same batch with back-off (250 ms doubling to 15 s) and commits nothing | Automatic when Redis returns; Kafka still holds the events |
| Redis down (recommender) | ?source=redis requests return 503; ?source=profile still works | Automatic |
| Redis data lost | Live profiles are gone | Restore a backup, stop ingest, rewind the consumer group to the backup time, start ingest (below) |
| Kafka down | Producers queue and then block (back-pressure); ingest consumers report waiting and retry every 3 s | Automatic within retention (7 days locally) |
| Ingest crash | Uncommitted batch is lost from memory | On restart, consumers resume at the last commit; the offset guard skips what was already applied |
| Unreadable message | Counted as poison, skipped, offset committed | Fix the producer; check kaleido.gre.ingest.messages{outcome=poison} |
| Schema Registry down | Decode fails with a non-format error; batch retried, nothing dropped | Automatic |
| Profile service down | Batch recommendations return 502; crowd builds from profile fail | Automatic; live path unaffected |
| Taxonomy API down | Catalogue builds fail (marked Failed); the recommender keeps serving its in-memory catalogue | Re-run the build |
| Postgres down (recommender) | Requests keep working from memory until the suppression list is 60 s stale, then every list is empty (fail closed) | Automatic on recovery |
| Service restart mid-build | Builds left Running or Queued are marked Failed, "Interrupted by a restart." | Start the build again |
| API restart mid-tagging | Items left Running go back to Queued | Automatic |
| Holdout test fails midway | finally rebuilds profile and crowd for "now" | If that also fails, the log says to run the builds from the Data step |
| AI provider slow or failing | Item fails after 6 minutes or with the provider's (shortened) error | retry-failed on the run |
Restoring Redis from an older backup
The profile state ignores events it has already applied, so reading too much is safe; reading too little loses data. The procedure therefore rewinds to a time at or before the backup.
Each ingestion has its own consumer group and offset guard, so repeat the rewind per ingestion.
Why nothing is lost or double-counted
- Commit happens strictly after Redis has the data: at-least-once delivery.
- The offset guard in the same atomic script makes replays no-ops: effectively once.
Known gap: duplicate bets upstream
The offset guard recognises a replay of the same Kafka message, not the same bet sent twice as two messages. If a producer can do that, a bet-id check is needed. The batch path does not have this gap: the activity service is idempotent by EventId.