Skip to main content

Failure handling

What happens when each part fails, and how the platform recovers. The live-path drills are automated in tests/Kaleido.Gre.Tests (KafkaPipelineTests.Redis_outage_loses_nothing_and_commits_nothing_until_it_is_back, StreamStateTests.Replaying_events_changes_nothing, StreamStateTests.Resuming_from_an_earlier_offset_after_a_restore_gives_the_same_state).

Failure matrix​

FailureEffectRecovery
Redis down (ingest)Batch apply fails; ingest retries the same batch with back-off (250 ms doubling to 15 s) and commits nothingAutomatic when Redis returns; Kafka still holds the events
Redis down (recommender)?source=redis requests return 503; ?source=profile still worksAutomatic
Redis data lostLive profiles are goneRestore a backup, stop ingest, rewind the consumer group to the backup time, start ingest (below)
Kafka downProducers queue and then block (back-pressure); ingest consumers report waiting and retry every 3 sAutomatic within retention (7 days locally)
Ingest crashUncommitted batch is lost from memoryOn restart, consumers resume at the last commit; the offset guard skips what was already applied
Unreadable messageCounted as poison, skipped, offset committedFix the producer; check kaleido.gre.ingest.messages{outcome=poison}
Schema Registry downDecode fails with a non-format error; batch retried, nothing droppedAutomatic
Profile service downBatch recommendations return 502; crowd builds from profile failAutomatic; live path unaffected
Taxonomy API downCatalogue builds fail (marked Failed); the recommender keeps serving its in-memory catalogueRe-run the build
Postgres down (recommender)Requests keep working from memory until the suppression list is 60 s stale, then every list is empty (fail closed)Automatic on recovery
Service restart mid-buildBuilds left Running or Queued are marked Failed, "Interrupted by a restart."Start the build again
API restart mid-taggingItems left Running go back to QueuedAutomatic
Holdout test fails midwayfinally rebuilds profile and crowd for "now"If that also fails, the log says to run the builds from the Data step
AI provider slow or failingItem fails after 6 minutes or with the provider's (shortened) errorretry-failed on the run

Restoring Redis from an older backup​

The profile state ignores events it has already applied, so reading too much is safe; reading too little loses data. The procedure therefore rewinds to a time at or before the backup.

Each ingestion has its own consumer group and offset guard, so repeat the rewind per ingestion.

Why nothing is lost or double-counted​

  • Commit happens strictly after Redis has the data: at-least-once delivery.
  • The offset guard in the same atomic script makes replays no-ops: effectively once.

Known gap: duplicate bets upstream​

The offset guard recognises a replay of the same Kafka message, not the same bet sent twice as two messages. If a producer can do that, a bet-id check is needed. The batch path does not have this gap: the activity service is idempotent by EventId.