Live streaming: from bet to recommendation in seconds
The batch path is fine for tuning and tests, but a real casino sends thousands of bets a second, and a player who just discovered a game should see that reflected quickly. The live path keeps a small, always-current note per customer in Redis and computes recommendations from it on read.
Platform / Simulator --> Kafka (transactions, sessions; key = customer) --> Ingest --> Redis c:{customer} --> Recommender
^
crowd build (scheduled): SCAN Redis --> co-play + popularity --+ (stored with the catalogue, held in memory)
End-to-end sequence
Partitioning and ordering
Every message is keyed by customer id, so one customer's events land on one partition and are consumed in order by exactly one ingest instance at a time. Running more ingest instances spreads the 48 partitions across them; there is no coordination beyond Kafka's consumer group.
The decayed counter
Each g:{game} field holds bets, stake, seconds, atMs: totals that are valid as of atMs and halve every half-life H (default 14 days).
Reading at time t: total(t) = total(atMs) x 0.5 ^ ((t - atMs) / H)
New event at time te >= atMs (normal case):
f = 0.5 ^ ((te - atMs) / H)
total' = total x f + delta, atMs' = te
Late event at time te < atMs (out of order):
f = 0.5 ^ ((atMs - te) / H)
total' = total + delta x f, atMs' = atMs (never moves back in time)
This replaces a fixed look-back window: old play fades smoothly, nothing has to be re-scanned or expired, and one event costs one hash update. A transaction adds bets = 1, stake = amount; a session adds seconds = duration; a payout adds nothing.
DecayedCounter.Apply in C# and the Lua script in ProfileStore implement the same rule, and StreamStateTests checks they agree.
The apply script
-- KEYS[1] = c:{customer}
-- ARGV: halfLifeMs, ttlSeconds, then groups of 7: source, offset, game, atMs, bets, stake, seconds
for each group:
last[src] = last[src] or HGET key 'o:'..src or -1
if offset > last[src] then
apply the decayed-counter rule to HGET key 'g:'..game
HSET key 'g:'..game "b,s,x,t"
last[src] = offset; applied++
for each src: HSET key 'o:'..src last[src]
if applied > 0 then HINCRBY key 'v' 1
EXPIRE key ttlSeconds
return applied
Properties that fall out of it:
| Property | Mechanism |
|---|---|
| Atomic per customer | One EVAL per customer per batch; Redis runs scripts without interleaving |
| Idempotent replays | An event whose offset is not above o:{source} is skipped, so re-reading Kafka after a crash or a rewind is harmless |
| Multi-cluster safe | Offsets are tracked per source and topic (t, s, t:{id}, s:{id}), since two clusters have unrelated offsets |
| Cheap change detection | v increments only when something was applied; the recommender's result cache is keyed by it |
| Memory follows activity | EXPIRE 30 days after the last write; about 820 bytes per customer with 8 games |
Message formats
Publishers must follow the schemas in src/Kaleido.Gre.Stream/Schemas/.
| Format | Without registry | With Confluent Schema Registry |
|---|---|---|
| JSON | JsonFormat (camelCase, enums as strings) | not supported (validation refuses it) |
| Avro | AvroFormat, plain binary in transaction.avsc / session.avsc | RegistryAvroDecoder: Confluent wire format; the writer's schema is fetched by id, so the sender may use its own schema if field names match |
| Protobuf | ProtobufFormat, plain bytes of gre_events.proto messages | RegistryProtobufDecoder: magic byte, schema id, message index |
A message that cannot be read raises FormatException and is counted as poison and skipped. Any other failure, such as the registry being down, is retried so no event is dropped for an infrastructure reason. The simulator only writes plain (non-registry) messages.
Reading on the recommender
Measured on one laptop
Kafka, Redis, all services and the load generator on the same machine:
| Measure | Result |
|---|---|
| Produced and consumed | 10,000 events/s, ingest batch p99 about 60 ms |
| Backlog drain | about 55,000 events/s |
| Recommendation reads while ingesting 10,000 events/s | 5,900/s, p99 about 6 ms |
| Redis per customer (8 games) | about 820 bytes |
These are single-machine numbers, not a capacity plan.
Try it
K="X-Api-Key: <SERVICE_API_KEY>"
curl -X POST localhost:8082/streams -H "$K" -H 'content-type: application/json' -d '{"customers":1000,"days":30,"tps":10000}'
curl localhost:8085/stats -H "$K"
Or use the Simulation page: select the stream box, choose an ingestion in Send to, and start.