Skip to main content

Live streaming: from bet to recommendation in seconds

The batch path is fine for tuning and tests, but a real casino sends thousands of bets a second, and a player who just discovered a game should see that reflected quickly. The live path keeps a small, always-current note per customer in Redis and computes recommendations from it on read.

Platform / Simulator --> Kafka (transactions, sessions; key = customer) --> Ingest --> Redis c:{customer} --> Recommender
^
crowd build (scheduled): SCAN Redis --> co-play + popularity --+ (stored with the catalogue, held in memory)

End-to-end sequence​

Partitioning and ordering​

Every message is keyed by customer id, so one customer's events land on one partition and are consumed in order by exactly one ingest instance at a time. Running more ingest instances spreads the 48 partitions across them; there is no coordination beyond Kafka's consumer group.

The decayed counter​

Each g:{game} field holds bets, stake, seconds, atMs: totals that are valid as of atMs and halve every half-life H (default 14 days).

Reading at time t: total(t) = total(atMs) x 0.5 ^ ((t - atMs) / H)

New event at time te >= atMs (normal case):
f = 0.5 ^ ((te - atMs) / H)
total' = total x f + delta, atMs' = te

Late event at time te < atMs (out of order):
f = 0.5 ^ ((atMs - te) / H)
total' = total + delta x f, atMs' = atMs (never moves back in time)

This replaces a fixed look-back window: old play fades smoothly, nothing has to be re-scanned or expired, and one event costs one hash update. A transaction adds bets = 1, stake = amount; a session adds seconds = duration; a payout adds nothing.

DecayedCounter.Apply in C# and the Lua script in ProfileStore implement the same rule, and StreamStateTests checks they agree.

The apply script​

-- KEYS[1] = c:{customer}
-- ARGV: halfLifeMs, ttlSeconds, then groups of 7: source, offset, game, atMs, bets, stake, seconds
for each group:
last[src] = last[src] or HGET key 'o:'..src or -1
if offset > last[src] then
apply the decayed-counter rule to HGET key 'g:'..game
HSET key 'g:'..game "b,s,x,t"
last[src] = offset; applied++
for each src: HSET key 'o:'..src last[src]
if applied > 0 then HINCRBY key 'v' 1
EXPIRE key ttlSeconds
return applied

Properties that fall out of it:

PropertyMechanism
Atomic per customerOne EVAL per customer per batch; Redis runs scripts without interleaving
Idempotent replaysAn event whose offset is not above o:{source} is skipped, so re-reading Kafka after a crash or a rewind is harmless
Multi-cluster safeOffsets are tracked per source and topic (t, s, t:{id}, s:{id}), since two clusters have unrelated offsets
Cheap change detectionv increments only when something was applied; the recommender's result cache is keyed by it
Memory follows activityEXPIRE 30 days after the last write; about 820 bytes per customer with 8 games

Message formats​

Publishers must follow the schemas in src/Kaleido.Gre.Stream/Schemas/.

FormatWithout registryWith Confluent Schema Registry
JSONJsonFormat (camelCase, enums as strings)not supported (validation refuses it)
AvroAvroFormat, plain binary in transaction.avsc / session.avscRegistryAvroDecoder: Confluent wire format; the writer's schema is fetched by id, so the sender may use its own schema if field names match
ProtobufProtobufFormat, plain bytes of gre_events.proto messagesRegistryProtobufDecoder: magic byte, schema id, message index

A message that cannot be read raises FormatException and is counted as poison and skipped. Any other failure, such as the registry being down, is retried so no event is dropped for an infrastructure reason. The simulator only writes plain (non-registry) messages.

Reading on the recommender​

Measured on one laptop​

Kafka, Redis, all services and the load generator on the same machine:

MeasureResult
Produced and consumed10,000 events/s, ingest batch p99 about 60 ms
Backlog drainabout 55,000 events/s
Recommendation reads while ingesting 10,000 events/s5,900/s, p99 about 6 ms
Redis per customer (8 games)about 820 bytes

These are single-machine numbers, not a capacity plan.

Try it​

K="X-Api-Key: <SERVICE_API_KEY>"
curl -X POST localhost:8082/streams -H "$K" -H 'content-type: application/json' -d '{"customers":1000,"days":30,"tps":10000}'
curl localhost:8085/stats -H "$K"

Or use the Simulation page: select the stream box, choose an ingestion in Send to, and start.