Skip to main content

Ingest service (Kaleido.Gre.Ingest, :8085)

The ingest service keeps every active customer's live profile in Redis up to date from Kafka. It never re-reads history: each event adds to a per-game running total that fades with time. Several instances share the Kafka partitions, and several Kafka clusters ("ingestions") can feed the same profiles at once.

API​

MethodPathPurpose
GET/statsAggregated throughput, batch time, lag and state per ingestion
GET/customers/recent?size=20Most recently active customers (in memory, capped at 2,000)
GET/sourcesIngestions without secrets, each with its live status
POST/sourcesCreate an ingestion
PUT/sources/{id}Change an ingestion; the local one can only be switched on or off
DELETE/sources/{id}Delete an ingestion (not the local one)
POST/sources/test?id=Check brokers, topics and Schema Registry without saving
GET/internal/sources/{id}/connectionFull connection with passwords, for the simulator only; 403 unless the ingestion allows simulation
DELETE/dataDelete every c:* key (simulation reset)

The gateway blocks /internal/... and lets only admins change /sources.

Consumer loop​

Each SourceConsumer runs on its own thread with its own consumer group and topics.

Key properties:

  • At-least-once, effectively once. Offsets are committed only after Redis has the data. A crash replays a few events, and the Lua script skips any event whose offset is not above the last applied offset for that source and topic.
  • Batching. Up to Ingest:MaxBatch (5,000) messages or Ingest:MaxWaitMs (50 ms), whichever first. Every customer's events in a batch go to Redis in one atomic call, and all customers run in parallel.
  • Poison messages (unreadable bytes) are counted and skipped, never retried forever. Faults that are not about the message itself, such as an unreachable Schema Registry, are retried.
  • Lag is measured every 5 seconds per topic from the high watermark and the consumer's position.

Redis state​

One hash per customer:

c:{customerId}
g:{gameRef} = "bets,stake,seconds,atMs" decayed totals as of atMs
o:{source} = last applied Kafka offset source = "t", "s", or "t:{id}", "s:{id}" for non-local ingestions
v = version, +1 whenever an event is applied

The update rule and its maths are in Live streaming.

Multiple ingestions​

An ingestion is one Kafka source: brokers, security, two topics, consumer group, start position, message format and optional Schema Registry. The local ingestion always exists and comes from the service's own configuration; admins add others in Settings, Kafka ingestion.

Because each consumer has its own topics and offset guard names (t:{id}, s:{id}), replacing one ingestion never disturbs another, and two clusters with unrelated offsets cannot shadow each other's events.

Ingestion settings​

FieldValues
BootstrapComma-separated host:port
SecurityProtocolplaintext, ssl, sasl_plaintext, sasl_ssl
SaslMechanismplain, scram-sha-256, scram-sha-512
SaslUsername, SaslPassword, SslCaPemCredentials and CA certificate
TransactionsTopic, SessionsTopicMust differ
GroupIdConsumer group, default profile-state
StartFromearliest or latest
CreateTopics, PartitionsCreate missing topics with this many partitions
Formatjson, avro, protobuf
RegistryUrl, RegistryUser, RegistryPasswordConfluent Schema Registry (Avro or Protobuf only)
Enabled, AllowSimulationRunning; whether the simulator may write test messages to it

Configuration​

KeyDefaultPurpose
Kafka:Bootstrap, Kafka:TopicPrefix, Kafka:GroupId, Kafka:StartFrom, Kafka:CreateTopics, Kafka:Partitions, Kafka:Formatlocalhost:9094, none, profile-state, earliest, true, 48, jsonThe local ingestion
Redis:Connectionlocalhost:6380Profile and settings store
Stream:HalfLifeDays, Stream:KeyTtlDays14, 30Decay and key expiry
Ingest:MaxBatch, Ingest:MaxWaitMs5000, 50Batch shape
Ingest:ConfigKeycfg:ingestRedis key prefix for ingestion settings

Traces are sampled at 1% by default for this service (defaultSampleRatio: 0.01), because it handles thousands of batches a second.