Ingest service (Kaleido.Gre.Ingest, :8085)
The ingest service keeps every active customer's live profile in Redis up to date from Kafka. It never re-reads history: each event adds to a per-game running total that fades with time. Several instances share the Kafka partitions, and several Kafka clusters ("ingestions") can feed the same profiles at once.
API
| Method | Path | Purpose |
|---|---|---|
| GET | /stats | Aggregated throughput, batch time, lag and state per ingestion |
| GET | /customers/recent?size=20 | Most recently active customers (in memory, capped at 2,000) |
| GET | /sources | Ingestions without secrets, each with its live status |
| POST | /sources | Create an ingestion |
| PUT | /sources/{id} | Change an ingestion; the local one can only be switched on or off |
| DELETE | /sources/{id} | Delete an ingestion (not the local one) |
| POST | /sources/test?id= | Check brokers, topics and Schema Registry without saving |
| GET | /internal/sources/{id}/connection | Full connection with passwords, for the simulator only; 403 unless the ingestion allows simulation |
| DELETE | /data | Delete every c:* key (simulation reset) |
The gateway blocks /internal/... and lets only admins change /sources.
Consumer loop
Each SourceConsumer runs on its own thread with its own consumer group and topics.
Key properties:
- At-least-once, effectively once. Offsets are committed only after Redis has the data. A crash replays a few events, and the Lua script skips any event whose offset is not above the last applied offset for that source and topic.
- Batching. Up to
Ingest:MaxBatch(5,000) messages orIngest:MaxWaitMs(50 ms), whichever first. Every customer's events in a batch go to Redis in one atomic call, and all customers run in parallel. - Poison messages (unreadable bytes) are counted and skipped, never retried forever. Faults that are not about the message itself, such as an unreachable Schema Registry, are retried.
- Lag is measured every 5 seconds per topic from the high watermark and the consumer's position.
Redis state
One hash per customer:
c:{customerId}
g:{gameRef} = "bets,stake,seconds,atMs" decayed totals as of atMs
o:{source} = last applied Kafka offset source = "t", "s", or "t:{id}", "s:{id}" for non-local ingestions
v = version, +1 whenever an event is applied
The update rule and its maths are in Live streaming.
Multiple ingestions
An ingestion is one Kafka source: brokers, security, two topics, consumer group, start position, message format and optional Schema Registry. The local ingestion always exists and comes from the service's own configuration; admins add others in Settings, Kafka ingestion.
Because each consumer has its own topics and offset guard names (t:{id}, s:{id}), replacing one ingestion never disturbs another, and two clusters with unrelated offsets cannot shadow each other's events.
Ingestion settings
| Field | Values |
|---|---|
Bootstrap | Comma-separated host:port |
SecurityProtocol | plaintext, ssl, sasl_plaintext, sasl_ssl |
SaslMechanism | plain, scram-sha-256, scram-sha-512 |
SaslUsername, SaslPassword, SslCaPem | Credentials and CA certificate |
TransactionsTopic, SessionsTopic | Must differ |
GroupId | Consumer group, default profile-state |
StartFrom | earliest or latest |
CreateTopics, Partitions | Create missing topics with this many partitions |
Format | json, avro, protobuf |
RegistryUrl, RegistryUser, RegistryPassword | Confluent Schema Registry (Avro or Protobuf only) |
Enabled, AllowSimulation | Running; whether the simulator may write test messages to it |
Configuration
| Key | Default | Purpose |
|---|---|---|
Kafka:Bootstrap, Kafka:TopicPrefix, Kafka:GroupId, Kafka:StartFrom, Kafka:CreateTopics, Kafka:Partitions, Kafka:Format | localhost:9094, none, profile-state, earliest, true, 48, json | The local ingestion |
Redis:Connection | localhost:6380 | Profile and settings store |
Stream:HalfLifeDays, Stream:KeyTtlDays | 14, 30 | Decay and key expiry |
Ingest:MaxBatch, Ingest:MaxWaitMs | 5000, 50 | Batch shape |
Ingest:ConfigKey | cfg:ingest | Redis key prefix for ingestion settings |
Traces are sampled at 1% by default for this service (defaultSampleRatio: 0.01), because it handles thousands of batches a second.