Observability
Every service sends traces, metrics and logs over OTLP to an OpenTelemetry collector in the stack. The collector adds Redis, Postgres and Kafka metrics (including consumer-group lag) and forwards everything to SigNoz when it runs. Services never talk to SigNoz directly, so they behave the same with or without it.
Traces
| Span | Where | Notes |
|---|---|---|
| HTTP server and client | every service | Named by route template: customer ids never reach telemetry |
| SQL | Npgsql | Every command |
| Redis | recommender, ingest | Only inside a sampled request or batch; GreSampler never starts a trace from a lone Redis or database call |
simulator.stream | simulator | Producer span of a stream run; its context travels in each message's traceparent header |
ingest.batch | ingest | Consumer span per batch; links to the distinct producer traces in the batch |
profile.build, recommender.build | profile, recommender | Root of everything a build does |
Sampling: 100% for web-facing services, 1% for ingest by default (OTEL_TRACES_SAMPLER_ARG or Telemetry:SampleRatio override). A child always follows its parent's decision.
Metrics
| Metric | Type | Tags | Use |
|---|---|---|---|
kaleido.gre.ingest.messages | counter | source, outcome (applied, replayed, ignored, poison) | Throughput and data quality |
kaleido.gre.ingest.lag | gauge | source, topic | Alert: messages waiting in Kafka |
kaleido.gre.ingest.partitions | gauge | source | Partitions owned by this instance |
kaleido.gre.ingest.batch.duration, .batch.size | histogram | source | Batch cost |
kaleido.gre.ingest.failures | counter | source | Retried batches (nothing lost) |
kaleido.gre.redis.apply.duration | histogram | One customer's Lua call | |
kaleido.gre.recommend.duration | histogram | source, cache, result | Latency; cache hit ratio; personalised vs cold-start vs suppressed |
kaleido.gre.catalogue.games, .links, .load.duration | gauge, histogram | kind | What the recommender holds |
kaleido.gre.suppression.age, .size, .refresh.failures | gauge, counter | Alert on age approaching 60 s | |
kaleido.gre.builds, kaleido.gre.build.duration | counter, histogram | service, outcome | Background builds |
kaleido.gre.activity.events | counter | kind, outcome | Accepted, duplicate, rejected |
kaleido.gre.simulator.stream.sent, .failed | counter | Simulator output |
Plus standard http.server.request.duration, http.client.request.duration, dotnet.*, db.client.*, and kafka.consumer_group.lag from the collector to cross-check ingest lag.
Two SigNoz dashboards are generated by deploy/signoz/dashboards/generate.py: kaleido-gre-business.json and kaleido-gre-technical.json.
Logs
Logs carry the trace id of the request that wrote them. Request lines and SQL are kept at warning and above, because they would be thousands of records a second.
Health and live view
- Every service answers
GET /healthwithout a key. deploy.sh statusshows container health and ingest lag.- The Simulation page's pipeline diagram is fed by
PipelineSamplerover SignalR (see Request flows). GET /statson ingest gives per-ingestion state (running,waiting for Kafka,retrying,invalid settings,disabled), throughput, batch time and lag.