Skip to main content

Observability

Every service sends traces, metrics and logs over OTLP to an OpenTelemetry collector in the stack. The collector adds Redis, Postgres and Kafka metrics (including consumer-group lag) and forwards everything to SigNoz when it runs. Services never talk to SigNoz directly, so they behave the same with or without it.

Traces​

SpanWhereNotes
HTTP server and clientevery serviceNamed by route template: customer ids never reach telemetry
SQLNpgsqlEvery command
Redisrecommender, ingestOnly inside a sampled request or batch; GreSampler never starts a trace from a lone Redis or database call
simulator.streamsimulatorProducer span of a stream run; its context travels in each message's traceparent header
ingest.batchingestConsumer span per batch; links to the distinct producer traces in the batch
profile.build, recommender.buildprofile, recommenderRoot of everything a build does

Sampling: 100% for web-facing services, 1% for ingest by default (OTEL_TRACES_SAMPLER_ARG or Telemetry:SampleRatio override). A child always follows its parent's decision.

Metrics​

MetricTypeTagsUse
kaleido.gre.ingest.messagescountersource, outcome (applied, replayed, ignored, poison)Throughput and data quality
kaleido.gre.ingest.laggaugesource, topicAlert: messages waiting in Kafka
kaleido.gre.ingest.partitionsgaugesourcePartitions owned by this instance
kaleido.gre.ingest.batch.duration, .batch.sizehistogramsourceBatch cost
kaleido.gre.ingest.failurescountersourceRetried batches (nothing lost)
kaleido.gre.redis.apply.durationhistogramOne customer's Lua call
kaleido.gre.recommend.durationhistogramsource, cache, resultLatency; cache hit ratio; personalised vs cold-start vs suppressed
kaleido.gre.catalogue.games, .links, .load.durationgauge, histogramkindWhat the recommender holds
kaleido.gre.suppression.age, .size, .refresh.failuresgauge, counterAlert on age approaching 60 s
kaleido.gre.builds, kaleido.gre.build.durationcounter, histogramservice, outcomeBackground builds
kaleido.gre.activity.eventscounterkind, outcomeAccepted, duplicate, rejected
kaleido.gre.simulator.stream.sent, .failedcounterSimulator output

Plus standard http.server.request.duration, http.client.request.duration, dotnet.*, db.client.*, and kafka.consumer_group.lag from the collector to cross-check ingest lag.

Two SigNoz dashboards are generated by deploy/signoz/dashboards/generate.py: kaleido-gre-business.json and kaleido-gre-technical.json.

Logs​

Logs carry the trace id of the request that wrote them. Request lines and SQL are kept at warning and above, because they would be thousands of records a second.

Health and live view​

  • Every service answers GET /health without a key.
  • deploy.sh status shows container health and ingest lag.
  • The Simulation page's pipeline diagram is fed by PipelineSampler over SignalR (see Request flows).
  • GET /stats on ingest gives per-ingestion state (running, waiting for Kafka, retrying, invalid settings, disabled), throughput, batch time and lag.