Evaluation: is it any good?
The simulator keeps the answer key: every simulated customer's persona and favourite games. Two evaluations use it.
Persona fit (POST /evaluate)
Asks the recommender for an evenly spread sample of customers (1 to 1,000, list size K 1 to 50, optional persona filter, 8 requests in parallel) and measures each list:
| Measure | Meaning |
|---|---|
| Type fit | Share of recommended games whose type the persona likes (persona affinity for the type at least 0.5) |
| Distinct studios | Diversity of each list |
| Average score | Mean recommendation score |
| Empty lists | Customers who got nothing |
| Distinct games | Catalogue coverage across all lists |
It is quick and useful for "did a setting change break something", but it cannot tell whether the recommender predicts what people will play.
Holdout accuracy test (POST /evaluate/holdout)
Like judging a weather forecaster: hide tomorrow, ask for a forecast, then look at what tomorrow really did.
The rebuild back to "now" runs in a finally, so the pipeline is restored even if scoring fails.
Why it is fair
The profile build asOf=cutoff only reads activity before the cutoff and decays from it, and the crowd build then learns co-play from those affinities. The recommender genuinely cannot see the hidden days. Every variant is scored on the same customers with the same affinities, so differences come from the blend alone.
Metrics
For one customer with list L (top k) and target set T:
hit = 1 if any game in L is in T
recall = |L intersect T| / |T|
DCG = sum over positions r (0-based) where L[r] in T of 1 / log2(r + 2)
IDCG = sum for r = 0 .. min(k, |T|) - 1 of 1 / log2(r + 2)
NDCG = DCG / IDCG
Averaged over judged customers, plus per variant: distinct games recommended, share of new releases, and average RTP of the recommended games.
Baseline
A score means nothing alone, so every run includes Popular only: everyone gets the same most popular games. If the blend cannot beat it, it is not really using what it knows about the customer.
Results so far
On 600 simulated customers (313 played something new in the hidden week):
| Blend | Hit rate | NDCG |
|---|---|---|
| The blend in use | 17.6% | 0.050 |
| Tagged alike only | 15.0% | 0.051 |
| Popular only (baseline) | 12.1% | 0.034 |
| Played together only | 9.9% | 0.033 |
The blend beats the baseline by about 1.5 times. When every game in play is tagged (taggedOnly), the hit rate rises to 43% against a baseline of 33%.
Ideas that did not help and were removed: a same-studio-and-type signal, discounting co-play pairs with little support, and raw rather than stretched scores. A 27-combination weight grid moved scores by less than the noise: the weights are not the bottleneck; the data (tag coverage) is.
Caveat
The customers are simulated, and their rules include "customers tend to like games tagged like their favourites". The test proves the machinery finds patterns that exist; it does not prove real customers behave that way. The same test gives the real answer once real play data flows in, because it only needs activity, not personas.