Skip to main content

Evaluation: is it any good?

The simulator keeps the answer key: every simulated customer's persona and favourite games. Two evaluations use it.

Persona fit (POST /evaluate)​

Asks the recommender for an evenly spread sample of customers (1 to 1,000, list size K 1 to 50, optional persona filter, 8 requests in parallel) and measures each list:

MeasureMeaning
Type fitShare of recommended games whose type the persona likes (persona affinity for the type at least 0.5)
Distinct studiosDiversity of each list
Average scoreMean recommendation score
Empty listsCustomers who got nothing
Distinct gamesCatalogue coverage across all lists

It is quick and useful for "did a setting change break something", but it cannot tell whether the recommender predicts what people will play.

Holdout accuracy test (POST /evaluate/holdout)​

Like judging a weather forecaster: hide tomorrow, ask for a forecast, then look at what tomorrow really did.

The rebuild back to "now" runs in a finally, so the pipeline is restored even if scoring fails.

Why it is fair​

The profile build asOf=cutoff only reads activity before the cutoff and decays from it, and the crowd build then learns co-play from those affinities. The recommender genuinely cannot see the hidden days. Every variant is scored on the same customers with the same affinities, so differences come from the blend alone.

Metrics​

For one customer with list L (top k) and target set T:

hit = 1 if any game in L is in T
recall = |L intersect T| / |T|
DCG = sum over positions r (0-based) where L[r] in T of 1 / log2(r + 2)
IDCG = sum for r = 0 .. min(k, |T|) - 1 of 1 / log2(r + 2)
NDCG = DCG / IDCG

Averaged over judged customers, plus per variant: distinct games recommended, share of new releases, and average RTP of the recommended games.

Baseline​

A score means nothing alone, so every run includes Popular only: everyone gets the same most popular games. If the blend cannot beat it, it is not really using what it knows about the customer.

Results so far​

On 600 simulated customers (313 played something new in the hidden week):

BlendHit rateNDCG
The blend in use17.6%0.050
Tagged alike only15.0%0.051
Popular only (baseline)12.1%0.034
Played together only9.9%0.033

The blend beats the baseline by about 1.5 times. When every game in play is tagged (taggedOnly), the hit rate rises to 43% against a baseline of 33%.

Ideas that did not help and were removed: a same-studio-and-type signal, discounting co-play pairs with little support, and raw rather than stretched scores. A 27-combination weight grid moved scores by less than the noise: the weights are not the bottleneck; the data (tag coverage) is.

Caveat​

The customers are simulated, and their rules include "customers tend to like games tagged like their favourites". The test proves the machinery finds patterns that exist; it does not prove real customers behave that way. The same test gives the real answer once real play data flows in, because it only needs activity, not personas.