Experiments
A/B test a game change on live servers — describe it in a sentence, get a clear verdict, a recommendation and results per cohort.
An experiment tests whether a game change moves a metric: a shorter tutorial, a cheaper starter pack, a new map. Results are measured in game metrics RoChart computes from the events every game sends: D1 and D7 retention, playtime, sessions, Robux and payer conversion per player. You can also use your own events. Describe what you want to test, review the plan RoChart drafts, and read the variant in your server code with RoChart.getFlag. Once it's live, results update continuously and tell you:
- the chance each variant is best;
- the lift over control, with a range it very likely falls in;
- a verdict: winner, keep control, inconclusive or needs more data;
- whether the split looks healthy, how much longer to run, and a one-line recommendation;
- the same numbers per cohort, such as new players vs power players.
You can check results as often as you like. Looking early doesn't invalidate them.
Experiment "Shorter tutorial" (running). Exposures: control 1475, test 1525.
Primary: D1 retention — control: 30.8% (n=1402); test: 39.1% (n=1449) lift +26.9% [15.0%, 39.8%], P(beats control) 100.0%. Verdict: winner (test).
Secondary: Playtime per player — control: 41.2 min (n=1475); test: 44.0 min (n=1525) lift +6.8% [-1.9%, 16.1%], P(beats control) 93.4%. Verdict: inconclusive.
Segment "Mobile players": D1 retention — … lift +61.2% [27.5%, 102.8%], P(beats control) 100.0%. Verdict: winner (test).
Recommendation: Ship "test": it is the best variant with 100.0% probability.Lifecycle
| Status | What happens |
|---|---|
| Draft | The experiment and its flag exist, but the flag is off, so nothing changes for players. |
| Running | Launch turns the flag on. Exposures and metrics count from here. |
| Stopped | No new players enter. Players already in keep counting until their metric window closes. |
| Concluded | The decision is recorded: the winner, the reason, and who decided. Optionally the winner is rolled out to everyone. |
Every step shows up in the activity log with the person or agent who took it. Experiments need the Feature flags product.