A/B test a game change

Track the goalCopy Anchor Link

An experiment needs an event that says "it worked". Here that's tutorial_completed, sent from the server when a player finishes. Revenue comes for free from $purchase.

RoChart.track(player, "tutorial_completed", { seconds = os.clock() - startedAt })

If you followed the onboarding guide, you already have it.

Create the experimentCopy Anchor Link

Experiments → New experiment, or describe it in a sentence:

Does a 3-step tutorial instead of the current 6 steps get more new players to finish it?

Check the plan RoChart drafts:

• Who sees it: everyone, split 50 / 50 between control and short, on a new flag short-tutorial.
• Primary metric: % of players who do tutorial_completed within 1 day.
• Secondary metrics: sum of $robux on $purchase per player within 7 days, so you notice if the shorter tutorial costs you revenue.
• Segments: platform, or a cohort such as Premium players, to see whether phones react differently.

Save it as a draft for now.

Read the variant on the serverCopy Anchor Link

Read the flag where the experience starts, when the tutorial begins, not when the player joins. Everyone who reads it counts as exposed, so reading it on join would dilute the result with players who never reached the tutorial.

local RoChart = require(game.ServerScriptService.RoChart)

local function startTutorial(player)
  if RoChart.getFlag(player, "short-tutorial") == "short" then
    runTutorial(player, SHORT_STEPS)
  else
    runTutorial(player, FULL_STEPS) -- "control", or the flag is off or unknown
  end
end

The flag is evaluated per player on the game server, so a player can't change their variant from the client. The same player always gets the same variant, on every server and device. Anything that isn't "short" takes the current path, so the game keeps working if the flag is off.

Publish the place, then Launch the experiment. Old servers keep running the old code until they shut down, so give it a few hours, or use Shut Down All Servers if you want everyone on the new version.

Read the resultsCopy Anchor Link

Results update continuously, and looking early doesn't invalidate them:

Experiment "Short tutorial" (running). Exposures: control 4,212, short 4,190.
Primary: tutorial_completed — control: 38.1% (n=4212); short: 47.9% (n=4190) lift +25.7% [19.8%, 31.9%], P(beats control) 100.0%. Verdict: winner (short).
Segment "Phone": tutorial_completed — … lift +34.0% [24.1%, 44.6%], P(beats control) 100.0%. Verdict: winner (short).
Recommendation: Ship "short": it is the best variant with 100.0% probability.

Check the secondary metric before you ship. A tutorial that more players finish but that skips the shop might bring in less Robux. See Read results.

Ship the winnerCopy Anchor Link

Conclude the experiment and roll short out to everyone through the same flag. The code doesn't change. Later, when you remove the old tutorial, delete the flag check too.