Create an experiment

Set it up step by stepCopy Anchor Link

Experiments → New experiment walks you through five short steps, one question each. The steps on the left tick off as you go, and the card under them sums up your choices. Press ↵ to continue, or click a finished step to go back. Your place and everything you've typed survive a page refresh.

1. Hypothesis: name the test, and optionally write We believe … because ….
2. Who sees it: Everyone, A cohort (pick it, and optionally enroll only a share of it) or A share of players. Choose the split (50 / 50, 70 / 30 or 3 variants) and rename variants if you like; control is the baseline. The test runs on a new flag whose key is suggested from the name (you can edit it; taken keys are flagged), or on an existing flag that isn't in another experiment. An existing flag's targeting and variants are replaced by the plan.
3. What success is: pick a game metric. D1 retention is suggested; the others are D7 retention, playtime, Robux, payer conversion and sessions per player. Or pick a custom event (% of players who do it, how many times or sum of a property) and its window. The forecast shows the baseline, players needed per variant and roughly how many days that takes at today's traffic.
4. Guardrails: metrics that must not get worse. New experiments start with Playtime per player and Robux per player, so a retention win can't quietly cost engagement or revenue. Add others, such as matches played, or remove them for a small change.
5. Review & launch: the plan as a sentence. Click any highlighted part to jump back to its step.

Split everyone 50/50 between control and test on short-tutorial new flag. Success = D1 retention — players who come back the day after they first see it. Needs ~5,157 players per variant to detect a 10% lift — about 6 days at today's traffic.

Below it, a checklist shows whether the name, flag and goal are ready, and warns when your code hasn't read the flag yet. Advanced opens every setting: notes, variants, metric filters, targeting, segments and the code snippet. Then choose:

• Save draft: creates the experiment and its flag (off). Launch it once your code reads the flag.
• Launch: creates it and turns the flag on right away.
• Hand this to your agent: copies the plan as instructions for Claude, ChatGPT or a coding agent.

Or describe itCopy Anchor Link

When the assistant is set up, step 1 also offers describe it in a sentence. Write it the way you'd say it to a colleague:

Does a three-step tutorial instead of eight bring more new players back the next day?

RoChart reads your project's events, cohorts and existing flags and drafts a complete plan (name, hypothesis, flag, variants, metrics, guardrails, cohorts to compare, a rationale and warnings), then opens it on Review. Drafts use game metrics first: D1 retention for most changes, payer conversion or Robux for shop changes, playtime for content, with playtime and Robux as guardrails. It never invents an event you don't track, and nothing is created until you save or launch.

Choose good metricsCopy Anchor Link

The primary metric decides the verdict. Every metric is measured per exposed player, from the moment they first read the flag.

Game metrics need no extra tracking:

Days are in your project's timezone. D1 and D7 only count players whose day has passed: a player exposed today joins the D1 numbers the day after tomorrow, so D1 needs two days before the first result. The exposure counts at the top still include everyone.

Custom events cover what your game tracks with RoChart.track:

• Conversion: the share of exposed players who did an event at least once. Use it for yes/no outcomes: finished the tutorial, joined a clan.
• Mean: the average per exposed player of an event count (such as matches played) or the sum of a numeric property (such as coins earned).

In averages, players who never did it count as 0, so "Robux per player" includes players who bought nothing. Shares of players are analysed with a Beta-binomial model, and averages with a Normal model.

Each metric except D1/D7 has a window: how many days after their first exposure a player's actions count. Everyone gets the same window, however late they joined. Match it to the behaviour: a day for finishing the tutorial, about 7 days for playtime, Robux or a first purchase.

Secondary metrics get full results but don't decide the verdict. Use them as:

• guardrails: things that mustn't get worse, like playtime, Robux or matches per player;
• value: Robux per player next to a retention or payer-conversion primary.

Minimum lift (minimum detectable effect) is the smallest relative improvement worth acting on. It only affects the sample size estimate. Smaller lifts need far more players: halving it needs about four times the sample. Rare goals, like a 2% purchase rate, need many more players than common ones.

Decide on the primary metric before launch. You can add a guardrail later, but swapping the primary halfway through is a way to cherry-pick a winner.

Put the variant in your codeCopy Anchor Link

The experiment only gets data when your game reads its flag:

local function offerStarterPack(player)
  local productId = STARTER_PACK
  if RoChart.getFlag(player, "starter-pack-discount") == "test" then
    productId = STARTER_PACK_DISCOUNTED
  end
  MarketplaceService:PromptProductPurchase(player, productId)
end

Websites and backends read flags with the browser and Node SDKs.

• Read the flag where the experience starts, such as when the offer is shown and not when the player joins. Everyone who reads the flag counts as exposed.
• Show control for anything else, including undefined.
• Game metrics need nothing else: sessions, playtime and retention come from the SDK, and purchases are recorded for you by processReceipt. For a custom-event metric, send the event, such as tutorial_completed.

See Flags in your code for details.

Launch, stop, deleteCopy Anchor Link

• Launch turns the flag on and starts counting.
• Stop lets no new players in. The flag keeps serving variants until you conclude or switch it off.
• Conclude records the decision. See Read results.
• Delete removes the experiment, and its flag too if the flag is off.