# Create an experiment (/docs/experiments/create)

<CopyPrompt preset="experiment">
  Want an agent to run the whole test? This prompt walks it through schema, cohorts, experiment, implementation, launch, results and decision, and it checks with you before anything goes live.
</CopyPrompt>

## Set it up step by step [#set-it-up-step-by-step]

**Experiments → New experiment** walks you through five short steps, one question each. The steps on the left tick off as you go, and the card under them sums up your choices. Press <kbd>↵</kbd> to continue, or click a finished step to go back. Your place and everything you've typed survive a page refresh.

1. **Hypothesis**: name the test, and optionally write *We believe … because …*.
2. **Who sees it**: **Everyone**, **A cohort** (pick it, and optionally enroll only a share of it) or **A share** of players. Choose the split (**50 / 50**, **70 / 30** or **3 variants**) and rename variants if you like; `control` is the baseline. The test runs on a **new flag** whose key is suggested from the name (you can edit it; taken keys are flagged), or on an **existing flag** that isn't in another experiment. An existing flag's targeting and variants are replaced by the plan.
3. **What success is**: pick a **game metric**. **D1 retention** is suggested; the others are D7 retention, playtime, Robux, payer conversion and sessions per player. Or pick a **custom event** (**% of players who do it**, **how many times** or **sum of a property**) and its window. The forecast shows the baseline, players needed per variant and roughly how many days that takes at today's traffic.
4. **Guardrails**: metrics that must not get worse. New experiments start with **Playtime per player** and **Robux per player**, so a retention win can't quietly cost engagement or revenue. Add others, such as matches played, or remove them for a small change.
5. **Review & launch**: the plan as a sentence. Click any highlighted part to jump back to its step.

> Split **everyone** **50/50** between control and test on **short-tutorial** new flag. Success = **D1 retention** — players who come back the day after they first see it. Needs ~5,157 players per variant to detect a **10%** lift — about 6 days at today's traffic.

Below it, a checklist shows whether the name, flag and goal are ready, and warns when your code hasn't read the flag yet. **Advanced** opens every setting: notes, variants, metric filters, targeting, segments and the code snippet. Then choose:

- **Save draft**: creates the experiment and its flag (off). Launch it once your code reads the flag.
- **Launch**: creates it and turns the flag on right away.
- **Hand this to your agent**: copies the plan as instructions for Claude, ChatGPT or a coding agent.

## Or describe it [#or-describe-it]

When the assistant is set up, step 1 also offers **describe it in a sentence**. Write it the way you'd say it to a colleague:

> Does a three-step tutorial instead of eight bring more new players back the next day?

RoChart reads your project's events, cohorts and existing flags and drafts a complete plan (name, hypothesis, flag, variants, metrics, guardrails, cohorts to compare, a rationale and warnings), then opens it on **Review**. Drafts use game metrics first: D1 retention for most changes, payer conversion or Robux for shop changes, playtime for content, with playtime and Robux as guardrails. It never invents an event you don't track, and **nothing is created** until you save or launch.

## Choose good metrics [#choose-good-metrics]

The **primary metric** decides the verdict. Every metric is measured per exposed player, from the moment they first read the flag.

**Game metrics** need no extra tracking:

| Metric | Per exposed player | Analysed as |
| --- | --- | --- |
| **D1 retention** | Played (a `$session_start`) on the day after their exposure day | share of players |
| **D7 retention** | Played on the 7th day after their exposure day | share of players |
| **Playtime per player** | Minutes played within the window (session time, as in Game Analytics) | average |
| **Sessions per player** | `$session_start`s within the window | average |
| **Robux per player** | Sum of `$robux` on `$purchase` within the window | average |
| **Payer conversion** | Bought anything within the window | share of players |

Days are in your project's timezone. D1 and D7 only count players whose day has passed: a player exposed today joins the D1 numbers the day after tomorrow, so D1 needs two days before the first result. The exposure counts at the top still include everyone.

**Custom events** cover what your game tracks with `RoChart.track`:

- **Conversion**: the share of exposed players who did an event at least once. Use it for yes/no outcomes: finished the tutorial, joined a clan.
- **Mean**: the average per exposed player of an event count (such as matches played) or the sum of a numeric property (such as coins earned).

In averages, players who never did it count as 0, so "Robux per player" includes players who bought nothing. Shares of players are analysed with a Beta-binomial model, and averages with a Normal model.

Each metric except D1/D7 has a **window**: how many days after their first exposure a player's actions count. Everyone gets the same window, however late they joined. Match it to the behaviour: a day for finishing the tutorial, about 7 days for playtime, Robux or a first purchase.

**Secondary metrics** get full results but don't decide the verdict. Use them as:

- **guardrails**: things that mustn't get worse, like playtime, Robux or matches per player;
- **value**: Robux per player next to a retention or payer-conversion primary.

<Callout type="warn" title="Money metrics are noisy">
  A few big spenders can swing Robux per player. Make Robux a secondary metric, and use **Payer conversion** as the primary for shop changes.
</Callout>

**Minimum lift** (minimum detectable effect) is the smallest relative improvement worth acting on. It only affects the sample size estimate. Smaller lifts need far more players: halving it needs about four times the sample. Rare goals, like a 2% purchase rate, need many more players than common ones.

Decide on the primary metric before launch. You can add a guardrail later, but swapping the primary halfway through is a way to cherry-pick a winner.

## Put the variant in your code [#put-the-variant-in-your-code]

The experiment only gets data when your game reads its flag:

```luau
local function offerStarterPack(player)
  local productId = STARTER_PACK
  if RoChart.getFlag(player, "starter-pack-discount") == "test" then
    productId = STARTER_PACK_DISCOUNTED
  end
  MarketplaceService:PromptProductPurchase(player, productId)
end
```

Websites and backends read flags with the [browser and Node SDKs](/docs/feature-flags/in-code#websites-and-backends).

- **Read the flag where the experience starts**, such as when the offer is shown and not when the player joins. Everyone who reads the flag counts as exposed.
- **Show control for anything else**, including `undefined`.
- **Game metrics need nothing else**: sessions, playtime and retention come from the SDK, and purchases are recorded for you by [`processReceipt`](/docs/getting-started/install#record-purchases). For a custom-event metric, send the event, such as `tutorial_completed`.

See [Flags in your code](/docs/feature-flags/in-code) for details.

## Launch, stop, delete [#launch-stop-delete]

- **Launch** turns the flag on and starts counting.
- **Stop** lets no new players in. The flag keeps serving variants until you conclude or switch it off.
- **Conclude** records the decision. See [Read results](/docs/experiments/results#conclude).
- **Delete** removes the experiment, and its flag too if the flag is off.