How ConfigDirector computes results

Once you attach metrics to an experiment, the Results panel on its Experiment tab compares each variation with the control. This page explains where every number on that panel comes from: what ConfigDirector counts, what it leaves out, how it compares the variations and what it does not do. It also explains how the How long to run card estimates the length of a run.

The examples follow checkout-cta-copy from the Experiments guide, which compares two labels for a checkout button: control and treatment-a.

What is counted

Exposures

A context is exposed to a variation when the experiment's percentage rollout serves it that variation during a run. It counts once per variation, however often it evaluates the experiment, so results compare contexts rather than page views or evaluations.

ConfigDirector tells contexts apart by their context kind and id. user 42 and device 42 are two contexts, and so are a person signed in as a user and the same person's device before signing in: ConfigDirector does not merge them. Contexts without an id, or marked anonymous, are never counted. Give every context a stable identifier explains what they are served instead.

Evaluations served by a conditional rule's single value or by the default targeting rule are not exposures, because those contexts were not assigned a variation at random.

Results are per percentage rollout. A run with two rollouts, such as one inside a conditional rule for plan is one of pro, team and one for everyone else, gets separate results for each, and they are never combined.

Metrics

A metric is an event your application reports with track, defined once per project on the Metrics page. It has a direction, Higher is better or Lower is better, and the metric kinds it reports. You attach metrics to an experiment on its Settings tab, each with a role: one primary metric and kind decide the experiment, secondary metrics are supporting evidence, and a guardrail is a metric that must not get worse.

Each metric kind turns the events of a variation's contexts into one number:

A context that reported nothing counts as zero in both averages, so they are averages over every exposed context, not only over the ones that reported the event. An event reported without a value counts toward Count per context and adds nothing to Value per context.

A metric's results cover the whole run, even when you attach it, or tick one of its kinds, after the run started. The one exception is a run that had already counted 100 other event names, the most a run counts, before you attached the metric: its event then counts from that moment on.

Which events count

An event counts toward a run when it happened while the run was open, at or after its context's first exposure in that run. From that first exposure to the end of the run, every event the context reports counts. There is no shorter window, and nothing after the run ends counts.

SDKs report evaluations in batches, about every 30 seconds, and ConfigDirector takes the start of the batch as the moment of exposure. So an event up to about 30 seconds before the exposure can count.

An event names no experiment. It counts toward every run open in its environment that exposed its context: when two experiments run at once in Production, the checkout_completed of a context exposed to both counts in each.

If a context's exposure reaches ConfigDirector more than five minutes after its event, for example from a phone that was offline, the event may not be counted. Events dropped by a tracked event limit are never counted.

What is left out

Your team's own accounts, test accounts and bots are counted like any other context unless you keep them out of the split, as Keep internal users and bots out shows.

Crossovers

A crossover is a context exposed to more than one variation of the same rollout during a run. A context's share of the split depends only on its context kind and id, so it normally sees one variation for the whole run. A crossover happens when a variation's value is edited during a run.

Counting a crossover under each variation it saw would pull the variations' numbers toward each other. So ConfigDirector leaves it out of every metric, from every variation: its exposures, and every event it reported, before and after it crossed over. The panel says how many contexts crossed over, and each variation they saw counts them out, so the metrics count fewer contexts than the Exposures card.

Variations outside the split

A variation with a 0% share takes no part in the results, and neither does one that was not in the split when the run started.

The control

Every variation is compared with the rollout's control: the variation the default targeting rule served when the run started, if the split gives it a share. Otherwise it is the first variation in the split that has a share. That's why the Default variation you pick when you create an experiment is usually your control.

The control is fixed when the run starts, so changing the default targeting rule value during the run does not change it. It comes first in each table, tagged Control, with no lift or chance to beat control of its own.

How the variations are compared

ConfigDirector uses a Bayesian method with flat priors. It starts with no opinion about any variation, so every number comes from your run's own data, and it answers the question a team asks directly: given what has happened so far, how likely is this variation to be better than the control, and by how much?

Each variation's row shows these numbers:

• The metric's value, such as 14.1% with 176 of 1,246 beneath it for Conversion: the share or average described in Metrics, over that variation's exposed contexts.
• Lift: how much higher or lower the variation's value is than the control's, relative to the control's. A lift of +10% on a control conversion of 12% means about 13.2%. For a metric where lower is better, a negative lift is the good direction.
• Likely range: where the real lift very likely is, given the data so far. The real lift is as likely to be above the range as below it. A range that spans zero means the variation could still turn out better or worse than the control. The range narrows as the run gathers contexts.
• Chance to beat control: the chance that the variation's real value is better than the control's, in the metric's direction. It answers whether the variation is better, not by how much: the lift and its range answer that. While no context has reported the event, it is 50% for Count per context and Value per context, and about 50% for Conversion when the variations have similar exposures.
• Chance to be best, with three or more variations: the chance that each variation, the control included, is the best of them all in the metric's direction. Across the variations, the chances add up to 100%.

The panel rounds the lift, its range and the chances to whole percentages, and shows a chance that rounds to 0% as <1% and one that rounds to 100% as >99%.

The details behind each number

• For Conversion, the lift and its range compare the rates after counting one extra context that reported the event and one that did not in each variation. That is what starting with no opinion means. With hundreds of conversions it barely moves the lift away from the ratio of the rates shown; with only a few, it can move it a lot.
• For Count per context and Value per context, the lift compares the averages shown. It is left out when the control's average is zero or below, because there is nothing to take a relative change of, and the range is left out while the control's average is too uncertain to tell apart from zero. The chance to beat control and the verdict still show.
• Chance to be best is worked out by simulating the comparison many times. The same counts on the same day always give the same number, so reloading the page does not change it, and once a run has its final results, its chance to be best no longer changes.

A worked example

checkout-cta-copy has one primary metric, Checkout, on the checkout_completed event, with Conversion as its primary kind. Partway through the run in Production, the Results panel shows:

treatment-a converts at 14.1% against the control's 12.7%, a lift of +11%. The likely range, −10% to +36%, still spans zero: the data so far allows treatment-a to be somewhat worse than the control, or a lot better. Its chance to beat control is 84%. That sounds high, but it leaves about a one-in-six chance that treatment-a is not better, which is not enough to call it likely better, so its verdict is No clear difference. With two variations, there is no chance to be best.

If the gap held as more contexts were exposed, the range would narrow and the chance to beat control would climb, until the verdict read Likely better.

When results appear

The panel shows each variation's exposures and value from the start of the run. The lift, the likely range, the chances and the verdict appear once every variation in the rollout has enough exposures, not counting crossovers. Until then, the panel says the run needs more exposures.

The minimum is the same for every metric, and high enough that the Exposures card can already check the split against your shares, so a result never shows while that check still needs more exposures. In a run with two rollouts, one can show results while the other is still waiting.

What the verdict means

Each variation gets a verdict for each metric kind, in the metric's direction:

The bar is high, the same in both directions and the same for every metric, and it cannot be changed. A Likely better verdict still leaves a small chance that the variation is not better.

A verdict informs a decision and never makes one: nothing ships or stops on its own, and your team decides. When you ship a variation, the Ship dialog shows the primary metric's verdict for it and warns you when a guardrail is likely worse.

How long a run needs

The How long to run card on the Experiment tab estimates how many exposures a run needs before a variation has a good chance of reading Likely better on the primary metric, in the kind it decides on, and how many days that takes. It gives a row for each of five lifts: 2%, 5%, 10%, 20% and 50%, in the primary metric's better direction, so they read −2% to −50% when lower is better.

A row's lift is one a variation might really have, not the one measured so far. The smaller it is, the longer the run: half the lift takes about four times the exposures. Pick the smallest lift that would be worth shipping, and read its row.

Before a run

While no run is open in the environment, the card estimates from two numbers you type for Conversion: the control's conversion today, and how many distinct contexts would reach the experiment each day. Each row gives the exposures needed, all variations together, and the days it takes to expose that many contexts. When the environment has an ended run with results, both numbers start from the most recent one: what its control converted, and how many contexts it exposed a day. You can change them, and nothing you type is saved.

For Count per context and Value per context, the estimate also needs how much the number varies from one context to the next, which nobody knows before measuring it. So for those kinds the card estimates before a run only once an earlier run in the environment has results. It uses what that run's control averaged and how much it varied, and only the contexts a day can be typed.

While a run is open

During a run, the card estimates from the run's own numbers for each percentage rollout: the control's value so far, the run's shares, and the exposures so far, without the crossovers. Each row gives the exposures so far against those needed, and the days remaining at the run's recent pace: the new exposures a day over about the last 7 days, or since the run started when it is younger than that. The card says since when.

A row that says Enough exposures means the run is long enough for that lift: a variation that really is that much better has a good chance of reading Likely better by now. The estimate appears once the variations can be compared, as When results appear describes.

What the estimate assumes

• The control's numbers. Before a run, they are what you type or what the last run measured. During a run, they are what the control has measured so far, which moves early in the run, and the estimate moves with it.
• A good chance, not a certainty. At the exposures a row gives, a variation whose real lift is that large has a good chance of reading Likely better, but it can still read No clear difference. A variation whose real lift is smaller needs a longer run.
• The pace holds. The days assume contexts keep arriving at the pace you typed or the run measured. New contexts usually arrive more slowly as a run goes on, because the contexts that come back often were exposed in its first days. So take the days as the least the run needs.
• The split. Before a run, the card assumes an even split between the experiment's variations. The smallest share decides, so an uneven split, or another variation, needs more exposures.

A lift that would take the conversion to 100% or past it gets no row. During a run, while the control's conversion or average is zero, as when none of its contexts has reported the event yet, there is nothing to take a lift of, and the card says no estimate can be made.

Whatever the estimate says, run for at least one full week, so every day of the week is in the result.

An example

Before starting checkout-cta-copy in Production, the team expects the control to convert 12.7%, and about 400 contexts a day to reach the experiment, split 50/50 between control and treatment-a. The card reads:

A lift of +10% is the smallest the team would ship for, so they plan a run of about 45 days, rounded up to seven full weeks.

Partway through the run, at the point of the worked example, the run has 2,400 exposures, the control converts 147 of 1,154, and the run exposes about 400 new contexts a day. The card now reads the control's own numbers:

treatment-a reads No clear difference so far, with a lift of +11%. The run has enough exposures for a lift of +50%, but not yet for one of +10%, so the team keeps to its plan.

How fresh the numbers are

While a run is going, its results can be up to 25 minutes behind what your applications report, and the panel's heading says so.

The daily chart of each metric kind adds a point when a day ends in UTC, with the value for the whole run up to that moment. Its last point holds the current numbers, the same as the table above it: a hollow point labeled Now while the run is going, and a point labeled Run ended once the run has ended. Hover over a day's point to see when that day ended in your own time zone, and over Run ended to see when the run ended.

When a run ends, its results keep taking in the last reports for up to 25 minutes. Then the panel shows the Final results, as of the moment the run ended, and they no longer change: an event that arrives later is not added. A run that ended with no exposures, or before ConfigDirector had results, has no final results, and the panel shows its dates instead. The Ship dialog shows results that can be up to 25 minutes behind, so the final results can differ slightly from what you saw when you shipped.

What the method does not do

• No sequential test. ConfigDirector does not ask you to fix the run's length in advance or adjust its numbers for how often you look. Its chances mean what they say each time you read them, but early in a run they can swing a lot. See Reading results well.
• No variance reduction. Results use only what happened during the run, not what each context did before it, so the ranges narrow only as the run gathers contexts.
• No correction for several metrics. Each metric and kind is compared on its own. The more you attach, the more likely one of them reads Likely better or Likely worse by chance alone. That's why the primary metric decides and secondary metrics only support it.

Reading results well

• Pick the primary metric before you start the run. Choosing it after you have seen the results makes it easy to pick whichever metric happens to look good.
• Don't stop on the first Likely better day. The chance to beat control can cross the line and fall back, especially early in a run. Decide before you start how long the run will go, and read the verdict then. How long a run needs helps you size it, in full weeks so that every day of the week is in it.
• Watch the guardrails. A guardrail that reads Likely worse is a reason not to ship, even when the primary metric reads Likely better. The panel and the Ship dialog both warn you.
• Check the split first. When the Exposures card says the split does not match the configured shares, the run's results are not reliable.