# AI: A/B test two prompts

> Serve prompt or model variant A vs B by rollout, deterministically per user, and measure the winner in your own analytics.

Run two system prompts (or two models) side by side and let a percentage
rollout decide who sees which. Switchbox delivers the split, deterministically
per user; you measure which one wins in your own analytics. This is the
server-side A/B pattern, built on the same two-value flags as the
[model config recipe](/docs/recipes/ai-model-config).

One boundary up front, because it's the whole shape of this recipe: **Switchbox
delivers the variant, it does not measure it.** It has no view of token cost,
latency, or answer quality, and it does not compute which variant won. Pair it
with your eval or analytics tool (Langfuse, PostHog, Helicone, ...) to judge the
result. If you want the vendor to run the statistics for you, that's a different
kind of product.

## What you'll build

One `json` flag, `assistant_prompt`, whose two values are your control and your
treatment. Bundling the variant label into the value means your code can both
use the prompt and report which arm the user landed in, without guessing:

- **off value** (`default_value`): the control, `{ "variant": "control", "system_prompt": "..." }`.
- **on value** (`enabled_value`): the treatment, `{ "variant": "treatment", "system_prompt": "..." }`.
- **Rollout**: `50%`, so half your users get the treatment and half stay on control.

Create the flag in the dashboard with those two values, then wire it in.

## 1. Create the client once

Create one client at startup and reuse it (see the
[model config recipe](/docs/recipes/ai-model-config) for the lifecycle detail).

```python
from switchbox import Switchbox

client = Switchbox(sdk_key="your-server-sdk-key")
```

## 2. Read the variant and log the exposure

Read the flag with the user context, use its prompt, and record **which variant
this user saw** in your analytics. That exposure event is what lets you tie an
outcome back to an arm later.

```python
CONTROL = {"variant": "control", "system_prompt": "You are a helpful assistant."}

def answer(user_id: str, messages: list[dict]) -> str:
    user = {"user_id": user_id}

    cfg = client.get_value("assistant_prompt", user=user, default=CONTROL)

    # Log the exposure: this user was assigned this arm. You measure, not us.
    analytics.capture(
        distinct_id=user_id,
        event="assistant_prompt_exposure",
        properties={"variant": cfg["variant"]},
    )

    return llm.chat(
        system=cfg["system_prompt"],
        messages=messages,
    )
```

Because the value carries `variant`, the assignment and the content stay in one
place: no separate lookup to work out which arm a user is in, and no fragile
string comparison against the prompt text.

## 3. Record the outcome

Send whatever you're optimising for, a thumbs-up, a resolved ticket, a token
cost, to the same analytics tool, keyed by the same `user_id`. Your tool joins
the outcome to the exposure and tells you which arm won. Switchbox is not in this
step at all.

```python
analytics.capture(
    distinct_id=user_id,
    event="assistant_answer_rated",
    properties={"helpful": True},
)
```

## 4. Ship the winner

When the numbers are in, there's no separate "conclude experiment" action. It's
the same rollout dial you already have:

- Treatment won? Set `assistant_prompt` to `100%`, everyone gets it.
- Control won? Set it to `0%`, everyone stays on control.
- Inconclusive? Leave it, or set it to `0%` and try a new treatment value.

Every server picks up the change within ~30 seconds, no redeploy.

## Same split in every SDK

Assignment is a deterministic hash of `user_id` and the flag key, and it is
**identical across the Python and JavaScript SDKs** (pinned by the shared parity
vectors). So a user who reads the flag from your Python backend and again from a
Node service lands in the same arm both times. A given user never flips variant
between polls or between services, which is exactly what you want for a clean
experiment.

## Guardrails

- **You measure, we deliver.** No experiment analytics, no LLM observability.
  Judge the result in your own tool.
- **Server-side only.** Flag JSON is world-readable by its SDK key, so read
  prompts from your backend, not a browser bundle where they'd be public.
- **Config, not secrets.** Never put a model API key in a flag.

## Where to next

- [AI: model config in Python](/docs/recipes/ai-model-config) — the model kill switch and gradual migration companion.
- [Percentage rollouts](/docs/concepts/rollouts) — how deterministic bucketing works.
- [Switchbox for AI apps](/ai) — the full picture of the AI wedge.