AI: A/B test two prompts

Serve prompt or model variant A vs B by rollout, deterministically per user, and measure the winner in your own analytics.

Run two system prompts (or two models) side by side and let a percentage rollout decide who sees which. Switchbox delivers the split, deterministically per user; you measure which one wins in your own analytics. This is the server-side A/B pattern, built on the same two-value flags as the model config recipe.

One boundary up front, because it's the whole shape of this recipe: Switchbox delivers the variant, it does not measure it. It has no view of token cost, latency, or answer quality, and it does not compute which variant won. Pair it with your eval or analytics tool (Langfuse, PostHog, Helicone, ...) to judge the result. If you want the vendor to run the statistics for you, that's a different kind of product.

What you'll build

One json flag, assistant_prompt, whose two values are your control and your treatment. Bundling the variant label into the value means your code can both use the prompt and report which arm the user landed in, without guessing:

  • off value (default_value): the control, { "variant": "control", "system_prompt": "..." }.
  • on value (enabled_value): the treatment, { "variant": "treatment", "system_prompt": "..." }.
  • Rollout: 50%, so half your users get the treatment and half stay on control.

Create the flag in the dashboard with those two values, then wire it in.

1. Create the client once

Create one client at startup and reuse it (see the model config recipe for the lifecycle detail).

from switchbox import Switchbox


client = Switchbox(sdk_key="your-server-sdk-key")

2. Read the variant and log the exposure

Read the flag with the user context, use its prompt, and record which variant this user saw in your analytics. That exposure event is what lets you tie an outcome back to an arm later.

CONTROL = {"variant": "control", "system_prompt": "You are a helpful assistant."}


def answer(user_id: str, messages: list[dict]) -> str:
    user = {"user_id": user_id}


    cfg = client.get_value("assistant_prompt", user=user, default=CONTROL)


    # Log the exposure: this user was assigned this arm. You measure, not us.
    analytics.capture(
        distinct_id=user_id,
        event="assistant_prompt_exposure",
        properties={"variant": cfg["variant"]},
    )


    return llm.chat(
        system=cfg["system_prompt"],
        messages=messages,
    )

Because the value carries variant, the assignment and the content stay in one place: no separate lookup to work out which arm a user is in, and no fragile string comparison against the prompt text.

3. Record the outcome

Send whatever you're optimising for, a thumbs-up, a resolved ticket, a token cost, to the same analytics tool, keyed by the same user_id. Your tool joins the outcome to the exposure and tells you which arm won. Switchbox is not in this step at all.

analytics.capture(
    distinct_id=user_id,
    event="assistant_answer_rated",
    properties={"helpful": True},
)

4. Ship the winner

When the numbers are in, there's no separate "conclude experiment" action. It's the same rollout dial you already have:

  • Treatment won? Set assistant_prompt to 100%, everyone gets it.
  • Control won? Set it to 0%, everyone stays on control.
  • Inconclusive? Leave it, or set it to 0% and try a new treatment value.

Every server picks up the change within ~30 seconds, no redeploy.

Same split in every SDK

Assignment is a deterministic hash of user_id and the flag key, and it is identical across the Python and JavaScript SDKs (pinned by the shared parity vectors). So a user who reads the flag from your Python backend and again from a Node service lands in the same arm both times. A given user never flips variant between polls or between services, which is exactly what you want for a clean experiment.

Guardrails

  • You measure, we deliver. No experiment analytics, no LLM observability. Judge the result in your own tool.
  • Server-side only. Flag JSON is world-readable by its SDK key, so read prompts from your backend, not a browser bundle where they'd be public.
  • Config, not secrets. Never put a model API key in a flag.

Where to next