# AI: model config in Python

> Serve model choice and prompt parameters as edge config, with a rollout-based gradual model migration and a kill switch.

Run your AI app's model choice and parameters as edge config instead of code.
Swap models, tune the system prompt, or roll a new model out to a slice of
traffic, all from the dashboard, with no redeploy. Your backend reads the config
from Cloudflare's edge and calls the model itself. Switchbox is never in your
inference path.

This is a **server-side** recipe. Read the [guardrails](#guardrails) first: flag
JSON is world-readable by its key, so this belongs in your backend, not a browser
bundle.

## What you'll build

Two flags drive a chat endpoint:

- **`chat_model`** — a `string` flag that holds the model name. Its **off value**
  is your current model and its **on value** is the one you're migrating to, so a
  rollout percentage moves traffic from one to the other.
- **`assistant_params`** — a `json` flag holding the tunable knobs: system prompt,
  temperature, and token cap.

Create both in the dashboard first (see the [Quickstart](/docs/quickstart) if you
haven't made a flag yet), then wire them in.

## 1. Create the client once

Create one client when your process starts and reuse it. It fetches the config on
creation, then refreshes in the background every 30 seconds.

```python
from switchbox import Switchbox

client = Switchbox(sdk_key="your-server-sdk-key")
```

## 2. Read the config and call the model

Both reads are in-memory lookups against the cached config, no network call.
Pass the `user` context so targeting rules and rollouts can bucket per user.

```python
# Safe fallback params in case the flag is ever missing at read time.
DEFAULT_PARAMS = {
    "system_prompt": "You are a concise, helpful assistant.",
    "temperature": 0.7,
    "max_tokens": 1024,
}

def answer(user_id: str, messages: list[dict]) -> str:
    user = {"user_id": user_id}

    model = client.get_value("chat_model", user=user, default="gpt-4o")
    params = client.get_value("assistant_params", user=user, default=DEFAULT_PARAMS)

    # `llm` is your own provider client (OpenAI, Anthropic, ...).
    # Switchbox delivers the config; your code makes the call.
    return llm.chat(
        model=model,
        system=params["system_prompt"],
        temperature=params["temperature"],
        max_tokens=params["max_tokens"],
        messages=messages,
    )
```

`get_value()` returns the resolved value for any flag type: the string for
`chat_model`, the parsed dict for `assistant_params`. The `default` is returned
only if the flag doesn't exist, so a config that hasn't loaded yet or a typo'd
key degrades safely instead of raising.

## 3. Migrate models gradually with a rollout

Say you're on `gpt-4o` and want to move to `claude-sonnet-4-6`. Set the
`chat_model` flag's values and dial the rollout:

- **off value** (`default_value`): `gpt-4o` — everyone starts here.
- **on value** (`enabled_value`): `claude-sonnet-4-6` — the target model.
- **Rollout**: `0%` → `10%` → `50%` → `100%`.

At 10%, one deterministic tenth of your users get `claude-sonnet-4-6`; the rest
stay on `gpt-4o`. Bucketing is by `user_id`, so a given user is stable across
polls and across every server in your fleet: they don't flip model mid-session.
Watch your own quality and cost metrics between steps, and if the new model
regresses, set the rollout back to `0%`. Every server picks up the change within
~30 seconds. No deploy, no incident call.

> You measure, we deliver. Switchbox ships the config and records who changed it
> in the audit log, but it has no view of token cost, latency, or answer quality.
> Judge the migration in your own eval or analytics tool (Langfuse, PostHog,
> Helicone, ...).

## 4. Keep a kill switch

Add a boolean `ai_assistant_enabled` flag and check it before you call the model.
If a provider has an outage or starts burning money, flip it off and every server
falls back to a canned response within ~30 seconds:

```python
def answer(user_id: str, messages: list[dict]) -> str:
    if not client.enabled("ai_assistant_enabled", user={"user_id": user_id}):
        return "The assistant is briefly unavailable. Please try again shortly."
    ...
```

Because the config serves from Cloudflare's edge, this switch works even if the
Switchbox API, database, and the rest of our backend are down. That's the point:
a model kill switch that can't itself go down.

## Guardrails

A few boundaries to design around, not around:

- **Config, not secrets.** Never put a model API key (OpenAI, Anthropic, ...) in a
  flag. Switchbox is delivery config, not a secrets manager. Keep keys in your own
  environment or secret store.
- **Server-side only.** Flag JSON is world-readable by its SDK key. Read it from
  your backend, where the key and your prompts stay private, not from a browser
  bundle where both would be public.
- **A few prompts, not a library.** The config is static edge JSON, ideal for a
  handful of prompts and parameters. A large versioned prompt library per
  environment is outside the architecture's sweet spot.

## Where to next

- [Percentage rollouts](/docs/concepts/rollouts) — how deterministic bucketing works.
- [Flags & flag types](/docs/concepts/flags) — string and json flags in detail.
- [Python SDK reference](/docs/sdk/python) — every method and option.
- [Switchbox for AI apps](/ai) — the full picture of the AI wedge.