AI: model config in Python

Serve model choice and prompt parameters as edge config, with a rollout-based gradual model migration and a kill switch.

Run your AI app's model choice and parameters as edge config instead of code. Swap models, tune the system prompt, or roll a new model out to a slice of traffic, all from the dashboard, with no redeploy. Your backend reads the config from Cloudflare's edge and calls the model itself. Switchbox is never in your inference path.

This is a server-side recipe. Read the guardrails first: flag JSON is world-readable by its key, so this belongs in your backend, not a browser bundle.

What you'll build

Two flags drive a chat endpoint:

  • chat_model — a string flag that holds the model name. Its off value is your current model and its on value is the one you're migrating to, so a rollout percentage moves traffic from one to the other.
  • assistant_params — a json flag holding the tunable knobs: system prompt, temperature, and token cap.

Create both in the dashboard first (see the Quickstart if you haven't made a flag yet), then wire them in.

1. Create the client once

Create one client when your process starts and reuse it. It fetches the config on creation, then refreshes in the background every 30 seconds.

from switchbox import Switchbox


client = Switchbox(sdk_key="your-server-sdk-key")

2. Read the config and call the model

Both reads are in-memory lookups against the cached config, no network call. Pass the user context so targeting rules and rollouts can bucket per user.

# Safe fallback params in case the flag is ever missing at read time.
DEFAULT_PARAMS = {
    "system_prompt": "You are a concise, helpful assistant.",
    "temperature": 0.7,
    "max_tokens": 1024,
}


def answer(user_id: str, messages: list[dict]) -> str:
    user = {"user_id": user_id}


    model = client.get_value("chat_model", user=user, default="gpt-4o")
    params = client.get_value("assistant_params", user=user, default=DEFAULT_PARAMS)


    # `llm` is your own provider client (OpenAI, Anthropic, ...).
    # Switchbox delivers the config; your code makes the call.
    return llm.chat(
        model=model,
        system=params["system_prompt"],
        temperature=params["temperature"],
        max_tokens=params["max_tokens"],
        messages=messages,
    )

get_value() returns the resolved value for any flag type: the string for chat_model, the parsed dict for assistant_params. The default is returned only if the flag doesn't exist, so a config that hasn't loaded yet or a typo'd key degrades safely instead of raising.

3. Migrate models gradually with a rollout

Say you're on gpt-4o and want to move to claude-sonnet-4-6. Set the chat_model flag's values and dial the rollout:

  • off value (default_value): gpt-4o — everyone starts here.
  • on value (enabled_value): claude-sonnet-4-6 — the target model.
  • Rollout: 0% → 10% → 50% → 100%.

At 10%, one deterministic tenth of your users get claude-sonnet-4-6; the rest stay on gpt-4o. Bucketing is by user_id, so a given user is stable across polls and across every server in your fleet: they don't flip model mid-session. Watch your own quality and cost metrics between steps, and if the new model regresses, set the rollout back to 0%. Every server picks up the change within ~30 seconds. No deploy, no incident call.

You measure, we deliver. Switchbox ships the config and records who changed it in the audit log, but it has no view of token cost, latency, or answer quality. Judge the migration in your own eval or analytics tool (Langfuse, PostHog, Helicone, ...).

4. Keep a kill switch

Add a boolean ai_assistant_enabled flag and check it before you call the model. If a provider has an outage or starts burning money, flip it off and every server falls back to a canned response within ~30 seconds:

def answer(user_id: str, messages: list[dict]) -> str:
    if not client.enabled("ai_assistant_enabled", user={"user_id": user_id}):
        return "The assistant is briefly unavailable. Please try again shortly."
    ...

Because the config serves from Cloudflare's edge, this switch works even if the Switchbox API, database, and the rest of our backend are down. That's the point: a model kill switch that can't itself go down.

Guardrails

A few boundaries to design around, not around:

  • Config, not secrets. Never put a model API key (OpenAI, Anthropic, ...) in a flag. Switchbox is delivery config, not a secrets manager. Keep keys in your own environment or secret store.
  • Server-side only. Flag JSON is world-readable by its SDK key. Read it from your backend, where the key and your prompts stay private, not from a browser bundle where both would be public.
  • A few prompts, not a library. The config is static edge JSON, ideal for a handful of prompts and parameters. A large versioned prompt library per environment is outside the architecture's sweet spot.

Where to next