Skip to content

Soak test a release candidate

A Switchyard soak test sends sustained traffic through a release-candidate server long enough to expose failures that short tests miss. Run it for 48 hours before code freeze when a release changes libsy, the Rust server, routing, streaming, translation, or server lifecycle behavior.

The test sends closed-loop traffic through:

  • OpenAI Chat Completions (/v1/chat/completions)
  • Anthropic Messages (/v1/messages)
  • OpenAI Responses (/v1/responses)
  • streaming and non-streaming responses
  • four repeated prompt prefixes

The runner also checks /health and /metrics every minute. Every five minutes, it sends an invalid Chat Completions request, expects HTTP 400, and then confirms the server is still live.

Prepare the server

Run the exact commit, build, route bundle, backend, and model planned for the release. Do not use a development server in front of a different Switchyard build.

For the Python server:

uv sync
uv run switchyard --routing-profiles release-routes.yaml -- serve --port 4000 \
  > switchyard-soak.log 2>&1 &
SOAK_SERVER_PID=$!

For the Rust server and its libsy algorithms:

cargo build --release -p switchyard-server
target/release/switchyard-server --config release-routes.toml \
  > switchyard-soak.log 2>&1 &
SOAK_SERVER_PID=$!

Wait for GET http://127.0.0.1:4000/health to return HTTP 200 with the JSON body {"status": "ok"}; the runner requires both before it starts. Check GET http://127.0.0.1:4000/v1/models and choose the model id that represents the release workload.

Run the server and test from a dedicated host, job scheduler, or terminal multiplexer that will stay alive for the full test. Confirm that the host will not suspend or restart and has enough disk space for the server log.

Run the 48-hour test

Choose concurrency from the release capacity plan. Increase it in short rehearsals until you find the highest expected steady load that remains below the backend's rate limit. Use that load for the 48-hour run. An overload test that spends most of its time throttled does not measure release stability.

uv run python scripts/soak_test.py \
  --base-url http://127.0.0.1:4000 \
  --model RELEASE_MODEL_ID \
  --duration 48h \
  --concurrency 16 \
  --server-pid "$SOAK_SERVER_PID" \
  --max-rss-growth-mib 512

The runner keeps 16 requests in flight until the test ends. This can generate large usage charges against a metered backend. Use a dedicated test deployment, estimate the request volume first with a short run, and get approval for any paid-provider cost.

Use a five-minute rehearsal to confirm the route and result files:

uv run python scripts/soak_test.py \
  --base-url http://127.0.0.1:4000 \
  --model RELEASE_MODEL_ID \
  --duration 5m \
  --concurrency 4 \
  --report-interval 10

If the Switchyard endpoint requires a bearer token, pass the environment variable name instead of putting the token on the command line:

export SWITCHYARD_SOAK_TOKEN="..."
uv run python scripts/soak_test.py \
  --api-key-env SWITCHYARD_SOAK_TOKEN \
  --model RELEASE_MODEL_ID

Pass criteria

The command exits with status 0 only when:

  • the requested duration completes;
  • at least one inference request completes;
  • the inference error rate stays at or below --max-error-rate (default 0, which means no inference request may fail);
  • every periodic liveness check passes;
  • every /metrics read returns both Switchyard request counters;
  • every requested process sample returns RSS and CPU data;
  • every invalid-request recovery check passes;
  • the server request counter never resets; and
  • RSS growth stays within --max-rss-growth-mib when that limit is set.

If the release plan permits transient failures from a remote provider, set an explicit error budget with --max-error-rate. Record the reason for that exception in the release record.

The RSS limit is deployment-specific. Set it from an approved baseline for the same model, concurrency, and worker count. Omit --server-pid and --max-rss-growth-mib when the server runs on another host, then collect memory and restart data from that host's monitoring system.

Review the results

Each run creates a timestamped directory under soak-results/:

  • config.json records the non-secret test inputs.
  • intervals.csv records request rate, errors, latency percentiles, health, Switchyard counters, RSS, and CPU once per reporting interval. cpu_percent is the ps lifetime-average CPU for the process, not the interval's usage, so read it as a long-run average rather than a spike detector.
  • errors.jsonl records up to 10,000 request and canary failures.
  • summary.json records the final pass result and any failed gates.

Before approving the release, check intervals.csv for late failures, falling throughput, increasing p95 or p99 latency, and steady RSS growth. Compare the first and last several hours, not only the run-wide averages. Attach summary.json, the interval chart, the Switchyard log, the tested commit, and the route bundle to the release record.