Run an Audit Locally¶

This tutorial walks through running a single audit end-to-end with the NeMo Auditor plugin SDK. You will persist an audit configuration and a target, run the audit in-process, and inspect the resulting report artifacts.

What you will learn:

Initialize the NeMo Platform SDK and reach the client.auditor resource.
Create an AuditConfig selecting which garak probes to run.
Create an AuditTarget pointing at a model through the Inference Gateway.
Execute the audit locally with client.auditor.run(...).
Read the JSONL, HTML, and hitlog report artifacts the run produces.

Tip

This tutorial takes approximately 10 minutes to complete, plus however long garak takes to run the selected probes against your target.

Prerequisites¶

Install and start NeMo Platform using the Setup guide.
Install garak in a Python virtual environment so the plugin can shell out to it. By default the plugin invokes ~/.auditor/.venv/bin/python -m garak. Override the interpreter path with NEMO_AUDITOR_GARAK_PYTHON if your install lives elsewhere.
Configure at least one Inference Gateway provider — this tutorial uses a build provider named build that routes to NVIDIA-build models, but any chat-completion-compatible provider works. See Inference Gateway for details on the nmp_uri_spec block used below.

Key Concepts¶

AuditConfig — Selects probes, detectors, and reporting settings for a garak run. Persisted in the entity store.
AuditTarget — Identifies the model under test (generator class, model identifier, endpoint options). Persisted in the entity store.
In-process run — client.auditor.run(...) shells out to garak on the host running the SDK call, writes report artifacts under a temporary directory, and returns their file paths. There is no remote job submission.

1. Initialize the SDK¶

Connect to the running platform and reach the auditor sub-resource:

import os

from nemo_platform import NeMoPlatform


client = NeMoPlatform(
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
    workspace="default",
)
auditor = client.auditor

print(auditor.plugin_status())
# {'plugin': 'auditor', 'status': 'ok', ...}

If plugin_status() raises a connection error, the platform is not running or NMP_BASE_URL is misconfigured. Refer back to Setup.

2. Create an Audit Configuration¶

Build a configuration that runs a small slice of garak's latent-injection probes with three generations per probe:

from nemo_auditor.entities import (
    AuditPluginsData,
    AuditReportData,
    AuditRunData,
    AuditSystemData,
)


config = auditor.configs.create(
    workspace="default",
    name="quick-scan",
    description="Lite latentinjection scan, 3 generations per probe.",
    system=AuditSystemData(lite=True, parallel_attempts=4),
    run=AuditRunData(generations=3),
    plugins=AuditPluginsData(probe_spec="latentinjection", detector_spec="auto"),
    reporting=AuditReportData(report_prefix="quick-scan"),
)
print(config.model_dump_json(indent=2))

For the full set of options on each sub-block, see Configuration Schema. For probe selection syntax, see Selecting Probes.

3. Create an Audit Target¶

Define the model under test. This example uses garak's nim.NVOpenAIChat generator pointed at a provider registered in the Inference Gateway:

target = auditor.targets.create(
    workspace="default",
    name="llama-31-8b",
    type="nim.NVOpenAIChat",
    model="meta/llama-3.1-8b-instruct",
    options={
        "nim": {
            "max_tokens": 1024,
            "nmp_uri_spec": {
                "inference_gateway": {
                    "workspace": "default",
                    "provider": "build",
                },
            },
        },
    },
)
print(target.model_dump_json(indent=2))

The nmp_uri_spec sentinel inside options.nim tells the plugin to resolve a concrete URI from the Inference Gateway provider at run time. See Inference Gateway for the full conflict rules and resolution behavior.

4. Run the Audit Locally¶

run() accepts either inline entities or name strings that reference entities in the entity store. Pass the names you persisted above:

result = auditor.run(
    config="quick-scan",
    target="llama-31-8b",
    workspace="default",
)

print(result["status"], result["returncode"])
# completed 0

for name, ref in result["results"].items():
    print(f"{name}: {ref['artifact_url']}")
# report-jsonl: file:///var/folders/.../results/report-jsonl
# report-html: file:///var/folders/.../results/report-html
# report-hitlog-jsonl: file:///var/folders/.../results/report-hitlog-jsonl

If you prefer to skip the entity-store roundtrip, pass the inline AuditConfig and AuditTarget objects from the previous two steps directly:

result = auditor.run(config=config, target=target, workspace="default")

5. Read the Results¶

run() returns file:// URLs for whichever of the three report types garak produced. Parse them with urllib.parse.urlparse to recover local paths, then load them as needed.

Load the JSONL probe-by-probe summary:

import json
from pathlib import Path
from urllib.parse import urlparse


def url_to_path(url: str) -> Path:
    return Path(urlparse(url).path)


jsonl_path = url_to_path(result["results"]["report-jsonl"]["artifact_url"])
records = [json.loads(line) for line in jsonl_path.read_text().splitlines() if line.strip()]
print(f"Loaded {len(records)} report entries.")

Summarize hits per probe from the hitlog:

from collections import Counter


hitlog_ref = result["results"].get("report-hitlog-jsonl")
if hitlog_ref is None:
    print("No hits recorded — every probe passed.")
else:
    hitlog_path = url_to_path(hitlog_ref["artifact_url"])
    hits = [json.loads(line) for line in hitlog_path.read_text().splitlines() if line.strip()]
    by_probe = Counter(entry.get("probe") for entry in hits)
    for probe, count in by_probe.most_common():
        print(f"{probe}: {count} hits")

The HTML report at result["results"]["report-html"]["artifact_url"] is the most human-friendly summary — open it in a browser to see grouped pass/fail counts and per-probe details.

6. Clean Up¶

Delete the entities you created:

auditor.configs.delete(workspace="default", name="quick-scan")
auditor.targets.delete(workspace="default", name="llama-31-8b")

Report artifacts are left in place under the scheduler's temporary directory; the OS reaps them eventually.

Troubleshooting¶

FileNotFoundError: garak interpreter not found at ...: The plugin couldn't find a garak install. Either install garak at ~/.auditor/.venv/bin/python, or set NEMO_AUDITOR_GARAK_PYTHON to the absolute path of a Python interpreter that has garak installed.
RuntimeError: Failed to resolve inference gateway provider '<workspace>/<provider>': The nmp_uri_spec block in your target's options references an Inference Gateway provider that doesn't exist. List your providers with client.inference.providers.list(workspace="default") and update the target to reference an existing one.
returncode != 0 with empty results: garak started but failed early, usually because the target endpoint is unreachable. Inspect result["stderr_tail"] for the error message, and verify the model is reachable through the provider you configured.
Where do the report files live?: The scheduler writes garak output under <temp-dir>/garak/<reporting.report_dir>/<reporting.report_prefix>.*, then copies the produced files into the local results directory referenced by each artifact_url. For a local run this is a temporary directory under the system's $TMPDIR.

Summary and Next Steps¶

You created an AuditConfig and AuditTarget, ran a single audit locally with client.auditor.run(...), and loaded the resulting reports.

For the full SDK surface — including async variants — see SDK Resources.
For more probe selection options, see Selecting Probes.
For other generator types (OpenAI-compatible endpoints, REST targets, test targets), see Audit Targets.