Multimodal Configuration Reference¶

MultimodalSDGConfig defines the configuration for run_multimodal_sdg. The module command loads the same fields from YAML. Unknown configuration fields are rejected. For a first run, use the step-by-step guide.

Run Configuration¶

Paths in the configuration resolve from the current working directory. Only image paths inside source records resolve relative to their JSONL file.

The plugin groups requests by resolved model settings, response schema, and presence of images. Each batch runs a Data Designer structured-response column. Matching requests reuse cached responses keyed by their prompt, schema, model settings, and image content.

missing_response_attempts limits attempts for missing or invalid rows after a batch completes normally. Valid responses are cached before retrying the missing rows. Runtime exceptions stop the run after preserving available completed rows; continuing requires explicit resume. Unexpected or duplicate response identities stop the run before new responses from that attempt are cached.

This limit is separate from Data Designer's structured-response correction and provider retry behavior. Batch size and retry settings do not define a total API-call, cost, or elapsed-time budget. Refer to Resume an Interrupted Run for recovery.

Continue After Failed Records¶

For a new run, add this to the existing YAML configuration:

failed_record_policy: skip

The default error policy stops when required structured responses remain unavailable. With skip, the structured column records a diagnostic after Data Designer's native validation retries are exhausted. This adds no retry loop and does not change the native retry budget. missing_response_attempts keeps its separate meaning; a recorded failure is terminal and is not retried by that loop or by resume. Unexplained missing rows, duplicate identities, provider errors, and integrity errors are never converted into skipped records.

The original corpus and quality thresholds are preserved. If no queries pass, export still fails. Inspect generation_failures.json in the run directory and portable bundle for stage, request ID, source/context/query identity where applicable, validation detail, native attempt limit, and the last invalid response when available. The portable ledger is included in the manifest's checksums. report.json.generation_failures counts failures by stage, failed requested slots, and excluded query IDs. Failed query-generation slots contribute to requested_slots, separately from model abstentions and quality rejections. Earlier planning failures have no known query-slot count. Queries that fail a later judgment or localization still count in funnel.generated and the retention_fraction denominator, once per query. For example, three generated queries with one localization failure and two accepted queries report three generated and a retention fraction of 2/3.

The same policy is supported by RetrievalStructuredColumnConfig. In skip mode, non-null output cells are JSON strings containing either a validated response or a _retrieval_sdg_failure diagnostic. The multimodal workflow decodes these before its typed consumers. Direct column users must handle the diagnostic before treating a cell as a model response. Failed records have None in any configured trace and reasoning-content columns, including when every record in a batch fails, so downstream expressions can still reference those columns.

Resume preserves both cached successes and recorded failures. Changing the policy or upgrading the implementation changes the frozen run identity; use a new output directory. Failure rates and exclusions can affect sample composition, so include them when comparing generation yield.

Source Records¶

Each line in sources_file is one JSON object describing an independently retrievable unit. Unknown fields are rejected. Keep these fields in the source file, separate from run configuration:

The loader rejects empty files, duplicate unit IDs, missing image files, and records containing neither text nor an image. A single unit is not split during request fitting; prepare smaller units before generation when needed.

For programmatic validation, use the source loader:

from data_designer_retrieval_sdg.multimodal.source_file import load_retrieval_sources

sources = load_retrieval_sources("sources.jsonl")
print(len(sources))

RetrievalSource and SplitRatios are defined in data_designer_retrieval_sdg.multimodal.source_models. Query grouping and split assignment belong to the same multimodal subpackage. These source records are also the output of multimodal directory preparation. Text QA uses its own seed source configuration and generation workflow.

Directory Input¶

source_directory prepares source records from local UTF-8 text, Markdown, and static PNG, JPEG, or WebP images. Use it instead of sources_file. For Python configuration, import SourceDirectoryConfig from data_designer_retrieval_sdg.multimodal. Configure directory input with these fields:

Text Preparation¶

Sentence mode uses NLTK's default English sentence tokenizer on each nonempty paragraph, then joins consecutive sentences with spaces and no overlap. Chunks can cross paragraph boundaries. These retrieval units are independent of generation context grouping.

Sentence chunking can remove Markdown heading boundaries. For section-aware planning, use sources_file with units prepared at section or page boundaries. Whole-file mode preserves internal formatting and strips surrounding whitespace. Both modes accept UTF-8 with or without a byte-order mark.

Sentence mode downloads missing NLTK punkt and punkt_tab resources on first use. To prepare an offline environment, download them beforehand:

python -m nltk.downloader punkt punkt_tab

Whole-file mode does not require NLTK resources.

Token Limits¶

By default, sentence mode packs complete sentences into chunks of at most 1,024 tokens with no sentence-count cap. Both limits are upper bounds when sentences_per_chunk is set: for example, sentences_per_chunk: 5 and max_tokens_per_chunk: 1024 produce chunks of at most five sentences and 1,024 tokens, whichever limit is reached first. Set max_tokens_per_chunk: null to disable the token cap. With both limits set to null, sentence mode joins all sentences in each file into one unit.

Counts use the assembled text with padding and truncation disabled, including configured special tokens plus reserved_tokens. Set reserved_tokens for downstream prefixes or other tokens that the tokenizer does not already count.

The default tokenizer is nvidia/Nemotron-3.5-Embed-1B-BF16. Set tokenizer to another Hugging Face model ID or a local directory saved with save_pretrained. A model ID downloads tokenizer assets on first use and uses the Hugging Face cache thereafter. Model weights are not loaded, and remote custom code is disabled. For offline preparation, save the tokenizer beforehand and use that directory:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("nvidia/Nemotron-3.5-Embed-1B-BF16")
tokenizer.save_pretrained("./embedding-tokenizer")

source_directory:
  path: ./documents
  tokenizer: ./embedding-tokenizer

Tokenizers with nonzero BPE dropout are rejected. Save an inference tokenizer with dropout set to null or 0 so token counts and prepared units are repeatable. Generator and judge request limits remain separate. Images and explicit JSONL units are unaffected; image-only directories do not load the embedding tokenizer.

An oversized sentence fails unless split_oversized_sentences: true. This option splits the original text into fragments that fit, without truncation or overlap. It strips surrounding whitespace and can split words or separate supporting evidence. Preparation fails if no fragment fits. Whole-file mode validates the entire file without splitting it.

File Discovery and Identity¶

Files are sorted lexicographically by corpus-relative path. Each text file is one document; each image is a separate document with one image unit. Document IDs are URL-escaped relative paths. Text unit IDs append #chunk-000001, #chunk-000002, and so on, including in whole-file mode. Image unit IDs append #image. Relocating unchanged inputs preserves IDs; renaming files or changing chunking settings can change them. Update explicit context memberships accordingly.

The loader does not infer text/image pairs, page numbers, or groups of files belonging to one document. Use sources_file for those relationships. Extract PDF text, perform OCR, and resolve Markdown image links before loading sources.

Hidden files and directories, directory symlinks, and unsupported extensions are ignored. Selected symlink files, unreadable files, invalid UTF-8, empty text, and damaged or animated images fail explicitly. An empty selection also fails.

The run saves reusable prepared inputs before model calls. To move them, copy the whole inputs/ directory and load inputs/sources.jsonl through sources_file. Resume rejects changed selected files, preparation settings, or tokenizer identity, and verifies completed artifacts. Start a new run after changing chunking settings.

Model Settings¶

Configure each model role explicitly. Model identifiers and endpoint capabilities come from your serving deployment; the plugin does not choose a replacement model.

For endpoints that support JSON-object mode, add the following configuration under both generator and judge:

extra_body:
  response_format:
    type: json_object

This option requests JSON formatting; it does not verify source grounding or factual correctness. Follow the model's recommended sampling settings. For reasoning models, max_tokens can include both reasoning and the final response.

To enable local token preflight for a role, set context_window_tokens, tokenizer_file, and request_overhead_tokens. Also set image_tokens_per_image for the generator, relevance judge, and localization judge when the corpus contains images. Text-only judgment stages do not require an image allowance. Use values appropriate to the deployment. The check includes the structured-response instructions, source input, image allowances, and output reserve.

Without local token preflight, Data Designer's rendered-prompt limit still applies, but it is not a model context-window guarantee. Provider context and image limits remain applicable.

Before freezing a new run, the workflow calls Data Designer's check_models() for the generator, enabled judges, the fallback judge when automatic TOC confirmation is enabled, and hosted summary embeddings configured for combinations. Identical serving configurations share one check. Each check makes a real request using the configured credentials and inference options, so provider latency and usage charges apply.

For example, an invalid credential or unavailable model for a later judge stops the run before summary and query generation. Data Designer's native errors propagate to the caller. Fix the deployment or configuration and rerun; a failed check does not create the run directory.

Incomplete resumes repeat the checks after validating the saved run identity. Completed resumes verify the saved artifacts without credentials or model calls. Generation batches do not repeat the checks.

These are basic text-response and embedding probes; they do not verify image support or judge output schemas. Providers must accept Data Designer's built-in Hello! probe with the configured inference options. To disable probes, set DATA_DESIGNER_SKIP_MODEL_HEALTH_CHECKS=1, Data Designer's native opt-out.

Judge Model Overrides¶

Use judge for the default judgment model. Override individual stages under judges when a different deployment or sampling configuration suits the task:

An omitted or null stage uses judge. Each override is a complete Model Settings object. Required fields must be supplied; optional fields use their own defaults rather than inheriting values from judge. This includes sampling parameters, output limits, credentials, and local token budgets. Configure vision-capable deployments for image-bearing requests; the plugin does not infer model capabilities or switch models on errors.

This configuration uses one text judge for summary and query checks, and the default vision judge for relevance and support localization. Replace the model identifiers and endpoints with your serving deployments, and set both credential environment variables before running:

sources_file: sources.jsonl
output_dir: runs/stage-judges
dataset_id: service-manuals
generator:
  model: your-generation-model
  endpoint: https://generation.example.com/v1
  credential_env: GENERATION_API_KEY
judge:
  model: your-vision-judge-model
  endpoint: https://vision.example.com/v1
  credential_env: JUDGE_API_KEY
  temperature: 0
  max_tokens: 4096
judges:
  summary: &text_judge
    model: your-text-judge-model
    endpoint: https://text.example.com/v1
    credential_env: JUDGE_API_KEY
    temperature: 0
    max_tokens: 1024
  query_metadata: *text_judge
  self_sufficiency: *text_judge

You can also configure judges.relevance and judges.localization independently. For Python configuration, import JudgeModels from data_designer_retrieval_sdg.multimodal. config.model_for("judge", "relevance") returns the effective settings for that stage.

Automatic section planning (context_strategy: sections without contexts_file) uses the fallback judge for text-only table-of-contents confirmation. This judge remains active even when all five named stages are overridden or judge_summaries: false. Its local token budget is validated before creating the run, and its tokenizer content hash is included in the frozen run identity. Explicit contexts bypass this confirmation step.

judge_summaries: false disables the summary judgment stage, including its model-budget validation. The run manifest records the fallback under models.judge and the effective models for enabled stages under models.judges. shared_generator_judge is true only when every active judge, including the fallback used for TOC confirmation, uses the generator's model identifier and endpoint.

Cached responses include the resolved model settings and tokenizer content hash. Changing one override cannot reuse that stage's incompatible response. Changing settings in an existing run still violates its frozen identity; use a new output directory. Omitting overrides preserves the original judge request cache keys. Stage configuration does not establish better judgment quality; compare reviewed examples when choosing models.

Context Planning¶

A context specifies the units available for one generation request. These fields control automatic grouping and the bounds applied to planned contexts:

For explicit memberships, supply contexts_file with records such as:

{"context_id":"compare-manuals","unit_ids":["manual-a-text","manual-b-page"],"language":"en"}

Unknown source IDs and repeated IDs within a context are rejected. Membership controls available evidence, not query split groups. Disable semantic combinations with combination_iterations: 0 and leave related_contexts_per_context: 0 when you want only your supplied memberships, subject to selection and request bounds.

Section planning preserves whole units in document/language input order. It uses nonrepeated Markdown headings and model-confirmed table-of-contents boundaries when they can be mapped to source units. Page-offset mapping requires explicit, strictly increasing page_number metadata; IDs and filenames are not page numbers. Unsupported or ambiguous structure falls back to headings or fixed-size sections. The boundary and coverage decisions are recorded in planning/section_boundaries.json.

With context_strategy: sections, visual enrichment describes non-prose content such as charts and diagrams. It does not retain descriptions of images that the model classifies as ordinary text. Supply extracted text for scanned prose, or use context_strategy: unit to summarize the original page image directly.

Oversized contexts split on whole-unit boundaries. The workflow also checks judgment-request sizes after query generation. If a generated query would require an oversized judgment request, it records the superseded slots and generates new queries for smaller contexts. It does not judge a multi-unit query against only a fragment. An indivisible oversized unit fails and needs preprocessing.

Related Section Combinations¶

Semantic combinations group related section summaries using embeddings and clustering. Install the optional dependencies from the repository root:

uv pip install './plugins/data-designer-retrieval-sdg[multimodal]'

The following fields configure this optional stage:

For hosted embeddings, configure options supported by that model, including its input type and truncation policy. The endpoint receives summary text, not images. For local embeddings, clear the endpoint and hosted options, select the model and revision, and allow for local weights and memory requirements.

Semantic combinations require at least 12 section summaries in a language group; smaller groups skip both embedding requests and combinations. The stage can form pairs or triples across documents. It cannot be enabled together with lexical related_contexts_per_context expansion.

Summary Selection and Query Profiles¶

The following fields control which summaries reach generation and what queries are requested:

With both fields omitted, the sampler draws one weighted profile from each text, figure, and table pool for each selected context. The profiles vary query type, format, and full or partial answerability using the run seed and context ID. This requests three queries; unsupported tasks can abstain and quality checks can reject generated queries.

A custom instructions list replaces the default profiles rather than extending them. Each selected profile requests one query or abstention. Setting instructions_per_context samples entries from the configured list instead of using the built-in weighted sampler. Omit both fields to restore that sampler; instructions: [] is invalid. Refer to Customize Query Instructions for reasons to tune the profiles and an example.

Custom profiles use the following fields:

Labels guide generation rather than enforce requested style or answerability. Refer to quality checks for what acceptance establishes.

Exact summary membership/language deduplication is always active. Optional near deduplication also requires high source-unit overlap, matching document sets, and compatible numeric and negation content. Selection caps prioritize visual multi-document combinations, visual single-document combinations, visual sections, and then the corresponding nonvisual groups. Selection reasons remain in context_outcomes.json; source units are not removed from the corpus.

Python API¶

The public API accepts the same configuration as the module command. This example loads sdg.yaml and returns the portable handoff path:

from pathlib import Path

import yaml

from data_designer_retrieval_sdg.multimodal import MultimodalSDGConfig, run_multimodal_sdg

settings = yaml.safe_load(Path("sdg.yaml").read_text())
config = MultimodalSDGConfig.model_validate(settings)
handoff = run_multimodal_sdg(config)
print(handoff)

Output Files¶

A successful run writes the following artifacts under its output directory:

Views are created only when they have eligible corpus units, and can have empty query splits. The pipeline exports text, image, image_and_text, and mixed views. There is no run configuration field for selecting export views; downstream consumers select from the completed bundle. Refer to evidence and export views for their eligibility rules.