🎨 Data Designer Tutorial: Providing Images as Context for Vision-Based Data Generation¶

📚 What you'll learn¶

This notebook demonstrates how to provide images as context to generate text descriptions using vision-language models.

✨ Visual Document Processing: Converting images to chat-ready format for model consumption
🔍 Vision-Language Generation: Using vision models to generate detailed summaries from images

If this is your first time using Data Designer, we recommend starting with the first notebook in this tutorial series.

📦 Import Data Designer¶

data_designer.config provides access to the configuration API.
DataDesigner is the main interface for data generation.

In [1]:

Copied!





# Standard library imports
import base64
import io
import uuid

# Third-party imports
import pandas as pd
import rich
from datasets import load_dataset
from IPython.display import display
from rich.panel import Panel

# Data Designer imports
import data_designer.config as dd
from data_designer.interface import DataDesigner
# Standard library imports
import base64
import io
import uuid

# Third-party imports
import pandas as pd
import rich
from datasets import load_dataset
from IPython.display import display
from rich.panel import Panel

# Data Designer imports
import data_designer.config as dd
from data_designer.interface import DataDesigner

⚙️ Initialize the Data Designer interface¶

DataDesigner is the main object responsible for managing the data generation process.
When initialized without arguments, the default model providers are used.

In [2]:

Copied!

data_designer = DataDesigner()
data_designer = DataDesigner()

🎛️ Define model configurations¶

Each ModelConfig defines a model that can be used during the generation process.
The "model alias" is used to reference the model in the Data Designer config (as we will see below).
The "model provider" is the external service that hosts the model (see the model config docs for more details).
By default, we use build.nvidia.com as the model provider.

In [3]:

Copied!





# This name is set in the model provider configuration.
MODEL_PROVIDER = "nvidia"

model_configs = [
    dd.ModelConfig(
        alias="vision",
        model="meta/llama-4-scout-17b-16e-instruct",
        provider=MODEL_PROVIDER,
        inference_parameters=dd.ChatCompletionInferenceParams(
            temperature=0.60,
            top_p=0.95,
            max_tokens=2048,
        ),
    ),
]
# This name is set in the model provider configuration.
MODEL_PROVIDER = "nvidia"

model_configs = [
    dd.ModelConfig(
        alias="vision",
        model="meta/llama-4-scout-17b-16e-instruct",
        provider=MODEL_PROVIDER,
        inference_parameters=dd.ChatCompletionInferenceParams(
            temperature=0.60,
            top_p=0.95,
            max_tokens=2048,
        ),
    ),
]

🏗️ Initialize the Data Designer Config Builder¶

The Data Designer config defines the dataset schema and generation process.
The config builder provides an intuitive interface for building this configuration.
The list of model configs is provided to the builder at initialization.

In [4]:

Copied!

config_builder = dd.DataDesignerConfigBuilder(model_configs=model_configs)
config_builder = dd.DataDesignerConfigBuilder(model_configs=model_configs)

🌱 Seed Dataset Creation¶

In this section, we'll prepare our visual documents as a seed dataset for summarization:

Loading Visual Documents: We use the ColPali dataset containing document images
Image Processing: Convert images to base64 format for vision model consumption
Metadata Extraction: Preserve relevant document information (filename, page number, source, etc.)

The seed dataset will be used to generate detailed text summaries of each document image.

In [5]:

Copied!





# Dataset processing configuration
IMG_COUNT = 512  # Number of images to process
BASE64_IMAGE_HEIGHT = 512  # Standardized height for model input

# Load ColPali dataset for visual documents
img_dataset_cfg = {"path": "vidore/colpali_train_set", "split": "train", "streaming": True}
# Dataset processing configuration
IMG_COUNT = 512  # Number of images to process
BASE64_IMAGE_HEIGHT = 512  # Standardized height for model input

# Load ColPali dataset for visual documents
img_dataset_cfg = {"path": "vidore/colpali_train_set", "split": "train", "streaming": True}

In [6]:

Copied!





def resize_image(image, height: int):
    """
    Resize image while maintaining aspect ratio.

    Args:
        image: PIL Image object
        height: Target height in pixels

    Returns:
        Resized PIL Image object
    """
    original_width, original_height = image.size
    width = int(original_width * (height / original_height))
    return image.resize((width, height))


def convert_image_to_chat_format(record, height: int) -> dict:
    """
    Convert PIL image to base64 format for chat template usage.

    Args:
        record: Dataset record containing image and metadata
        height: Target height for image resizing

    Returns:
        Updated record with base64_image and uuid fields
    """
    # Resize image for consistent processing
    image = resize_image(record["image"], height)

    # Convert to base64 string
    img_buffer = io.BytesIO()
    image.save(img_buffer, format="PNG")
    byte_data = img_buffer.getvalue()
    base64_encoded_data = base64.b64encode(byte_data)
    base64_string = base64_encoded_data.decode("utf-8")

    # Return updated record
    return record | {"base64_image": base64_string, "uuid": str(uuid.uuid4())}
def resize_image(image, height: int):
    """
    Resize image while maintaining aspect ratio.

    Args:
        image: PIL Image object
        height: Target height in pixels

    Returns:
        Resized PIL Image object
    """
    original_width, original_height = image.size
    width = int(original_width * (height / original_height))
    return image.resize((width, height))


def convert_image_to_chat_format(record, height: int) -> dict:
    """
    Convert PIL image to base64 format for chat template usage.

    Args:
        record: Dataset record containing image and metadata
        height: Target height for image resizing

    Returns:
        Updated record with base64_image and uuid fields
    """
    # Resize image for consistent processing
    image = resize_image(record["image"], height)

    # Convert to base64 string
    img_buffer = io.BytesIO()
    image.save(img_buffer, format="PNG")
    byte_data = img_buffer.getvalue()
    base64_encoded_data = base64.b64encode(byte_data)
    base64_string = base64_encoded_data.decode("utf-8")

    # Return updated record
    return record | {"base64_image": base64_string, "uuid": str(uuid.uuid4())}

In [7]:

Copied!





# Load and process the visual document dataset
print("📥 Loading and processing document images...")

img_dataset_iter = iter(
    load_dataset(**img_dataset_cfg).map(convert_image_to_chat_format, fn_kwargs={"height": BASE64_IMAGE_HEIGHT})
)
img_dataset = pd.DataFrame([next(img_dataset_iter) for _ in range(IMG_COUNT)])

print(f"✅ Loaded {len(img_dataset)} images with columns: {list(img_dataset.columns)}")
# Load and process the visual document dataset
print("📥 Loading and processing document images...")

img_dataset_iter = iter(
    load_dataset(**img_dataset_cfg).map(convert_image_to_chat_format, fn_kwargs={"height": BASE64_IMAGE_HEIGHT})
)
img_dataset = pd.DataFrame([next(img_dataset_iter) for _ in range(IMG_COUNT)])

print(f"✅ Loaded {len(img_dataset)} images with columns: {list(img_dataset.columns)}")

📥 Loading and processing document images...

✅ Loaded 512 images with columns: ['image', 'image_filename', 'query', 'answer', 'source', 'options', 'page', 'model', 'prompt', 'answer_type', 'base64_image', 'uuid']

In [8]:

Copied!

img_dataset.head()
img_dataset.head()

Out[8]:

	image	image_filename	query	answer	source	options	page	model	prompt	answer_type	base64_image	uuid
0	<PIL.JpegImagePlugin.JpegImageFile image mode=...	images/1810.07757_2.jpg	Comparing panels a, b, c, and d, which stateme...	D	arxiv_qa	['A. The variance of the data decreases from p...		gpt4V		None	iVBORw0KGgoAAAANSUhEUgAAAUAAAAIACAIAAAB8QiIMAA...	e0b94c2c-5d27-40df-9107-e39c777f38d4
1	<PIL.JpegImagePlugin.JpegImageFile image mode=...	data/scrapped_pdfs_split/pages_extracted/energ...	What is the duration of the course mentioned i...	['five to ten hours, not including field trips']	pdf	None	9	sonnet	\n You are an assistant specialized in ...	None	iVBORw0KGgoAAAANSUhEUgAAAYsAAAIACAIAAAD8HddaAA...	416ae7a9-1455-4dd6-a6f7-780606f54419
2	<PIL.JpegImagePlugin.JpegImageFile image mode=...	data/scrapped_pdfs_split/pages_extracted/energ...	What is the primary purpose of the PTC in lith...	['protect against external short circuits']	pdf	None	414	sonnet	\n You are an assistant specialized in ...	None	iVBORw0KGgoAAAANSUhEUgAAAZgAAAIACAIAAAAwhO2xAA...	d1aa3ce8-1156-4967-8d1a-d11200c860e2
3	<PIL.PngImagePlugin.PngImageFile image mode=L ...	0fd47b51ae9248ef36669b8619b1223f268edae3e7a44a...	What is the date?\nYour answer should be very ...	OCTOBER 17, 1995.	docvqa	None	None	None	None	None	iVBORw0KGgoAAAANSUhEUgAAAX0AAAIACAAAAABLRuMPAA...	849b6334-f4dd-4b4c-bef7-619b325f4cb3
4	<PIL.PngImagePlugin.PngImageFile image mode=L ...	b335cfb9d442f8925ea41a064cb445a5395577f2345d52...	What is Bert Shulimson's title?\nYour response...	EXECUTIVE SECRETARY.	docvqa	None	None	None	None	None	iVBORw0KGgoAAAANSUhEUgAAAY8AAAIACAAAAABf/7+rAA...	0b247f41-379f-430c-ab16-2969c3976930

In [9]:

Copied!

# Add the seed dataset containing our processed images
df_seed = pd.DataFrame(img_dataset)[["uuid", "image_filename", "base64_image", "page", "options", "source"]]
config_builder.with_seed_dataset(dd.DataFrameSeedSource(df=df_seed))
# Add the seed dataset containing our processed images
df_seed = pd.DataFrame(img_dataset)[["uuid", "image_filename", "base64_image", "page", "options", "source"]]
config_builder.with_seed_dataset(dd.DataFrameSeedSource(df=df_seed))

Out[9]:

DataDesignerConfigBuilder()

In [10]:

Copied!





# Add a column to generate detailed document summaries
config_builder.add_column(
    dd.LLMTextColumnConfig(
        name="summary",
        model_alias="vision",
        prompt=(
            "Provide a detailed summary of the content in this image in Markdown format. "
            "Start from the top of the image and then describe it from top to bottom. "
            "Place a summary at the bottom."
        ),
        multi_modal_context=[
            dd.ImageContext(
                column_name="base64_image",
                data_type=dd.ModalityDataType.BASE64,
                image_format=dd.ImageFormat.PNG,
            )
        ],
    )
)

data_designer.validate(config_builder)
# Add a column to generate detailed document summaries
config_builder.add_column(
    dd.LLMTextColumnConfig(
        name="summary",
        model_alias="vision",
        prompt=(
            "Provide a detailed summary of the content in this image in Markdown format. "
            "Start from the top of the image and then describe it from top to bottom. "
            "Place a summary at the bottom."
        ),
        multi_modal_context=[
            dd.ImageContext(
                column_name="base64_image",
                data_type=dd.ModalityDataType.BASE64,
                image_format=dd.ImageFormat.PNG,
            )
        ],
    )
)

data_designer.validate(config_builder)

[00:04:17] [INFO] ✅ Validation passed

🔁 Iteration is key – preview the dataset!¶

Use the preview method to generate a sample of records quickly.
Inspect the results for quality and format issues.
Adjust column configurations, prompts, or parameters as needed.
Re-run the preview until satisfied.

In [11]:

Copied!

preview = data_designer.preview(config_builder, num_records=2)
preview = data_designer.preview(config_builder, num_records=2)

[00:04:17] [INFO] 🎥 Preview generation in progress

[00:04:17] [INFO] ✅ Validation passed

[00:04:17] [INFO] ⛓️ Sorting column configs into a Directed Acyclic Graph

[00:04:17] [INFO] 🩺 Running health checks for models...

[00:04:17] [INFO]   |-- 👀 Checking 'meta/llama-4-scout-17b-16e-instruct' in provider named 'nvidia' for model alias 'vision'...

[00:04:18] [INFO]   |-- ✅ Passed!

[00:04:18] [INFO] 🌱 Sampling 2 records from seed dataset

[00:04:18] [INFO]   |-- seed dataset size: 512 records

[00:04:18] [INFO]   |-- sampling strategy: ordered

[00:04:18] [INFO] 📝 llm-text model config for column 'summary'

[00:04:18] [INFO]   |-- model: 'meta/llama-4-scout-17b-16e-instruct'

[00:04:18] [INFO]   |-- model alias: 'vision'

[00:04:18] [INFO]   |-- model provider: 'nvidia'

[00:04:18] [INFO]   |-- inference parameters:

[00:04:18] [INFO]   |  |-- generation_type=chat-completion

[00:04:18] [INFO]   |  |-- max_parallel_requests=4

[00:04:18] [INFO]   |  |-- temperature=0.60

[00:04:18] [INFO]   |  |-- top_p=0.95

[00:04:18] [INFO]   |  |-- max_tokens=2048

[00:04:18] [INFO] ⚡️ Processing llm-text column 'summary' with 4 concurrent workers

[00:04:18] [INFO] ⏱️ llm-text column 'summary' will report progress after each record

[00:04:21] [INFO]   |-- 🌗 llm-text column 'summary' progress: 1/2 (50%) complete, 1 ok, 0 failed, 0.28 rec/s, eta 3.6s

[00:04:22] [INFO]   |-- 🌕 llm-text column 'summary' progress: 2/2 (100%) complete, 2 ok, 0 failed, 0.49 rec/s, eta 0.0s

[00:04:22] [INFO] 📊 Model usage summary:

[00:04:22] [INFO]   |-- model: meta/llama-4-scout-17b-16e-instruct

[00:04:22] [INFO]   |-- tokens: input=1394, output=645, total=2039, tps=452

[00:04:22] [INFO]   |-- requests: success=2, failed=0, total=2, rpm=26

[00:04:22] [INFO] 📐 Measuring dataset column statistics:

[00:04:22] [INFO]   |-- 📝 column: 'summary'

[00:04:22] [INFO] 🎆 Preview complete!

In [12]:

Copied!

# Run this cell multiple times to cycle through the 2 preview records.
preview.display_sample_record()
# Run this cell multiple times to cycle through the 2 preview records.
preview.display_sample_record()

                                                                                                                   
                                                   Seed Columns                                                    
┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name           ┃ Value                                                                                          ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ uuid           │ e0b94c2c-5d27-40df-9107-e39c777f38d4                                                           │
├────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────┤
│ image_filename │ images/1810.07757_2.jpg                                                                        │
├────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────┤
│ base64_image   │ iVBORw0KGgoAAAANSUhEUgAAAUAAAAIACAIAAAB8QiIMAAEAAElEQVR4nOy9edRt2VUX+vvNufY+53zNbauvVJdKSEUQQ… │
├────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────┤
│ page           │                                                                                                │
├────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────┤
│ options        │ ['A. The variance of the data decreases from panel a to panel d.', 'B. The variance of the     │
│                │ data increases from panel a to panel d.', 'C. The data presents no variance in any of the      │
│                │ panels.', 'D. The variance of the data is inconsistent across the panels.', '-']               │
├────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────┤
│ source         │ arxiv_qa                                                                                       │
└────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────┘
                                                                                                                   
                                                                                                                   
                                                 Generated Columns                                                 
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name    ┃ Value                                                                                                 ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ summary │ The image presents a series of 8 heatmaps, arranged in two columns of four rows each. The heatmaps    │
│         │ are labeled from "a" to "h" and display a range of colors, including blue, yellow, orange, and green. │
│         │                                                                                                       │
│         │ **Color Legend:**                                                                                     │
│         │ A color legend is provided at the top center of the image, featuring a gradient that transitions from │
│         │ dark blue (0.2) to yellow (1). The legend indicates that the colors represent values ranging from 0.2 │
│         │ to 1.                                                                                                 │
│         │                                                                                                       │
│         │ **Heatmaps:**                                                                                         │
│         │ Each heatmap has a distinct pattern, with some displaying:                                            │
│         │                                                                                                       │
│         │ * Horizontal bands of color (e.g., "a" and "e")                                                       │
│         │ * Diagonal bands of color (e.g., "f", "g", and "h")                                                   │
│         │ * Randomly scattered colors (e.g., "b", "c", and "d")                                                 │
│         │                                                                                                       │
│         │ The x-axis of each heatmap is labeled "t(μs)" and ranges from 0 to 0.8. The y-axis is labeled "Δf     │
│         │ (MHz)" and ranges from -60 to 20.                                                                     │
│         │                                                                                                       │
│         │ **Summary:**                                                                                          │
│         │ The image appears to be a visual representation of data related to frequency shifts (Δf) over time    │
│         │ (t) in microseconds, with the color legend indicating a correlation coefficient or similarity measure │
│         │ (τ) ranging from 0.2 to 1. The heatmaps display various patterns, suggesting different types of       │
│         │ frequency shift behaviors or correlations.                                                            │
└─────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────┘
                                                                                                                   
                                                    [index: 0]

In [13]:

Copied!

# The preview dataset is available as a pandas DataFrame.
preview.dataset
# The preview dataset is available as a pandas DataFrame.
preview.dataset

Out[13]:

	uuid	image_filename	base64_image	page	options	source	summary
0	e0b94c2c-5d27-40df-9107-e39c777f38d4	images/1810.07757_2.jpg	iVBORw0KGgoAAAANSUhEUgAAAUAAAAIACAIAAAB8QiIMAA...		['A. The variance of the data decreases from p...	arxiv_qa	The image presents a series of 8 heatmaps, arr...
1	416ae7a9-1455-4dd6-a6f7-780606f54419	data/scrapped_pdfs_split/pages_extracted/energ...	iVBORw0KGgoAAAANSUhEUgAAAYsAAAIACAIAAAD8HddaAA...	9	None	pdf	The image presents a document with the title "...

📊 Analyze the generated data¶

Data Designer automatically generates a basic statistical analysis of the generated data.
This analysis is available via the analysis property of generation result objects.

In [14]:

Copied!

# Print the analysis as a table.
preview.analysis.to_report()
# Print the analysis as a table.
preview.analysis.to_report()

──────────────────────────────────────── 🎨 Data Designer Dataset Profile ─────────────────────────────────────────

                                                                                                                   
                                                 Dataset Overview                                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ number of records               ┃ number of columns               ┃ percent complete records                    ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 2                               │ 1                               │ 100.0%                                      │
└─────────────────────────────────┴─────────────────────────────────┴─────────────────────────────────────────────┘
                                                                                                                   
                                                                                                                   
                                                📝 LLM-Text Columns                                                
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                  ┃               ┃                              ┃       prompt tokens ┃       completion tokens ┃
┃ column name      ┃     data type ┃         number unique values ┃          per record ┃              per record ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ summary          │        string │                   2 (100.0%) │        38.0 +/- 0.0 │          323.5 +/- 43.1 │
└──────────────────┴───────────────┴──────────────────────────────┴─────────────────────┴─────────────────────────┘
                                                                                                                   
                                                                                                                   
╭────────────────────────────────────────────────── Table Notes ──────────────────────────────────────────────────╮
│                                                                                                                 │
│  1. All token statistics are based on a sample of max(1000, len(dataset)) records.                              │
│  2. Tokens are calculated using tiktoken's cl100k_base tokenizer.                                               │
│                                                                                                                 │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
                                                                                                                   
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────

🔎 Visual Inspection¶

Let's compare the original document image with the generated summary to validate quality:

In [15]:

Copied!





# Compare original document with generated summary
index = 0  # Change this to view different examples

# Merge preview data with original images for comparison
comparison_dataset = preview.dataset.merge(pd.DataFrame(img_dataset)[["uuid", "image"]], how="left", on="uuid")

# Extract the record for display
record = comparison_dataset.iloc[index]

print("📄 Original Document Image:")
display(resize_image(record.image, BASE64_IMAGE_HEIGHT))

print("\n📝 Generated Summary:")
rich.print(Panel(record.summary, title="Document Summary", title_align="left"))
# Compare original document with generated summary
index = 0  # Change this to view different examples

# Merge preview data with original images for comparison
comparison_dataset = preview.dataset.merge(pd.DataFrame(img_dataset)[["uuid", "image"]], how="left", on="uuid")

# Extract the record for display
record = comparison_dataset.iloc[index]

print("📄 Original Document Image:")
display(resize_image(record.image, BASE64_IMAGE_HEIGHT))

print("\n📝 Generated Summary:")
rich.print(Panel(record.summary, title="Document Summary", title_align="left"))

📄 Original Document Image:

No description has been provided for this image

📝 Generated Summary:

╭─ Document Summary ──────────────────────────────────────────────────────────────────────────────────────────────╮
│ The image presents a series of 8 heatmaps, arranged in two columns of four rows each. The heatmaps are labeled  │
│ from "a" to "h" and display a range of colors, including blue, yellow, orange, and green.                       │
│                                                                                                                 │
│ **Color Legend:**                                                                                               │
│ A color legend is provided at the top center of the image, featuring a gradient that transitions from dark blue │
│ (0.2) to yellow (1). The legend indicates that the colors represent values ranging from 0.2 to 1.               │
│                                                                                                                 │
│ **Heatmaps:**                                                                                                   │
│ Each heatmap has a distinct pattern, with some displaying:                                                      │
│                                                                                                                 │
│ * Horizontal bands of color (e.g., "a" and "e")                                                                 │
│ * Diagonal bands of color (e.g., "f", "g", and "h")                                                             │
│ * Randomly scattered colors (e.g., "b", "c", and "d")                                                           │
│                                                                                                                 │
│ The x-axis of each heatmap is labeled "t(μs)" and ranges from 0 to 0.8. The y-axis is labeled "Δf (MHz)" and    │
│ ranges from -60 to 20.                                                                                          │
│                                                                                                                 │
│ **Summary:**                                                                                                    │
│ The image appears to be a visual representation of data related to frequency shifts (Δf) over time (t) in       │
│ microseconds, with the color legend indicating a correlation coefficient or similarity measure (τ) ranging from │
│ 0.2 to 1. The heatmaps display various patterns, suggesting different types of frequency shift behaviors or     │
│ correlations.                                                                                                   │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

🆙 Scale up!¶

Happy with your preview data?
Use the create method to submit larger Data Designer generation jobs.

In [16]:

Copied!

results = data_designer.create(config_builder, num_records=10, dataset_name="tutorial-4")
results = data_designer.create(config_builder, num_records=10, dataset_name="tutorial-4")

[00:04:22] [INFO] 🎨 Creating Data Designer dataset

[00:04:22] [INFO] ✅ Validation passed

[00:04:22] [INFO] ⛓️ Sorting column configs into a Directed Acyclic Graph

[00:04:22] [INFO] 🩺 Running health checks for models...

[00:04:22] [INFO]   |-- 👀 Checking 'meta/llama-4-scout-17b-16e-instruct' in provider named 'nvidia' for model alias 'vision'...

[00:04:23] [INFO]   |-- ✅ Passed!

[00:04:23] [INFO] ⏳ Processing batch 1 of 1

[00:04:23] [INFO] 🌱 Sampling 10 records from seed dataset

[00:04:23] [INFO]   |-- seed dataset size: 512 records

[00:04:23] [INFO]   |-- sampling strategy: ordered

[00:04:23] [INFO] 📝 llm-text model config for column 'summary'

[00:04:23] [INFO]   |-- model: 'meta/llama-4-scout-17b-16e-instruct'

[00:04:23] [INFO]   |-- model alias: 'vision'

[00:04:23] [INFO]   |-- model provider: 'nvidia'

[00:04:23] [INFO]   |-- inference parameters:

[00:04:23] [INFO]   |  |-- generation_type=chat-completion

[00:04:23] [INFO]   |  |-- max_parallel_requests=4

[00:04:23] [INFO]   |  |-- temperature=0.60

[00:04:23] [INFO]   |  |-- top_p=0.95

[00:04:23] [INFO]   |  |-- max_tokens=2048

[00:04:23] [INFO] ⚡️ Processing llm-text column 'summary' with 4 concurrent workers

[00:04:23] [INFO] ⏱️ llm-text column 'summary' will report progress after each record

[00:04:26] [INFO]   |-- 🥚 llm-text column 'summary' progress: 1/10 (10%) complete, 1 ok, 0 failed, 0.28 rec/s, eta 31.6s

[00:04:26] [INFO]   |-- 🥚 llm-text column 'summary' progress: 2/10 (20%) complete, 2 ok, 0 failed, 0.55 rec/s, eta 14.5s

[00:04:28] [INFO]   |-- 🐣 llm-text column 'summary' progress: 3/10 (30%) complete, 3 ok, 0 failed, 0.59 rec/s, eta 11.9s

[00:04:30] [INFO]   |-- 🐣 llm-text column 'summary' progress: 4/10 (40%) complete, 4 ok, 0 failed, 0.58 rec/s, eta 10.4s

[00:04:34] [INFO]   |-- 🐥 llm-text column 'summary' progress: 5/10 (50%) complete, 5 ok, 0 failed, 0.46 rec/s, eta 10.9s

[00:04:34] [INFO]   |-- 🐥 llm-text column 'summary' progress: 6/10 (60%) complete, 6 ok, 0 failed, 0.55 rec/s, eta 7.3s

[00:04:35] [INFO]   |-- 🐥 llm-text column 'summary' progress: 7/10 (70%) complete, 7 ok, 0 failed, 0.58 rec/s, eta 5.2s

[00:04:35] [INFO]   |-- 🐤 llm-text column 'summary' progress: 8/10 (80%) complete, 8 ok, 0 failed, 0.64 rec/s, eta 3.1s

[00:04:41] [INFO]   |-- 🐤 llm-text column 'summary' progress: 9/10 (90%) complete, 9 ok, 0 failed, 0.49 rec/s, eta 2.0s

[00:04:46] [INFO]   |-- 🐔 llm-text column 'summary' progress: 10/10 (100%) complete, 10 ok, 0 failed, 0.44 rec/s, eta 0.0s

[00:04:46] [INFO] 📊 Model usage summary:

[00:04:46] [INFO]   |-- model: meta/llama-4-scout-17b-16e-instruct

[00:04:46] [INFO]   |-- tokens: input=8130, output=5179, total=13309, tps=570

[00:04:46] [INFO]   |-- requests: success=10, failed=0, total=10, rpm=25

[00:04:46] [INFO] 📐 Measuring dataset column statistics:

[00:04:46] [INFO]   |-- 📝 column: 'summary'

In [17]:

Copied!

# Load the generated dataset as a pandas DataFrame.
dataset = results.load_dataset()

dataset.head()
# Load the generated dataset as a pandas DataFrame.
dataset = results.load_dataset()

dataset.head()

Out[17]:

	uuid	image_filename	base64_image	page	options	source	summary
0	e0b94c2c-5d27-40df-9107-e39c777f38d4	images/1810.07757_2.jpg	iVBORw0KGgoAAAANSUhEUgAAAUAAAAIACAIAAAB8QiIMAA...		['A. The variance of the data decreases from p...	arxiv_qa	The image presents a series of 2D heatmaps, ar...
1	416ae7a9-1455-4dd6-a6f7-780606f54419	data/scrapped_pdfs_split/pages_extracted/energ...	iVBORw0KGgoAAAANSUhEUgAAAYsAAAIACAIAAAD8HddaAA...	9	<NA>	pdf	The image presents a document with the title "...
2	d1aa3ce8-1156-4967-8d1a-d11200c860e2	data/scrapped_pdfs_split/pages_extracted/energ...	iVBORw0KGgoAAAANSUhEUgAAAZgAAAIACAIAAAAwhO2xAA...	414	<NA>	pdf	The image presents a page from a technical doc...
3	849b6334-f4dd-4b4c-bef7-619b325f4cb3	0fd47b51ae9248ef36669b8619b1223f268edae3e7a44a...	iVBORw0KGgoAAAANSUhEUgAAAX0AAAIACAAAAABLRuMPAA...	<NA>	<NA>	docvqa	The image presents a document with the title "...
4	0b247f41-379f-430c-ab16-2969c3976930	b335cfb9d442f8925ea41a064cb445a5395577f2345d52...	iVBORw0KGgoAAAANSUhEUgAAAY8AAAIACAAAAABf/7+rAA...	<NA>	<NA>	docvqa	The image presents a letter from Volunteers Ag...

In [18]:

Copied!

# Load the analysis results into memory.
analysis = results.load_analysis()

analysis.to_report()
# Load the analysis results into memory.
analysis = results.load_analysis()

analysis.to_report()

──────────────────────────────────────── 🎨 Data Designer Dataset Profile ─────────────────────────────────────────

                                                                                                                   
                                                 Dataset Overview                                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ number of records               ┃ number of columns               ┃ percent complete records                    ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 10                              │ 1                               │ 100.0%                                      │
└─────────────────────────────────┴─────────────────────────────────┴─────────────────────────────────────────────┘
                                                                                                                   
                                                                                                                   
                                                📝 LLM-Text Columns                                                
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                  ┃               ┃                              ┃       prompt tokens ┃       completion tokens ┃
┃ column name      ┃     data type ┃         number unique values ┃          per record ┃              per record ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ summary          │        string │                  10 (100.0%) │        38.0 +/- 0.0 │         526.5 +/- 234.8 │
└──────────────────┴───────────────┴──────────────────────────────┴─────────────────────┴─────────────────────────┘
                                                                                                                   
                                                                                                                   
╭────────────────────────────────────────────────── Table Notes ──────────────────────────────────────────────────╮
│                                                                                                                 │
│  1. All token statistics are based on a sample of max(1000, len(dataset)) records.                              │
│  2. Tokens are calculated using tiktoken's cl100k_base tokenizer.                                               │
│                                                                                                                 │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
                                                                                                                   
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────

⏭️ Next Steps¶

Now that you've learned how to use visual context for image summarization in Data Designer, explore more:

Experiment with different vision models for specific document types
Try different prompt variations to generate specialized descriptions (e.g., technical details, key findings)
Combine vision-based summaries with other column types for multi-modal workflows
Apply this pattern to other vision tasks like image captioning, OCR validation, or visual question answering