🎨 Data Designer Tutorial: Providing Images as Context for Vision-Based Data Generation¶

📚 What you'll learn¶

This notebook demonstrates how to provide images as context to generate text descriptions using vision-language models.

✨ Visual Document Processing: Converting images to chat-ready format for model consumption
🔍 Vision-Language Generation: Using vision models to generate detailed summaries from images

If this is your first time using Data Designer, we recommend starting with the first notebook in this tutorial series.

📦 Import the essentials¶

The essentials module provides quick access to the most commonly used objects.

In [1]:

Copied!





# Standard library imports
import base64
import io
import uuid

# Third-party imports
import pandas as pd
import rich
from datasets import load_dataset
from IPython.display import display
from rich.panel import Panel

# Data Designer imports
from data_designer.essentials import (
    ChatCompletionInferenceParams,
    DataDesigner,
    DataDesignerConfigBuilder,
    DataFrameSeedSource,
    ImageContext,
    ImageFormat,
    LLMTextColumnConfig,
    ModalityDataType,
    ModelConfig,
)
# Standard library imports
import base64
import io
import uuid

# Third-party imports
import pandas as pd
import rich
from datasets import load_dataset
from IPython.display import display
from rich.panel import Panel

# Data Designer imports
from data_designer.essentials import (
    ChatCompletionInferenceParams,
    DataDesigner,
    DataDesignerConfigBuilder,
    DataFrameSeedSource,
    ImageContext,
    ImageFormat,
    LLMTextColumnConfig,
    ModalityDataType,
    ModelConfig,
)

⚙️ Initialize the Data Designer interface¶

DataDesigner is the main object is responsible for managing the data generation process.
When initialized without arguments, the default model providers are used.

In [2]:

Copied!

data_designer = DataDesigner()
data_designer = DataDesigner()

🎛️ Define model configurations¶

Each ModelConfig defines a model that can be used during the generation process.
The "model alias" is used to reference the model in the Data Designer config (as we will see below).
The "model provider" is the external service that hosts the model (see the model config docs for more details).
By default, we use build.nvidia.com as the model provider.

In [3]:

Copied!





# This name is set in the model provider configuration.
MODEL_PROVIDER = "nvidia"

model_configs = [
    ModelConfig(
        alias="vision",
        model="meta/llama-4-scout-17b-16e-instruct",
        provider=MODEL_PROVIDER,
        inference_parameters=ChatCompletionInferenceParams(
            temperature=0.60,
            top_p=0.95,
            max_tokens=2048,
        ),
    ),
]
# This name is set in the model provider configuration.
MODEL_PROVIDER = "nvidia"

model_configs = [
    ModelConfig(
        alias="vision",
        model="meta/llama-4-scout-17b-16e-instruct",
        provider=MODEL_PROVIDER,
        inference_parameters=ChatCompletionInferenceParams(
            temperature=0.60,
            top_p=0.95,
            max_tokens=2048,
        ),
    ),
]

🏗️ Initialize the Data Designer Config Builder¶

The Data Designer config defines the dataset schema and generation process.
The config builder provides an intuitive interface for building this configuration.
The list of model configs is provided to the builder at initialization.

In [4]:

Copied!

config_builder = DataDesignerConfigBuilder(model_configs=model_configs)
config_builder = DataDesignerConfigBuilder(model_configs=model_configs)

🌱 Seed Dataset Creation¶

In this section, we'll prepare our visual documents as a seed dataset for summarization:

Loading Visual Documents: We use the ColPali dataset containing document images
Image Processing: Convert images to base64 format for vision model consumption
Metadata Extraction: Preserve relevant document information (filename, page number, source, etc.)

The seed dataset will be used to generate detailed text summaries of each document image.

In [5]:

Copied!





# Dataset processing configuration
IMG_COUNT = 512  # Number of images to process
BASE64_IMAGE_HEIGHT = 512  # Standardized height for model input

# Load ColPali dataset for visual documents
img_dataset_cfg = {"path": "vidore/colpali_train_set", "split": "train", "streaming": True}
# Dataset processing configuration
IMG_COUNT = 512  # Number of images to process
BASE64_IMAGE_HEIGHT = 512  # Standardized height for model input

# Load ColPali dataset for visual documents
img_dataset_cfg = {"path": "vidore/colpali_train_set", "split": "train", "streaming": True}

In [6]:

Copied!





def resize_image(image, height: int):
    """
    Resize image while maintaining aspect ratio.

    Args:
        image: PIL Image object
        height: Target height in pixels

    Returns:
        Resized PIL Image object
    """
    original_width, original_height = image.size
    width = int(original_width * (height / original_height))
    return image.resize((width, height))


def convert_image_to_chat_format(record, height: int) -> dict:
    """
    Convert PIL image to base64 format for chat template usage.

    Args:
        record: Dataset record containing image and metadata
        height: Target height for image resizing

    Returns:
        Updated record with base64_image and uuid fields
    """
    # Resize image for consistent processing
    image = resize_image(record["image"], height)

    # Convert to base64 string
    img_buffer = io.BytesIO()
    image.save(img_buffer, format="PNG")
    byte_data = img_buffer.getvalue()
    base64_encoded_data = base64.b64encode(byte_data)
    base64_string = base64_encoded_data.decode("utf-8")

    # Return updated record
    return record | {"base64_image": base64_string, "uuid": str(uuid.uuid4())}
def resize_image(image, height: int):
    """
    Resize image while maintaining aspect ratio.

    Args:
        image: PIL Image object
        height: Target height in pixels

    Returns:
        Resized PIL Image object
    """
    original_width, original_height = image.size
    width = int(original_width * (height / original_height))
    return image.resize((width, height))


def convert_image_to_chat_format(record, height: int) -> dict:
    """
    Convert PIL image to base64 format for chat template usage.

    Args:
        record: Dataset record containing image and metadata
        height: Target height for image resizing

    Returns:
        Updated record with base64_image and uuid fields
    """
    # Resize image for consistent processing
    image = resize_image(record["image"], height)

    # Convert to base64 string
    img_buffer = io.BytesIO()
    image.save(img_buffer, format="PNG")
    byte_data = img_buffer.getvalue()
    base64_encoded_data = base64.b64encode(byte_data)
    base64_string = base64_encoded_data.decode("utf-8")

    # Return updated record
    return record | {"base64_image": base64_string, "uuid": str(uuid.uuid4())}

In [7]:

Copied!





# Load and process the visual document dataset
print("📥 Loading and processing document images...")

img_dataset_iter = iter(
    load_dataset(**img_dataset_cfg).map(convert_image_to_chat_format, fn_kwargs={"height": BASE64_IMAGE_HEIGHT})
)
img_dataset = pd.DataFrame([next(img_dataset_iter) for _ in range(IMG_COUNT)])

print(f"✅ Loaded {len(img_dataset)} images with columns: {list(img_dataset.columns)}")
# Load and process the visual document dataset
print("📥 Loading and processing document images...")

img_dataset_iter = iter(
    load_dataset(**img_dataset_cfg).map(convert_image_to_chat_format, fn_kwargs={"height": BASE64_IMAGE_HEIGHT})
)
img_dataset = pd.DataFrame([next(img_dataset_iter) for _ in range(IMG_COUNT)])

print(f"✅ Loaded {len(img_dataset)} images with columns: {list(img_dataset.columns)}")

📥 Loading and processing document images...

✅ Loaded 512 images with columns: ['image', 'image_filename', 'query', 'answer', 'source', 'options', 'page', 'model', 'prompt', 'answer_type', 'base64_image', 'uuid']

In [8]:

Copied!

img_dataset.head()
img_dataset.head()

Out[8]:

	image	image_filename	query	answer	source	options	page	model	prompt	answer_type	base64_image	uuid
0	<PIL.JpegImagePlugin.JpegImageFile image mode=...	images/1810.07757_2.jpg	Comparing panels a, b, c, and d, which stateme...	D	arxiv_qa	['A. The variance of the data decreases from p...		gpt4V		None	iVBORw0KGgoAAAANSUhEUgAAAUAAAAIACAIAAAB8QiIMAA...	130906e7-ab6b-40be-9550-66c84a6ddc69
1	<PIL.JpegImagePlugin.JpegImageFile image mode=...	data/scrapped_pdfs_split/pages_extracted/energ...	What is the duration of the course mentioned i...	['five to ten hours, not including field trips']	pdf	None	9	sonnet	\n You are an assistant specialized in ...	None	iVBORw0KGgoAAAANSUhEUgAAAYsAAAIACAIAAAD8HddaAA...	bfd43029-c099-403a-8b61-023cc640018f
2	<PIL.JpegImagePlugin.JpegImageFile image mode=...	data/scrapped_pdfs_split/pages_extracted/energ...	What is the primary purpose of the PTC in lith...	['protect against external short circuits']	pdf	None	414	sonnet	\n You are an assistant specialized in ...	None	iVBORw0KGgoAAAANSUhEUgAAAZgAAAIACAIAAAAwhO2xAA...	76c07916-b47a-4328-a23a-ec2612dc4ef8
3	<PIL.PngImagePlugin.PngImageFile image mode=L ...	0fd47b51ae9248ef36669b8619b1223f268edae3e7a44a...	What is the date?\nYour answer should be very ...	OCTOBER 17, 1995.	docvqa	None	None	None	None	None	iVBORw0KGgoAAAANSUhEUgAAAX0AAAIACAAAAABLRuMPAA...	0be006d7-a6dd-4118-84e4-a80b9a66c44a
4	<PIL.PngImagePlugin.PngImageFile image mode=L ...	b335cfb9d442f8925ea41a064cb445a5395577f2345d52...	What is Bert Shulimson's title?\nYour response...	EXECUTIVE SECRETARY.	docvqa	None	None	None	None	None	iVBORw0KGgoAAAANSUhEUgAAAY8AAAIACAAAAABf/7+rAA...	74f6f759-512b-4cdf-8bd8-6ba09d965c59

In [9]:

Copied!

# Add the seed dataset containing our processed images
df_seed = pd.DataFrame(img_dataset)[["uuid", "image_filename", "base64_image", "page", "options", "source"]]
config_builder.with_seed_dataset(DataFrameSeedSource(df=df_seed))
# Add the seed dataset containing our processed images
df_seed = pd.DataFrame(img_dataset)[["uuid", "image_filename", "base64_image", "page", "options", "source"]]
config_builder.with_seed_dataset(DataFrameSeedSource(df=df_seed))

Out[9]:

DataDesignerConfigBuilder()

In [10]:

Copied!





# Add a column to generate detailed document summaries
config_builder.add_column(
    LLMTextColumnConfig(
        name="summary",
        model_alias="vision",
        prompt=(
            "Provide a detailed summary of the content in this image in Markdown format. "
            "Start from the top of the image and then describe it from top to bottom. "
            "Place a summary at the bottom."
        ),
        multi_modal_context=[
            ImageContext(
                column_name="base64_image",
                data_type=ModalityDataType.BASE64,
                image_format=ImageFormat.PNG,
            )
        ],
    )
)
# Add a column to generate detailed document summaries
config_builder.add_column(
    LLMTextColumnConfig(
        name="summary",
        model_alias="vision",
        prompt=(
            "Provide a detailed summary of the content in this image in Markdown format. "
            "Start from the top of the image and then describe it from top to bottom. "
            "Place a summary at the bottom."
        ),
        multi_modal_context=[
            ImageContext(
                column_name="base64_image",
                data_type=ModalityDataType.BASE64,
                image_format=ImageFormat.PNG,
            )
        ],
    )
)

Out[10]:

DataDesignerConfigBuilder(
    seed_dataset: df seed
    llm_text_columns: ['summary']
)

🔁 Iteration is key – preview the dataset!¶

Use the preview method to generate a sample of records quickly.
Inspect the results for quality and format issues.
Adjust column configurations, prompts, or parameters as needed.
Re-run the preview until satisfied.

In [11]:

Copied!

preview = data_designer.preview(config_builder, num_records=2)
preview = data_designer.preview(config_builder, num_records=2)

[21:55:27] [INFO] 👀 Preview generation in progress

[21:55:28] [INFO] ✅ Validation passed

[21:55:28] [INFO] ⛓️ Sorting column configs into a Directed Acyclic Graph

[21:55:28] [INFO] 🩺 Running health checks for models...

[21:55:28] [INFO]   |-- 👀 Checking 'meta/llama-4-scout-17b-16e-instruct' in provider named 'nvidia' for model alias 'vision'...

[21:55:29] [INFO]   |-- ✅ Passed!

[21:55:29] [INFO] 🌱 Sampling 2 records from seed dataset

[21:55:29] [INFO]   |-- seed dataset size: 512 records

[21:55:29] [INFO]   |-- sampling strategy: ordered

[21:55:29] [INFO] 📝 llm-text model config for column 'summary'

[21:55:29] [INFO]   |-- model: 'meta/llama-4-scout-17b-16e-instruct'

[21:55:29] [INFO]   |-- model alias: 'vision'

[21:55:29] [INFO]   |-- model provider: 'nvidia'

[21:55:29] [INFO]   |-- inference parameters: generation_type=chat-completion, max_parallel_requests=4, temperature=0.60, top_p=0.95, max_tokens=2048

[21:55:29] [INFO] 🐙 Processing llm-text column 'summary' with 4 concurrent workers

[21:55:38] [INFO] 📊 Model usage summary:
{
    "meta/llama-4-scout-17b-16e-instruct": {
        "token_usage": {
            "input_tokens": 1396,
            "output_tokens": 770,
            "total_tokens": 2166
        },
        "request_usage": {
            "successful_requests": 2,
            "failed_requests": 0,
            "total_requests": 2
        },
        "tokens_per_second": 252,
        "requests_per_minute": 13
    }
}

[21:55:38] [INFO] 📐 Measuring dataset column statistics:

[21:55:38] [INFO]   |-- 📝 column: 'summary'

[21:55:38] [INFO] 🥳 Preview complete!

In [12]:

Copied!

# Run this cell multiple times to cycle through the 2 preview records.
preview.display_sample_record()
# Run this cell multiple times to cycle through the 2 preview records.
preview.display_sample_record()

                                                                                                                   
                                                   Seed Columns                                                    
┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name           ┃ Value                                                                                          ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ uuid           │ 130906e7-ab6b-40be-9550-66c84a6ddc69                                                           │
├────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────┤
│ image_filename │ images/1810.07757_2.jpg                                                                        │
├────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────┤
│ base64_image   │ iVBORw0KGgoAAAANSUhEUgAAAUAAAAIACAIAAAB8QiIMAAEAAElEQVR4nOy9edRt2VUX+vvNufY+53zNbauvVJdKSEUQQ… │
├────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────┤
│ page           │                                                                                                │
├────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────┤
│ options        │ ['A. The variance of the data decreases from panel a to panel d.', 'B. The variance of the     │
│                │ data increases from panel a to panel d.', 'C. The data presents no variance in any of the      │
│                │ panels.', 'D. The variance of the data is inconsistent across the panels.', '-']               │
├────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────┤
│ source         │ arxiv_qa                                                                                       │
└────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────┘
                                                                                                                   
                                                                                                                   
                                                 Generated Columns                                                 
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name    ┃ Value                                                                                                 ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ summary │ ## Image Summary                                                                                      │
│         │                                                                                                       │
│         │ The image presents a collection of 8 heatmap graphs, arranged in two columns labeled a) to h). A      │
│         │ color bar at the top indicates the correlation coefficient (τ) values, ranging from 0.2 (blue) to 1   │
│         │ (yellow).                                                                                             │
│         │                                                                                                       │
│         │ ### Top Row: a) and e)                                                                                │
│         │                                                                                                       │
│         │ * Graph a) displays a predominantly blue horizontal band centered around 0 MHz, with minimal          │
│         │ variation along the time axis (t).                                                                    │
│         │ * Graph e) shows a similar blue band, but with a more pronounced curvature, indicating a change in    │
│         │ frequency over time.                                                                                  │
│         │                                                                                                       │
│         │ ### Second Row: b) and f)                                                                             │
│         │                                                                                                       │
│         │ * Graph b) features a random, pixelated pattern with no clear trend, spanning from -60 to 20 MHz.     │
│         │ * Graph f) exhibits a diagonal, curved pattern with a clear trend, starting from the top-left and     │
│         │ moving towards the bottom-right.                                                                      │
│         │                                                                                                       │
│         │ ### Third Row: c) and g)                                                                              │
│         │                                                                                                       │
│         │ * Graph c) displays a random, pixelated pattern similar to graph b), with frequencies ranging from    │
│         │ -60 to 20 MHz.                                                                                        │
│         │ * Graph g) shows a curved, diagonal pattern with a clear trend, similar to graph f).                  │
│         │                                                                                                       │
│         │ ### Bottom Row: d) and h)                                                                             │
│         │                                                                                                       │
│         │ * Graph d) features a random, pixelated pattern with frequencies ranging from -60 to 20 MHz.          │
│         │ * Graph h) exhibits a curved, diagonal pattern with a clear trend, similar to graphs f) and g).       │
│         │                                                                                                       │
│         │ ## Summary                                                                                            │
│         │                                                                                                       │
│         │ The image presents a series of heatmap graphs illustrating the relationship between frequency (Δ) and │
│         │ time (t) for various correlation coefficients (τ). The graphs can be divided into two categories:     │
│         │ those with random, pixelated patterns (b, c, d) and those with curved, diagonal patterns (e, f, g,    │
│         │ h). The color bar at the top indicates the correlation coefficient values, with higher values         │
│         │ corresponding to yellow and lower values corresponding to blue. Overall, the image suggests a complex │
│         │ relationship between frequency, time, and correlation coefficient.                                    │
└─────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────┘
                                                                                                                   
                                                    [index: 0]

In [13]:

Copied!

# The preview dataset is available as a pandas DataFrame.
preview.dataset
# The preview dataset is available as a pandas DataFrame.
preview.dataset

Out[13]:

	uuid	image_filename	base64_image	page	options	source	summary
0	130906e7-ab6b-40be-9550-66c84a6ddc69	images/1810.07757_2.jpg	iVBORw0KGgoAAAANSUhEUgAAAUAAAAIACAIAAAB8QiIMAA...		['A. The variance of the data decreases from p...	arxiv_qa	## Image Summary\n\nThe image presents a colle...
1	bfd43029-c099-403a-8b61-023cc640018f	data/scrapped_pdfs_split/pages_extracted/energ...	iVBORw0KGgoAAAANSUhEUgAAAYsAAAIACAIAAAD8HddaAA...	9	None	pdf	## Document Summary\nThe document is a page fr...

📊 Analyze the generated data¶

Data Designer automatically generates a basic statistical analysis of the generated data.
This analysis is available via the analysis property of generation result objects.

In [14]:

Copied!

# Print the analysis as a table.
preview.analysis.to_report()
# Print the analysis as a table.
preview.analysis.to_report()

──────────────────────────────────────── 🎨 Data Designer Dataset Profile ─────────────────────────────────────────

                                                                                                                   
                                                 Dataset Overview                                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ number of records               ┃ number of columns               ┃ percent complete records                    ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 2                               │ 1                               │ 100.0%                                      │
└─────────────────────────────────┴─────────────────────────────────┴─────────────────────────────────────────────┘
                                                                                                                   
                                                                                                                   
                                                📝 LLM-Text Columns                                                
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                  ┃               ┃                              ┃       prompt tokens ┃       completion tokens ┃
┃ column name      ┃     data type ┃         number unique values ┃          per record ┃              per record ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ summary          │        string │                   2 (100.0%) │        38.0 +/- 0.0 │          384.5 +/- 20.5 │
└──────────────────┴───────────────┴──────────────────────────────┴─────────────────────┴─────────────────────────┘
                                                                                                                   
                                                                                                                   
╭────────────────────────────────────────────────── Table Notes ──────────────────────────────────────────────────╮
│                                                                                                                 │
│  1. All token statistics are based on a sample of max(1000, len(dataset)) records.                              │
│  2. Tokens are calculated using tiktoken's cl100k_base tokenizer.                                               │
│                                                                                                                 │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
                                                                                                                   
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────

🔎 Visual Inspection¶

Let's compare the original document image with the generated summary to validate quality:

In [15]:

Copied!





# Compare original document with generated summary
index = 0  # Change this to view different examples

# Merge preview data with original images for comparison
comparison_dataset = preview.dataset.merge(pd.DataFrame(img_dataset)[["uuid", "image"]], how="left", on="uuid")

# Extract the record for display
record = comparison_dataset.iloc[index]

print("📄 Original Document Image:")
display(resize_image(record.image, BASE64_IMAGE_HEIGHT))

print("\n📝 Generated Summary:")
rich.print(Panel(record.summary, title="Document Summary", title_align="left"))
# Compare original document with generated summary
index = 0  # Change this to view different examples

# Merge preview data with original images for comparison
comparison_dataset = preview.dataset.merge(pd.DataFrame(img_dataset)[["uuid", "image"]], how="left", on="uuid")

# Extract the record for display
record = comparison_dataset.iloc[index]

print("📄 Original Document Image:")
display(resize_image(record.image, BASE64_IMAGE_HEIGHT))

print("\n📝 Generated Summary:")
rich.print(Panel(record.summary, title="Document Summary", title_align="left"))

📄 Original Document Image:

No description has been provided for this image

📝 Generated Summary:

╭─ Document Summary ──────────────────────────────────────────────────────────────────────────────────────────────╮
│ ## Image Summary                                                                                                │
│                                                                                                                 │
│ The image presents a collection of 8 heatmap graphs, arranged in two columns labeled a) to h). A color bar at   │
│ the top indicates the correlation coefficient (τ) values, ranging from 0.2 (blue) to 1 (yellow).                │
│                                                                                                                 │
│ ### Top Row: a) and e)                                                                                          │
│                                                                                                                 │
│ * Graph a) displays a predominantly blue horizontal band centered around 0 MHz, with minimal variation along    │
│ the time axis (t).                                                                                              │
│ * Graph e) shows a similar blue band, but with a more pronounced curvature, indicating a change in frequency    │
│ over time.                                                                                                      │
│                                                                                                                 │
│ ### Second Row: b) and f)                                                                                       │
│                                                                                                                 │
│ * Graph b) features a random, pixelated pattern with no clear trend, spanning from -60 to 20 MHz.               │
│ * Graph f) exhibits a diagonal, curved pattern with a clear trend, starting from the top-left and moving        │
│ towards the bottom-right.                                                                                       │
│                                                                                                                 │
│ ### Third Row: c) and g)                                                                                        │
│                                                                                                                 │
│ * Graph c) displays a random, pixelated pattern similar to graph b), with frequencies ranging from -60 to 20    │
│ MHz.                                                                                                            │
│ * Graph g) shows a curved, diagonal pattern with a clear trend, similar to graph f).                            │
│                                                                                                                 │
│ ### Bottom Row: d) and h)                                                                                       │
│                                                                                                                 │
│ * Graph d) features a random, pixelated pattern with frequencies ranging from -60 to 20 MHz.                    │
│ * Graph h) exhibits a curved, diagonal pattern with a clear trend, similar to graphs f) and g).                 │
│                                                                                                                 │
│ ## Summary                                                                                                      │
│                                                                                                                 │
│ The image presents a series of heatmap graphs illustrating the relationship between frequency (Δ) and time (t)  │
│ for various correlation coefficients (τ). The graphs can be divided into two categories: those with random,     │
│ pixelated patterns (b, c, d) and those with curved, diagonal patterns (e, f, g, h). The color bar at the top    │
│ indicates the correlation coefficient values, with higher values corresponding to yellow and lower values       │
│ corresponding to blue. Overall, the image suggests a complex relationship between frequency, time, and          │
│ correlation coefficient.                                                                                        │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

🆙 Scale up!¶

Happy with your preview data?
Use the create method to submit larger Data Designer generation jobs.

In [16]:

Copied!

results = data_designer.create(config_builder, num_records=10, dataset_name="tutorial-4")
results = data_designer.create(config_builder, num_records=10, dataset_name="tutorial-4")

[21:55:38] [INFO] 🎨 Creating Data Designer dataset

[21:55:38] [INFO] ✅ Validation passed

[21:55:38] [INFO] ⛓️ Sorting column configs into a Directed Acyclic Graph

[21:55:38] [INFO] 🩺 Running health checks for models...

[21:55:38] [INFO]   |-- 👀 Checking 'meta/llama-4-scout-17b-16e-instruct' in provider named 'nvidia' for model alias 'vision'...

[21:55:38] [INFO]   |-- ✅ Passed!

[21:55:38] [INFO] ⏳ Processing batch 1 of 1

[21:55:38] [INFO] 🌱 Sampling 10 records from seed dataset

[21:55:38] [INFO]   |-- seed dataset size: 512 records

[21:55:38] [INFO]   |-- sampling strategy: ordered

[21:55:38] [INFO] 📝 llm-text model config for column 'summary'

[21:55:38] [INFO]   |-- model: 'meta/llama-4-scout-17b-16e-instruct'

[21:55:38] [INFO]   |-- model alias: 'vision'

[21:55:38] [INFO]   |-- model provider: 'nvidia'

[21:55:38] [INFO]   |-- inference parameters: generation_type=chat-completion, max_parallel_requests=4, temperature=0.60, top_p=0.95, max_tokens=2048

[21:55:38] [INFO] 🐙 Processing llm-text column 'summary' with 4 concurrent workers

[21:56:07] [INFO] 📊 Model usage summary:
{
    "meta/llama-4-scout-17b-16e-instruct": {
        "token_usage": {
            "input_tokens": 8140,
            "output_tokens": 5037,
            "total_tokens": 13177
        },
        "request_usage": {
            "successful_requests": 10,
            "failed_requests": 0,
            "total_requests": 10
        },
        "tokens_per_second": 458,
        "requests_per_minute": 20
    }
}

[21:56:07] [INFO] 📐 Measuring dataset column statistics:

[21:56:07] [INFO]   |-- 📝 column: 'summary'

In [17]:

Copied!

# Load the generated dataset as a pandas DataFrame.
dataset = results.load_dataset()

dataset.head()
# Load the generated dataset as a pandas DataFrame.
dataset = results.load_dataset()

dataset.head()

Out[17]:

	uuid	image_filename	base64_image	page	options	source	summary
0	130906e7-ab6b-40be-9550-66c84a6ddc69	images/1810.07757_2.jpg	iVBORw0KGgoAAAANSUhEUgAAAUAAAAIACAIAAAB8QiIMAA...		['A. The variance of the data decreases from p...	arxiv_qa	## Image Summary The image presents a collecti...
1	bfd43029-c099-403a-8b61-023cc640018f	data/scrapped_pdfs_split/pages_extracted/energ...	iVBORw0KGgoAAAANSUhEUgAAAYsAAAIACAIAAAD8HddaAA...	9	<NA>	pdf	## Detailed Summary of the Content in the Imag...
2	76c07916-b47a-4328-a23a-ec2612dc4ef8	data/scrapped_pdfs_split/pages_extracted/energ...	iVBORw0KGgoAAAANSUhEUgAAAZgAAAIACAIAAAAwhO2xAA...	414	<NA>	pdf	## Detailed Summary of Lithium Batteries Conte...
3	0be006d7-a6dd-4118-84e4-a80b9a66c44a	0fd47b51ae9248ef36669b8619b1223f268edae3e7a44a...	iVBORw0KGgoAAAANSUhEUgAAAX0AAAIACAAAAABLRuMPAA...	<NA>	<NA>	docvqa	## Document Summary The document appears to be...
4	74f6f759-512b-4cdf-8bd8-6ba09d965c59	b335cfb9d442f8925ea41a064cb445a5395577f2345d52...	iVBORw0KGgoAAAANSUhEUgAAAY8AAAIACAAAAABf/7+rAA...	<NA>	<NA>	docvqa	## Detailed Summary of the Content in the Imag...

In [18]:

Copied!

# Load the analysis results into memory.
analysis = results.load_analysis()

analysis.to_report()
# Load the analysis results into memory.
analysis = results.load_analysis()

analysis.to_report()

──────────────────────────────────────── 🎨 Data Designer Dataset Profile ─────────────────────────────────────────

                                                                                                                   
                                                 Dataset Overview                                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ number of records               ┃ number of columns               ┃ percent complete records                    ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 10                              │ 1                               │ 100.0%                                      │
└─────────────────────────────────┴─────────────────────────────────┴─────────────────────────────────────────────┘
                                                                                                                   
                                                                                                                   
                                                📝 LLM-Text Columns                                                
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                  ┃               ┃                              ┃       prompt tokens ┃       completion tokens ┃
┃ column name      ┃     data type ┃         number unique values ┃          per record ┃              per record ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ summary          │        string │                  10 (100.0%) │        38.0 +/- 0.0 │         486.0 +/- 180.8 │
└──────────────────┴───────────────┴──────────────────────────────┴─────────────────────┴─────────────────────────┘
                                                                                                                   
                                                                                                                   
╭────────────────────────────────────────────────── Table Notes ──────────────────────────────────────────────────╮
│                                                                                                                 │
│  1. All token statistics are based on a sample of max(1000, len(dataset)) records.                              │
│  2. Tokens are calculated using tiktoken's cl100k_base tokenizer.                                               │
│                                                                                                                 │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
                                                                                                                   
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────

⏭️ Next Steps¶

Now that you've learned how to use visual context for image summarization in Data Designer, explore more:

Experiment with different vision models for specific document types
Try different prompt variations to generate specialized descriptions (e.g., technical details, key findings)
Combine vision-based summaries with other column types for multi-modal workflows
Apply this pattern to other vision tasks like image captioning, OCR validation, or visual question answering