🎨 Data Designer Tutorial: Providing Images as Context for Vision-Based Data Generation¶

📚 What you'll learn¶

This notebook demonstrates how to provide images as context to generate text descriptions using vision-language models.

✨ Visual Document Processing: Converting images to chat-ready format for model consumption
🔍 Vision-Language Generation: Using vision models to generate detailed summaries from images

If this is your first time using Data Designer, we recommend starting with the first notebook in this tutorial series.

📦 Import Data Designer¶

data_designer.config provides access to the configuration API.
DataDesigner is the main interface for data generation.

In [1]:

Copied!





# Standard library imports
import base64
import io
import uuid

# Third-party imports
import pandas as pd
import rich
from datasets import load_dataset
from IPython.display import display
from rich.panel import Panel

# Data Designer imports
import data_designer.config as dd
from data_designer.interface import DataDesigner
# Standard library imports
import base64
import io
import uuid

# Third-party imports
import pandas as pd
import rich
from datasets import load_dataset
from IPython.display import display
from rich.panel import Panel

# Data Designer imports
import data_designer.config as dd
from data_designer.interface import DataDesigner

⚙️ Initialize the Data Designer interface¶

DataDesigner is the main object is responsible for managing the data generation process.
When initialized without arguments, the default model providers are used.

In [2]:

Copied!

data_designer = DataDesigner()
data_designer = DataDesigner()

🎛️ Define model configurations¶

Each ModelConfig defines a model that can be used during the generation process.
The "model alias" is used to reference the model in the Data Designer config (as we will see below).
The "model provider" is the external service that hosts the model (see the model config docs for more details).
By default, we use build.nvidia.com as the model provider.

In [3]:

Copied!





# This name is set in the model provider configuration.
MODEL_PROVIDER = "nvidia"

model_configs = [
    dd.ModelConfig(
        alias="vision",
        model="meta/llama-4-scout-17b-16e-instruct",
        provider=MODEL_PROVIDER,
        inference_parameters=dd.ChatCompletionInferenceParams(
            temperature=0.60,
            top_p=0.95,
            max_tokens=2048,
        ),
    ),
]
# This name is set in the model provider configuration.
MODEL_PROVIDER = "nvidia"

model_configs = [
    dd.ModelConfig(
        alias="vision",
        model="meta/llama-4-scout-17b-16e-instruct",
        provider=MODEL_PROVIDER,
        inference_parameters=dd.ChatCompletionInferenceParams(
            temperature=0.60,
            top_p=0.95,
            max_tokens=2048,
        ),
    ),
]

🏗️ Initialize the Data Designer Config Builder¶

The Data Designer config defines the dataset schema and generation process.
The config builder provides an intuitive interface for building this configuration.
The list of model configs is provided to the builder at initialization.

In [4]:

Copied!

config_builder = dd.DataDesignerConfigBuilder(model_configs=model_configs)
config_builder = dd.DataDesignerConfigBuilder(model_configs=model_configs)

🌱 Seed Dataset Creation¶

In this section, we'll prepare our visual documents as a seed dataset for summarization:

Loading Visual Documents: We use the ColPali dataset containing document images
Image Processing: Convert images to base64 format for vision model consumption
Metadata Extraction: Preserve relevant document information (filename, page number, source, etc.)

The seed dataset will be used to generate detailed text summaries of each document image.

In [5]:

Copied!





# Dataset processing configuration
IMG_COUNT = 512  # Number of images to process
BASE64_IMAGE_HEIGHT = 512  # Standardized height for model input

# Load ColPali dataset for visual documents
img_dataset_cfg = {"path": "vidore/colpali_train_set", "split": "train", "streaming": True}
# Dataset processing configuration
IMG_COUNT = 512  # Number of images to process
BASE64_IMAGE_HEIGHT = 512  # Standardized height for model input

# Load ColPali dataset for visual documents
img_dataset_cfg = {"path": "vidore/colpali_train_set", "split": "train", "streaming": True}

In [6]:

Copied!





def resize_image(image, height: int):
    """
    Resize image while maintaining aspect ratio.

    Args:
        image: PIL Image object
        height: Target height in pixels

    Returns:
        Resized PIL Image object
    """
    original_width, original_height = image.size
    width = int(original_width * (height / original_height))
    return image.resize((width, height))


def convert_image_to_chat_format(record, height: int) -> dict:
    """
    Convert PIL image to base64 format for chat template usage.

    Args:
        record: Dataset record containing image and metadata
        height: Target height for image resizing

    Returns:
        Updated record with base64_image and uuid fields
    """
    # Resize image for consistent processing
    image = resize_image(record["image"], height)

    # Convert to base64 string
    img_buffer = io.BytesIO()
    image.save(img_buffer, format="PNG")
    byte_data = img_buffer.getvalue()
    base64_encoded_data = base64.b64encode(byte_data)
    base64_string = base64_encoded_data.decode("utf-8")

    # Return updated record
    return record | {"base64_image": base64_string, "uuid": str(uuid.uuid4())}
def resize_image(image, height: int):
    """
    Resize image while maintaining aspect ratio.

    Args:
        image: PIL Image object
        height: Target height in pixels

    Returns:
        Resized PIL Image object
    """
    original_width, original_height = image.size
    width = int(original_width * (height / original_height))
    return image.resize((width, height))


def convert_image_to_chat_format(record, height: int) -> dict:
    """
    Convert PIL image to base64 format for chat template usage.

    Args:
        record: Dataset record containing image and metadata
        height: Target height for image resizing

    Returns:
        Updated record with base64_image and uuid fields
    """
    # Resize image for consistent processing
    image = resize_image(record["image"], height)

    # Convert to base64 string
    img_buffer = io.BytesIO()
    image.save(img_buffer, format="PNG")
    byte_data = img_buffer.getvalue()
    base64_encoded_data = base64.b64encode(byte_data)
    base64_string = base64_encoded_data.decode("utf-8")

    # Return updated record
    return record | {"base64_image": base64_string, "uuid": str(uuid.uuid4())}

In [7]:

Copied!





# Load and process the visual document dataset
print("📥 Loading and processing document images...")

img_dataset_iter = iter(
    load_dataset(**img_dataset_cfg).map(convert_image_to_chat_format, fn_kwargs={"height": BASE64_IMAGE_HEIGHT})
)
img_dataset = pd.DataFrame([next(img_dataset_iter) for _ in range(IMG_COUNT)])

print(f"✅ Loaded {len(img_dataset)} images with columns: {list(img_dataset.columns)}")
# Load and process the visual document dataset
print("📥 Loading and processing document images...")

img_dataset_iter = iter(
    load_dataset(**img_dataset_cfg).map(convert_image_to_chat_format, fn_kwargs={"height": BASE64_IMAGE_HEIGHT})
)
img_dataset = pd.DataFrame([next(img_dataset_iter) for _ in range(IMG_COUNT)])

print(f"✅ Loaded {len(img_dataset)} images with columns: {list(img_dataset.columns)}")

📥 Loading and processing document images...

✅ Loaded 512 images with columns: ['image', 'image_filename', 'query', 'answer', 'source', 'options', 'page', 'model', 'prompt', 'answer_type', 'base64_image', 'uuid']

In [8]:

Copied!

img_dataset.head()
img_dataset.head()

Out[8]:

	image	image_filename	query	answer	source	options	page	model	prompt	answer_type	base64_image	uuid
0	<PIL.JpegImagePlugin.JpegImageFile image mode=...	images/1810.07757_2.jpg	Comparing panels a, b, c, and d, which stateme...	D	arxiv_qa	['A. The variance of the data decreases from p...		gpt4V		None	iVBORw0KGgoAAAANSUhEUgAAAUAAAAIACAIAAAB8QiIMAA...	143b4c94-24cc-47be-94b9-ab9d867722b8
1	<PIL.JpegImagePlugin.JpegImageFile image mode=...	data/scrapped_pdfs_split/pages_extracted/energ...	What is the duration of the course mentioned i...	['five to ten hours, not including field trips']	pdf	None	9	sonnet	\n You are an assistant specialized in ...	None	iVBORw0KGgoAAAANSUhEUgAAAYsAAAIACAIAAAD8HddaAA...	f0a218f1-4454-452f-9e3a-6024b7a10e7e
2	<PIL.JpegImagePlugin.JpegImageFile image mode=...	data/scrapped_pdfs_split/pages_extracted/energ...	What is the primary purpose of the PTC in lith...	['protect against external short circuits']	pdf	None	414	sonnet	\n You are an assistant specialized in ...	None	iVBORw0KGgoAAAANSUhEUgAAAZgAAAIACAIAAAAwhO2xAA...	ea705ead-c07a-47fb-b083-5a619b310cfa
3	<PIL.PngImagePlugin.PngImageFile image mode=L ...	0fd47b51ae9248ef36669b8619b1223f268edae3e7a44a...	What is the date?\nYour answer should be very ...	OCTOBER 17, 1995.	docvqa	None	None	None	None	None	iVBORw0KGgoAAAANSUhEUgAAAX0AAAIACAAAAABLRuMPAA...	edbce50b-b2f7-4fdd-a5ac-9d8a1b55f69a
4	<PIL.PngImagePlugin.PngImageFile image mode=L ...	b335cfb9d442f8925ea41a064cb445a5395577f2345d52...	What is Bert Shulimson's title?\nYour response...	EXECUTIVE SECRETARY.	docvqa	None	None	None	None	None	iVBORw0KGgoAAAANSUhEUgAAAY8AAAIACAAAAABf/7+rAA...	ca099aaf-0e70-48dc-abf1-8c26ae29222d

In [9]:

Copied!

# Add the seed dataset containing our processed images
df_seed = pd.DataFrame(img_dataset)[["uuid", "image_filename", "base64_image", "page", "options", "source"]]
config_builder.with_seed_dataset(dd.DataFrameSeedSource(df=df_seed))
# Add the seed dataset containing our processed images
df_seed = pd.DataFrame(img_dataset)[["uuid", "image_filename", "base64_image", "page", "options", "source"]]
config_builder.with_seed_dataset(dd.DataFrameSeedSource(df=df_seed))

Out[9]:

DataDesignerConfigBuilder()

In [10]:

Copied!





# Add a column to generate detailed document summaries
config_builder.add_column(
    dd.LLMTextColumnConfig(
        name="summary",
        model_alias="vision",
        prompt=(
            "Provide a detailed summary of the content in this image in Markdown format. "
            "Start from the top of the image and then describe it from top to bottom. "
            "Place a summary at the bottom."
        ),
        multi_modal_context=[
            dd.ImageContext(
                column_name="base64_image",
                data_type=dd.ModalityDataType.BASE64,
                image_format=dd.ImageFormat.PNG,
            )
        ],
    )
)
# Add a column to generate detailed document summaries
config_builder.add_column(
    dd.LLMTextColumnConfig(
        name="summary",
        model_alias="vision",
        prompt=(
            "Provide a detailed summary of the content in this image in Markdown format. "
            "Start from the top of the image and then describe it from top to bottom. "
            "Place a summary at the bottom."
        ),
        multi_modal_context=[
            dd.ImageContext(
                column_name="base64_image",
                data_type=dd.ModalityDataType.BASE64,
                image_format=dd.ImageFormat.PNG,
            )
        ],
    )
)

Out[10]:

DataDesignerConfigBuilder(
    seed_dataset: df seed
    llm_text_columns: ['summary']
)

🔁 Iteration is key – preview the dataset!¶

Use the preview method to generate a sample of records quickly.
Inspect the results for quality and format issues.
Adjust column configurations, prompts, or parameters as needed.
Re-run the preview until satisfied.

In [11]:

Copied!

preview = data_designer.preview(config_builder, num_records=2)
preview = data_designer.preview(config_builder, num_records=2)

[03:46:18] [INFO] 🔁 Preview generation in progress

[03:46:20] [INFO] ✅ Validation passed

[03:46:20] [INFO] ⛓️ Sorting column configs into a Directed Acyclic Graph

[03:46:20] [INFO] 🩺 Running health checks for models...

[03:46:20] [INFO]   |-- 👀 Checking 'meta/llama-4-scout-17b-16e-instruct' in provider named 'nvidia' for model alias 'vision'...

[03:46:20] [INFO]   |-- ✅ Passed!

[03:46:20] [INFO] 🌱 Sampling 2 records from seed dataset

[03:46:20] [INFO]   |-- seed dataset size: 512 records

[03:46:20] [INFO]   |-- sampling strategy: ordered

[03:46:20] [INFO] 📝 llm-text model config for column 'summary'

[03:46:20] [INFO]   |-- model: 'meta/llama-4-scout-17b-16e-instruct'

[03:46:20] [INFO]   |-- model alias: 'vision'

[03:46:20] [INFO]   |-- model provider: 'nvidia'

[03:46:20] [INFO]   |-- inference parameters: generation_type=chat-completion, max_parallel_requests=4, temperature=0.60, top_p=0.95, max_tokens=2048

[03:46:20] [INFO] 🐙 Processing llm-text column 'summary' with 4 concurrent workers

[03:46:20] [INFO] 🧭 llm-text column 'summary' will report progress every 1 record(s).

[03:46:25] [INFO]   |-- ⛅ llm-text column 'summary' progress: 1/2 (50%) complete, 1 ok, 0 failed, 0.21 rec/s, eta 4.9s

[03:46:29] [INFO]   |-- ☀️ llm-text column 'summary' progress: 2/2 (100%) complete, 2 ok, 0 failed, 0.24 rec/s, eta 0.0s

[03:46:29] [INFO] 📊 Model usage summary:
{
    "meta/llama-4-scout-17b-16e-instruct": {
        "token_usage": {
            "input_tokens": 1394,
            "output_tokens": 633,
            "total_tokens": 2027
        },
        "request_usage": {
            "successful_requests": 2,
            "failed_requests": 0,
            "total_requests": 2
        },
        "tokens_per_second": 232,
        "requests_per_minute": 13
    }
}

[03:46:29] [INFO] 📐 Measuring dataset column statistics:

[03:46:29] [INFO]   |-- 📝 column: 'summary'

[03:46:29] [INFO] 🍾 Preview complete!

In [12]:

Copied!

# Run this cell multiple times to cycle through the 2 preview records.
preview.display_sample_record()
# Run this cell multiple times to cycle through the 2 preview records.
preview.display_sample_record()

                                                                                                                   
                                                   Seed Columns                                                    
┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name           ┃ Value                                                                                          ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ uuid           │ 143b4c94-24cc-47be-94b9-ab9d867722b8                                                           │
├────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────┤
│ image_filename │ images/1810.07757_2.jpg                                                                        │
├────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────┤
│ base64_image   │ iVBORw0KGgoAAAANSUhEUgAAAUAAAAIACAIAAAB8QiIMAAEAAElEQVR4nOy9edRt2VUX+vvNufY+53zNbauvVJdKSEUQQ… │
├────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────┤
│ page           │                                                                                                │
├────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────┤
│ options        │ ['A. The variance of the data decreases from panel a to panel d.', 'B. The variance of the     │
│                │ data increases from panel a to panel d.', 'C. The data presents no variance in any of the      │
│                │ panels.', 'D. The variance of the data is inconsistent across the panels.', '-']               │
├────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────┤
│ source         │ arxiv_qa                                                                                       │
└────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────┘
                                                                                                                   
                                                                                                                   
                                                 Generated Columns                                                 
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name    ┃ Value                                                                                                 ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ summary │ The image presents a series of 2D heatmaps, arranged in two columns and four rows, with each heatmap  │
│         │ labeled with a letter from "a" to "h".                                                                │
│         │                                                                                                       │
│         │ ### Top Section: Color Bar                                                                            │
│         │ At the top of the image, a color bar is displayed, featuring a gradient that transitions from blue to │
│         │ yellow. The color bar is accompanied by a numerical scale ranging from 0.2 to 1, with increments of   │
│         │ 0.1.                                                                                                  │
│         │                                                                                                       │
│         │ ### Heatmaps                                                                                          │
│         │ The heatmaps are rectangular in shape and exhibit a range of colors, including blue, green, yellow,   │
│         │ and orange. Each heatmap has a distinct pattern, with some displaying more vibrant colors than        │
│         │ others. The x-axis of each heatmap is labeled with values ranging from 0 to 0.8, with increments of   │
│         │ 0.2, and is denoted as "t(μs)". The y-axis is labeled with values ranging from -60 to 20, with        │
│         │ increments of 20, and is denoted as "Δf(MHz)".                                                        │
│         │                                                                                                       │
│         │ ### Summary                                                                                           │
│         │ In summary, the image showcases a collection of 2D heatmaps, each with its own unique                 │
│         │ characteristics, arranged in a grid-like pattern. The color bar at the top provides a visual          │
│         │ reference for interpreting the heatmaps, which appear to represent some type of frequency or signal   │
│         │ data.                                                                                                 │
└─────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────┘
                                                                                                                   
                                                    [index: 0]

In [13]:

Copied!

# The preview dataset is available as a pandas DataFrame.
preview.dataset
# The preview dataset is available as a pandas DataFrame.
preview.dataset

Out[13]:

	uuid	image_filename	base64_image	page	options	source	summary
0	143b4c94-24cc-47be-94b9-ab9d867722b8	images/1810.07757_2.jpg	iVBORw0KGgoAAAANSUhEUgAAAUAAAAIACAIAAAB8QiIMAA...		['A. The variance of the data decreases from p...	arxiv_qa	The image presents a series of 2D heatmaps, ar...
1	f0a218f1-4454-452f-9e3a-6024b7a10e7e	data/scrapped_pdfs_split/pages_extracted/energ...	iVBORw0KGgoAAAANSUhEUgAAAYsAAAIACAIAAAD8HddaAA...	9	None	pdf	The image presents a document with the title "...

📊 Analyze the generated data¶

Data Designer automatically generates a basic statistical analysis of the generated data.
This analysis is available via the analysis property of generation result objects.

In [14]:

Copied!

# Print the analysis as a table.
preview.analysis.to_report()
# Print the analysis as a table.
preview.analysis.to_report()

──────────────────────────────────────── 🎨 Data Designer Dataset Profile ─────────────────────────────────────────

                                                                                                                   
                                                 Dataset Overview                                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ number of records               ┃ number of columns               ┃ percent complete records                    ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 2                               │ 1                               │ 100.0%                                      │
└─────────────────────────────────┴─────────────────────────────────┴─────────────────────────────────────────────┘
                                                                                                                   
                                                                                                                   
                                                📝 LLM-Text Columns                                                
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                  ┃               ┃                              ┃       prompt tokens ┃       completion tokens ┃
┃ column name      ┃     data type ┃         number unique values ┃          per record ┃              per record ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ summary          │        string │                   2 (100.0%) │        38.0 +/- 0.0 │          315.5 +/- 64.3 │
└──────────────────┴───────────────┴──────────────────────────────┴─────────────────────┴─────────────────────────┘
                                                                                                                   
                                                                                                                   
╭────────────────────────────────────────────────── Table Notes ──────────────────────────────────────────────────╮
│                                                                                                                 │
│  1. All token statistics are based on a sample of max(1000, len(dataset)) records.                              │
│  2. Tokens are calculated using tiktoken's cl100k_base tokenizer.                                               │
│                                                                                                                 │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
                                                                                                                   
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────

🔎 Visual Inspection¶

Let's compare the original document image with the generated summary to validate quality:

In [15]:

Copied!





# Compare original document with generated summary
index = 0  # Change this to view different examples

# Merge preview data with original images for comparison
comparison_dataset = preview.dataset.merge(pd.DataFrame(img_dataset)[["uuid", "image"]], how="left", on="uuid")

# Extract the record for display
record = comparison_dataset.iloc[index]

print("📄 Original Document Image:")
display(resize_image(record.image, BASE64_IMAGE_HEIGHT))

print("\n📝 Generated Summary:")
rich.print(Panel(record.summary, title="Document Summary", title_align="left"))
# Compare original document with generated summary
index = 0  # Change this to view different examples

# Merge preview data with original images for comparison
comparison_dataset = preview.dataset.merge(pd.DataFrame(img_dataset)[["uuid", "image"]], how="left", on="uuid")

# Extract the record for display
record = comparison_dataset.iloc[index]

print("📄 Original Document Image:")
display(resize_image(record.image, BASE64_IMAGE_HEIGHT))

print("\n📝 Generated Summary:")
rich.print(Panel(record.summary, title="Document Summary", title_align="left"))

📄 Original Document Image:

No description has been provided for this image

📝 Generated Summary:

╭─ Document Summary ──────────────────────────────────────────────────────────────────────────────────────────────╮
│ The image presents a series of 2D heatmaps, arranged in two columns and four rows, with each heatmap labeled    │
│ with a letter from "a" to "h".                                                                                  │
│                                                                                                                 │
│ ### Top Section: Color Bar                                                                                      │
│ At the top of the image, a color bar is displayed, featuring a gradient that transitions from blue to yellow.   │
│ The color bar is accompanied by a numerical scale ranging from 0.2 to 1, with increments of 0.1.                │
│                                                                                                                 │
│ ### Heatmaps                                                                                                    │
│ The heatmaps are rectangular in shape and exhibit a range of colors, including blue, green, yellow, and orange. │
│ Each heatmap has a distinct pattern, with some displaying more vibrant colors than others. The x-axis of each   │
│ heatmap is labeled with values ranging from 0 to 0.8, with increments of 0.2, and is denoted as "t(μs)". The    │
│ y-axis is labeled with values ranging from -60 to 20, with increments of 20, and is denoted as "Δf(MHz)".       │
│                                                                                                                 │
│ ### Summary                                                                                                     │
│ In summary, the image showcases a collection of 2D heatmaps, each with its own unique characteristics, arranged │
│ in a grid-like pattern. The color bar at the top provides a visual reference for interpreting the heatmaps,     │
│ which appear to represent some type of frequency or signal data.                                                │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

🆙 Scale up!¶

Happy with your preview data?
Use the create method to submit larger Data Designer generation jobs.

In [16]:

Copied!

results = data_designer.create(config_builder, num_records=10, dataset_name="tutorial-4")
results = data_designer.create(config_builder, num_records=10, dataset_name="tutorial-4")

[03:46:29] [INFO] 🎨 Creating Data Designer dataset

[03:46:29] [INFO] ✅ Validation passed

[03:46:29] [INFO] ⛓️ Sorting column configs into a Directed Acyclic Graph

[03:46:29] [INFO] 🩺 Running health checks for models...

[03:46:29] [INFO]   |-- 👀 Checking 'meta/llama-4-scout-17b-16e-instruct' in provider named 'nvidia' for model alias 'vision'...

[03:46:30] [INFO]   |-- ✅ Passed!

[03:46:30] [INFO] ⏳ Processing batch 1 of 1

[03:46:30] [INFO] 🌱 Sampling 10 records from seed dataset

[03:46:30] [INFO]   |-- seed dataset size: 512 records

[03:46:30] [INFO]   |-- sampling strategy: ordered

[03:46:30] [INFO] 📝 llm-text model config for column 'summary'

[03:46:30] [INFO]   |-- model: 'meta/llama-4-scout-17b-16e-instruct'

[03:46:30] [INFO]   |-- model alias: 'vision'

[03:46:30] [INFO]   |-- model provider: 'nvidia'

[03:46:30] [INFO]   |-- inference parameters: generation_type=chat-completion, max_parallel_requests=4, temperature=0.60, top_p=0.95, max_tokens=2048

[03:46:30] [INFO] 🐙 Processing llm-text column 'summary' with 4 concurrent workers

[03:46:30] [INFO] 🧭 llm-text column 'summary' will report progress every 1 record(s).

[03:46:36] [INFO]   |-- 🌧️ llm-text column 'summary' progress: 1/10 (10%) complete, 1 ok, 0 failed, 0.15 rec/s, eta 58.4s

[03:46:37] [INFO]   |-- 🌧️ llm-text column 'summary' progress: 2/10 (20%) complete, 2 ok, 0 failed, 0.29 rec/s, eta 27.3s

[03:46:38] [INFO]   |-- 🌦️ llm-text column 'summary' progress: 3/10 (30%) complete, 3 ok, 0 failed, 0.38 rec/s, eta 18.4s

[03:46:45] [INFO]   |-- 🌦️ llm-text column 'summary' progress: 4/10 (40%) complete, 4 ok, 0 failed, 0.26 rec/s, eta 22.8s

[03:46:45] [INFO]   |-- ⛅ llm-text column 'summary' progress: 5/10 (50%) complete, 5 ok, 0 failed, 0.33 rec/s, eta 15.3s

[03:46:47] [INFO]   |-- ⛅ llm-text column 'summary' progress: 6/10 (60%) complete, 6 ok, 0 failed, 0.34 rec/s, eta 11.7s

[03:46:49] [INFO]   |-- ⛅ llm-text column 'summary' progress: 7/10 (70%) complete, 7 ok, 0 failed, 0.37 rec/s, eta 8.1s

[03:46:53] [INFO]   |-- 🌤️ llm-text column 'summary' progress: 8/10 (80%) complete, 8 ok, 0 failed, 0.34 rec/s, eta 5.9s

[03:46:57] [INFO]   |-- 🌤️ llm-text column 'summary' progress: 9/10 (90%) complete, 9 ok, 0 failed, 0.34 rec/s, eta 3.0s

[03:47:00] [INFO]   |-- ☀️ llm-text column 'summary' progress: 10/10 (100%) complete, 10 ok, 0 failed, 0.33 rec/s, eta 0.0s

[03:47:00] [INFO] 📊 Model usage summary:
{
    "meta/llama-4-scout-17b-16e-instruct": {
        "token_usage": {
            "input_tokens": 8130,
            "output_tokens": 4531,
            "total_tokens": 12661
        },
        "request_usage": {
            "successful_requests": 10,
            "failed_requests": 0,
            "total_requests": 10
        },
        "tokens_per_second": 413,
        "requests_per_minute": 19
    }
}

[03:47:00] [INFO] 📐 Measuring dataset column statistics:

[03:47:00] [INFO]   |-- 📝 column: 'summary'

In [17]:

Copied!

# Load the generated dataset as a pandas DataFrame.
dataset = results.load_dataset()

dataset.head()
# Load the generated dataset as a pandas DataFrame.
dataset = results.load_dataset()

dataset.head()

Out[17]:

	uuid	image_filename	base64_image	page	options	source	summary
0	143b4c94-24cc-47be-94b9-ab9d867722b8	images/1810.07757_2.jpg	iVBORw0KGgoAAAANSUhEUgAAAUAAAAIACAIAAAB8QiIMAA...		['A. The variance of the data decreases from p...	arxiv_qa	The image presents a series of heatmaps, arran...
1	f0a218f1-4454-452f-9e3a-6024b7a10e7e	data/scrapped_pdfs_split/pages_extracted/energ...	iVBORw0KGgoAAAANSUhEUgAAAYsAAAIACAIAAAD8HddaAA...	9	<NA>	pdf	The image presents a document with the title "...
2	ea705ead-c07a-47fb-b083-5a619b310cfa	data/scrapped_pdfs_split/pages_extracted/energ...	iVBORw0KGgoAAAANSUhEUgAAAZgAAAIACAIAAAAwhO2xAA...	414	<NA>	pdf	The image presents a page from a technical doc...
3	edbce50b-b2f7-4fdd-a5ac-9d8a1b55f69a	0fd47b51ae9248ef36669b8619b1223f268edae3e7a44a...	iVBORw0KGgoAAAANSUhEUgAAAX0AAAIACAAAAABLRuMPAA...	<NA>	<NA>	docvqa	The image presents a document with a white bac...
4	ca099aaf-0e70-48dc-abf1-8c26ae29222d	b335cfb9d442f8925ea41a064cb445a5395577f2345d52...	iVBORw0KGgoAAAANSUhEUgAAAY8AAAIACAAAAABf/7+rAA...	<NA>	<NA>	docvqa	The image presents a letter from Volunteers Ag...

In [18]:

Copied!

# Load the analysis results into memory.
analysis = results.load_analysis()

analysis.to_report()
# Load the analysis results into memory.
analysis = results.load_analysis()

analysis.to_report()

──────────────────────────────────────── 🎨 Data Designer Dataset Profile ─────────────────────────────────────────

                                                                                                                   
                                                 Dataset Overview                                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ number of records               ┃ number of columns               ┃ percent complete records                    ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 10                              │ 1                               │ 100.0%                                      │
└─────────────────────────────────┴─────────────────────────────────┴─────────────────────────────────────────────┘
                                                                                                                   
                                                                                                                   
                                                📝 LLM-Text Columns                                                
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                  ┃               ┃                              ┃       prompt tokens ┃       completion tokens ┃
┃ column name      ┃     data type ┃         number unique values ┃          per record ┃              per record ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ summary          │        string │                  10 (100.0%) │        38.0 +/- 0.0 │         433.0 +/- 117.0 │
└──────────────────┴───────────────┴──────────────────────────────┴─────────────────────┴─────────────────────────┘
                                                                                                                   
                                                                                                                   
╭────────────────────────────────────────────────── Table Notes ──────────────────────────────────────────────────╮
│                                                                                                                 │
│  1. All token statistics are based on a sample of max(1000, len(dataset)) records.                              │
│  2. Tokens are calculated using tiktoken's cl100k_base tokenizer.                                               │
│                                                                                                                 │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
                                                                                                                   
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────

⏭️ Next Steps¶

Now that you've learned how to use visual context for image summarization in Data Designer, explore more:

Experiment with different vision models for specific document types
Try different prompt variations to generate specialized descriptions (e.g., technical details, key findings)
Combine vision-based summaries with other column types for multi-modal workflows
Apply this pattern to other vision tasks like image captioning, OCR validation, or visual question answering