NewRead the blog

Molt

Easy to modify. Built to scale.

A Hugging Face-native framework for agentic RL research.

News & Blog

The latest.

All posts

Architecture

One loop.

Ray coordinates three kinds of processes: agent workers, vLLM rollout engines, and one trainer. In asynchronous mode the six stages overlap.

Rollout

vLLM engines

Generate tokens for the agents.

Your program

Agent workers

3Call tools and the model in turn, then return a reward.

Ray

Queue

Buffers finished trajectories and bounds policy lag.

Training

Trainer

NeMo AutoModel + FSDP2 update the policy.

6weight refit: new weights return to the engines

NeMo AutoModel

Parallelism, applied from outside.

The policy stays an ordinary PyTorch module in the Hugging Face layout. FSDP2 shards its parameters, and three forms of parallelism divide the computation. Each is one flag.

The trainer, as released

2.8T

parameters, fully fine-tuned on 256 GPUs with NeMo AutoModel. Molt uses that trainer unchanged, so a trillion-parameter policy takes the same command as a 4B one.

--fsdp.tp_size

Tensor parallelism

Splits each weight matrix across GPUs.

--fsdp.ep_size

Expert parallelism

Places MoE experts on different GPUs and routes each token to the GPU that holds its expert.

--fsdp.cp_size

Context parallelism

Splits one long sequence across GPUs, for trajectories of tens of thousands of tokens.

Hugging Face-native

A new model needs no bridge.

The checkpoint loads directly, the rollout engines read the same directory, and the model code contains no parallel logic.

Bridged, the Megatron pathcode written for each model family
Hugging Face checkpoint Translate the configuration Convert weights to Megatron’s layout Megatron model Convert back at every weight refit Rollout engine
Native, the Molt path
Hugging Face checkpoint Ordinary PyTorch module in the Hugging Face layoutFSDP2 + TP / EP / CP wrap it from outside Rollout engine

Easy to modify

One path. Flat code. Small files.

One training backend, one rollout engine, and command-line flags as the only configuration. The function you edit is the function that runs.

Code map

Where a research change goes.

Each mechanism is one Python file of a few hundred lines, small enough to read in a sitting or hand whole to a coding agent.

To changeEditLines
Advantage estimatortrainer/algorithm/advantage.py249
Policy lossmodels/loss.py393
Token captureagents/_chat_server.py379
Training looptrainer/rl_trainer.py735
Algorithms

Pick an estimator by name.

  • REINFORCE
  • GRPO
  • Dr. GRPO
  • RLOO
  • GAE (PPO)
  • FlashREINFORCE
  • On-policy distillation
Agents & rewards

Reward is any Python you write.

  • Graders
  • Multi-turn tools
  • VLM environments
  • LLM-as-judge
  • Gymnasium-style Env
  • ChatAgent
Models

Train them as released.

  • Qwen3
  • DeepSeek-V3
  • GLM-5
  • Kimi K2
  • Gemma 4
  • Llama
  • Nemotron 3
  • 60+ families
Training controls

Stay stable at scale.

  • Importance sampling
  • MoE routing replay
  • Router freeze
  • KL regularization
  • LoRA
  • Adam CPU offload

Benchmarks and results in the technical report

Quick start

From clone to rollout.

A run is one command that names the checkpoint, the agent module, and the advantage estimator.

git clone https://github.com/NVIDIA-NeMo/labs-molt.git
cd labs-molt

# recommended: the prebuilt container with the whole CUDA 13 stack
docker pull hijkzzz/molt:latest
from openai import AsyncOpenAI
from molt.agents import ChatAgent, ChatAgentRunner, ChatContext, Result


class MyAgent(ChatAgent):
    async def run(self, ctx: ChatContext) -> Result:
        # ctx.base_url carries the session id and auto-captures the token
        # trace — no extra_body, no logprobs=True, no session plumbing.
        client = AsyncOpenAI(base_url=ctx.base_url, api_key=ctx.api_key)
        resp = await client.chat.completions.create(
            model=ctx.model_name,
            messages=[{"role": "user", "content": ctx.prompt}],
            max_tokens=ctx.sampling_params.max_tokens,
            temperature=ctx.sampling_params.temperature,
        )
        return Result(reward=grade(resp.choices[0].message.content, ctx.label))


class AgentRunner(ChatAgentRunner):
    def __init__(self):
        super().__init__(MyAgent)
python3 -m molt.cli.train_rl_ray \
  --actor.model_name_or_path /path/to/automodel \
  --data.prompt_dataset /path/to/prompts.jsonl \
  --data.input_key input \
  --data.label_key answer \
  --train.agent_path examples/python/agents/math.py \
  --vllm.num_engines 2 \
  --vllm.tensor_parallel_size 2 \
  --rollout.batch_size 128 \
  --train.batch_size 128 \
  --train.micro_batch_size 1 \
  --algo.advantage.estimator reinforce \
  --fsdp.attn_implementation te \
  --ckpt.output_dir ./ckpt/rl
torchrun --standalone --nproc_per_node=8 -m molt.cli.train_sft \
  --model.model_name_or_path /path/to/automodel \
  --data.dataset /path/to/sft.jsonl \
  --data.input_key input \
  --data.output_key output \
  --ckpt.output_dir ./ckpt/sft \
  --fsdp.attn_implementation te