2.8T
parameters, fully fine-tuned on 256 GPUs with NeMo AutoModel. Molt uses that trainer unchanged, so a trillion-parameter policy takes the same command as a 4B one.
Easy to modify. Built to scale.
A Hugging Face-native framework for agentic RL research.
News & Blog
Architecture
Ray coordinates three kinds of processes: agent workers, vLLM rollout engines, and one trainer. In asynchronous mode the six stages overlap.
Generate tokens for the agents.
3Call tools and the model in turn, then return a reward.
Buffers finished trajectories and bounds policy lag.
NeMo AutoModel + FSDP2 update the policy.
Redirect an OpenAI or Anthropic client to Molt. When the agent rewrites its own context, the rollout is segmented automatically.
Batches come from completed groups while other rollouts stay in flight. Stale samples are corrected with the log-probabilities recorded at sampling time.
Screenshots and token sequences live in each node’s Ray object store. The queue carries only references.
NeMo AutoModel
The policy stays an ordinary PyTorch module in the Hugging Face layout. FSDP2 shards its parameters, and three forms of parallelism divide the computation. Each is one flag.
2.8T
parameters, fully fine-tuned on 256 GPUs with NeMo AutoModel. Molt uses that trainer unchanged, so a trillion-parameter policy takes the same command as a 4B one.
--fsdp.tp_size
Splits each weight matrix across GPUs.
--fsdp.ep_size
Places MoE experts on different GPUs and routes each token to the GPU that holds its expert.
--fsdp.cp_size
Splits one long sequence across GPUs, for trajectories of tens of thousands of tokens.
The checkpoint loads directly, the rollout engines read the same directory, and the model code contains no parallel logic.
Easy to modify
One training backend, one rollout engine, and command-line flags as the only configuration. The function you edit is the function that runs.
Each mechanism is one Python file of a few hundred lines, small enough to read in a sitting or hand whole to a coding agent.
| To change | Edit | Lines |
|---|---|---|
| Advantage estimator | trainer/algorithm/advantage.py | 249 |
| Policy loss | models/loss.py | 393 |
| Token capture | agents/_chat_server.py | 379 |
| Training loop | trainer/rl_trainer.py | 735 |
Quick start
A run is one command that names the checkpoint, the agent module, and the advantage estimator.
git clone https://github.com/NVIDIA-NeMo/labs-molt.git
cd labs-molt
# recommended: the prebuilt container with the whole CUDA 13 stack
docker pull hijkzzz/molt:latest
from openai import AsyncOpenAI
from molt.agents import ChatAgent, ChatAgentRunner, ChatContext, Result
class MyAgent(ChatAgent):
async def run(self, ctx: ChatContext) -> Result:
# ctx.base_url carries the session id and auto-captures the token
# trace — no extra_body, no logprobs=True, no session plumbing.
client = AsyncOpenAI(base_url=ctx.base_url, api_key=ctx.api_key)
resp = await client.chat.completions.create(
model=ctx.model_name,
messages=[{"role": "user", "content": ctx.prompt}],
max_tokens=ctx.sampling_params.max_tokens,
temperature=ctx.sampling_params.temperature,
)
return Result(reward=grade(resp.choices[0].message.content, ctx.label))
class AgentRunner(ChatAgentRunner):
def __init__(self):
super().__init__(MyAgent)
python3 -m molt.cli.train_rl_ray \
--actor.model_name_or_path /path/to/automodel \
--data.prompt_dataset /path/to/prompts.jsonl \
--data.input_key input \
--data.label_key answer \
--train.agent_path examples/python/agents/math.py \
--vllm.num_engines 2 \
--vllm.tensor_parallel_size 2 \
--rollout.batch_size 128 \
--train.batch_size 128 \
--train.micro_batch_size 1 \
--algo.advantage.estimator reinforce \
--fsdp.attn_implementation te \
--ckpt.output_dir ./ckpt/rl
torchrun --standalone --nproc_per_node=8 -m molt.cli.train_sft \
--model.model_name_or_path /path/to/automodel \
--data.dataset /path/to/sft.jsonl \
--data.input_key input \
--data.output_key output \
--ckpt.output_dir ./ckpt/sft \
--fsdp.attn_implementation te
Installation guide Example scripts Agent contract Every RL flag