Nemo-Skills

Benchmarks and rollouts are moving to NeMo-Gym

We encourage you to move your rollout workflows over to NeMo-Gym, and to implement new benchmarks there rather than in Nemo-Skills.

Nemo-Skills will continue to add support for running NeMo-Gym benchmarks, so you can run them with the same cluster configs, server configurations, and slurm setup you already use. The benchmark implementation itself should live in NeMo-Gym. An early version of this exists today as ns nemo_gym_rollouts, but the interface is experimental and will likely change.

Nemo-Skills is a collection of pipelines to improve "skills" of large language models (LLMs). We support everything needed for LLM development, from synthetic data generation, to model training, to evaluation on a wide range of benchmarks. Start developing on a local workstation and move to a large-scale Slurm cluster with just a one-line change.

Here are some of the features we support:

• Flexible LLM inference:
  • Seamlessly switch between API providers, local server and large-scale Slurm jobs for LLM inference.
  • Host models (on 1 or many nodes) with TensorRT-LLM, vLLM, sglang or Megatron.
  • Scale SDG jobs from 1 GPU on a local machine all the way to tens of thousands of GPUs on a Slurm cluster.
• Model evaluation:
  • Evaluate your models on many popular benchmarks.
    • Math (natural language): e.g. aime24, aime25, hmmt_feb25
    • Math (formal language): e.g. minif2f, proofnet, putnam-bench
    • Code: e.g. swe-bench, livecodebench, bird
    • Scientific knowledge: e.g., hle, scicode, gpqa.
    • Instruction following: e.g. ifbench, ifeval
    • Long-context: e.g. ruler, mrcr, aalcr
    • Tool-calling: e.g. bfcl_v3
    • Multilingual: e.g. mmlu-prox, flores-200, wmt24pp
    • Speech & Audio: e.g. asr-leaderboard, mmau-pro
    • Vision-Language Models (VLM): e.g. mmmu-pro
  • Easily parallelize each evaluation across many Slurm jobs, self-host LLM judges, bring your own prompts or change benchmark configuration in any other way.
• Model training: Train models using NeMo-RL or verl.

To get started, follow these steps, browse available pipelines or run ns --help to see all available commands and their options.

You can find more examples of how to use Nemo-Skills in the tutorials page.

We've built and released many popular models and datasets using Nemo-Skills. See all of them in the Papers & Releases documentation.

We support many popular benchmarks and it's easy to add new in the future. The following categories of benchmarks are supported