Finetune MiniMax-M2.7 in NeMo Automodel

[HuiyingLi]

MiniMax-M2.7 is supported in NeMo-Automodel thanks to @hemildesai !
MiniMaxAI/MiniMax-M2.7 is MiniMax's latest open-source large Mixture-of-Experts language model featuring linear attention for efficient long-context inference:

• Mixture of Experts (MoE): 256 routed experts with 8 active per token, enabling efficient scaling and specialization.
• 229B total parameters, with a compact activated footprint per token.
• 62 layers, hidden size 3072, with 48 attention heads and 8 KV heads (GQA).
• ~196k context window (max_position_embeddings=196,608).
• Model Self-Evolution: Autonomously optimized programming scaffold over 100+ rounds, achieving 30% performance improvement.
• Agent Teams: Native multi-agent collaboration with stable role identity.
• Tool Calling: Supports complex skill execution and dynamic tool search.

Parallel Setup

We provide a fine-tuning recipe for MiniMax-M2.7 that scales training using Expert Parallelism and Pipeline Parallelism. The configuration runs with EP=16 and PP=2 across 4 nodes (8× H100 GPUs per node).

distributed:
  strategy: fsdp2
  tp_size: 1
  cp_size: 1
  pp_size: 2
  dp_replicate_size: 1
  ep_size: 16
  sequence_parallel: false

  pipeline:
    pp_schedule: interleaved1f1b
    pp_microbatch_size: 1
    round_virtual_stages_to_pp_multiple: down
    scale_grads_in_schedule: false
    patch_inner_model: false
    patch_causal_lm_model: false
    layers_per_stage: 2

Data

We use the HellaSwag dataset as an example. HellaSwag is a commonsense reasoning benchmark for evaluating language models on sentence completion tasks.

Below is the loss curve obtained when fine-tuning on HellaSwag with this recipe: