Train

train.py

usage: train.py [-h] [-config CONFIG] [-save_config SAVE_CONFIG] -tasks TASKS
                [-transforms {denoising,prefix,filterfeats,inferfeats,switchout,tokendrop,tokenmask,filtertoolong,filterwordratio,filterrepetitions,filterterminalpunct,filternonzeronumerals,sentencepiece,bpe} [{denoising,prefix,filterfeats,inferfeats,switchout,tokendrop,tokenmask,filtertoolong,filterwordratio,filterrepetitions,filterterminalpunct,filternonzeronumerals,sentencepiece,bpe} ...]]
                -src_vocab SRC_VOCAB [-tgt_vocab TGT_VOCAB]
                [-src_vocab_size SRC_VOCAB_SIZE]
                [-tgt_vocab_size TGT_VOCAB_SIZE] [-use_hf_tokenizer]
                [-add_language_tokens] [-decoder_start_with_eos]
                [--permute_sent_ratio PERMUTE_SENT_RATIO]
                [--rotate_ratio ROTATE_RATIO] [--insert_ratio INSERT_RATIO]
                [--random_ratio RANDOM_RATIO] [--mask_ratio MASK_RATIO]
                [--mask_length {subword,word,span-poisson}]
                [--poisson_lambda POISSON_LAMBDA] [--replace_length {-1,0,1}]
                [--denoising_objective {bart,mass}]
                [--reversible_tokenization {joiner,spacer}]
                [--prior_tokenization]
                [-switchout_temperature SWITCHOUT_TEMPERATURE]
                [-tokendrop_temperature TOKENDROP_TEMPERATURE]
                [-tokenmask_temperature TOKENMASK_TEMPERATURE]
                [--src_seq_length_max SRC_SEQ_LENGTH_MAX]
                [--tgt_seq_length_max TGT_SEQ_LENGTH_MAX]
                [--src_seq_length_min SRC_SEQ_LENGTH_MIN]
                [--tgt_seq_length_min TGT_SEQ_LENGTH_MIN]
                [--word_ratio_threshold WORD_RATIO_THRESHOLD]
                [--rep_threshold REP_THRESHOLD] [--rep_min_len REP_MIN_LEN]
                [--rep_max_len REP_MAX_LEN]
                [--punct_threshold PUNCT_THRESHOLD]
                [--nonzero_threshold NONZERO_THRESHOLD]
                [-src_subword_model SRC_SUBWORD_MODEL]
                [-tgt_subword_model TGT_SUBWORD_MODEL]
                [-src_subword_nbest SRC_SUBWORD_NBEST]
                [-tgt_subword_nbest TGT_SUBWORD_NBEST]
                [-src_subword_alpha SRC_SUBWORD_ALPHA]
                [-tgt_subword_alpha TGT_SUBWORD_ALPHA]
                [-src_subword_type {sentencepiece,bpe,none}]
                [-tgt_subword_type {sentencepiece,bpe,none}]
                [-src_subword_vocab SRC_SUBWORD_VOCAB]
                [-tgt_subword_vocab TGT_SUBWORD_VOCAB]
                [-src_vocab_threshold SRC_VOCAB_THRESHOLD]
                [-tgt_vocab_threshold TGT_VOCAB_THRESHOLD] [-share_vocab]
                [--enable_embeddingless] [--share_encoder_decoder_embeddings]
                [--model_type {text}] [--model_dtype {fp32,fp16,bf16}]
                [--enc_layers ENC_LAYERS [ENC_LAYERS ...]]
                [--dec_layers DEC_LAYERS [DEC_LAYERS ...]]
                [--model_dim MODEL_DIM] [--enc_model_dim ENC_MODEL_DIM]
                [--dec_model_dim DEC_MODEL_DIM]
                [--enc_sliding_window ENC_SLIDING_WINDOW]
                [--dec_sliding_window DEC_SLIDING_WINDOW]
                [--enc_global_attn_every_n_layers ENC_GLOBAL_ATTN_EVERY_N_LAYERS]
                [--dec_global_attn_every_n_layers DEC_GLOBAL_ATTN_EVERY_N_LAYERS]
                [--enc_global_rope_theta ENC_GLOBAL_ROPE_THETA]
                [--enc_local_rope_theta ENC_LOCAL_ROPE_THETA]
                [--dec_global_rope_theta DEC_GLOBAL_ROPE_THETA]
                [--dec_local_rope_theta DEC_LOCAL_ROPE_THETA]
                [--max_relative_positions MAX_RELATIVE_POSITIONS]
                [--heads HEADS] [--ff_mult FF_MULT]
                [--ff_activation {swiglu,gelu}] [--rotary_pos_emb]
                [--post_emb_norm] [--attn_dropout ATTN_DROPOUT]
                [--ff_dropout FF_DROPOUT] [--generator_function {softmax}]
                [--loss_scale LOSS_SCALE] [--apex_opt_level {O0,O1,O2,O3}]
                [--hidden_ab_size HIDDEN_AB_SIZE]
                [--ab_fixed_length AB_FIXED_LENGTH]
                [--ab_layers [{lin,simple,transformer,perceiver,feedforward} ...]]
                [--ab_layer_norm {none,rmsnorm,layernorm}]
                [--ab_heads AB_HEADS] [--dropout DROPOUT [DROPOUT ...]]
                [--attention_dropout ATTENTION_DROPOUT [ATTENTION_DROPOUT ...]]
                [--save_model SAVE_MODEL]
                [--save_checkpoint_steps SAVE_CHECKPOINT_STEPS]
                [--keep_checkpoint KEEP_CHECKPOINT]
                [--save_strategy {steps,best_only,best_and_last,best_n}]
                [--metric_for_best_model METRIC_FOR_BEST_MODEL]
                [--greater_is_better GREATER_IS_BETTER]
                [--train_steps TRAIN_STEPS] [--epochs EPOCHS]
                [--valid_steps VALID_STEPS] [--valid_start VALID_START]
                [--valid_timeout VALID_TIMEOUT]
                [--valid_decode_timeout VALID_DECODE_TIMEOUT]
                [--valid_max_length VALID_MAX_LENGTH] [--valid_at_start]
                [--valid_max_batches VALID_MAX_BATCHES]
                [--valid_metrics [VALID_METRICS ...]]
                [--early_stopping EARLY_STOPPING]
                [--early_stopping_criteria [EARLY_STOPPING_CRITERIA ...]]
                [--gpu_ranks [GPU_RANKS ...]] [--n_nodes N_NODES] --node_rank
                NODE_RANK [--world_size WORLD_SIZE]
                [--gpu_backend GPU_BACKEND]
                [--gpu_verbose_level GPU_VERBOSE_LEVEL]
                [--master_ip MASTER_IP] [--master_port MASTER_PORT]
                [--queue_size QUEUE_SIZE]
                [--prefetch_buffer_size PREFETCH_BUFFER_SIZE] --seed SEED
                [--param_init PARAM_INIT] [--param_init_glorot]
                [--train_from TRAIN_FROM] [--override_checkpoint_vocab]
                [--reset_optim {none,all,states,keep_states}]
                [--yes_i_messed_with_the_checkpoint] [--freeze_encoder]
                [--freeze_decoder] [--freeze_encoder_embeddings]
                [--freeze_decoder_embeddings] [--freeze_cross_attention]
                [--freeze_attention_bridge]
                [--freeze_decoder_except_cross_attention]
                [--batch_size BATCH_SIZE]
                [--valid_batch_size VALID_BATCH_SIZE]
                [--batch_size_multiple BATCH_SIZE_MULTIPLE]
                [--batch_type {sents,tokens}] [--pad_to_max_length]
                --max_length MAX_LENGTH
                [--task_distribution_strategy {weighted_sampling,roundrobin}]
                [-lookahead_minibatches LOOKAHEAD_MINIBATCHES]
                [-max_look_ahead_sentences MAX_LOOK_AHEAD_SENTENCES]
                [--optim {sgd,adagrad,adadelta,adam,adamw,adafactor}]
                [--adagrad_accumulator_init ADAGRAD_ACCUMULATOR_INIT]
                [--max_grad_norm MAX_GRAD_NORM] [--weight_decay WEIGHT_DECAY]
                [--dropout_steps DROPOUT_STEPS [DROPOUT_STEPS ...]]
                [--adam_beta1 ADAM_BETA1] [--adam_beta2 ADAM_BETA2]
                [--label_smoothing LABEL_SMOOTHING]
                [--average_decay AVERAGE_DECAY]
                [--average_every AVERAGE_EVERY]
                [--normalization {sents,tokens}]
                [--accum_count ACCUM_COUNT [ACCUM_COUNT ...]]
                [--accum_steps ACCUM_STEPS [ACCUM_STEPS ...]]
                [--learning_rate LEARNING_RATE]
                [--learning_rate_decay LEARNING_RATE_DECAY]
                [--start_decay_steps START_DECAY_STEPS]
                [--decay_steps DECAY_STEPS]
                [--decay_method {noam,noamwd,rsqrt,linear_warmup,none}]
                [--warmup_steps WARMUP_STEPS] [--log_file LOG_FILE]
                [--structured_log_file STRUCTURED_LOG_FILE]
                [--log_file_level {CRITICAL,ERROR,WARNING,INFO,DEBUG,NOTSET,50,40,30,20,10,0}]
                [--verbose] [--verbose_dataloader] [--log_model_structure]
                [--report_every REPORT_EVERY] [--exp_host EXP_HOST]
                [--exp EXP] [--tensorboard]
                [--tensorboard_log_dir TENSORBOARD_LOG_DIR]
                [--report_stats_from_parameters] [--report_training_accuracy]
                [--report_tflops] [--beam_size BEAM_SIZE] [--ratio RATIO]
                [--random_sampling_topk RANDOM_SAMPLING_TOPK]
                [--random_sampling_topp RANDOM_SAMPLING_TOPP]
                [--random_sampling_temp RANDOM_SAMPLING_TEMP]
                [--length_penalty {none,wu,avg}] [--alpha ALPHA]
                [--coverage_penalty {none,wu,summary}] [--beta BETA]
                [--stepwise_penalty] [--min_length MIN_LENGTH]
                [--max_sent_length] [--block_ngram_repeat BLOCK_NGRAM_REPEAT]
                [--ignore_when_blocking IGNORE_WHEN_BLOCKING [IGNORE_WHEN_BLOCKING ...]]
                [--replace_unk] [--ban_unk_token] [--kv_cache [KV_CACHE]]
                [--phrase_table PHRASE_TABLE]

Configuration

-config, --config

Path of the main YAML config file.

-save_config, --save_config

Path where to save the config.

Data/Tasks

-tasks, --tasks

List of datasets and their specifications. See examples/*.yaml for further details.

-transforms, --transforms

Possible choices: denoising, prefix, filterfeats, inferfeats, switchout, tokendrop, tokenmask, filtertoolong, filterwordratio, filterrepetitions, filterterminalpunct, filternonzeronumerals, sentencepiece, bpe

Default transform pipeline to apply to data. Can be specified in each corpus of data to override.

Default: []

Vocab

-src_vocab, --src_vocab

Path to src (or shared) vocabulary file. Format: one <word> or <word> <count> per line.

-tgt_vocab, --tgt_vocab

Path to tgt vocabulary file. Format: one <word> or <word> <count> per line.

-src_vocab_size, --src_vocab_size

Maximum size of the source vocabulary; will silently truncate your vocab file if longer.

-tgt_vocab_size, --tgt_vocab_size

Maximum size of the target vocabulary; will silently truncate your vocab file if longer.

-use_hf_tokenizer, --use_hf_tokenizer

Use HuggingFace tokenizers instead of traditional vocab files. Vocab paths should point to .json tokenizer files.

Default: False

-add_language_tokens, --add_language_tokens

Automatically add language prefix tokens (src_prefix/tgt_prefix from tasks) to HuggingFace tokenizers. Only applies when use_hf_tokenizer is True. Default: True

Default: True

-decoder_start_with_eos, --decoder_start_with_eos

BART-style decoder start: target sequences begin with </s> (EOS) followed by <s> (BOS). At inference, primes the decoder with </s> instead of <s>. Required for BART checkpoints.

Default: False

Transform/Denoising AE

--permute_sent_ratio, -permute_sent_ratio

Permute this proportion of sentences (boundaries defined by [‘.’, ‘?’, ‘!’]) in all inputs.

Default: 0.0

--rotate_ratio, -rotate_ratio

Rotate this proportion of inputs.

Default: 0.0

--insert_ratio, -insert_ratio

Insert this percentage of additional random tokens.

Default: 0.0

--random_ratio, -random_ratio

Instead of using <mask>, use random token this often. Incompatible with MASS

Default: 0.0

--mask_ratio, -mask_ratio

Fraction of words/subwords that will be masked.

Default: 0.0

--mask_length, -mask_length

Possible choices: subword, word, span-poisson

Length of masking window to apply.

Default: 'subword'

--poisson_lambda, -poisson_lambda

Lambda for Poisson distribution to sample span length if -mask_length set to span-poisson.

Default: 3.0

--replace_length, -replace_length

Possible choices: -1, 0, 1

When masking N tokens, replace with 0, 1, or N tokens. (use -1 for N)

Default: -1

--denoising_objective

Possible choices: bart, mass

choose between BART-style or MASS-style denoising objectives

Default: 'bart'

Transform/InferFeats

--reversible_tokenization, -reversible_tokenization

Possible choices: joiner, spacer

Type of reversible tokenization applied on the tokenizer.

Default: 'joiner'

--prior_tokenization, -prior_tokenization

Whether the input has already been tokenized.

Default: False

Transform/SwitchOut

-switchout_temperature, --switchout_temperature

Sampling temperature for SwitchOut. \(\tau^{-1}\) in [WPDN18]. Smaller value makes data more diverse.

Default: 1.0

Transform/Token_Drop

-tokendrop_temperature, --tokendrop_temperature

Sampling temperature for token deletion.

Default: 1.0

Transform/Token_Mask

-tokenmask_temperature, --tokenmask_temperature

Sampling temperature for token masking.

Default: 1.0

Transform/Length filter

--src_seq_length_max, -src_seq_length_max

Maximum source sequence length.

Default: 200

--tgt_seq_length_max, -tgt_seq_length_max

Maximum target sequence length.

Default: 200

--src_seq_length_min, -src_seq_length_min

Minimum source sequence length.

Default: 1

--tgt_seq_length_min, -tgt_seq_length_min

Minimum target sequence length.

Default: 1

Transform/Word ratio filter

--word_ratio_threshold, -word_ratio_threshold

Threshold for discarding sentences based on word ratio.

Default: 3

Transform/Repetitions filter

--rep_threshold, -rep_threshold

Number of times the substring is repeated.

Default: 2

--rep_min_len, -rep_min_len

Minimum length of the repeated pattern.

Default: 3

--rep_max_len, -rep_max_len

Maximum length of the repeated pattern.

Default: 100

Transform/Terminal punctuation filter

--punct_threshold, -punct_threshold

Minimum penalty score for discarding sentences based on their terminal punctuation signs

Default: -2

Transform/Non-zero numerals filter

--nonzero_threshold, -nonzero_threshold

Threshold for discarding sentences based on numerals between the segments with zeros removed

Default: 0.5

Transform/Subword/Common

Attention

Common options shared by all subword transforms. Including options for indicate subword model path, Subword Regularization/BPE-Dropout, and Vocabulary Restriction.

-src_subword_model, --src_subword_model

Path of subword model for src (or shared).

-tgt_subword_model, --tgt_subword_model

Path of subword model for tgt.

-src_subword_nbest, --src_subword_nbest

Number of candidates in subword regularization. Valid for unigram sampling, invalid for BPE-dropout. (source side)

Default: 1

-tgt_subword_nbest, --tgt_subword_nbest

Number of candidates in subword regularization. Valid for unigram sampling, invalid for BPE-dropout. (target side)

Default: 1

-src_subword_alpha, --src_subword_alpha

Smoothing parameter for sentencepiece unigram sampling, and dropout probability for BPE-dropout. (source side)

Default: 0

-tgt_subword_alpha, --tgt_subword_alpha

Smoothing parameter for sentencepiece unigram sampling, and dropout probability for BPE-dropout. (target side)

Default: 0

-src_subword_type, --src_subword_type

Possible choices: sentencepiece, bpe, none

Type of subword tokenization for source side. Used by denoising transforms to determine word boundaries.

Default: 'sentencepiece'

-tgt_subword_type, --tgt_subword_type

Possible choices: sentencepiece, bpe, none

Type of subword tokenization for target side. Used by denoising transforms to determine word boundaries.

Default: 'sentencepiece'

-src_subword_vocab, --src_subword_vocab

Path to the vocabulary file for src subword. Format: <word> <count> per line.

Default: ''

-tgt_subword_vocab, --tgt_subword_vocab

Path to the vocabulary file for tgt subword. Format: <word> <count> per line.

Default: ''

-src_vocab_threshold, --src_vocab_threshold

Only produce src subword in src_subword_vocab with frequency >= src_vocab_threshold.

Default: 0

-tgt_vocab_threshold, --tgt_vocab_threshold

Only produce tgt subword in tgt_subword_vocab with frequency >= tgt_vocab_threshold.

Default: 0

-share_vocab, --share_vocab

use the same model for both sides

Default: False

Model- Embeddings

--enable_embeddingless, -enable_embeddingless

Enable the use of byte-based embeddingless models(Shaham et. al, 2021) https://aclanthology.org/2021.naacl-main.17/

Default: False

--share_encoder_decoder_embeddings, -share_encoder_decoder_embeddings

Share token embeddings between encoder and decoder. Only works when encoder and decoder use the same vocabulary and have matching model dimensions.

Default: False

Model- Encoder-Decoder

--model_type, -model_type

Possible choices: text

Type of source model to use. Allows the system to incorporate non-text inputs. Options are [text].

Default: 'text'

--model_dtype, -model_dtype

Possible choices: fp32, fp16, bf16

Data type of the model. For LUMI (MI250X based), bf16 is recommended. For Puhti (V100 based), fp32 is recommended.

Default: 'fp32'

--enc_layers, -enc_layers

Number of layers in each encoder module

--dec_layers, -dec_layers

Number of layers in each decoder module

--model_dim, -model_dim

Size of Transformer representations (shared for encoder and decoder). Mutually exclusive with enc_model_dim / dec_model_dim.

Default: -1

--enc_model_dim, -enc_model_dim

Encoder hidden dimension. Must be set together with dec_model_dim. Mutually exclusive with model_dim.

--dec_model_dim, -dec_model_dim

Decoder hidden dimension. Must be set together with enc_model_dim. Mutually exclusive with model_dim.

Model- Attention

--enc_sliding_window, -enc_sliding_window

Sliding window size for encoder attention (total context, symmetric). -1 = disabled (full attention). Requires Flash Attention 2. Reduces memory for long sequences. Example: 128 = attend to ±64 tokens.

Default: -1

--dec_sliding_window, -dec_sliding_window

Sliding window size for decoder attention (total context, symmetric). -1 = disabled (full attention). Requires Flash Attention 2. Causal masking is automatically applied. Example: 128 = attend to ±64 tokens.

Default: -1

--enc_global_attn_every_n_layers, -enc_global_attn_every_n_layers

Encoder: every Nth layer uses global attention (full context), others use sliding window. Interleaved pattern from ModernBERT. Layer 0, N, 2N, … get global attention. Only applies if enc_sliding_window > 0. Default: 3

Default: 3

--dec_global_attn_every_n_layers, -dec_global_attn_every_n_layers

Decoder: every Nth layer uses global attention (full context), others use sliding window. Interleaved pattern from ModernBERT. Layer 0, N, 2N, … get global attention. Only applies if dec_sliding_window > 0. Default: 3

Default: 3

--enc_global_rope_theta, -enc_global_rope_theta

Encoder: RoPE theta for global attention layers (every Nth layer). Higher values allow for longer context modeling. Default: 160000.0 (ModernBERT style). Only applies if enc_sliding_window > 0 and rotary_pos_emb is enabled.

Default: 160000.0

--enc_local_rope_theta, -enc_local_rope_theta

Encoder: RoPE theta for local sliding window attention layers. Standard value for most transformer models. Default: 10000.0. Only applies if enc_sliding_window > 0 and rotary_pos_emb is enabled.

Default: 10000.0

--dec_global_rope_theta, -dec_global_rope_theta

Decoder: RoPE theta for global attention layers (every Nth layer). Higher values allow for longer context modeling. Default: 160000.0 (ModernBERT style). Only applies if dec_sliding_window > 0 and rotary_pos_emb is enabled.

Default: 160000.0

--dec_local_rope_theta, -dec_local_rope_theta

Decoder: RoPE theta for local sliding window attention layers. Standard value for most transformer models. Default: 10000.0. Only applies if dec_sliding_window > 0 and rotary_pos_emb is enabled.

Default: 10000.0

--max_relative_positions, -max_relative_positions

Maximum distance between inputs in relative positions representations. For more detailed information, see: https://arxiv.org/pdf/1803.02155.pdf

Default: 0

--heads, -heads

Number of attention heads. model_dim must be divisible by heads.

Default: 8

--ff_mult, -ff_mult

FFN inner dimension multiplier: ff_inner = model_dim * ff_mult. If unset, the default is chosen from ff_activation: 2.67 (≈ 8/3, matches parameter count of a 4× GELU FFN) for swiglu, 4.0 for gelu.

--ff_activation, -ff_activation

Possible choices: swiglu, gelu

Activation function for feed-forward layers in native transformer blocks. Choices: swiglu (default), gelu. SWiGLU uses 3 matrices (gate, up, down), GELU uses 2 matrices (up, down).

Default: 'swiglu'

--rotary_pos_emb, -rotary_pos_emb

Use rotary positional embeddings (RoPE) instead of absolute positional embeddings.

Default: False

--post_emb_norm, -post_emb_norm

Apply RMSNorm after the token embedding.

Default: False

--attn_dropout, -attn_dropout

Dropout probability inside attention (applied to the attention weights).

Default: 0.0

--ff_dropout, -ff_dropout

Dropout probability inside the feed-forward sublayer.

Default: 0.0

Generator

--generator_function, -generator_function

Possible choices: softmax

Which function to use for generating probabilities over the target vocabulary (choices: softmax)

Default: 'softmax'

--loss_scale, -loss_scale

For FP16 training, the static loss scale to use. If not set, the loss scale is dynamically computed.

Default: 0.0

--apex_opt_level, -apex_opt_level

Possible choices: O0, O1, O2, O3

For FP16 training, the opt_level to use. See https://nvidia.github.io/apex/amp.html#opts-levels.

Default: 'O1'

Attention bridge

--hidden_ab_size, -hidden_ab_size

Size of attention bridge hidden states

Default: 2048

--ab_fixed_length, -ab_fixed_length

Number of attention heads in attention bridge (fixed length of output)

Default: 50

--ab_layers, -ab_layers

Possible choices: lin, simple, transformer, perceiver, feedforward

Composition of the attention bridge

Default: []

--ab_layer_norm, -ab_layer_norm

Possible choices: none, rmsnorm, layernorm

Use layer normalization after lin, simple and feedforward bridge layers

Default: 'layernorm'

--ab_heads, -ab_heads

Number of heads for transformer self-attention. Semi-obsolete: not used for x-transformers, only used for some attention bridge configuations.

Default: 8

--dropout, -dropout

Dropout probability; Legacy: applied in the attention bridge

Default: [0.3]

--attention_dropout, -attention_dropout

Attention Dropout probability; Legacy: applied in the attention bridge

Default: [0.1]

General

--save_model, -save_model

Model filename (the model will be saved as <save_model>_N.pt where N is the number of steps

Default: 'model'

--save_checkpoint_steps, -save_checkpoint_steps

Save a checkpoint every X steps. Required when save_strategy=’steps’. Ignored for metric-based strategies (best_*), which save at validation time (–valid_steps).

Default: 0

--keep_checkpoint, -keep_checkpoint

Keep X checkpoints (negative: keep all). Works with –save_strategy to determine rotation behavior: FIFO for ‘steps’, metric-based for ‘best_only’/’best_and_last’/’best_n’.

Default: -1

--save_strategy, -save_strategy

Possible choices: steps, best_only, best_and_last, best_n

Checkpoint saving strategy:

‘steps’: Save every save_checkpoint_steps (FIFO rotation) ‘best_only’: Keep only the best checkpoint by metric (saves at validation time) ‘best_and_last’: Keep best + keep_checkpoint most recent (saves at validation time) ‘best_n’: Keep top keep_checkpoint checkpoints by metric (saves at validation time)

Note: Metric-based strategies (best_*) save at validation frequency (–valid_steps), not –save_checkpoint_steps. Use –keep_checkpoint to control how many to keep.

Default: 'steps'

--metric_for_best_model, -metric_for_best_model

Metric to use for determining best checkpoint. Options: ‘ppl’ (perplexity), ‘accuracy’, ‘bleu’, ‘chrf’, or custom metric name

Default: 'bleu'

--greater_is_better, -greater_is_better

Whether higher metric values are better. If None, auto-inferred: False for ppl, True for accuracy/bleu

--train_steps, -train_steps

Number of training steps

Default: 100000

--epochs, -epochs

Deprecated epochs see train_steps

Default: 0

--valid_steps, -valid_steps

Perfom validation every X steps

Default: 10000

--valid_start, -valid_start

Start regular validation at X steps

Default: 0

--valid_timeout, -valid_timeout

Timeout for validation runs in seconds

--valid_decode_timeout, -valid_decode_timeout

Timeout for decoding onebatch during validation runs (in seconds)

--valid_max_length, -valid_max_length

Maximum sequence length during validation.

--valid_at_start, -valid_at_start

Perform validation before training starts

Default: False

--valid_max_batches, -valid_max_batches

Maximum number of batches used for validation.

--valid_metrics, -valid_metrics

List of names of additional validation metrics

Default: []

--early_stopping, -early_stopping

Number of validation steps without improving.

Default: 0

--early_stopping_criteria, -early_stopping_criteria

Criteria to use for early stopping.

Computation Environment

--gpu_ranks, -gpu_ranks

list of ranks of each process.

Default: []

--n_nodes, -n_nodes

total number of training nodes.

Default: 1

--node_rank, -node_rank

index of current node (0-based). When using non-distributed training (CPU, single-GPU), set to 0

--world_size, -world_size

total number of distributed processes.

Default: 1

--gpu_backend, -gpu_backend

Type of torch distributed backend

Default: 'nccl'

--gpu_verbose_level, -gpu_verbose_level

Gives more info on each process per GPU.

Default: 0

--master_ip, -master_ip

IP of master for torch.distributed training.

Default: 'localhost'

--master_port, -master_port

Port of master for torch.distributed training.

Default: 10000

--queue_size, -queue_size

Size of queue for each process in producer/consumer

Default: 80

--prefetch_buffer_size, -prefetch_buffer_size

Size of prefetch buffer for background thread batch prefetching. Larger values hide longer disk I/O latency but use more memory. Recommended: 16-32 for datasets with occasional slow batches.

Default: 16

Reproducibility

--seed, -seed

Set random seed used for better reproducibility between experiments. Mandatory for multi-gpu training, and for convenience required for all.

Initialization

--param_init, -param_init

Legacy opt for attention bridge. Parameters are initialized over uniform distribution with support (-param_init, param_init). Use 0 to not use initialization

Default: 0.1

--param_init_glorot, -param_init_glorot

Legacy opt for attention bridge. Init parameters with xavier_uniform.

Default: False

--train_from, -train_from

If training from a checkpoint then this is the path to the pretrained model’s state_dict.

Default: ''

--override_checkpoint_vocab, -override_checkpoint_vocab

When training from checkpoint, override the checkpoint’s vocabulary with the one specified in config. This allows using different tokenizers than the ones baked into the checkpoint. Default: False (use checkpoint vocabulary).

Default: False

--reset_optim, -reset_optim

Possible choices: none, all, states, keep_states

Controls which parts of optimizer to load from checkpoint when using –train_from. This is a 2x2 matrix of [Optimizer State] × [Training Step]: ‘none’: KEEP state + KEEP step → True resumption after interruption. Loads optimizer state (momentum, variance buffers) and continues from checkpoint’s training step. CRITICAL: Your config’s ‘optim’ must match the optimizer used to create the checkpoint. ‘keep_states’: KEEP state + RESET step to 1 → Restart LR schedule with same optimizer. Useful when you want to change learning rate schedule but preserve optimizer momentum. ‘states’: RESET state + KEEP step → Switch optimizer mid-training. Loads checkpoint’s training step but builds fresh optimizer state. Use when switching optimizer types (e.g., Adam → Adafactor). ‘all’: RESET state + RESET step to 1 → Fine-tuning from scratch. Only loads model weights, starts fresh training with new optimizer and step=1. IMPORTANT: Checkpoints do NOT save optimizer configuration (optim, learning_rate, etc). The optimizer type is ALWAYS from your config file, regardless of reset_optim value. Checkpoints only save: (1) model architecture, (2) model weights, (3) optimizer state buffers.

Default: 'none'

--yes_i_messed_with_the_checkpoint, -yes_i_messed_with_the_checkpoint

Only set this if you know what you are doing.

Default: False

Fine-grained Parameter Freezing

--freeze_encoder, -freeze_encoder

Freeze all encoder parameters (attention and feedforward layers). Does not affect embeddings unless –freeze_encoder_embeddings is also set. Adapters remain trainable if configured.

Default: False

--freeze_decoder, -freeze_decoder

Freeze all decoder parameters (self-attention, cross-attention, feedforward, sandwich-norm post-branch norms where present, and the vocabulary output projection). Does not affect embeddings unless –freeze_decoder_embeddings is also set. Adapters remain trainable if configured. Can be combined with –freeze_cross_attention=false to keep only cross-attention trainable.

Default: False

--freeze_encoder_embeddings, -freeze_encoder_embeddings

Freeze encoder token embeddings and the shared post-embedding norm. Can be used independently or with –freeze_encoder.

Default: False

--freeze_decoder_embeddings, -freeze_decoder_embeddings

Freeze decoder token embeddings and the shared post-embedding norm. Can be used independently or with –freeze_decoder.

Default: False

--freeze_cross_attention, -freeze_cross_attention

Freeze only cross-attention layers in the decoder. Useful for adapting decoder to new encoder representations while preserving the encoder-decoder alignment learned during pretraining. Mutually exclusive with –freeze_decoder (decoder freeze takes precedence).

Default: False

--freeze_attention_bridge, -freeze_attention_bridge

Freeze the attention bridge between encoder and decoder if present. Only applies when attention bridge is configured (ab_layers is not empty) and encoder/decoder dimensions match. Has no effect if attention bridge does not exist.

Default: False

--freeze_decoder_except_cross_attention, -freeze_decoder_except_cross_attention

Freeze decoder self-attention, feedforward, sandwich-norm post-branch norms (where present), and the vocabulary output projection, but keep cross-attention trainable. Useful for adapting how decoder attends to encoder representations while keeping decoder’s internal processing frozen. Adapters remain trainable if configured. Mutually exclusive with –freeze_decoder and –freeze_cross_attention.

Default: False

Batching

--batch_size, -batch_size

Maximum batch size for training

Default: 64

--valid_batch_size, -valid_batch_size

Maximum batch size for validation

Default: 32

--batch_size_multiple, -batch_size_multiple

Batch size multiple for token batches.

--batch_type, -batch_type

Possible choices: sents, tokens

Batch grouping for batch_size. Standard is sents. Tokens will do dynamic batching

Default: 'sents'

--pad_to_max_length, -pad_to_max_length

Pad all minibatches to max_length instead of to the length of the longest sequence in the minibatch. Using this together with batch_type=sents results in tensors of a fixed shape.

Default: False

--max_length, -max_length

Maximum sequence length. This parameter serves three purposes: (1) sets the positional embedding size when RoPE is not used; (2) controls the maximum number of tokens generated during inference.(3) sets the padded batch length when –pad_to_max_length is enabled;

Default: 256

--task_distribution_strategy, -task_distribution_strategy

Possible choices: weighted_sampling, roundrobin

Strategy for the order in which tasks (e.g. language pairs) are scheduled for training

Default: 'weighted_sampling'

-lookahead_minibatches, --lookahead_minibatches

The number of minibatches that SimpleLookAheadBucketing will read into a maxibatch, pessimisticly sort by length, split into minibatches, and yield in one go. Recommended value: same as accum_count, or at least a multiple of it.

Default: 4

-max_look_ahead_sentences, --max_look_ahead_sentences

(Maximum) number of sentence pairs that SimpleLookAheadBucketing can attempt to add to the maxibatch. This is mainly a failsafe in case some corpus contains very short examples.

Default: 2048

--optim, -optim

Possible choices: sgd, adagrad, adadelta, adam, adamw, adafactor

Optimization method.

Default: 'sgd'

--adagrad_accumulator_init, -adagrad_accumulator_init

Initializes the accumulator values in adagrad. Mirrors the initial_accumulator_value option in the tensorflow adagrad (use 0.1 for their default).

Default: 0.0

--max_grad_norm, -max_grad_norm

If the norm of the gradient vector exceeds this, renormalize it to have the norm equal to max_grad_norm

Default: 1.0

--weight_decay, -weight_decay

L2 penalty (weight decay) regularizer

Default: 0.0

--dropout_steps, -dropout_steps

Steps at which dropout changes.

Default: [0]

--adam_beta1, -adam_beta1

The beta1 parameter used by Adam. Almost without exception a value of 0.9 is used in the literature, seemingly giving good results, so we would discourage changing this value from the default without due consideration.

Default: 0.9

--adam_beta2, -adam_beta2

The beta2 parameter used by Adam. Typically a value of 0.999 is recommended, as this is the value suggested by the original paper describing Adam, and is also the value adopted in other frameworks such as Tensorflow and Keras, i.e. see: https://www.tensorflow.org/api_docs/python/tf/train/AdamOptimizer or https://keras.io/optimizers/ . Whereas recently the paper “Attention is All You Need” suggested a value of 0.98 for beta2, this parameter may not work well for normal models / default baselines.

Default: 0.999

--label_smoothing, -label_smoothing

Label smoothing value epsilon. Probabilities of all non-true labels will be smoothed by epsilon / (vocab_size - 1). Set to zero to turn off label smoothing. For more detailed information, see: https://arxiv.org/abs/1512.00567

Default: 0.0

--average_decay, -average_decay

Moving average decay. Set to other than 0 (e.g. 1e-4) to activate. Similar to Marian NMT implementation: http://www.aclweb.org/anthology/P18-4020 For more detail on Exponential Moving Average: https://en.wikipedia.org/wiki/Moving_average

Default: 0.0

--average_every, -average_every

Step for moving average. Default is every update, if -average_decay is set.

Default: 1

--normalization, -normalization

Possible choices: sents, tokens

Normalization method of the gradient.

Default: 'sents'

--accum_count, -accum_count

Accumulate gradient this many times. Approximately equivalent to updating batch_size * accum_count batches at once. Recommended for Transformer.

Default: [1]

--accum_steps, -accum_steps

Steps at which accum_count values change

Default: [0]

--learning_rate, -learning_rate

Starting learning rate.

Default: 1.0

--learning_rate_decay, -learning_rate_decay

If update_learning_rate, decay learning rate by this much if steps have gone past start_decay_steps

Default: 0.5

--start_decay_steps, -start_decay_steps

Start decaying every decay_steps after start_decay_steps

Default: 50000

--decay_steps, -decay_steps

Decay every decay_steps

Default: 10000

--decay_method, -decay_method

Possible choices: noam, noamwd, rsqrt, linear_warmup, none

Use a custom decay rate.

Default: 'none'

--warmup_steps, -warmup_steps

Number of warmup steps for custom decay.

Default: 4000

Logging

--log_file, -log_file

Output logs to a file under this path.

Default: ''

--structured_log_file, -structured_log_file

Output machine-readable structured logs to a file under this path.

Default: ''

--log_file_level, -log_file_level

Possible choices: CRITICAL, ERROR, WARNING, INFO, DEBUG, NOTSET, 50, 40, 30, 20, 10, 0

Default: '0'

--verbose, -verbose

Print data loading and statistics for all process (default only log the first process shard)

Default: False

--verbose_dataloader, -verbose_dataloader

Show DataLoader line checking info from all devices (default: only master device logs). Useful for debugging dataset continuation across distributed training.

Default: False

--log_model_structure, -log_model_structure

Print the entire model structure when building the model. Verbose, but useful for debugging.

Default: False

--report_every, -report_every

Print stats at this interval.

Default: 50

--exp_host, -exp_host

Send logs to this crayon server.

Default: ''

--exp, -exp

Name of the experiment for logging.

Default: ''

--tensorboard, -tensorboard

Use tensorboard for visualization during training. Must have the library tensorboard >= 1.14.

Default: False

--tensorboard_log_dir, -tensorboard_log_dir

Log directory for Tensorboard. This is also the name of the run.

Default: 'runs/mammoth'

--report_stats_from_parameters, -report_stats_from_parameters

Report parameter-level statistics in tensorboard. This has a huge impact on performance: only use for debugging.

Default: False

--report_training_accuracy, -report_trainig_accuracy

Report accuracy for training batches. This has an impact on performance: only use for debugging, or if no validation set exists.

Default: False

--report_tflops, -report_tflops

Report TFLOPs/GPU throughput during training. Uses analytical FLOP counting (Megatron-LM style). Enabled by default.

Default: True

Beam Search

--beam_size, -beam_size

Beam size

Default: 1

--ratio, -ratio

Ratio based beam stop condition

Default: -0.0

Random Sampling

--random_sampling_topk, -random_sampling_topk

Set this to -1 to do random sampling from full distribution. Set this to value k>1 to do random sampling restricted to the k most likely next tokens. Set this to 1 to use argmax.

Default: 0

--random_sampling_topp, -random_sampling_topp

Probability for top-p/nucleus sampling. Restrict tokens to the most likely until the cumulated probability is over p. In range [0, 1]. https://arxiv.org/abs/1904.09751

Default: 0.0

--random_sampling_temp, -random_sampling_temp

If doing random sampling, divide the logits by this before computing softmax during decoding.

Default: 1.0

--beam_size, -beam_size

Beam size

Default: 1

Penalties

Note

Coverage Penalty is not available in sampling.

--length_penalty, -length_penalty

Possible choices: none, wu, avg

Length Penalty to use.

Default: 'none'

--alpha, -alpha

Google NMT length penalty parameter (higher = longer generation)

Default: 0.0

--coverage_penalty, -coverage_penalty

Possible choices: none, wu, summary

Coverage Penalty to use. Only available in beam search.

Default: 'none'

--beta, -beta

Coverage penalty parameter

Default: -0.0

--stepwise_penalty, -stepwise_penalty

Apply coverage penalty at every decoding step. Helpful for summary penalty.

Default: False

Decoding tricks

Tip

Following options can be used to limit the decoding length or content.

--min_length, -min_length

Minimum prediction length

Default: 0

--max_sent_length, -max_sent_length

Deprecated, use -max_length instead

--block_ngram_repeat, -block_ngram_repeat

Block repetition of ngrams during decoding.

Default: 0

--ignore_when_blocking, -ignore_when_blocking

Ignore these strings when blocking repeats. You want to block sentence delimiters.

Default: []

--replace_unk, -replace_unk

Replace the generated UNK tokens with the source token that had highest attention weight. If phrase_table is provided, it will look up the identified source token and give the corresponding target token. If it is not provided (or the identified source token does not exist in the table), then it will copy the source token.

Default: False

--ban_unk_token, -ban_unk_token

Prevent unk token generation by setting unk proba to 0

Default: False

--kv_cache, -kv_cache

Use the decoder KV cache (true/false, default true). With false, the full target prefix is re-encoded at every step: much slower, only useful for debugging or for checking that cached decoding gives the same output.

Default: True

--phrase_table, -phrase_table

If phrase_table is provided (with replace_unk), it will look up the identified source token and give the corresponding target token. If it is not provided (or the identified source token does not exist in the table), then it will copy the source token.

Default: ''