Data Loaders

Dataset

class mammoth.inputters.dataset.ParallelCorpus(src_file, tgt_file, src_vocab, tgt_vocab, transforms, device='cpu', stride=None, offset=None, is_train=False, task=None, max_length=None, line_idx_restore=None, model_max_seq_len=None, verbose_dataloader=False, device_rank=0)

Bases: IterableDataset

Torch-style dataset

Data loading

class mammoth.inputters.dataloader.DynamicDatasetIter(task_queue_manager, opts, corpora_info, transforms_cls, vocabs_dict, is_train, batch_type, batch_size, batch_size_multiple, max_look_ahead_sentences=20480, lookahead_minibatches=4, line_idx_restore=None)

Bases: object

Yield batch from (multiple) plain text corpus.

Parameters:

• corpora (dict[str, ParallelCorpus]) – collections of corpora to iterate;

• corpora_info (dict[str, dict]) – corpora infos correspond to corpora;

• transforms (dict[str, Transform]) – transforms may be used by corpora;

• fields (dict[str, Field]) – fields dict for convert corpora into Tensor;

• is_train (bool) – True when generate data for training;

• batch_type (str) – batching type to count on, choices=[tokens, sents];

• batch_size (int) – numbers of examples in a batch;

• batch_size_multiple (int) – make batch size multiply of this;

• max_look_ahead_sentences (int) – accum this number of examples in a dynamic dataset;

• skip_empty_level (str) – security level when encouter empty line;

• stride (int) – iterate data files with this stride;

• offset (int) – iterate data files with this offset.

Variables:

• dataset_adapter (DatasetAdapter) – organize raw corpus to tensor adapt;

• mixer (MixingStrategy) – the strategy to iterate corpora.

classmethod from_opts(task_queue_manager, transforms_cls, vocabs_dict, opts, is_train, line_idx_restore)

Initilize DynamicDatasetIter with options parsed from opts.

class mammoth.inputters.dataloader.SimpleLookAheadBucketing(dataset, max_look_ahead_sentences, lookahead_minibatches, batch_size, score_fn=None)

Bases: object

Parameters:

• dataset – mammoth.inputters.ParallelCorpus

• max_look_ahead_sentences – The maximum number of sentence pairs to read before yielding minibatches. Limits the time spent looping if there is a corpus with unexpectedly short sentences.

• lookahead_minibatches – The number of minibatches that will be yielded once bucketing is complete. Recommended value: same as accum_count, or at least a multiple of it. Setting lookahead_minibatches == accum_count means that each accumulated batch uses up the whole buffer. All tasks stay in sync concerning the length sorting: each task begins with the smallest minibatch and ends with the largest just before accumulation ends.

• batch_size – The maximum size of each minibatch in tokens. Note that the maximum batch size can not be guaranteed if the data contains examples that exceed the limit on their own. Use filtertoolong to avoid such examples.

• score_fn – Compute the size estimate (single integer) for sorting examples.

class mammoth.inputters.dataloader.InferenceBatcher(dataset, batch_size)

Bases: object

Iterator for inference

Vocab

class mammoth.inputters.vocab.Vocab(path, items=None, tag='', size=None, specials=[])

Bases: object

classmethod merge(*vocabs, size=None)

Merge vocabs.