Vaclav Kosar's

Software & Machine Learning Blog

MassiveText Dataset introduced for pre-training of DeepMind's Gopher

Private diverse 10-lingual textual dataset composed of web, Github, news, Wikipedia, Books, C4.

MassiveText dataset composition table

Introduced in DeepMind’s Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Disk size is 10.5 TB.
Token count is around 5T tokens.
Document count is 2.32B with average 2k tokens per document.
Dataset is private - not open-source licenced as of 2022-03-20.
Similar but public dataset is The Pile (diverse 800GB text dataset)

MassiveText Contents

Dataset language composition: 99% English, then 10 other languages.
Google SafeSearch filter
Dataset is composed of subsets: web, Github, news, Wikipedia, Books, C4 (web-text)

MassiveText non-english composition

Papers Using MassiveText

Since it is private only DeepMind uses the dataset
DeepMind’s RETRO Transformer used multi-lingual version of the dataset
DeepMind’s Gopher used only english version of the dataset

Don’t Confuse MassiveText with Amazon Massive Dataset

Amazon Massive is not MassiveText
Amazon Massive is:
- released by Amazon on 2022-04-20
- multilingual natural-language understanding
- 51-language dataset

Created on 20 Mar 2022. Updated on: 14 May 2022.

Let's connect

You'll love also...

How Computers Understood Humans

Catch on with this 7-slide introduction to deep natural language processing of 2022, featuring TF-IDF, Word2vec, knowledge graphs, and transformers.

DeepMind's RETRO Retrieval-Enhanced Transformer

Retrieval-Enhanced Language Model cross-attends trillions of tokens for SoTA on Wikitext103 and The Pile with 25x fewer parameters.

StarSpace - Embeddings For Documents, Users, and Words

Create vectors of various entities in a single space with this general-purpose embedding model from Facebook AI.

Google's Pathways Language Model and Chain-of-Thought Prompting

PaLM, the largest model as of early 2022, outperforms average human on grade school logic and math (BIG-bench) by simulating reasoning steps.

FastText Word Embeddings

How FastText works, word embeddings, n-grams, OOV words, and visualize embedding norms.

Wav2vec: Semi-supervised and Unsupervised Speech Recognition

Word2vec for audio quantizes phonemes, transforms, GAN trains on text and audio from Facebook AI.

About Vaclav Kosar How many days left in this quarter? Twitter Bullet Points to Copy & Paste Privacy Policy

Copyright © Vaclav Kosar. All rights reserved. Not investment, financial, medical, or any other advice. No guarantee of information accuracy.