Natural Language

The speech + text data comes from professional audiobooks produced by Usborne Publishing. about 6.5 hours of British English speech data in from a single female talke

The Blog Authorship Corpus consists of the collected posts of 19,320 bloggers gathered from blogger.com in August 2004. The corpus incorporates a total of 681,288 posts and over 140 million words - or approximately 35 posts and 7250 words per person.

Structured data from Wikipedia

With millions of images in our library and billions of user-submitted keywords, we work hard at Shutterstock to make sure that bad words don't show up in places they shouldn't. This repo contains a list of words that we use to filter results from our autocomplete server and recommendation engine.

Collection of political speeches from the German Presidency, Presidency of the Bundestag, Chancellery, and Ministry of Foreign Affairs

Generated based on the public available Gigaword dataset using Paragraph Vectors

Speech dataset consisting of 13,100 short audio clips of a single speaker reading passages from 7 non-fiction books. A transcription is provided for each clip. Clips vary in length from 1 to 10 seconds and have a total length of approximately 24 hours.

Concept Extraction (from Archive.org)

Named Entity rEcognition and Linking

Dataset based on Meta's metadata for mined bitext. Contains word-aligned text in 148 English-centric and 1465 non-English-centric language pairs.

Clean and noisy parallel speech database. The database was designed to train and test speech enhancement methods that operate at 48kHz. A more detailed description can be found in the papers associated with the database.

The Personae corpus was collected for experiments in Authorship Attribution and Personality Prediction. It consists of 145 Dutch-language essays, written by 145 different students (BA in Linguistics and Literature at the University of Antwerp, Belgium). Each student also took an online MBTI…

1,551 Russian given names (809 male, 742 female) with origin, short meaning, editorial etymology notes, diminutive and international forms, and popularity ranks among newborns based on open data from Moscow civil registry offices (ZAGS). Useful as a gazetteer for NER, transliteration and onomastics…

The SMS Spam Collection is a public set of SMS labeled messages that have been collected for mobile phone spam research.

Dated historical event records from 3000 BCE to the present. Every record is a single sentence quoted verbatim from a specific English Wikipedia revision and carries that revision id, the article title and the section anchor, so any claim built on a row can be checked against its source. Rows are…

News/Blogs in multiple languages

Wikipedia databases

40 Million Entities in Context

Open, de-identified database of vocabulary development from 84,138 children and 93,955 CDI administrations, across 38 languages and 78 instruments. The data managers of this project request that all data be accessed through the R package wordbankr - https://github.com/langcog/wordbankr

a corpus of manually-constructed explanation graphs, explanatory role ratings, and associated semistructured tablestore for most publicly available elementary science exam questions in the US

The thirty nayin (纳音) of the Chinese sexagenary calendar with English translations, as JSON and CSV. Each of the 30 entries gives the Chinese term, tone-marked pinyin, an English rendering, the Five-Element class, and the two stem-branch (ganzhi) pairs it covers, so all 60 calendar pairs are keyed…

Bilingual (English + Japanese) judgment-eliciting Q&A corpus (851 examples) encoding the documented decisions of a research program on AI agent design, with per-example metadata for line, source, language, and shape. Directly downloadable as JSONL; mirrored on Hugging Face Datasets and DOI-archived…

Word origins and etymological data covering millions of entries across multiple languages with historical roots, language family mappings, and borrowing chains. One of the most downloaded linguistics datasets on HuggingFace with 4.38 million downloads.

Structured dataset of world languages covering ISO 639 codes, speaker population estimates, language families, geographic distribution by country, and writing systems. Useful as a reference for NLP preprocessing and multilingual research.