Computer Networks

A source-linked dataset of ten documented crawler and control tokens from OpenAI, Anthropic, Perplexity, and Google, classifying search discovery, model training, and user-triggered retrieval behavior with primary documentation URLs and review dates.

A robots.txt census of the Tranco top 1,000,000 domains measuring what share of the web blocks major AI crawlers (GPTBot, CCBot, ClaudeBot, Bytespider, Amazonbot, Google-Extended, and more). Finds that 9.33% of all domains, and 14.8% of the 63% that serve a robots.txt, fully block at least one…

A monthly measurement of which machine-readable discovery and consent files public websites actually serve -- robots.txt, sitemap.xml, llms.txt, llms-full.txt, ai.txt, rsl.xml, /.well-known/tdmrep.json, /.well-known/gpc.json, /.well-known/mcp.json, /.well-known/api-catalog and further routes -- and…

1B web pages

733M web pages

A reachability census of the top 10 million domains (DomCop top-10M ranking), probed with both a polite-bot client and a real-browser (Chrome TLS/JA3) client. Classifies every domain as alive, dead (DNS failure), blocked (anti-bot/WAF), or redirect, and finds roughly 14% of the web's most popular…

Enterprise datacenter hardware end-of-sale and end-of-service-life dates by part number (Cisco, Dell, HPE, Fortinet, IBM, Juniper and more), each record carrying the URL of the manufacturer's own end-of-life bulletin. CSV and JSON, refreshed weekly.

Daily aggregated round-trip latency measured from an independent probe network, plus routing anomalies detected from those measurements. Vantage points include regions under-represented in public measurement datasets. CSV/JSON with manifest and checksums, updated daily.

Monthly aggregated download, upload, and ping statistics by country and ISP, computed from internet speed tests run by visitors to internetspeedtest.net. Each monthly JSON file provides average, median, p25, and p75 globally and per country and ISP. This is observational data, not a representative…

A repeated robots.txt measurement of a fixed panel of 600 German .de domains (300 highest-ranked and a systematic sample of 300 with Tranco rank above 50,000). Records for each domain whether each of 14 AI crawler user-agents is disallowed from / under RFC 9309, including the wildcard fallback…

MIRAGE-2019 is a human-generated dataset for mobile traffic analysis with associated ground-truth, having the goal of advancing the state-of-the-art in mobile app traffic analysis. MIRAGE-2019 takes into consideration the traffic generated by more than 280 experimenters using 40 mobile apps via 3…

Contains 4B+ DNS records accross 700 million unique hostnames. Includes A, AAAA, ANAME, CAA, CNAME, HINFO, HTTPS, MX, NAPTR, NS, PTR, SOA, SRV, SSHFP, SVCB, TLSA and TXT entries.

Internet censorship data

ASNs of Singapore's carriers (Singtel, M1, StarHub, Simba) with a role for each (mobile, fixed broadband, carrier core or unknown), a confidence level and the evidence behind it, plus the IPv4 and IPv6 prefixes each ASN announces. Prefixes are refreshed weekly from RIPEstat BGP data. Available as…

Aggregate tables from a 2026 census of US franchise car dealership domains collected from manufacturer dealer locators. Mail gateway table: DNS MX classification of 17,659 domains into receiving providers (Microsoft 365, Google Workspace, GoDaddy, Mimecast, Barracuda, Proofpoint and others)…

Live and historical internet censorship measurements aggregated from OONI and partner probe networks. Covers 19.6M data points across 126 countries (live JSON) plus a 1.6M-row 10-year historical archive in Parquet. Includes per-country block rates, per-domain reachability, ISP/ASN signals, and…