Billion Triples Challenge 2009 Dataset

Description

The major part of the dataset was crawled during February/March 2009 based on datasets provided by Falcon-S, Sindice, Swoogle, SWSE, and Watson using the MultiCrawler/SWSE framework. To ensure wide coverage, we also included a (bounded) breadth-first crawl of depth 50 starting from http://www.w3.org/People/Berners-Lee/card.

The downloaded content was parsed using the Redland toolkit with rdfxml, rss-tag-soup, rdfa parsers. We rewrote blank node identifiers to include the data source in order to provide unique blank nodes for each data source, and appended the data source to the output file. The data is encoded in NQuads format and split into chunks of 10m statements each.

Citation

@misc{btc-2009,
  author = {Andreas Harth},
  title = {{Billion Triples Challenge} data set}, 
  howpublished = {Downloaded from http://km.aifb.kit.edu/projects/btc-2009/},
  year = 2009
}

Download

To fetch the content of the entire directory, download the 000-CONTENTS file and do $ wget -i 000-CONTENTS

Please note that the size of the combined dataset (gzipped) is around 17GB. A smaller crawl useful for testing is available at btc-2009-small.nq.gz.

For questions about data format, server issues, or download problems contact the BTC dataset coordinator harth@kit.edu.

Enjoy!

History

2012-06-15

Completed move from vmlion25.deri.ie to KIT server

2009-06-19

Fixed minor unicode and blank node syntax issues

2009-04-16

Dataset posted

General statistics (~1,14 billion statements)

We calculated statistics over a 1 % random sample of the dataset, and scaled the derived numbers up accordingly. While the sampling process introduces an error in the final numbers, we still think the numbers are a sufficiently good estimate of the overall characteristics of the dataset.

• distinct nodes: 1,464,829,200
• distinct resources: 866,094,100
• distinct blank nodes: 352,032,700
• literals: 246,702,400

The stats were created with the NxParser library statistics package by Jürgen Umbrich.

Top-50 classes and their frequency

For resolving the prefixes with their namespaces please use the service provided by prefix.cc.

Top-50 RDF Properties and their frequency

For resolving the prefixes with their namespaces please use the service provided by prefix.cc.

Top-50 PLDs of sources and their frequency in the data set

"A pay-level domain (PLD)is any domain that requires payment at a TLD or cc-TLD registrar. PLDs are usually one level below the corresponding TLD (e.g., amazon.com), with certain exceptions for cc-TLDs (e.g., ebay.co.uk, det.wa.edu.au)"[IRLBot]