world’s largest Hadoop application producing data used by the Yahoo! Search Webmap
From my first blog, oslutions.com (2005 to 2008), recovered from the Internet Archive and republished as it was. Links are from that time and many no longer work. Original: http://oslutions.com/blog/2008/02/20/worlds-largest-hadoop-application-producing-data-used-by-the-yahoo-search-webmap/.
Yahoo announced it is running the world’s largest Hadoop application, a 10,000 core Linux cluster producing data used by the Yahoo! Search Webmap.”. the numbers are really impressive:
- number of links between pages in the index: roughly 1 trillion links
- size of output: over 300 TB, compressed!
- number of cores used to run a single map-reduce job: over 10,000
- raw disk used in the production cluster: over 5 petabytes