Science Library Pad

billions of photos

Facebook says

we've added 20% more photo servers and 50% more upload servers to process this year's Halloween traffic. We've also packed in 40 terabytes of additional storage (that's 40,000,000,000,000 bytes). By comparison, the words of all 20 million books in the Library of Congress could be digitized in about 20 terabytes of text. So we're making room for the equivalent of two Libraries of Congress.

The popularity of Facebook's photo application, home of more than 10 billion photos, has compelled us to think big for a while.

Click or Treat: Halloween Photo Bonanza - Facebook blog - October 31, 2008

Wow, that's a lot of photos of drunk college students...

It's also an order of magnitude increase from last year.  In an infrastructure posting from 2007, they said

We have:

    * 1.7 billion user photos
    * 2.2 billion friends tagged in user photos
    * 160 terabytes of photo storage used with an extra 60 terabytes available
    * 60+ million photos added each week which take up 5 terabytes of disk space
    * 3+ billion photo images served to users every day
    * 100,000+ images served per second during our peak traffic windows

Facebook Photos Infrastructure - Facebook blog - May 21, 2007

I discovered the Facebook has a Statistics page in its press area

* More than 120 million active users

* More than 10 billion photos uploaded to the site
* More than 30 million photos uploaded daily

Let's see 30x30 = almost a billion photos a month, so yeah, the math works out.

Flickr, photo-sharing darling (which always has had a buzz disproportionate to its actual number of users and photos, due its "look, folksonomy stuff" usefulness for the technorati), Flickr reports its 3 billionth photo.

Take the pinky of either your left or right hand, hold it next to the corner of your mouth and say “3 beelleeeeon photos…”

3 Billion! - Flickr blog - November 3, 2008

I don't think any of us quite know what it means to have photos at the giga-scale, using storage in the tera- and peta-scale.  (It's quite possible evolution has not equipped us to think at such scales, although it appears the Maya were quite comfortable thinking about time at the mega- and even giga-year scales.)

UPDATE: As libraries, we also need to think about what this means in terms of our position as information managers.  Pre-digital, the central library in your town or city probably did contain the largest (or nearly the largest) collection and concentration of information available to you.  My organisation has millions of items (articles mostly) that it can make available to thousands of patrons.  But compare that to billions of items and tens or hundreds of millions of patrons.  Or compare that to home users who may have terabytes of storage and tens of thousands of photos.  Can libraries still claim to be the centre of knowledge management expertise when they are tiny players in the digital environment?

Amazon S3: Storage Service

Amazon S3 is storage for the Internet. It is designed to make web-scale computing easier for developers.

Amazon S3 provides a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. ...

Amazon S3 Functionality

Amazon S3 is intentionally built with a minimal feature set.

• Write, read, and delete objects containing from 1 byte to 5 gigabytes of data each. The number of objects you can store is unlimited.
• Each object is stored and retrieved via a unique, developer-assigned key.
• Authentication mechanisms are provided to ensure that data is kept secure from unauthorized access. Objects can be made private or public, and rights can be granted to specific users.
• Uses standards-based REST and SOAP interfaces designed to work with any Internet-development toolkit.
• Built to be flexible so that protocol or functional layers can easily be added.  Default download protocol is HTTP.  A BitTorrent (TM) protocol interface is provided to lower costs for high-scale distribution.  Additional interfaces will be added in the future.

Pricing

• Pay only for what you use. There is no minimum fee, and no start-up cost.
• $0.15 per GB-Month of storage used.
• $0.20 per GB of data transferred.

via panlibus

Alexa Web Search Platform... with an actual business model

The Alexa Web Search Platform provides public access to the vast web crawl collected by Alexa Internet. Users can search and process billions of documents -- even create their own search engines -- using Alexa's search and publication tools. Alexa provides compute and storage resources that allow users to quickly process and store large amounts of web data. Users can view the results of their processes interactively, transfer the results to their home machine, or publish them as a new web service.

but here's the beauty part: a Price Guide (US dollars)

• $1 per cpu hour ($0.50 for reserved but unused hours)
• $1 per GB/year of user storage
• $1 per 50 GB processed
• $1 per GB uploaded/downloaded
• $1 for every 4,000 user-published web service requests

In the world of free or "you may only use this x times per day for non-commercial purposes" services, I think this is a real breakthrough.

via Slashdot Alexa Web Search Platform Released /.

Euro Fedora User Meeting 2005 - Researching Fedora to Serve as Central Repository System of the State and University Library

Researching Fedora to Serve as Central Repository System of the State and University Library
Stephan Drescher (PowerPoint)
and Birte Christensen-Dalsgaard (PowerPoint)

* National Library - digital preservation
* Storage Preservation
- as many different systems as different objects (currently)
- including both digitized and born-digital (IR and Webarchive)

working on automatic ingest
working on national infrastructure for institutional repositories, with infrastructure in charge of
preservation - national infrastructure for storage with redundancy etc.

Preferred preservation strategy: migration

working with Royal Library in Holland who have an emulation program running with IBM

Stephan: Archiving of Denmark's Broadcasting of Radio and Television (BART)

* covering 24/7/365
* 220 GB a day
* data needs to be evaluated and eventually corrected after 48 hours
* automatically ingested into repository

How to fit Fedora within Bart's resource workflow

Uses Linux and off-the-shelf.

80-100 TB a year

Q: are you going to put this data into the repository?
A: into the repository only references (to the raw data) will be input

Digital Preservation Coalition - Reports - Storage

The Digital Preservation Coalition (DPC) was established in 2001 to foster joint action to address the urgent challenges of securing the preservation of digital resources in the UK and to work with others internationally to secure our global digital memory and knowledge base.

They have a number of reports, including their latest: Technology Watch Report 04-03 - The Large-scale Archival Storage of Digital Objects (PDF).  It includes a case study of a storage system chosen by the British Library.  The storage system is part of their Digital Object Management initiative.  You can see a bit of their storage architecture on their solutions page.

via digitizationblog

Survivor's Guide to info tech 2005

Network Computing publishes an annual "Survivor's Guide" issue.
I linked previously to just the Business Applications part.

The full issue is Network Computing December 16, 2004. Topics include Security, Storage and Servers and many others.

storage commentary, serial ATA

The big storage companies want you to buy big, expensive storage.  While I laud their capitalist ambition, they have cranked out quite a bit of FUD over the years.

In meeting after meeting they told us we must have a SAN (Storage Area Network) for databases, for reliability, for whatever.

We listened politely and after extensive research, went instead with a NAS (Network-Attached Storage).

They will also tell you that SCSI and Fibre Channel (FC) drives are "enterprise class", whatever that means.  However, lower-cost Serial ATA drives are now quite close to the specs of SCSI drives from a few years ago.  Network Computing storage blog has some numbers:

Comparing yesterday's "primary storage" with today's Serial ATA

Collaboration for Knowledge Management and Campus Informatics

Collaboration for Knowledge Management and Campus Informatics By Sarah M. Pritchard,  August 23, 2004

http://www.library.ucsb.edu/informatics/informatics/documents/UCSB_Campus_Informatics_Project_Report.pdf 

Project home page:

http://www.library.ucsb.edu/informatics/

seen on Rocky Northern Shepherd, a blog by Peter Levesque of the SSHRC.

Particularly relevant channels from that blog are Knowledge Mobilization and Community Research.
(The rest of it is pretty cool too.)