Science Library Pad

Google and the web of structured data

Google has announced it will be using (some) microformats and RDFa to enrich search results.  They call this Rich Snippets.

To display Rich Snippets, Google looks for markup formats (microformats and RDFa) that you can easily add to your own web pages. In most cases, it's as quick as wrapping the existing data on your web pages with some additional tags.

Official Google Webmaster Blog - Introducing Rich Snippets - May 12, 2009

O'Reilly Radar says

Moving toward the Semantic Web will allow our searching technologies to become more intelligent and will set the stage for the next revolution in which computing systems can become more aware of the "meaningfulness of data".

We've already seen a shift toward "semantic search": Google has already been augmenting search results with Google Maps, limited catalog searches, and more recent entries into the search market such as Amazon's A9 and the yet to be released Wolfram Alpha differentiate themselves by the structured data and content that can be extracted from a search result. We have yet to a see a compelling reason for web masters to place RDFa or microformats into a site to enable this semantic data to be mined until today, until Google provided a social incentive for site designers.

O'Reilly Radar - Google Announces Support for Microformats and RDFa - May 12, 2009

Google has a help file to get you started: Marking up structured data.

Incidentally, if you're thinking, "why didn't someone tell me this structured data thing was coming?" I should mention that actually I did try to tell people, whether it was in my presentation to Allen Press in 2007 (where I talked about the need for microformats and semantic enrichment) or in my keynote to NISO Discovery last year (where I talked specifically about Yahoo SearchMonkey using semantic information).

Previously:
September 8, 2008  semantic search thoughts

Wolfram Alpha and the web of structured data

The good news is, WolframAlpha can understand a query like "unemployment in Ottawa".

The bad news is, as you can see in the Result, that while it knows what data it needs to compute the answer, "(data not available)".

This is as clear an example of why we need open data as I can think of.  If you want your computers to start providing you with smarter answers, you have to give them better information to work with.

WolframAlpha provides two channels to try to address this problem: Contribute Structured Data and Suggest Data Sources.  You can also contribute individual facts, but that won't scale very well.

in which I realise Twitter can make money

(I was going to say something about "monetization" but that term is getting a bit old.  Can I make it through the entire post with out using it?  Let's see...)

Ok, social networks are hard places to make money.  Sure you know a lot about the users, but they're busy flirting with one another or posting pictures of their cats or digging up embarassing images of people passed out last night.  They're not searching for information, they're interacting with people.  Trying to wedge commerce into friendship, as Facebook is trying to do, trying to make money from social interactions - just doesn't work well.  Wrong mode.  See previous posting the difficulty of (making money from) Web 2.0 eyeballs.

So I thought Twitter was probably hopeless in terms of a business.
But then I realized that Twitter is also real-time search.
And search we do know how to make money on.  Boy howdy do we.  To the tune of billions of dollars.

So Twitter actually has it easy.

1. Do the usual adsense matching ads to keywords thing on search.twitter.com
2. When you can't parse the keywords well (as may be the case with hashtags, e.g. #lotf09), build a metadata directory to figure out what the hashtag maps to and then match the ads to that - in which case, ideas like tagalus are going to be worth lots of money - you heard it here first

So Twitter should be fine.  Should be big dollars fine, in fact, as real-time search grows in popularity.

This is also a very simple lesson to all you Twitter-wannabes out there (are you listening, Facebook and FriendFeed?) : you want to make money from the real-time web dynamic whatever - fine, just do it in a separate search interface, not in your main user interface.  When they're in separate search, you know they have intentionality.  Intentionality, you can make money from.  When people are expressing a desire, you can use ads to meet that desire.  That's like, probably the whole underlying concept of ads (yeah, I watch Mad Men).

Whereas ads while people are socially sharing links, pictures, and whatever, not so much intentional seeking, not so much any opportunity to make money.

UPDATE 2009-04-07: I should also mention a current approach that Twitter is experimenting with

3. Subject-specific portals

Since Twitter provides aggregated real-time conversations, you can try to find various ways to slice the conversations up by topic or other  attribute.  The two approaches I am aware of so far are:

* to slice by "authority" or identity - ExecTweets is a portal to "executive" Tweeters (which I think is a bit nonsense, in an age in which authority emanates from your ideas, not your position) - the portal is sponsored by Microsoft - there are other portals that slice out particular celebrity twits etc.

* to slice by content - MarchTweetness/TitleTweets is a portal around the US March Madness basketball thing, sponsored by AT&T - one can imagine a similar soccer-oriented site for the international audience

So one can imagine e.g. a site built around the launch of the SF series Caprica, that captures all the #caprica tweets and attempts to target people interested in science fiction, selling BSG merchandise etc.

I personally think this idea is pretty weak overall, because subject-specific portals generally fail.  Google beat Yahoo.  People jump from topic to topic as they explore the web, as the impulse strikes them.  No matter how much info your portal pulls in, there's always going to be some rival portal on the same topic that people want to check as well.  You cannot hold people on a specific website.  Also, unless you do this in an automated fashion for all topics (in which case it will look generic), it is a lot of work to do a custom siloed portal for each and every particular topic you want to make money from.

I can imagine though they will continue the TitleTweets trend of searching for the "big eyeball" sweet spots like the SuperBowl for custom Twitter portals.

Twitter modes: real-time news and event reporting, topic rendezvous, mobile and power interfaces

There is one core use-case for Twitter which I have discussed before, that of using Twitter as a "social network maintenance" tool, to engage in an ongoing conversation with friends and colleagues, across distance and time (some of the people I tweet with are in England, others in California).

In discussion on FriendFeed, one of those far-away contacts, Deepak, reminds me that I should discuss some of the other modes, the other use-cases for Twitter.

1) Real-time news: if something happens, you should be able to pick out reports about it (with the right keywords) by using search.twitter.com

2) Similarly, through the use of hashtags, you can get reports from events and about topics.

For example, if you want to know what people are saying at the UK Serials Group conference, a search of #uksg09 will help (I don't know what you will find there, as at the time of this writing, the March 30 event hasn't started yet - right now you just see people doing their final arrangements to get there).

If you want to see a view post-event, try e.g. #jisc09 to see what people had to say during and after the annual JISC conference.

You can also see an example of this in my previous posting, about the Twittering of Obama's first visit to Canada.

And it was by watching event reporting on the #ali social media conference in Ottawa on Twitter that I got connected into the community that is now working on #changecampottawa.  In fact, a lot of the ChangeCamp Ottawa planning and information sharing takes place through Twitter.

Likewise, if you're interested in a particular topic, say #nasa space programs, a search will lead you to other interested people and links about that topic.

You can see a small list of Ottawa social media tags in my posting connecting to Ottawa social media community through Twitter.

3) Mobile and the constraints it brings

A big deal is made out of "only 140 characters".  Well, keep in mind that it's possible the desktop computer may one day be considered as obsolete as the deskset telephone.  For many people, their entire telephone experience is mobile, and it may be that our main computer interface in the US and Canada becomes mobile as well, as it already is in other places in the world.  Mobile means two things: small screens and terrible keyboards.  As anyone who is unskilled in cellphone text entry can attest, 140 characters starts to look like a lot when you're trying to use a numeric keyboard to enter words.  As well, trying to read more than a sentence or two at a time on many cellphone screens is simply not possible.

The reason both Twitter and Facebook have short status messages, is that this supports SMS as transport, and that many of the users only ever see those messages through the tiny window of a cellphone screen; they never use big screens on desktop computers for their social networking.

4) Other interfaces

The Twitter web interface is pretty basic.  There are better clients on mobile phones and on desktops.  In particular, TweetDeck is one popular power-user Twitter interface.  Here's a selected view showing some live searches (you can click through to Flickr then select All Sizes to see it at original size):

Using Twitter in this way is really a qualitatively different experience.  Not only is it much easier to do some things (such as retweet), you also can get a big picture of what is going on - incoming filtered tweets next to your incoming @ replies, next to direct messages, with the option to have people sorted into columns by group (work, personal, etc.) as well as to be continuously running realtime searches on topics of interest.  This is a very powerful window on the world of information, if you want to be grand about it, it's a sort of realtime world dashboard.

UPDATE: 5) Search and the Twitter ecosystem

I should mention a couple things:

First about search -

* Twitter search is (mostly) an exact match.  If you search e.g. social, you won't see hits for socialize.

* If you use Twitter search from the home page (or add &source=serp to the search URL), Twitter will show matching user accounts as well, and those are sliding matches.  e.g. here is a search on #gov that shows matching accounts for various US Governors.

Secondly, because Twitter was smart enough to provide an API, a huge ecosystem has grown up around Twitter - this includes tools I have mentioned before, but many many more.  See e.g. this giant list.

Change-tweets come to Ottawa

(A post rescued from draft, somewhat dated but helps to explain a bit about the aspect of Twitter for news and event reporting.)

This graph from TwitScoop shows the activity on Twitter for the hashtag #obamawa - the tag people used for President Obama's visit to Ottawa on February 19, 2009.  Graph captured around 4 PM.  You can see a peak as he arrives in Ottawa, specifically when he gets to Parliament Hill, and a second peak as the joint press conference takes place.

The use of hashtags and the powerful near-realtime search provided by search.twitter.com is a powerful engine for connecting people together who would otherwise never communicate, and for connecting across many different communities.  If you haven't been paying attention to Twitter, you may be surprised at a sampling of the news and government organisations that made postings ("tweets"):

• the US embassy in Ottawa - yes, the real US embassy officially live-tweeted the visit, including the visit to Byward Market
  
• CBC
• TVO's The Agenda

as well of course, many Canadian citizens just going about their lives in Ottawa and elsewhere posted their thoughts.  Whatever you may think about some of the trivialities that people post on Twitter (which are to be expected, since some of Twitter is small talk, which is usually not exactly groundshaking), I think you will agree that the first visit of President Obama to Canada was an event worth people's time and online words.

Here's what the scoop looked like at 5:30 PM that day

If we're talking about laments for democracy and citizenship and news reporting - what this means is people were engaged, they were excited, and they were participating in a shared experience as equals with major news organisations. This is what democracy is all about. I'm not saying that means they have an equal level of expertise, I'm saying the technology enables them to have the same level of participation, and to share their own observations.  Even if we wanted citizens to be passive consumers of broadcast news (and surely we didn't?) that age is gone.  The new era is about the newsmakers, the newsmedia, and the newscitizenry all engaging in a discussion together.  Surely that's a more profound engagement in our own democracy than sitting on a couch watching Peter Mansbridge, or skimming the Globe over morning coffee and then tossing it in the recycling bin.

CISTI in Google Scholar and Facebook - official news

I always have to walk a fine line because it takes a while for official messages to be approved in the Government of Canada. We've got some approved releases, so therefore I'm happy to now be able to safely enter

<marketing-mode>

Ottawa, ON, March 23, 2009 – NRC Canada Institute for Scientific and Technical Information (NRC-CISTI) is meeting the research community where they work through our new collaboration with Google Scholar.

CISTI Media Room - Google Scholar

Ottawa, ON, March 23, 2009 – In an effort to generate awareness about the products and services offered by NRC Canada Institute for Scientific and Technical Information (NRC-CISTI), NRC-CISTI is now on Facebook.

CISTI Media Room - NRC-CISTI on Facebook

</marketing-mode>

Libraries in Computers - March 31, 2009

I was thinking about the post title and it occurs to me - no offense to InfoToday - that "Computers in Libraries" is kind of an old way of thinking about things.  The reality today is Libraries in Computers.
This is not a new idea - Mark Weiser was writing about ubiquitous computing in Scientific American in 1991.

The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it.

Is your library becoming a part of your patrons' everyday life?  All this to introduce CIL 2009 session

D203: Embedding Services: Go Where the Client Is

coming up on March 31, featuring my colleague Natalie Collins.  She'll talk about the process that got us to "allowing discovery to happen in the user’s regular work environment".  If you want more details, a previous post about a thousand words may give a hint.

a thousand words?

http://scholar.google.ca/scholar?q=a+thousand+words&hl=en&lr=

Note: AFAIK you must use either scholar.google.ca or be in Canada to see all aspects of this result.

UPDATE: If you want a search that will take you more directly to the PDF option http://scholar.google.ca/scholar?q=Simulations+of+the+2.5D+inviscid+primitive+equations+in+a+limited+domain&hl=en&lr=&btnG=Search

UPDATE 2009-02-27: If you want to see a third type of result, try http://scholar.google.ca/scholar?q=Biochemistry+and+molecular+cell+biology+of+diabetic+complications&hl=en&lr=

10 years of Google

TVO's The Agenda with Steve Paikin had a good segment on 10 years of Google.

You can watch Flash video of it on the site,
they also have podcasts in audio and video flavours.

You can also follow them on Twitter

twitter.com/theagenda

twitter.com/spaikin

semantic search thoughts

Thomas Tague has asked for input to help better define semantic search.

So, let’s deconstruct semantic search into it’s constituent components and talk a bit about how and whether semantic technologies might actually make it better.  The results of the dissection are here on the table….

   1. What kinds of questions can we ask? Can we embed logic in our questions? Do we expect inference in our results?
   2. How can we ask them – keywords, natural language and all that jazz.
   3. Generating the “right” result set for the query.
   4. Displaying the result set in the most effective manner
   5. Making money from doing all that

Here at CISTI we see Semantic Search as a key advanced technology to help the library support researchers, and in particular to help researchers get at the meaning locked up in our millions of locally-loaded journal articles.

We think of it as three pieces:

1. Domain-specific semantic extraction, in which the different types of scientific information are identified and indexed for search (e.g. chemical names, mathematical formulas, genes, etc.)
2. Straightforward search frontends, which use basic keyword mapping to provide an enhanced discovery interface (e.g. you search for water and you get any documents with the various possible representations for water)
3. Semantic search frontends (possibly also domain-specific) in which a researcher enters a search in terminology that's meaningful to them - for example drawing a chemical structure, or asking a complex question (e.g. what X are affected by Y when conditions A and B hold)

For all of these areas we see partnering as a key, as no single organisation is going to have the domain-specific expertise necessary, or the technology expertise necessary.

I'd be happy to hear other opinions on semantic search, particularly as it relates to science.
Should we separate it into several subareas?  Perhaps

• Semantic embedding (the document is already semantically enriched)
• semantic extraction (terms are extracted from a document that isn't semantically encoded)
• semantic indexing (constructing search indexes using semantic information)
• semantic search (search engines that use semantic information)

I see I don't even have a category for Semantic Web, so I guess it's time to make one.

in which net searching gets doubly disrupted

Internet searching is by no means a stable space.
Remember simple old Google?  Type in keyword, get a link?

Well now it's type in keyword, get suggested related searches, a link, a bunch of subpages of the link, and (sometimes) a site-specific search box.

As if that wasn't enough, along comes Firefox 3, with character matching in both URLs and page titles, when you type in the main URL bar.

Is Firefox 3 matching going to mean less searches sent to Google, as people navigate within their own set of URLs?  Is it going to change the page titles we use, so that we get better matches?  Why does it rank VALA ahead of ALA in the pattern match?

availability, discovery, and delivery - redux

• Availability does not equal accessibility: researchers’ top concern about scholarly communication is that they cannot access all the content they wish to access
• Researchers tend to use tried-and-tested discovery tools, or those which their library specifically trains them to use. Google and other web search engines remain the most-used search tools for work-related information. The main problem with discovery is coming up against an access barrier
• Researchers do not always know how to seek out a freely-available copy of an article that they want and which they have discovered behind a toll barrier

Key concerns within the scholarly communication process: report to the JISC Scholarly Communication Group, March 2008 [Word document]

via Lorcan Dempsey

I find it interesting that the focus of concerns is around delivery, not discovery (perhaps this is how the questions were framed).

I think the academic library faces two challenges:

1. Ensure that your researchers can always get from their chosen discovery environment easily to
1. Get to your licensed resources
2. Get to free copies, if licensed versions aren't available
3. Get to purchase options, if free copies aren't available
2. Ensure that as many of your resources as possible can actually be discovered

I'm not convinced that we're doing a particularly good job of addressing these fundamental challenges even after years of working on proxies, federated search, link resolvers, and "live in your environment" plugins and external website settings.

It seems to me that librarians were so focused on trying to control the discovery experience, trying to make people discover resources "properly", following established librarian search protocols, that the simple challenges above were not addressed.

I think we need to spend a lot of time with researchers letting them search however they want, and seeing whether they dead-end either by not being able to get to a resource at all, or by landing at a paywall for a resource we license or can get to for free.  Fix that first.

And I'm not entirely convinced we have all the tools we need to fix that problem right now.

Then once you have addressed consistent delivery, work on improving discovery.  I think discovery is much harder to fix.  And to some extent, there should be vendor pushback.  I don't care how rich or comprehensive a licensed resource is, if my users can never discover it, then the message to the vendor should be "enable easy ways for my user to discover your resources within their preferred discovery environments, or next year we're not licensing your content".

Previously:
Lorcan and I had a bit of a back-and-forth about discovery and delivery in 2006.

August 08, 2006  Discovery and disclosure - Lorcan Dempsey
August 09, 2006  online library role in discovering and delivering - Science Library Pad

Yahoo - the SearchMonkey cometh

Well, as long as it isn't a flying monkey, ok.

Enter the Yahoo Open Strategy (YOS). ...

There’s a massive, latent social network within Yahoo, and we’re going to bring it to the surface. We’re making Yahoo more social, but we’re not building yet another social network. We already have an incredible social network… we just need to unlock it.

...

A first taste of our strategy is SearchMonkey, which will let developers mash up helpful data with our search engine results.  ... launch party May 15 [2008]

http://ycorpblog.com/2008/04/24/developer-welcome-mat/
(Exclamations removed because I refuse to put exclamations all over the place, including inside of acronyms.)

SearchMonkey is what I previously described as "semantically-enriched search results".

General story of Yahoo Open Strategy very widely reported, e.g. Globe - Associated Press - Yahoo plans social makeover.  SearchMonkey bit via Search Engine Watch blog.

Building SkyNet for Science - presentation for NISO Discovery Tools Forum

My presentation is available at

http://www.slideshare.net/scilib/building-skynet-for-science-discovering-new-frontiers-using-embedded-knowledge/

A lot of it is conceptual, so you may want to wait until the audio is available from the http://www.niso.org/news/events_workshops/discovery08/ NISO site - http://www.niso.org/news/events/2008/discovery08/ (hopefully next week).

UPDATE 2009-04-16: I don't think audio was made available.  ENDUPDATE

UPDATE 2008-03-28: I forgot to mention that all some of the supporting links for the presentation (will be) are available at http://www.connotea.org/user/scilib/tag/nisodiscovery2008  ENDUPDATE

I thought it went well, although as first speaker up there is a disadvantage of not seeing how other people set up.  I was in a bit of a rush to get started so that I would finish on time, so I didn't do a great job of attaching my mike and I just held the wireless transmitter in my hand.  With the transmitter in one hand and my laser pointer gripped in the other for the entire 50 minutes, it's possible I looked a bit of a prat.

I usually try to remember to keep my hands free for presentations so that I can use more natural body language, anyway lesson learned.

I also forgot to say "The future is not set.  There's no fate but what we make for ourselves." before the last slide.

There are some common themes emerging from the presentations, I'm always amazed when a bunch of people develop presentations in isolation and then they actually all fit together when presented.

I've posted some photos of the presenters in my Flickr under nisodiscovery2008 (my cameraphone can upload directly to Flickr over WiFi), they also show up because of the machine tag linkage on the Upcoming page.  No pics of Chapel Hill yet as I don't have a car and it turns out while we're only about 4km from the town, the most direct route for me to get there I think would be to walk beside a six-lane divided highway, which is not too appealing.

UPDATE 2008-03-28: The carbon offset for my flights from myclimate.org (including the trip to Open Repositories) was about C$118.

Google Book Search API

The Google Book Search Book Viewability API enables developers to:

                   

• Link to Books in Google Book Search using ISBNs, LCCNs, and OCLC numbers
• Know whether Google Book Search has a specific title and what the viewability of that title is
• Generate links to a thumbnail of the cover of a book
• Generate links to an informational page about a book
• Generate links to a preview of a book

http://code.google.com/apis/books/

via LibraryThing blog - Google Books in LibraryThing - March 13, 2008

We need APIs everywhere for everything.

Semantically-enriched search results coming from Yahoo

In an upcoming talk I will be continuing a theme I started at Allen Press, calling for more semantic enrichment of scientific information online (I am of course, only one of many making such calls).

It is therefore timely to see Yahoo offering an open platform for harvesting and returning semantically-enhanced search.

There was a pre-announcement on TechCrunch, followed by the official word on the Yahoo Search Blog

In the coming weeks, we'll be releasing more detailed specifications that will describe our support of semantic web standards. Initially, we plan to support a number of microformats, including hCard, hCalendar, hReview, hAtom, and XFN. Yahoo! Search will work with the web community to evolve the vocabulary framework for embedding structured data. For starters, we plan to support vocabulary components from Dublin Core, Creative Commons, FOAF, GeoRSS, MediaRSS, and others based on feedback. And, we will support RDFa and eRDF markup to embed these into existing HTML pages. Finally, we are announcing support for the OpenSearch specification, with extensions for structured queries to deep web data sources.

Yahoo Search Blog - The Yahoo! Search Open Ecosystem - March 13, 2008

You can sign up for more information at

http://tools.search.yahoo.com/newsearch/open.html

So what would an appropriate set of semantic information be for a scientific article, what would your ideal search display include?  # of citations?  Impact Factor?  Chemical and gene sequences?  Price?  (Sometimes information wants to be expensive...)  How much can we fit into a couple of lines that will help to select one article over another in results?

UPDATE: And Yahoo is just one player in this space, as Paul Miller indicates in his posting Looking for a dominant Semantic Web search engine.

via Twitter mostly

Next Generation Discovery - NISO Forum - March 2008

http://www.niso.org/news/events_workshops/discovery08/ Next Generation Discovery: New Tools, Aging Standards - http://www.niso.org/news/events/2008/discovery08/
March 27-28, 2008
Chapel Hill, NC

Discovering scholarly information and data is essential for research and use of the content that the information community is producing and making available. The development of knowledge bases, web systems, repositories, and other sources for this information brings the need for effective discovery -- search-driven discovery and network (or browse) driven discovery -- tools to the forefront. With new tools and systems emerging, however, are standards keeping pace with the next generation of tools?

I will be presenting.
I also (after fighting with Yahoo accounts) managed to make an Upcoming event.

Other confirmed presenters include Peter Murray, Karen Hawkins (talking about scitopia.org), and Eric Schnell & Dave Munger (talking about researchblogging.org and the BPR3 initiative).

I'm proposing a tag: nisodiscovery2008

UPDATE 2008-03-10: And a Twitter Hashtag #ndf08

Retrevo - searching for reviews

I know the search engine experts out there can probably list dozens of similar sites, but I liked the execution of this one: Retrevo.

If you search Google, you get a flat information space, which you have to narrow with keywords.  I like Google a lot for bringing relevant results to the top - but if it doesn't know why you're searching, of course it has problems.  In particular, if you search for just a particular part number, it will often list page after page of sites selling that item, with very few links to reviews or blog postings.

Retrevo specifically separates its results out into "Reviews & Articles", "Forums & Blogs", and "Shopping".  Nicely done.  Of course, the fact it ranks me first doesn't hurt...

http://www.retrevo.com/s/dpl700?sub.x=0&sub.y=0&sub=Search

Internet Archive 20th Century Search - DLF developers' preconference - Nov 3, 2007

Before the main event there was a preconference (with a fraction of the main attendees) exploring technical challenges and possible collaborations.  There were also a couple brief presentations, here are my raw notes on Kris Carpenter Negulescu, the Director of the Internet Archive Web Group talking about

"20th Century Find" using Amazon S3 & EC2

Internet Archive stats

3.5 PB
1.5 million downloads/day

this project is about providing full-text search of their web archive for the 20th century 1996-2000, ~22TB

NutchWAX = Nutch + Hadoop
focus moving to Nutch with plugins

Amazon S3
Amazon EC2 (beta)

there are now more EC2 node options: small (default), large, extra large 8 times small performance and better I/O ($0.80/cpu hour)

indexing began in October 2006
1996: indexed via 20 EC2 in ~36 hours
1997: 100+ EC2 nodes
1998: 300+ EC2 nodes

1999 was attempted in September 2007 using cluster of ~270 EC2 nodes but halted due to lack of
consistent CPU/IO across nodes.

deployed (alpha) index is 1.35TB in size, no compression, ~600 mill docs

Enhancements
* multiple instances of a page
* improved ranking of results
* handle dimension of time
* easy UI

Why Amazon Web Services
* pay as you go
* simple to provision
* committed to support
* indeal for indexing Web pages, providing offsite storage, reliable hosting
* great platform for experimentation, iteration
* geographically disperse from Internet Archive ?Data Repository?

Cost Effective, budgeted $20k

* Note: fees can add up fast if not vigilant

Working Well

* APIs
* tech support
* S3
* fee structure
* speed of provisioning
* S3 uniformity of nodes

Challenges - S3

Oct 2006 - June 2007

* (internal?) bandwidth availability
* no specific guarantees for data preservation
* issues related to popularity of the service

Fall 2007

* available bandwidth consistent (~4h to move 7.5TB into EC2)

Challenges - EC2

lots of issues Oct 2006 - June 2007
* location of S3 nodes relative to EC2 was a significant factor for large-scale data processing

July 2007 - present

* working well but hitting IO and CPU constraints on small (basic, default) nodes;
however will continue to use these small nodes

Consider Using AWS When

S3
* need cost-effective backup for data
* multi-provider preservation, geographically diverse

EC2
* if you have spiky computing needs (e.g. spikes in demand)
* you have available R&D resources

Will experiment with AWS for crawling and harvesting, starting Jan 2008, Heritrix/AWS.

Scirus Topic Pages now online

http://topics.scirus.com/

"curated" pages with various resources on a specific topic, e.g. Evidence-Based Laboratory Medicine

Previously:
June 28, 2007  Elsevier pre-announces Scirus Topic Pages

Casey Bisson on Scriblio and OpenLibrary

In the BIGWIG library social software showcase, Casey Bisson writes about his proposal to further enhance Scriblio.  Scriblio is a vision of a modern library web presence that is about finding books, not being a book inventory.

With its partners, the Internet Archive will build an open, structured website that includes tools to provide public and institutional access to library resources around the world. OpenLibrary.org will offer a system of integrated tools that can be used individually or together to meet a library or patron’s needs, that is free to the user and the library. The overall goal of the project is to shorten the distance between initial query and document.

Elsevier pre-announces Scirus Topic Pages

Topic Pages are an innovative way for scientists to communicate in an informal and flexible way. Each Topic Page will provide researchers with summaries of a specific topic written by an authority in the particular subject area, with direct links to relevant scholarly papers, abstracts and citations, supplemented with relevant websites and other online resources from Scirus. In the initial phase, authors for Topic Pages are invited through an editorial process facilitated through Elsevier publishing staff. As more pages are developed, additional authoring options will be considered.

...

At the official Topic Page launch later this year, the functionality of the Topic Pages will allow scientists and researchers to alter the content and provide feedback, allowing each topic to be shaped by the suggestions made by the research community.

from Elsevier PR - Elsevier Announces Partnership with FAST to Deliver Topic Pages for Scientific Communication on the Web

This is presumably the social network that was indicated in Joris van Rossum's Science-specific Search - Scirus and beyond talk at IATUL 2007.

UPDATE 2007-07-20:

http://topics.scirus.com/

Science-specific Search - Scirus and beyond - Joris van Rossum - June 14 - IATUL 2007

Joris van Rossum
Scirus, Amsterdam, The Netherlands

Science-specific Search: Bridging the gap in dissemination of and access to information

http://www.scirus.com/

Scirus is the free science search engine
he previously worked as a senior product manager at Scopus
(the pay science search engine http://www.scopus.com/ )

Content [Overview]
* How has content provision changed?
* How has information retrieval changed?
* Future trends

Internet has made journal publishing just one of many options for scientific content communication

why is there still journal publishing?
* stamp of authority
* versioning challenge - journal has version of record
* archiving - publishers ensure the article will always be available online

Increasing amount of content available online
* high amount of published content
- Scopus has 30 million abstracts
- ScienceDirect has 8 million articles
* amount of scholarly web content even higher
- Scirus currently indexes over 400 million scholarly web pages
(feel they need to add another 400 million on top of that in order to be complete at this moment)
* size of general web has exploded
- as of August 2005 Yahoo indexed over 19 billion pages
- Google says it indexes 3x more than that

Different content discovery methods
* browsing
* linking
* alerting
* searching
* user collaboration/sharing (potentially very strong, even replacing searching)

based on analysis from ScienceDirect usage logs

Browsing
* used to keep abreast of latest developmetns in subject area
* 31% of all full text article use on ScienceDirect is a result of journal browsing
* users that start this way download on average 1.9 articles

Linking
* very effective content discovery method
* publishers are collaborating through CrossRef to ensure correct reference linking
* 8% of all full text article use on ScienceDirect comes from reference linking
- Same is expect from 'cited-by' links
* next to reference and cited-by links in official literature there is
- web ref and cite-by
- patent ref and cite-by
- clustering
- author linking

citation paradigm applies beyond the official literature

Alerting
* journal issue [TOC] (RSS)
* top articles
* citation
* search

2-4% of downloads come from alerting

Search is the main driver of journal article use
* exponential growth of PubMed and ScienceDirect searches
- growth rates between 20-110% from 2001-2005
* search has overtaken browsing
[missed rest of slide]

search is important because it yields more than just journal results

* general web search (Google) often 1st choice for scientists (66%) and physicians (55%)

* subject-specific search platforms remain important
- avg. # of full text article downloads per sesson from PubMed is 3, from general search is 1.5

screenshot of Scirus, specialised science features

role of librarians in improving integration:
- search on library homepage, OpenURL (Scirus Library Partners)

*** User Collaboration and Sharing - The Future of Information Sharing ***

Combining browsing, linking, alerting and search in a community and network-driven system

... Scirus will launch a new community service in a couple weeks

Q: researchers aren't always searching for articles - in computer science we are interested in
aggregates e.g. projects and research programs
A: the new service will offer topic-based collected search and resources relevant to a particular area

Q (Jens): journal publishing enduring... authority good - versions ? - archival not true!
A:

archiving ScienceDirect - The Hague - for free - even if Elsevier disappears, info will still be available

why is Elsevier concerned about archiving
- helps provide confidence to move everyone to e-only
- gives authors confidence

meta: in which I make a claim

Just a posting to try yet again to claim the 2nd URL that Technorati sees for my blog

Technorati Profile

In other news, Technorati now presents a single search box, to search for tags specifically you'll have to go to advanced search.

UPDATE: It's working for both TypePad blog URLs now.  An interesting reminder about basing any decisions on a particular Technorati ranking result.

Science Library Pad      

Rank: 74,734 (84 links from 63 sites)

• URL: http://scilib.typepad.com/science_librar...

Science Library Pad

Rank: 56,727 (174 links from 82 sites)

• URL: http://scilib.typepad.com

Previously:
July 6, 2006  Technorati rank, updating, and multiple blog URLs

Amaznode related book visualization

Amaznode uses Flash and the Amazon API to extract and visualize related books for a search.

See e.g.

paris

and

gps

Note: it will continue pulling in results for a while (about a minute).

Via BlogSchmog.