Scholarly impact: from ranking to assessment - Johan Bollen - June 21 - ICSTI 2007 Nancy

This is some very interesting work and a huge project that should greatly enrich our understanding of the usage of scientific information.

Johan Bollen - Los Alamos National Laboratory
"Scholarly impact: from ranking to assessment"

Scholarly evaluation matters
- qualitative and quantitative indicators

many features in scholarly status space
- prestige
- novelty
- visibility
etc.

[MESUR project]

various opportunity to extract metrics in the scholarly life-scucle
- usage data
- review data
- citation data

usage data is available before citation data

public.lanl.gov/jbollen/Publications.html

From ranking to assessment

we're in mode ranking 0.6
- single data source
- single criterion

... to assessment 3.0 [yecch]

- situate item in value landscape
- multiple sources of scholarly information

question: which dimensions to choose?

1) MESUR project
- survey wide range of possible indicators
2) Peer review
- study peer review process

Marko Rodriguez and others

Can we improve on citation data and the impact factor?

- perhaps usage data applies to a larger subset of the scholarly community,
  capturing more scholarly objects and activities beyond journal articles

usage: COUNTER, IRS [?], SUSHI, CiteBase

MESUR: Metrics from Scholarly Usage of Resources

1 ontology to model the scholarly process
2 beg for usage data
3 dedupe
4 create semantic network

2/5th through the project

[ontology]

data 700million usage events and 1 bilion citations
10-15 billion triples

COUNTER logs, item-level data, SFX, etc.

link resolver data very good

[paper in JCDL 2006 about link resolver data gathering architecture]

they are using Franz's AllegroGraph triplestore

Network usage: usage graphs

"we should stop counting: we should look at relationships"

journal network - 50,000 journals

Example: Flow of information

[pretty network]

Metrics survey

many large organizations and sites are participating

U Texas case study...

[my comment: but isn't there an undergrad effect based on the articles they are assigned?]

principle component plot

[paper at JCDL 2007]

many issues and challenges