Science Library Pad

ICSTI video - "Assessing the quality and impact of research: practices and initiatives in scholarly information"

Videos of the presentations from the ICSTI 2007 Public Conference "Assessing the quality and impact of research: practices and initiatives in scholarly information" are now available.

You can see the programme with links to presentations, and the video page is separate, with links to each topic (RealVideo format - click on the camcorder icons on each sub-page, not on the presentation titles), I was in Quality, certification and peer review - Part 1, my presentation was Web tools for peer reviewers...and everyone (RealVideo, 35 minutes).  I encourage you to check out the other topics, I found there were some really informative presentations.

(As I said in my previous posting about my presentation, the title is not great, it's more like "categorizing the problem space of journal article exploration, and what new features or metrics we might use in this new space, as well as what new scholarly objects we might certify by applying peer review".)

My hair is, as usual, sticking out at some odd angle.  Fortunately for you, most of the time the video shows the slide I'm talking about. It's probably not entirely clear from the video, but about 14 minutes in the projector died, so I was left talking beside a blank screen (I could still see the slides on my own monitor) for about 5 minutes until everyone decided they'd had enough of that and we took a break and came back when the projector was fixed.

The next (closed, members-only) ICSTI meeting is coming up next week in Paris.  I don't know whether there is an accompanying public workshop.

UPDATE: The first version of this post ended up with an amusingly unfortunate URL.

WorldWideScience.org

The ICSTI 2007 Nancy public conference finished with a presentation of the

WorldWideScience.org

international federated search science portal.

data-driven science - Lee Dirks - June 22 - ICSTI 2007 Nancy

Lee Dirks - Director, Scholarly Communication - Microsoft
"Open access, data-driven science & the impact on research communication"

* basic research ACTIVITY unchanged
  but output options dramatically changed
  - blogs
  - wikis
  - scholarly journals
  - IRs
  - discipkine repositories
  - podcasts

Current Issues vs. Anticipated Trends

* OA to scientific content, specifically data, will become the norm

* international cross-discipline research facilitated by interoperable standards

* "evolved" methods of peer review will be adopted

* preservation of data will become a requirement

* services develop around scientific content and prevail over pure publishing
  - data analytics, publishing workflow tools, long term storage/access

EDUCAUSE "Horizon Report" 2007 - for higher education IT in USA

* key trends
  - academic review and rewards are increasingly out of sync with new forms of scholarship
  - the notions of collective intelligence and mass amateurization are pushing the boundaries of scholarship
* critical challenges
  - assessment of new forms of work
  - isses of IP and copyright continue to affect how scholarly work is done

OA momentum

... S.2695

Blogging
- example: useful chem
- recording experiments that fail

Wikis for Sharing Lab Protocols
- example: OpenWetWare

Bookmarks
- example: Connotea

IR
- 1400+ repositories worldwide

Influence of IRs

http://www.webometrics.info/top3000.asp

The Promise of Data Sharing

PLoS article - Sharing Detailed Research Data Is Associated with Increased Citation Rate

"this is going to radically change science"

ISSUES
- data integration and interop
- annotation
- provenance & quality
- exporting/publishing in agreed formats
- security

"an aspect of competitive differentiation"

Publications as Live Documents

MS will have some results on this later this year

* helps with reproducibility if you can get to the raw data, simulations etc.

Trend: The Rise of Mass Collaboration

* Novartis released all its raw data on genetics of type 2 diabetes

[missed the end of the presentation]

Community peer review in Wiki environment - Christine Chichester - June 22 - ICSTI 2007 Nancy

This is very cool stuff, if they ever manage to make it all work.

Christine Chichester - Knewco Inc.
"Community peer review in Wiki environment"

http://www.wikiprofessional.info/

? OmegaWiki

goal: distill down every unique scientific concept to a unique identifier (the "knowlet")

Many challenges in current biomedical research
* volume of data
* complexity
* distributed systems and databases
* incompatible data formats
* multi-disciplinary
** multi-linguality

[Brazilian Portugese sp etc]

*** ambiguity of terminology
* inability to share knowledge

"Too much to read" indicates major trends

* from reading to consulting
* from reading to meta analysis
* from texts to facts
  ... to central and community annotation

Synonym issue

difficult homonym disambiguation issue: use context
- first order symantic enrichment

a knowlet is a triple
* types
  - facts
  - wiki annotation
  - co-occurrence
  - concept profile match
  - sequence similarity
  - co-expression

build an association matrix for large data sources

- disambiguation of author names
[Dr. somebody has algorithms]

1 million disambiguated authors
- from MEDLINE

1 million for genes, drugs and ?proteins

Assignment of protein function and discovery of new nucleolar proteins based on automtic analysis of medline
Martijn Schuemie, Christine Chichester, Frederique Lisaceck, Yahoo Coute, Peter-Jan Roes, Jean Charles Sanches, Barend Mons
Special issue Systems Biology in Protemics, 2008 (in press)

put discovered hypotheses into WikiProf and then if approved into e.g. swissprot

WikiProteins
WikiMedical
WikiPhysics
Wiki Authors
...

databases
- uniprot
- GO gene ontology
- intact
- NLM UNMS???  UNLS?

runs on OmegaWiki which uses MediaWiki

[knowledge space knowlet thing]

Wikiproteins Peer Review: ??? automated selector/requestor for peer review of annotations ???

Institutions, repositories and research assessment - Tim Brody - June 22 - ICSTI 2007 Nancy

He presented a model by which future assessment could be more automated.

Tim Brody - University of Southampton
"Institutions, repositories and research assessment"

Intro to UK RAE

* RAE 2008
- submission deadline November 2007
- for 2009 funding onwards

http://www.rae.ac.uk/

Individuals' Measures
* subject-specific research outputs
* for most researchers: 4 self-selected published papers per research staff member
* "measures of esteem": editorships, awards, conferences

Submitting to RAE

* Scanned PDF or DOI

special deal with publishers to permit scanning to PDF and sending
if they don't have a paper copy,
they can order doc online from BL, but don't have rights to submit that PDF,
so they print it and scan it again

[a completely mad example of publisher rights insanity]

panel members read papers
e.g. 1000 papers per panel member

beyond 2008... mostly metrics based

* Open (Access) Research Metrics?

1. Researchers self-deposit or publish in OA journals
2. Metrics services harvest full text, citation links, and aggregate downloads
3. Funding agencies extract and generate reports
[4. PROFIT!]

[Tim Brody's page]

What Metrics
* if the data are *OPEN ACCESS* anyone can experiment
* page rank
* downloads/cites comparison

Experiments with Google Scholar
* experiment undertaken to provide some metrics for the ECS department's measures of esteem submission

Technical Implementation
* query Google Scholar
* etc.

[again unique identifiers are important]

From ad hoc evaluation to monitoring systems - Stefan Hornbostel - June 22 - ICSTI 2007 Nancy

Stefan Hornbostel - DFG, Institute for Research and Quality Assurance (IFQ)
"From ad hoc evaluation to monitoring systems"

http://www.forschungsinfo.de/

2005 IFQ

Types of Activities

* Funding Monitor
- database with web frontend
- including public information

reports generated from database
also plan to use it for internal project management

future plans
- store final report documents
- link to repositories
- generate a scientists directory

* ProFile online survey
- database of new PhDs
- career development
- etc.

research evaluation - Jerry Sheehan - June 22 - ICSTI 2007 Nancy

Jerry Sheehan - National Library of Medicine
"Research Evaluation: Evolving policies and practices for assessing impact"

The changing policy context for research evaluation

* Growing recognition of links between science, innovation, economic growth, health etc.

* Increased emphasis on evaluation of institutions and their research output

OECD Main Science and Technology Indicators

Changes in Governance of Public Research

* from funding basic research... to governing the science system
- increased priority setting
- increasing role of business and social groups
- new types of funding schemes
- targetting of collaborative activities
- new missions for research organisations
  - contributions to industry and society
  - knowledge translation

Increased emphasis on all levels of evaluation
- ex-ante, ex-post and in-process

- incorporation of evaluation results into policy making

Types of S&T Indicators are changing
- outputs and quality measures
- process indicators
- outcomes and impacts

Measuring Agency/Government Funding Impact

- does it change the research that is done, e.g. more challenging research

Evaluation at NIH

- Office of Portfolio Analysis and Strategic Objectives (OPASI)
* Office of Extramural Research
- Database to track grants from start to finish

Institutional and disciplinary archives a key element

* NLM's PMC and NIH Public Access Policy

* Knowledge infrastructure

http://www.oecd.org/sti/innovation

Scholarly impact: from ranking to assessment - Johan Bollen - June 21 - ICSTI 2007 Nancy

This is some very interesting work and a huge project that should greatly enrich our understanding of the usage of scientific information.

Johan Bollen - Los Alamos National Laboratory
"Scholarly impact: from ranking to assessment"

Scholarly evaluation matters
- qualitative and quantitative indicators

many features in scholarly status space
- prestige
- novelty
- visibility
etc.

[MESUR project]

various opportunity to extract metrics in the scholarly life-scucle
- usage data
- review data
- citation data

usage data is available before citation data

public.lanl.gov/jbollen/Publications.html

From ranking to assessment

we're in mode ranking 0.6
- single data source
- single criterion

... to assessment 3.0 [yecch]

- situate item in value landscape
- multiple sources of scholarly information

question: which dimensions to choose?

1) MESUR project
- survey wide range of possible indicators
2) Peer review
- study peer review process

Marko Rodriguez and others

Can we improve on citation data and the impact factor?

- perhaps usage data applies to a larger subset of the scholarly community,
  capturing more scholarly objects and activities beyond journal articles

usage: COUNTER, IRS [?], SUSHI, CiteBase

MESUR: Metrics from Scholarly Usage of Resources

1 ontology to model the scholarly process
2 beg for usage data
3 dedupe
4 create semantic network

2/5th through the project

[ontology]

data 700million usage events and 1 bilion citations
10-15 billion triples

COUNTER logs, item-level data, SFX, etc.

link resolver data very good

[paper in JCDL 2006 about link resolver data gathering architecture]

they are using Franz's AllegroGraph triplestore

Network usage: usage graphs

"we should stop counting: we should look at relationships"

journal network - 50,000 journals

Example: Flow of information

[pretty network]

Metrics survey

many large organizations and sites are participating

U Texas case study...

[my comment: but isn't there an undergrad effect based on the articles they are assigned?]

principle component plot

[paper at JCDL 2007]

many issues and challenges

quality evaluation of scientific publications - Denis Jérome - June 21 - ICSTI 2007 Nancy

Denis Jérome - CNRS, Académie des Sciences
"Evaluation based on scientific publications: experiences in physics"

* public funding is needed for basic research
* evaluation is needed
* can one use publications to evaluate research

[chart showing 64% of (european) physics letters published in US]

Paradox

* Europe is the first (largest) contributor to physics publications
* yet EU is a minor actor for scientific publications

A Mandatory Plurality

* an overwhelming concentration is dangerous
* need a variety of editorial policies

The need for evaluation

* peer review [of grants, and scientists]
* but also bibliometrics

* e.g. Impact Factor

IF is an indicator for publishers
*** misuse of IF for individual evaluation ***

Nature: 25% of articles receive 90% of citations

Nature & Science only have small number of physics papers therefore:
ban IF for evaluation

IF is about journal popularity, not about the actual citations

Need indications about quality

Physicists publish mostly in small number of journals listed in Web of Science

Databases

Google Scholar
Scopus
NASA ADS adsdoc.harvard.edu
ISI Thomson

Hirsh Index: H

Leo Egghe, 2006 "G" index

analysis: G seems to be more reliable than H

Grain of Salt

* clean scientist names [need unique scientist numbers]
* self-citations
* team work
* negative citations
* cronyism
* quality of citations

must be handled by scientists

my presentation at ICSTI 2007 Nancy

My presentation is on Slideshare

http://www.slideshare.net/scilib/web-tools-for-peer-reviewers-and-everyone

you can also download the PowerPoint directly from there.
It's in the Creative Commons.

The title is not so great.

It's mostly about categorizing the problem space of journal article exploration, and what new features or metrics we might use in this new space, as well as what new scholarly objects we might apply the certification of peer review to. 

All the links from the presentation are available at

http://www.connotea.org/user/scilib/tag/icsti2007akerman

The presentation file as well as video of the presentation should be up on the ICSTI site soon hopefully.
You'll get to watch me deal with the projector failing unrecoverably mid-presentation.

UPDATE 2009-02-17: The video is available.

open access publishing and collaborative peer review - Ulrich Pöschl - June 21 - ICSTI 2007 Nancy

Ulrich presented a very interesting open review model for publications, unfortunately his talk was a bit rushed due to factors outside his control.  Definitely an approach worth investigating further.

Ulrich Pöschl - Max Planck Society
"Interactive open access publishing and collaborative peer review for improved scientific communication and quality assurance"

www.mpch-mainz.mpg.de/~poeschl

* many motivations to do open access
- improve scientific quality assurance

with OA you can do collaborative peer review

problems with scientific publications
- fraud
- carelessness

speed vs quality
- but then neglect thorough review

Two-stage OA publication with collaborative peer review

www.egu.eu
* they [the journals] are financially viable
* they have good impact factor

Bernard F. Schultz - Albert Einstein Institute - Future styles? of assessment

Vision

- OA to high quality scientific publications
- documentation of scientific discussion (e.g. publish referee comments)
- demonstration of transparency and rationalism

Proposition

- prescribe OA to publically funded research
- transfer funds for subscription to OA
- foster OA publishing and collaborative peer review
- mere access is not enough (need to get all layers, data etc.)
- evaluate individual papers
- refine statistical parameters for citation, downloads, usage, interactive commenting and rating

scientific visibility and IF - Bruno Granier - June 21 - ICSTI 2007 Nancy

Bruno Granier - University of Western Brittany
"Impact of research assessment on scientific publication in earth sciences"

- Misuse of IF [impact factor]
citation: "The number that's devouring science"

- The Goal

* the only common goal is how visible you are...
because visibility is the qualitative factor used to assess your work

he started an OA journal - Notebooks in Geology

As an author
- (particularly in industry) you may want paper published ASAP
  - you may reiterate your message in other publications
- in academe you want impact factor

ways to increase visibiity
- slicing
- bogus signatures / invitation
- author names appear in alternate positions in similar papers
- selective or inexact quotations
- self-citation
- cutting and pasting
- lift information

Evaluators should use weighted averages for multi-author papers, 1st authorship worth much more

Question: How to detect frauds?

Answer: You need a good reviewer

As a reviewer

- the reviewer remains the sine qua non of the evaluation process

As an editor

- blacklist repeat offender authors
- use computer programs to detect plagiarism

- often citations are incorrect or not relevant

As a publisher

- OA gives happy google effect

- monitoring
  shall i use new tools/facilities (couunters) to discriminate the kind of papers that
  get the larger readership

Impact Factor

- a huge part of the scientific inofmration is not given any consideration,
since IF covers only well-established journals

The use and misuse of metrics is responsible for the death of many lab, museum etc. publications in
France and elsewhere.

Conclusion

- bibliometrics or not, the only goal remains to increase your visibility

- the Google effect is at our doors

peer review and research assessment - 2007 ICSTI Public Conference

The 2007 ICSTI Public Conference "Assessing the quality and impact of research: practices and initiatives in scholarly information" will be

June 21-22, 2007
Nancy, France

I am pleased to announce that I will be speaking about peer review and certification.
(Following Elsevier, the largest scientific publisher in the world, so, you know, no pressure.)

You can check out the preliminary programme; more info is available on the ICSTI site.

I will blog this event under category/tag ICSTI2007.

Following the event, there is the opportunity to tour INIST on the afternoon of the 22nd.

(Note that the ICSTI General Assembly, which is closed to the public, continues for two days after the public conference.)

Getting There

There is a new TGV line running direct Paris-Nancy that is supposed to open June 10, 2007; I can see the train schedules at voyages-sncf.com, but it won't let me book them.  The new line is TGV Est Européen, and the TGV Est site says tickets will be bookable starting April 10, 2007.  Also, if I'm reading correctly, there are discounted prices from June 10, 2007 to August 26, 2007.  If the TGV is running, it will only be 1.5 hours for the Paris-Nancy run, e.g.

Otherwise if you don't take a direct, it is showing times of about 3 hours in June.  Currently, without the new line, it is showing direct times of about three hours, and non-direct times of about 4 hours.  I think you can reserve and pick up your tickets in France.  Otherwise for Canada and USA, the site is Rail Europe, but it doesn't show the new TGV schedules for June.  For Canada see specifically

http://www.raileurope.ca/canada/

For booking the new TGV East from Canada there is a new page, but just to make things a hassle, it doesn't list Nancy as an option in the pull-down menus

http://www.raileurope.ca/canada/rail/tgvest/

UPDATE 2007-05-10: The TGV Est leaves from Gare de l'Est (PARIS EST).  To get there from Charles de Gaulle (CDG) Roissy airport, you can take the "Roissyrail" RER B train from the airport to Gare du Nord, and then walk to Gare de l'Est (it is just a few hundred metres away).  You can either go outside at Gare du Nord, or there is supposed to be a connecting hallway ready by the time the TGV Est starts running.

The RER B is easy to get from within the airport, you can buy tickets at automated kiosks or line up to buy them from the SNCF ticket counters, you can of course also order them in advance online.  It should only take about half an hour from CDG to Gare du Nord (although you might want to figure an hour, just to be safe).  There are two trains, a local and an express, they leave from opposite sides of the same platform.  Look at the signs beside each track - dots will indicate the stops the train makes.

ENDUPDATE

UPDATE 2007-06-17: You can check the status of the train lines, as well as individual trips, via

http://www.infolignes.com/

specifically you can check SNCF - Infolignes - Trafic Est

ENDUPDATE

UPDATE 2007-04-17: I have made a Google Earth KMZ of the suggested hotels and the INIST venue (which is about 5km from Nancy centre).

Download ICSTI2007.kmz

You can also view these placemarks in Google Maps.

By copying the placemarks using "Save to My Maps", I have also created a Google My Maps version.

The INIST site has information both on how to get to Nancy and how to get to INIST itself.

The tram1 runs directly from downtown to INIST.  The INIST site says to take it in direction Vandoeuvre CHU Brabois and get off at the Faisanderie stop.

See reseau-stan.com for more information on Nancy area transit.

Disclaimer: I'm approximating the location of INIST based on what I could find on their website; it's possible the location is off slightly as I've never been there.

ENDUPDATE

Previously:
January 22, 2007  ICSTI event on academic user behaviour
April 28, 2006  conference proceedings: ICSTI eScience, Bielefeld + Taiga + LtF on academic library future

Note:
ICSTI = International Council for Scientific and Technical Information
CISTI = Canada Institute for Scientific and Technical Information
INIST = L’Institut de l’Information Scientifique et Technique