CLEF 2008: Introduction

QA WSD @ CLEF 2008

Word Sense Disambiguation for Question Answering


Following the 2007 joint SemEval-CLEF task , a related pilot task is planned with the aim of exploring the contribution of Word Sense Disambiguation to Question Answering. In this case, the organizers of the task will provide document collections (from the news domain) and topics which have been automatically tagged with Word Senses from WordNet using several state-of-the-art Word Sense Disambiguation systems. Robust WSD CLIR @ CLEF is a closely related task on robust retrieval.

News

  • 2009: The CLEF QA exercise has changed, and thus the QA-WSD exercise won't be repeated.
  • Sep 13: working notes available

  • Apr. 24: New dates for submission (see below)

  • Apr. 14: Small bug in WSD results of UBC fixed. Please download again

Mailing list

  • You can browse the e-mail list. To join enter your e-mail in form to the left, and press subscribe.

Instructions for participation

Please note the following steps in order to participate:

1. Please read the main QA task guidelines and the specific guidelines for the QA WSD task (doc).
2. Registration is via the CLEF website and the QA website . Participants must sign an agreement restricting use of the data and regulating publication and dissemination of results. Registration closes 28 April.
3. Join the mailing list (see subscribe button to the left) for updates.
4. Download the document collection from 1 March.
5. Download the English and Spanish WordNets (see below)
6. Download the test topics from 19 May. (New date)
7. Submit results by 27 May (New date) (details for submission to be released).

IMPORTANT:

  • Remember that the results must be submitted WITHIN 5 days from the first test sets download, and anyway not later than May 27.
  • Thus, 5 days from downloading - and downloading must be done between 19 and 23 May.
  • Any submission after 5 days from the download, or after May 27, will not be accepted.

Time Schedule

- Registration Opens - 15 February 2008 (closes 28 April 2008)
- Release of documents with annotations - from 1 March 2008
- Question Release - 19 May 2008
- Submission of Runs by Participants - 23 May 2008, 12 a.m. (Central Europe Time)
- Release of Relevance Assessments and Individual Results - 1 July 2008
- Submission of Paper for Working Notes - 25 July, 12 a.m. (Central Europe Time) 2008
- Workshop - 17-19 September 2008

Description

The QA-WSD task will bring semantic and retrieval evaluation together. The participants will be offered the same queries and document collections from the main QA exercise which have been annotated by systems for word sense disambiguation (WSD). The goal of the task is to test whether WSD can be used beneficially for Question Answering.

The exercise scenario is event-targeted QA on a news document collection. In the QA-WSD track only English monolingual and Spanish to English bilingual tasks are offered, i.e. English will be the only target language, and queries will be available on both English and Spanish. The queries will be the same as for the main QA exercise, and the participation will follow the same process, except for the use of the sense-annotated data.

A large number of questions will be topic-related, i.e. clusters of questions which are related to the same topic and possibly contain anaphoric references between one question and the other questions. Contrary to the main QA task, Wikipedia articles are not included, and thus systems need to reply to the questions that have an answer in the news document collection.

The goal of this task is to evaluate whether word sense information can help in certain queries. For this reason, participants are required to send two runs for each of the monolingual/bilingual tasks where they participate: one which does not use sense annotations and another one which does use sense annotations. Whenever possible, the only difference between the two runs should be solely the use or not of the sense information. Participants which send a single run will be discarded from the evaluation.

The WSD data is based on WordNet version 1.6 and will be supplemented with data from the English and Spanish WordNets in order to test different expansion strategies. Several leading WSD experts will run their systems, and provide those WSD results for the participants to use.


Data formats, additional files, and source of WSD tags

These are the DTDs for the disambiguated queries and documents. Please check these sample disambiguated query (original ) and document (original).

Spanish topics will be disambiguated using the first sense heuristic. English topics and documents will be annotated with the following word sense disambiguation systems:

1. Agirre, Eneko & Lopez de Lacalle, Oier (2007). UBC-ALM: Combining k-NN with SVD for WSD. Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval 2007). pp. 341-345. Prague, Czech Republic.

2. Chan, Yee Seng, & Ng, Hwee Tou, & Zhong, Zhi (2007). NUS-PT: Exploiting Parallel Texts for Word Sense Disambiguation in the English All-Words Tasks. Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval 2007). pp. 253-256. Prague, Czech Republic.

In order to expand from WordNet synset numbers to words in English and Spanish, you will need the following:

  • The English WordNet version 1.6 available from here
  • The Spanish WordNet, available free for research from here
These are the ad-hoc collections (documents) which were available at CLEF 2001:
- LA Times 94 (with WSD data) : 72,027,935 tokens
- Glasgow Herald 95 (with WSD data): 27,731,946 tokens

Contact

Eneko Agirre, University of the Basque Country, e.agirre at ehu es
Anselmo PeƱas, UNED, anselmo at lsi uned es