5TH TEXTUAL ENTAILMENT CHALLENGE @ TAC 2009 TEXTUAL ENTAILMENT SEARCH PILOT Task Guidelines Version March 16, 2009 (Also see general TAC 2009 policies and guidelines at http://www.nist.gov/tac/2009/)
1. INTRODUCTION During the RTE4 Challenge workshop at TAC 2008 the need to move towards more realistic scenarios was stressed once again, both by organizers and participants. In the campaigns held so far, the necessity of getting acquainted with the textual entailment task had motivated proposing only test sets in which the pairs were artificially adapted in order to facilitate the study of the different entailment phenomena; however, the progress that has been made, now allows to make a step forward and start to test RTE systems against real data. In order to meet this demand, a Textual Entailment Search Pilot task has been set up, aimed principally at: 1) producing a data set which reflects the natural distribution of entailment in a corpus and presents all the problems that can arise while detecting textual entailment in a natural setting; 2) analyzing the potential impact of textual entailment recognition on a real NLP application task, namely the Summarization task as proposed by the Summarization community in the 2008 Text Analysis Conference at NIST. This document provides a definition of the Pilot Search Task and a description of the data set, together with the instructions on how to take part in the exercise.
2. TASK DESCRIPTION
The Textual Entailment Search task consists in finding all the sentences in a set of documents that entail a given Hypothesis. The task is situated in the Summarization application setting, where the Hypothesis (H) is taken from a Summary Content Unit1 (SCU), and the systems must find all the entailing sentences (Ts) in a corpus of 10 newswire documents about a common topic. We assume the standard definition of Textual Entailment [Dagan et al., 2006] as a directional relationship between two text fragments, which we term the Text (T) and the Hypothesis (H). We say that:

下一页