Graduate School of Informatics /Cognitive Sciences

In partial fulfillment of the requirements for the degree of Master of Science Faruk Büyüktekin will defend his thesis.

Title: MULTI-MODAL REFERENCE RESOLUTION IN SITUATED DIALOGUE THROUGH LINGUISTIC AND EXTRA-LINGUISTIC INFORMATION

Date: 31st August 2018

Time: 10:00 AM

Place: A-108

Thesis Abstract : This thesis investigates the effect of extra-linguistic information incorporated into linguistic information on multimodal reference resolution in Turkish. Identifying reference relations such as anaphora and coreference within texts is quite critical in the fields of natural language processing and human computer interaction since such applications require interpreting the text. Previous research on reference resolution has employed mostly hand-crafted rule-based approaches. However, due to the success of machine learning algorithms, there has been a shift towards corpus-based approaches. Therefore, investigations of reference resolution within real world situations have received a great deal of attention. In line with these developments, this study focuses on a Turkish corpus which is based on the dialogues of two participants collaboratively solving a Tangram puzzle. The corpus includes eye gaze and mouse operations of the participants along with their utterances. This study offers a model that identifies the referent (the object) of a referring expression in the real world by taking eye gaze and operations on puzzle pieces into account. The model is built making use of machine learning algorithms. The model also examines which type of extra linguistic information is more successful in determining the referents of pronouns, non-pronouns and both within a machine learning based-approach.