Invited Speaker: Goran Glavas

Title: Pitfalls in Evaluation of Multilingual Text Representations [slides] [video]

Abstract: Multilingual representation spaces, spanned by multilingual word embeddings or massively multilingual transformers, conceptually enable modeling of meaning across a wide range of languages and language transfer of task-specific NLP models from resource-rich to resource-lean languages. It is not yet clear, however, to which extent this conceptual promise holds in practice. Recent models, both cross-lingual word embedding models and multilingual transformers, have been praised for being able to induce multilingual representation spaces without any explicit supervision (i.e., without any word-level alignments or parallel corpora). In this talk, I will point to some prominent shortcomings and pitfalls of existing evaluations of multilingual representation spaces, which mask the limitations of state-of-the-art multilingual representation models. Remedying for some of these evaluation shortcomings, portrays meaning representation and language transfer capabilities of current state-of-the-art multilingual representation models in a less favorable light.

Invited Speaker: Ido Dagan

Title: Think out of the box! Conceptions and misconceptions in NLP evaluation [slides]

Abstract: Continuity is an important principle in empirical evaluation; otherwise -- how can we compare performance along time and measure progress? Yet, in some cases, we may reconsider common evaluation practices and examine whether they deserve an extension or a change, due to being either limiting or misleading. In this talk I’ll describe four such recent experiences regarding data collection and evaluation practices: a controlled crowdsourcing methodology for more challenging data collection tasks, revisiting an old “same length” assumption for reference and system summaries, extending summarization evaluation to the interactive summarization setting, and examining the practice of considering singleton annotations in cross-document coreference evaluation.

Invited Speaker: Asli Celikyilmaz

Title: Meta-Evalution of Text Generation

Abstract: Automatic text generation enables computers to summarize text, describe pictures to visually impaired, write stories or articles about an event, have conversations in customer-service, chit-chat with individuals, and other settings, etc. Recent advancements in deep learning have yielded tremendous improvements in many text generation tasks. While these models have dramatically improved the state of text generation, even state-of-the-art neural text generation models still face many challenges: a lack of diversity in generated text, commonsense violations in depicted situations, difficulties in determining if the generated text is coherent, factual, or makes any sense at all. In this talk I will briefly discuss current evaluation metrics for text generation focusing on factuality evaluation. Then I will talk about our recent work on building standards for evaluating different types of factuality metrics and conclude with our findings and avenues for future research in this direction.

Invited Speaker: William Wang

Title: Rethinking NLG Evaluation: From Metric Learning, Sample Variance, to Fact-Checking [slides] [video]

Abstract: Existing metrics for natural language generation evaluation focus on counting lexical overlaps between generated sentences and references. However, it is well-known that this evaluation paradigm has major issues in evaluating long stories, the stability of result reporting, and catching semantic correctness and faithfulness. In this talk, I will discuss some of our recent attempted solutions to mitigate these issues. Specifically, I will introduce (1) AREL, a new paradigm for joint metric learning and policy optimization in visual storytelling; (2) our in-depth study into the understanding of sample variance in visually-grounded language generation problems; (3) a new logicNLP task that requires logical reasoning to access the faithfulness of generation results.