Fact verification

Learning to generate answers with citations via factual consistency models

Rami Aly, Zhiqiang Tang, Samson Tan, George Karypis

ACL 2024

2024

Large Language Models (LLMs) frequently hallucinate, impeding their reliability in mission-critical situations. One approach to address this issue is to provide citations to relevant sources alongside generated content, enhancing the verifiability of generations. However, citing passages accurately in answers remains a substantial challenge. This paper proposes a weakly-supervised fine-tuning method leveraging

Conversational AI

Factual confidence of LLMs: On reliability and robustness of current estimators

Matéo Mahaut, Laura Aina, Paula Czarnowska, Momchil Hardalov, Thomas Müller, Lluís Marquez

ACL 2024

2024

Large Language Models (LLMs) tend to be unreliable in the factuality of their answers. To address this problem, NLP researchers have proposed a range of techniques to estimate LLM’s confidence over facts. However, due to the lack of a systematic comparison, it is not clear how the different methods compare to one another. To fill this gap, we present a survey and empirical comparison of estimators of factual

Conversational AI

Model assesses the validity of tips offered in product reviews

Lital Kuchy, Avihai Mejer

September 02, 2022

Method would enable customers to evaluate supporting evidence for tip reliability.

Conversational AI

FactGraph: Evaluating factuality in summarization with semantic graph representations

Leonardo Ribeiro, Mengwen Liu, Iryna Gurevych, Markus Dreyer, Mohit Bansal

2022

We propose FACTGRAPH, a method that decomposes the document and the summary into structured meaning representations (MR), which are more suitable for factuality evaluation. MRs describe core semantic concepts and their relations, aggregating the main content in both document and summary in a canonical form, and reducing data sparsity. FACTGRAPH encodes such graphs using a graph encoder augmented with structure-aware

Conversational AI

Paragraph-based transformer pretraining for multi-sentence inference

Luca Di Liello, Siddhant Garg, Luca Soldaini, Alessandro Moschitti

2022

Inference tasks such as answer sentence selection (AS2) or fact verification are typically solved by fine-tuning transformer-based models as individual sentence-pair classifiers. Recent studies show that these tasks benefit from modeling dependencies across multiple candidate sentences jointly. In this paper, we first show that popular pre-trained transformers perform poorly when used for fine-tuning on

Conversational AI

Paragraph-based transformer pre-training for multi-sentence inference

Luca Di Liello, Siddhant Garg, Luca Soldaini, Alessandro Moschitti

NAACL 2022

2022

Inference tasks such as answer sentence selection (AS2) or fact verification are typically solved by fine-tuning transformer-based models as individual sentence-pair classifiers. Recent studies show that these tasks benefit from modeling dependencies across multiple candidate sentences jointly. In this paper, we first show that popular pre-trained transformers perform poorly when used for fine-tuning on

Conversational AI

FactGraph: Evaluating factuality in summarization with semantic graph representations

Leonardo Ribeiro, Mengwen Liu, Iryna Gurevych, Markus Dreyer, Mohit Bansal

NAACL 2022

2022

Despite recent improvements in abstractive summarization, most current approaches generate summaries that are not factually consistent with the source document, severely restricting their trust and usage in real-world applications. Recent works have shown promising improvements in factuality error identification using text or dependency arc entailments; however, they do not consider the entire semantic

Conversational AI

FEVEROUS (2021)

Rami Aly, Zhijiang Guo, Michael Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, Arpit Mittal

2021

Fact verification has attracted a lot of attention in the machine learning and natural language processing communities, as it is one of the key methods for detecting misinformation. Existing large-scale benchmarks for this task have focused mostly on textual sources, i.e. unstructured information, and thus ignored the wealth of information available in structured formats, such as tables. In this paper we

Conversational AI

Christos Faloutsos awarded PAKDD’s “Most Influential Paper Award”

Staff writer

December 18, 2020

Amazon Scholar and Carnegie Mellon professor of artificial intelligence, Christos Faloutsos

Amazon Scholar and Carnegie Mellon professor of artificial intelligence and coauthors honored for paper that proposed a new approach to detecting anomalies in large, weighted graphs.

Information and knowledge management

FEVER 2.0 (2019)

James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, Arpit Mittal

2019

We present the results of the second Fact Extraction and VERification (FEVER2.0) Shared Task. The task challenged participants to both build systems to verify factoid claims using evidence retrieved from Wikipedia and to generate adversarial attacks against other participant’s systems. The shared task had three phases: building, breaking and fixing. There were 8 systems in the builder’s round, three of

Conversational AI

Fact verification

Work with us