Department of Language Modelling
MSCA Postdoctoral Fellowship

ERINIA – Evaluating the Robustness of Non-Credible Text Identification by Anticipating Adversarial Actions

The fellowship investigated how robust automatic misinformation detectors are against adversarial examples – small, meaning-preserving changes to a text that reverse a classifier's decision.

Lead Investigator: Piotr Przybyła

Grant Number
101060930
Start Date
1 October 2022
End Date
31 October 2024

Project factsheet
English name: Evaluating the Robustness of Non-Credible Text Identification by Anticipating Adversarial Actions
Project type: A Marie Skłodowska-Curie Actions Postdoctoral Fellowship (Horizon Europe, call HORIZON-MSCA-2021-PF-01)
Grant agreement number: 101060930
Duration: 1 November 2022 – 31 October 2024
Fellow: Piotr Przybyła
Host institution: TALN group, Universitat Pompeu Fabra, Barcelona, Spain
Project website: https://www.upf.edu/web/erinia
CORDIS information: https://cordis.europa.eu/project/id/101060930

In the paper presenting the project (2023), Piotr Przybyła is affiliated both with the TALN group at Universitat Pompeu Fabra and with the Institute of Computer Science, Polish Academy of Sciences. His earlier project on misinformation was HOMADOS.

Project summary

As challenges posed by misinformation become apparent in the modern digital society, state-of-the-art methods of Artificial Intelligence, especially Natural Language Processing (NLP) and Machine Learning, are considered as countermeasures. Indeed, previous research has shown that NLP solutions can detect phenomena such as fake news, social media bots or usage of propaganda techniques. However, little attention has been given to the robustness of these approaches, which is especially important in the case of deliberate misinformation, whose authors would likely attempt to deceive any automatic filtering algorithm to achieve their goals.

The goal of the ERINIA project is to explore the robustness of text classifiers in this application area by investigating methods for detecting adversarial examples. Such methods aim to perform small perturbations to a given text piece, so that its meaning is preserved, but the output of the investigated classifier is reversed. To that end, previously unexplored directions will be pursued, including training reinforcement learning solutions and leveraging research on simplification and style transfer. Finally, the developed tools will be used to check the robustness of the current state-of-the-art misinformation detection solutions.

The project includes a range of training activities for the researcher and a plan for dissemination of the obtained results to various research communities. It also takes into account the society at large, as the project outcomes can inform further discussion on whether automatic content filtering is a viable solution to the misinformation problem.

Results

The work in the project was organised into four stages:

  1. Assessing the vulnerability of common text classifiers to generic adversarial attacks. The project created BODEGA (Benchmark for Adversarial Example Generation in Credibility Assessment), an evaluation framework covering various text classification approaches, misinformation detection tasks (news bias assessment, propaganda detection, fact checking and rumour detection) and adversarial example generators. The experiments showed that popular text classifiers are very vulnerable: depending on the task, between 60% and 90% of text samples could be manipulated so that the victim’s credibility decision changed, and classifiers built by fine-tuning large language models were not more robust than their simpler antecedents.
  2. Attacks tuned specifically for misinformation detection. The project organised InCrediblAE, a shared task on the robustness of credibility assessment within the CheckThat! lab at CLEF 2024. Six university teams generated adversarial examples in an environment based on BODEGA, and their solutions outperformed the generic approaches, establishing a new state of the art for the task.
  3. Adaptive generation of adversarial examples with reinforcement learning. XARELLO is a generator that observes successful attacks and tunes its output to a particular victim classifier. It showed that an attacker who can interact with the victim model finds better adversarial examples with far fewer attempts – in the fact-checking task, a successful example was found in 5–7 queries, compared to 130–150 for the previously best approach.
  4. Improving meaning preservation with large language models. In the TREPAT approach, large generative models provide initial paraphrases, which are decomposed into small changes applied to the original text until an adversarial example is found. When only a limited number of queries to the victim model was allowed, TREPAT performed best, especially on long texts.

Selected publications