Hidden biases in unreliable news detection datasets

Automatic unreliable news detection is a research problem with great potential impact. Recently, several papers have shown promising results on large-scale news datasets with models that only use the article itself without resorting to any fact-checking mechanism or retrieving any supporting evidence. In this work, we take a closer look at these datasets. The code is tested on Python 3.7 and PyTorch 1.6.0. The experiments and results in our paper mainly involve two datasets: NELA and FakeNewsNet.