A Retrospective Analysis of the Fake News Challenge Stance-Detection Task
Abstract. Stance classification is a crucial first step in fake news detection. In this problem setting, the goal is to determine the relative stance of a source document to that of a target text. In 2017, the Fake News Challenge (FNC) addressed this particular task in a shared task. Given that there is yet no analysis paper, we reproduce the results of the top three systems from the FNC, critically assess their performance in an error, feature, and model analysis. We also discuss the dataset, the evaluation metric and the task definition itself and test the models for generalizability and transferability using a second, newly derived dataset. Based on our insights, we propose a novel feature-rich stacked LSTM model which achieves on par with the state-of-the-art systems, but is better able to correctly classify difficult cases.