DKPro Agreement
An Open-Source Java Library for Measuring Inter-Rater Agreement
Abstract. In this paper, we introduce a novel Java implementation of multiple inter-rater agreement measures, which we make available as open-source software. Besides assessing the reliability of coding tasks using S, π, κ, α, etc., we particularly support unitizing tasks by measuring αU as the agreement of the boundaries of the identified annotation units. We provide a unified interface and data model for both tasks as well as multiple diagnostic devices for analyzing the results.
Submitted: 16.05.2014 | Published: 23.08.2014