student.uva.nl
Welke opleiding volg je?
UvA Logo
Welke opleiding volg je?
Colloquiumpunten

Presentation Master's thesis - Nina Koperska - Psychological Methods

Colloquiumpunten

Presentation Master's thesis - Nina Koperska - Psychological Methods

Laatst gewijzigd op 19-10-2024 16:02
Automating open text quality assessment using an ensemble active learning algorithm
Toon informatie voor jouw opleiding
Welke opleiding volg je?
Startdatum
09-08-2024 11:00
Einddatum
09-08-2024 12:00
Locatie

Roeterseilandcampus - Building G, Street: Nieuwe Achtergracht 129-B, Room: G3.03

Data quality in the context of market research is one of the biggest current challenges of online research. While gathering data and collecting responses from large enough samples has become much easier over the last decade, low participant engagement and use of bots to automatically fill out online surveys prevails to be an issue. This study aimed at coming up with an automated way of classifying open-text survey responses as good or bad quality, based on the data labelled previously by human raters, and see whether it is possible to achieve human-like performance with this approach. To do this, an ensemble algorithm trained on word embeddings was proposed, together with an extension of an active learning paradigm, which utilized the query-by-committee approach to select the most uncertain predictions for relabelling by a human rater. This procedure was simulated for different methods of query selection, final prediction and query batch sizes. The final results showed that while a standard ensemble algorithm performed worse than humans, with interrater reliability at 92%, and the ensemble’s accuracy at 87%, the implementation of active learning improved the accuracy to 94%.