Informatie

student.uva.nl is vernieuwd. Lees meer...

Welke opleiding volg je?
Lezing

Presentation Master's thesis - Friedrich Ziehm - Psychological Research Methods

Laatst gewijzigd op 19-09-2022
Automatically testing the quality of freetext answers with Gpt-3
Toon informatie voor jouw opleiding
Nu zie je algemene informatie op deze site. Kies je opleiding om ook informatie te zien die specifiek voor jouw opleiding geldt, zoals deadlines, regelingen en contactgegevens.
Welke opleiding volg je?
Startdatum
23-09-2022 09:00
Einddatum
23-09-2022 10:00
Locatie

This is an online presentation. You can register hereExterne link

Poor data quality is a problem for industrial and academic research resulting in immense costs and unreliable research findings. Many survey participants answer questionnaires carelessly, thereby reducing data quality. Techniques have been developed to detect such participants, but these suffer several shortcomings. This study proposes implementing qualitative control questions (QCCs), which need to be answered in open text format as a quality measure. Because the interpretation of open-ended answers is performed mostly manually during the time of writing, it was investigated whether the quality of a question-answer pair can be evaluated automatically by Gpt-3. A dataset was generated consisting of reasonable and unreasonable question and answer pairs. 

One zero-shot model, receiving no training data, and two fine-tuned models, trained on 50 examples, were fitted. Model accuracies in correctly classifying the quality ranged from 85% to 92%. Results suggest that automatic classification of open-ended answers into high vs low quality is possible using Gpt-3. However, model classification accuracies are not high enough to propose full automatization. It is argued that researcher use it as supplemental tool and ensure the correctness of the results when applying QCCs.