Information

student.uva.nl has been renewed. Read more...

What is your study programme?
Talk

Presentation Master's thesis - Friedrich Ziehm - Psychological Research Methods

Last modified on 19-09-2022
Automatically testing the quality of freetext answers with Gpt-3
Show information for your study programme
You're currently viewing general information. Choose your study programme to see additional information that's specific to your study programme, such as deadlines, regulations and contact details.
What is your study programme?
Start date
23-09-2022 09:00
End date
23-09-2022 10:00
Location

This is an online presentation. You can register hereExternal link

Poor data quality is a problem for industrial and academic research resulting in immense costs and unreliable research findings. Many survey participants answer questionnaires carelessly, thereby reducing data quality. Techniques have been developed to detect such participants, but these suffer several shortcomings. This study proposes implementing qualitative control questions (QCCs), which need to be answered in open text format as a quality measure. Because the interpretation of open-ended answers is performed mostly manually during the time of writing, it was investigated whether the quality of a question-answer pair can be evaluated automatically by Gpt-3. A dataset was generated consisting of reasonable and unreasonable question and answer pairs. 

One zero-shot model, receiving no training data, and two fine-tuned models, trained on 50 examples, were fitted. Model accuracies in correctly classifying the quality ranged from 85% to 92%. Results suggest that automatic classification of open-ended answers into high vs low quality is possible using Gpt-3. However, model classification accuracies are not high enough to propose full automatization. It is argued that researcher use it as supplemental tool and ensure the correctness of the results when applying QCCs.