Data quality in the context of market research is one of the biggest current challenges of online research. While gathering data and collecting responses from large enough samples has become much easier over the last decade, low participant engagement and use of bots to automatically fill out online surveys prevails to be an issue. This study aimed at coming up with an automated way of classifying open-text survey responses as good or bad quality, based on the data labelled previously by human raters, and see whether it is possible to achieve human-like performance with this approach. To do this, an ensemble algorithm trained on word embeddings was proposed, together with an extension of an active learning paradigm, which utilized the query-by-committee approach to select the most uncertain predictions for relabelling by a human rater. This procedure was simulated for different methods of query selection, final prediction and query batch sizes. The final results showed that while a standard ensemble algorithm performed worse than humans, with interrater reliability at 92%, and the ensemble’s accuracy at 87%, the implementation of active learning improved the accuracy to 94%.