Teacher shortages and high workloads are a few of the many issues the educational sector faces. Automatic scoring which has a long history, initially being done with multiple choice questions is being used and further explored to remedy these issues. Modern large language models open up new opportunities for automatically scoring short open questions. We test different variations of an existing rule-based automatic scoring model over three different variables: spellchecking, preprocessing and choice of language model with the goal of obtaining overall results and results separated over six different subjects. In addition, we compare the existing rule-based automatic scoring model to a support vector machine, a popular text classification algorithm. Finally, we examine the relation between the response features and four metrics: Cohen’s kappa, proportion of scorable responses, sensitivity and specificity. We find that the gains in performance in variations across subjects, is greater than the averaged performance over subjects. Text and token length have a positive relation with the performance of the base model.