A new study published in Frontiers in Education suggests that how assessment criteria are written and presented to artificial intelligence systems can significantly improve the reliability of AI-assisted grading, at least in controlled settings. The research, led by Deepshikha of Queen Mary University of London and Navpreet Singh of QBYSS Ltd, examined whether a more explicit rubric and a carefully engineered marking prompt could bring AI-generated marks closer to those assigned by a human educator.
According to the study, the researchers compared educator-assigned marks for 40 graduate-level laboratory reports in Electrochemical Energy Storage Systems with marks generated by OpenAI's GPT-5 model under two different configurations. In the first configuration, the AI used a conventional rubric with standard descriptions. In the second, the researchers introduced an enhanced, criterion-referenced rubric and a zero-assumption, penalty-structured prompt designed to reduce ambiguity.
The results showed measurable improvements after calibration. The intraclass correlation coefficient — a statistical measure of agreement — rose from 0.818 to 0.928, indicating stronger concordance between AI and educator scores. The mean difference between AI and educator marks shrank from +1.625 to +1.041 marks, and no post-calibration report differed from the educator's mark by more than 10 marks. The study reported that the learning outcomes, task design, student submissions, and educator marks remained unchanged between the two conditions, allowing the researchers to attribute the improvement to the combined calibration strategy.
For university students, the findings carry implications for how assessment feedback might become more consistent and timely. If AI-assisted grading can be made more reliable through better specification of criteria, students could potentially receive faster and more detailed feedback on technical assignments without sacrificing fairness. However, the study's authors are careful to note the limitations of their work. The revised rubric and prompt were introduced together, making it impossible to isolate which component drove the improvement. Calibration drew on discrepancies observed in the same 40 reports used for post-calibration evaluation, meaning the results reflect improved in-sample agreement rather than generalised accuracy. Additionally, only one educator's marks served as the reference standard, so the study does not establish grading validity or interchangeability for autonomous summative decisions.
The research is part of a growing body of work exploring the role of assessment specification in AI grading. The study's introduction notes that much existing literature treats grading accuracy as a function of model capability, but emerging research suggests that how marking criteria, performance thresholds, and evidentiary requirements are translated into machine-interpretable form can materially influence outcomes. The authors cite studies showing that prompting strategies such as few-shot learning and chain-of-thought reasoning can enhance scoring accuracy, and that frameworks like "Grade Like a Human" have achieved improved concordance with expert markers through explicit rubric alignment.
Laboratory-based STEM assessment presents particular challenges for AI grading, the study notes. Effective marking requires simultaneous consideration of conceptual understanding, quantitative correctness, unit consistency, appropriate use of experimental data, and adherence to disciplinary norms. Human examiners routinely apply tacit judgement when deciding whether computational errors invalidate conclusions or how to penalise missing steps. Translating these practices into explicit rules suitable for AI systems is non-trivial, and insufficient formalisation risks systematic divergence between AI and educator judgements.
The study also draws a clear distinction between formative and summative uses of AI in assessment. While AI may be well-positioned to support assessment processes — for example, by providing detailed feedback that students can act on — the authors caution that AI's consistent bias patterns render it currently unsuitable as a sole grading tool for consequential summative decisions that determine student progression and certification. They cite research showing that AI systems tend to grade low-performing essays more leniently and high-performing ones more harshly, indicating systematic proportional bias.
For students, the practical takeaway is that AI-assisted grading is not yet ready to replace human judgement in high-stakes assessments, but the technology's reliability can be improved through careful design of the criteria and instructions it receives. As universities continue to experiment with AI in assessment, students may encounter more consistent feedback on technical assignments, though the final say on grades is likely to remain with human educators for the foreseeable future.