Anapproach to assess the content relevance of LLM-based chatbots in software testing education

Resumo

Large Language Model (LLM)-based chatbots have been increasingly adopted by students across educational levels as support tools for solving exercises, clarifying doubts, and deepening con ceptual understanding. In the context of software testing education, recent studies have explored the potential of these systems as mechanisms to enhance learning. However, as LLMs are integrated into educational activities, ensuring the reliability and conceptual accuracy of their responses becomes es sential. One major challenge is that poorly formulated or ambiguous questions can lead to conceptually incorrect answers. Moreover, LLMs are prone to hallucinations, producing linguistically coherent yet factually inaccurate responses, which poses a significant risk of misinformation. This paper proposes a systematic approach based on BERTScore to assess the domain relevance of responses generated by LLM-based chatbots in conversations about software testing. The approach semantically compares chatbot responses with a specialized conceptual reference, enabling the identification of domain drift and potential conceptual errors.