11th International Scientific Conference Technics, Informatics and Education – TIE 2026, (pp. 51-57)
AUTOR(I) / AUTHOR(S): Olga Dukić 
, Danijela Milošević and Jianxun Cui
DOI: https://doi.org/10.46793/TIE26.051D
SAŽETAK /ABSTRACT:
Artificial intelligence-supported educational chatbots are increasingly used to explain concepts, guide learners through tasks, support formative assessment, and recommend learning materials. Their reliability cannot be evaluated only through fixed dialog scenarios, because learners express the same educational need through paraphrases, incomplete messages, spelling errors, informal language, uncertainty, or direct requests for solutions. This paper proposes an extension of behavior-driven dialog testing with metamorphic relations for robust testing of educational chatbots. The approach defines controlled input transformations under which the chatbot should preserve or appropriately adapt its intended behavior. The proposed relations address paraphrasing, word-order variation, noisy input, formality variation, learner uncertainty, equivalent topic requests, requests for direct solutions, and repeated misunderstanding. Instead of requiring a single fixed response, the evaluation checks behavior-level properties such as intent, topic, dialog state, pedagogical strategy, and safe tutoring behavior. The framework is evaluated through an initial prototype case study in which one mathematics-help scenario is extended with seven metamorphic follow-up inputs and executed using automated pytest tests with trace information from the chatbot testing mode
KLJUČNE REČI / KEYWORDS:
metamorphic testing; behavior-driven development; educational chatbots; dialog systems; pedagogical consistency; robust artificial intelligence systems
PROJEKAT / ACKNOWLEDGEMENT:
LITERATURA / REFERENCES:
- Binamungu, L. P. & Maro, S. (2023). Behaviour driven development: A systematic mapping study. Journal of Systems and Software, 203, Article 111749. doi:10.1016/j.jss.2023.111749
- Chen, T. Y., Kuo, F.-C., Liu, H., Poon, P.-L., Towey, D., Tse, T. H. & Zhou, Z. Q. (2018). Metamorphic testing: A review of challenges and opportunities. ACM Computing Surveys, 51(1), Article 4, 4:1–4:27. doi:10.1145/3143561
- Božić, J. (2022). Ontology-based metamorphic testing for chatbots. Software Quality Journal, 30(1), 227–251. doi:10.1007/s11219-020-09544-9
- Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J. & Wang, H. (2024). Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology, 33(8), Article 220. doi:10.1145/3695988
- Li, Y., Liu, P., Wang, H., Chu, J. & Wong, W. E. (2025). Evaluating large language models for software testing. Computer Standards & Interfaces, 93, Article 103942. doi:10.1016/j.csi.2024.103942
- Guo, G., Aleti, A., Neelofar, N., Tantithamthavorn, C., Qi, Y. & Chen, T. Y. (2026). MORTAR: Multi-turn metamorphic testing for LLM-based dialogue systems. IEEE Transactions on Software Engineering, in press. doi:10.1109/TSE.2026.3701230
- Debets, T., Banihashem, S. K., Joosten-Ten Brinke, D., Vos, T. E. J., Maillette de Buy Wenniger, G. & Camp, G. (2025). Chatbots in education: A systematic review of objectives, underlying technology and theory, evaluation criteria, and impacts. Computers & Education, 234, Article 105323. doi:10.1016/j.compedu.2025.105323
- Macina, J., Daheim, N., Hakimi, I., Kapur, M., Gurevych, I. & Sachan, M. (2025). MathTutorBench: A benchmark for measuring open-ended pedagogical capabilities of LLM tutors. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 204–221. doi:10.18653/v1/2025.emnlp-main.11. Available at: arxiv.org/abs/2502.18940
- De Lara, J., Del Pozzo, A., Guerra, E. & Sánchez Cuadrado, J. (2026). Automated end-to-end testing for conversational agents. Journal of Systems and Software, 233, Article 112685. doi:10.1016/j.jss.2025.112685