Integration of Static Knowledge and Telemetry Data for an IoT-Based Customer Support AI Agent
DOI:
https://doi.org/10.35870/ijsecs.v6i3.7875Keywords:
AI Agents, Customer Support, Internet of Things, Knowledge Management, Retrieval-Augmented GenerationAbstract
The operational effectiveness of artificial intelligence agents in customer support is frequently attributed to the generative sophistication of underlying language models, although answer reliability primarily depends on the structure and quality of the referenced knowledge base. In complex technical domains such as the Internet of Things (IoT), static documentation alone often fails to resolve customer inquiries that involve real-time device states, sensor readings, and connectivity logs. This study investigates how progressive tiers of knowledge integration affect the response quality of an AI agent within an IoT customer support context. Employing a mixed-method pilot design combining literature synthesis, comparative analysis, experimental testing, and conceptual exploration, thirty representative questions were classified into informational, status, and diagnostic categories. These queries were evaluated across four operational scenarios: without retrieval-augmented generation (RAG), article-based RAG, article plus device metadata records, and a comprehensive hybrid framework integrating static articles, device records, and telemetry streams. Answer quality was measured using Cosine Similarity, BERTScore (F1), and a randomized blind human evaluation assessing relevance, correctness, and usefulness. The results demonstrate that while curated static documentation adequately addresses procedural informational queries, resolving status and diagnostic issues necessitates real-time operational context. The hybrid integration delivered the highest overall performance, achieving a human-evaluation score of 4.40 compared with 2.53 for article-based RAG (p = 0.0039). These empirical findings confirm that robust IoT support agents require dynamic knowledge governance that bridges versioned documentation, synchronized asset metadata, and live telemetry data within a unified retrieval framework.
Downloads
References
Alavi, M., & Leidner, D. E. (2001). Knowledge management and knowledge management systems: Conceptual foundations and research issues. MIS Quarterly, 25(1), 107–136. https://doi.org/10.2307/3250961
Dalkir, K. (2017). Knowledge management in theory and practice. The MIT Press.
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, & T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT 2019) (Vol. 1, pp. 4171–4186). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423
Dzenuska, E., & Rudzajs, P. (2025). Toward a knowledge management method for training customer support AI agents. In Proceedings of the BIR 2025 Workshops and Doctoral Consortium (CEUR Workshop Proceedings, Vol. 4034, pp. 70–83). CEUR-WS.org.
Edge, D., Trinh, H., Cheng, N., Bradley, J., Chao, A., Mody, A., Truitt, S., & Larson, J. (2024). From local to global: A graph RAG approach to query-focused summarization. arXiv. https://doi.org/10.48550/arXiv.2404.16130
Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2024). RAGAS: Automated evaluation of retrieval augmented generation. In Y. Graham & M. Purver (Eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2024): System Demonstrations (pp. 150–158). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.eacl-demo.16
Farahani, B., Firouzi, F., Chang, V., & Badaroglu, M. (2018). Towards fog-driven IoT eHealth: Promises and challenges of IoT in medicine and healthcare. Future Generation Computer Systems, 78(Part 2), 659–676. https://doi.org/10.1016/j.future.2017.04.036
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., & Wang, H. (2024). Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997. https://doi.org/10.48550/arXiv.2312.10997
Laskar, M. T. R., Tremblay, J. B., Fu, X.-Y., Chen, C., & Bhushan, S., TN. (2025). AI knowledge assist: An automated approach for the creation of knowledge bases for conversational AI agents. arXiv. https://doi.org/10.48550/arXiv.2510.08149
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, & H. Lin (Eds.), Advances in Neural Information Processing Systems (NeurIPS 2020) (Vol. 33, pp. 9459–9474). Curran Associates, Inc.
Nonaka, I., & Takeuchi, H. (1995). The knowledge-creating company: How Japanese companies create the dynamics of innovation. Oxford University Press.
Salemi, A., & Zamani, H. (2025). HELMET: How to evaluate long-context language models effectively and thoroughly. In Proceedings of the 13th International Conference on Learning Representations (ICLR 2025). OpenReview.net.
Shuster, K., Poff, S., Chen, M., Kiela, D., & Weston, J. (2021). Retrieval augmentation reduces hallucination in conversation. In M.-F. Moens, X. Huang, L. Specia, & S. W.-t. Yih (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2021 (pp. 3784–3803). Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.findings-emnlp.320
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, & R. Garnett (Eds.), Advances in Neural Information Processing Systems (NeurIPS 2017) (Vol. 30, pp. 5998–6008). Curran Associates, Inc.
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2020). BERTScore: Evaluating text generation with BERT. In Proceedings of the 8th International Conference on Learning Representations (ICLR 2020). OpenReview.net.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Jauhar Maknun Adib, Wahyu Purnama Magribi, Muhammad Fazly Qusyairy, Eric Julianto, Khusnul Fajri Rhomadon
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
1. Copyright Retention and Open Access License
Authors retain copyright of their work and grant the journal non-exclusive right of first publication under the Creative Commons Attribution 4.0 International License (CC BY 4.0).
This license allows unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
2. Rights Granted Under CC BY 4.0
Under this license, readers are free to:
- Share — copy and redistribute the material in any medium or format
- Adapt — remix, transform, and build upon the material for any purpose, including commercial use
- No additional restrictions — the licensor cannot revoke these freedoms as long as license terms are followed
3. Attribution Requirements
All uses must include:
- Proper citation of the original work
- Link to the Creative Commons license
- Indication if changes were made to the original work
- No suggestion that the licensor endorses the user or their use
4. Additional Distribution Rights
Authors may:
- Deposit the published version in institutional repositories
- Share through academic social networks
- Include in books, monographs, or other publications
- Post on personal or institutional websites
Requirement: All additional distributions must maintain the CC BY 4.0 license and proper attribution.
5. Self-Archiving and Pre-Print Sharing
Authors are encouraged to:
- Share pre-prints and post-prints online
- Deposit in subject-specific repositories (e.g., arXiv, bioRxiv)
- Engage in scholarly communication throughout the publication process
6. Open Access Commitment
This journal provides immediate open access to all content, supporting the global exchange of knowledge without financial, legal, or technical barriers.
