2026/07/18 by Seshu Kumar Damarla, Xiuli Zhu
Computer Science · #cs.IR
Industrial plants run many important machines such as pumps, turbines, and compressors. Although engineers can use their experience to identify and diagnose machine problems, transferring this reasoning ability to computer systems remains difficult. This work studies how well a retrieval-only method and an open-source large language model (LLM) perform failure-sensor diagnostic reasoning using the FailureSensorIQ benchmark, a multiple-choice question-answering task introduced by IBM Research. In the retrieval-only approach, each answer option is converted into an option-level query and scored using similar correct and incorrect records from the training data. TF-IDF, BM25, semantic search, and hybrid search are tested and compared. In the LLM-based approach, the Qwen2.5-7B-Instruct model is evaluated using zero-shot prompting, few-shot prompting, and QLoRA fine-tuning. The results show that semantic search and hybrid search perform better than pure keyword-matching techniques, indicating that meaning-based similarity is more important for industrial failure-sensor reasoning. Among the LLM-based methods, the fine-tuned model achieves the best performance and substantially improves over zero-shot and few-shot prompting. Error analysis shows that performance decreases as the number of answer options increases. Robustness analysis also shows that all methods are sensitive to option shuffling, changed labels, paraphrasing, and additional distractors.