Human Grounded Evaluation of Large Language Models for Optical Network Automation

2026/07/20 by Kiarash Rezaei, Omran Ayoub, Paolo Monti +1
#cs.NI #cs.AI

paper · pdf

Abstract

Large language models (LLMs) are increasingly adopted for network automation, yet their output quality and inference cost can vary substantially across LLM families. We present HuGLEN, a stepwise evaluation pipeline that uses an LLM-as-a-judge together with a small set of expert ratings to enable scalable and reproducible comparison of candidate LLMs, and to rank them using a quality efficiency score (QES). We demonstrate HuGLEN for translating outputs from an explainable artificial intelligence (XAI) model for the optical network quality of transmission (QoT) estimation task into operator-friendly explanations. Our results show that a medium-sized LLM (12B parameters) achieves the highest QES, indicating the best trade-off between explanation quality and efficiency. Overall, HuGLEN reduces the human-labeling burden while supporting consistent model selection for operator-facing automation tasks.

Citations

• LoRA: Low-Rank Adaptation of Large Language Models

Related

• When Large Language Models Meet Optical Networks: Paving the Way for Automation
• Explanation-Based Runtime Verification for Trustworthy ML-driven Optical Networks
• LM4Opt-RA: A Multi-Candidate LLM Framework with Structured Ranking for Automating Network Resource Allocation
• Reasoning AI Performance Degradation in 6G Networks with Large Language Models
• Synergistic Interplay of Large Language Model and Digital Twin for Autonomous Optical Networks: Field Demonstrations