2024/11/26 by Andor Diera, Diera, Andor, Lukas Galke +3 · 1 citation
Computer Science · Engineering · Materials Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning in Materials Science #Modular Robots and Swarm Intelligence #Software Engineering Research
paper · pdf · doi:10.48550/arxiv.2411.17538
openalex publication_date 2024/11/26 · openalex created_date 2024/12/05 · openalex updated_date 2026/07/28
Low isotropy in an embedding space impairs performance on tasks involving semantic inference. Our study investigates the impact of isotropy on semantic code search performance and explores post-processing techniques to mitigate this issue. We analyze various code language models, examine isotropy in their embedding spaces, and its influence on search effectiveness. We propose a modified ZCA whitening technique to control isotropy levels in embeddings. Our results demonstrate that Soft-ZCA whitening improves the performance of pre-trained code language models and can complement contrastive fine-tuning.