Fine Tuning Methods for Low-resource Languages

2025/10/05 by Tim Bakkenes, Daniel Wang, Bakkenes, Tim +2
Social Sciences · Computer Science · #Computational and Text Analysis Methods #Language and cultural evolution #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2510.04139

Abstract

The rise of Large Language Models has not been inclusive of all cultures. The models are mostly trained on English texts and culture which makes them underperform in other languages and cultural contexts. By developing a generalizable method for preparing culturally relevant datasets and post-training the Gemma 2 model, this project aimed to increase the performance of Gemma 2 for an underrepresented language and showcase how others can do the same to unlock the power of Generative AI in their country and preserve their cultural heritage.

Citations

• Tackling Bias in Pre-trained Language Models: Current Trends and Under-represented Societies
• LoRA: Low-Rank Adaptation of Large Language Models
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Related

• CultureLLM: Incorporating Cultural Differences into Large Language Models
• Scaling Cultural Resources for Improving Generative Models
• Cultural Fidelity in Large-Language Models: An Evaluation of Online Language Resources as a Driver of Model Performance in Value Representation
• Overcoming Data Scarcity in Generative Language Modelling for Low-Resource Languages: A Systematic Review
• LlamaTurk: Adapting Open-Source Generative Large Language Models for Low-Resource Language