AdaGradSelect: A Computationally Efficient and Memory-Optimized Fine-Tuning Method for Large Language Models

This paper introduces AdaGradSelect, a novel fine-tuning method for Large Language Models (LLMs) that offers significant computational efficiency and memory optimization. It trains about 12% faster and uses 35% less GPU memory while maintaining performance close to full fine-tuning, outperforming LoRA on certain benchmarks and providing a more effective and resource-efficient alternative to traditional fine-tuning.