[Untitled]

Mathematics, Computer Science

Thomas Yu

2025

Clarice Poon, Paris Giampouras

This poster investigates the effect of preconditioning in low-rank adaptation methods for large language models. We compare baseline LoRTA and a modified Preconditioned LoRTA implementation, both applied to RoBERTa fine-tuning on the MRPC task. Preconditioning yields smoother optimisation dynamics and improved long-term convergence, achieving lower evaluation loss compared to baseline LoRTA. The results highlight the potential of tensor-based preconditioning to enhance efficiency and stability in low-rank model adaptation.

Mathematics, Computer Science, Machine Learning, AI, Neural Networks, Optimisation, Fine Tuning, LLM