Upstage AI Lab 2기 [Day109] NLP 경진대회 - LoRA 공부

Upstage AI Lab 2기
2024년 5월 21일 (화) Day_109

 

LoRA (Low-Rank Adaptation)

LoRA: Low-Rank Adaptation of Large Language Models

 

LoRA: Low-Rank Adaptation of Large Language Models

https://arxiv.org/pdf/2106.09685

 

LoRA to transformer

Wk = W0k + BAk
Wq = W0q + BAq
Wv = W0v + BAv

 

 

 

 

QLORA: Efficient Finetuning of Quantized LLMs

https://arxiv.org/pdf/2305.14314

 

 

 

참고자료:

https://sebastianraschka.com/blog/2023/self-attention-from-scratch.html

https://velog.io/@kaiba0514/QLoRA-QLoRA-Efficient-Finetuning-of-Quantized-LLMs
https://dacon.io/codeshare/9692
https://www.kaggle.com/code/philculliton/fine-tuning-with-llama-2-qlora

 

https://huggingface.co/LDCC/LDCC-SOLAR-10.7B-GGUF