Upstage AI Lab 2기
Upstage AI Lab 2기 [Day109] NLP 경진대회 - LoRA 공부
hyj89han
2024. 5. 21. 07:32
Upstage AI Lab 2기
2024년 5월 21일 (화) Day_109
LoRA (Low-Rank Adaptation)
LoRA: Low-Rank Adaptation of Large Language Models
https://arxiv.org/pdf/2106.09685
LoRA to transformer
Wk = W0k + BAk
Wq = W0q + BAq
Wv = W0v + BAv
QLORA: Efficient Finetuning of Quantized LLMs
https://arxiv.org/pdf/2305.14314
참고자료:
https://sebastianraschka.com/blog/2023/self-attention-from-scratch.html
https://velog.io/@kaiba0514/QLoRA-QLoRA-Efficient-Finetuning-of-Quantized-LLMs
https://dacon.io/codeshare/9692
https://www.kaggle.com/code/philculliton/fine-tuning-with-llama-2-qlora
https://huggingface.co/LDCC/LDCC-SOLAR-10.7B-GGUF