prikaz prve stranice dokumenta Generative Pre-Trained Transformers: Architecture, Pre-Training and Fine-Tuning
  Download
PDF 13.12 MB
undergraduate thesis
Generative Pre-Trained Transformers: Architecture, Pre-Training and Fine-Tuning
Osijek: Josip Juraj Strossmayer University of Osijek, School of Applied Mathematics and Informatics, 2024. urn:nbn:hr:126:086357
Sušac, Ivo
Josip Juraj Strossmayer University of Osijek
School of Applied Mathematics and Informatics

Institutional repository: Repository of School of Applied Mathematics and Informatics

Cite this document

Sušac, I. (2024). Generative Pre-Trained Transformers: Architecture, Pre-Training and Fine-Tuning (Undergraduate thesis). Osijek: Josip Juraj Strossmayer University of Osijek, School of Applied Mathematics and Informatics. Retrieved from https://urn.nsk.hr/urn:nbn:hr:126:086357

Sušac, Ivo. "Generative Pre-Trained Transformers: Architecture, Pre-Training and Fine-Tuning." Undergraduate thesis, Josip Juraj Strossmayer University of Osijek, School of Applied Mathematics and Informatics, 2024. https://urn.nsk.hr/urn:nbn:hr:126:086357

Sušac, Ivo. "Generative Pre-Trained Transformers: Architecture, Pre-Training and Fine-Tuning." Undergraduate thesis, Josip Juraj Strossmayer University of Osijek, School of Applied Mathematics and Informatics, 2024. https://urn.nsk.hr/urn:nbn:hr:126:086357

Sušac, I. (2024). 'Generative Pre-Trained Transformers: Architecture, Pre-Training and Fine-Tuning', Undergraduate thesis, Josip Juraj Strossmayer University of Osijek, School of Applied Mathematics and Informatics, accessed 16 November 2024, https://urn.nsk.hr/urn:nbn:hr:126:086357

Sušac I. Generative Pre-Trained Transformers: Architecture, Pre-Training and Fine-Tuning [Undergraduate thesis]. Osijek: Josip Juraj Strossmayer University of Osijek, School of Applied Mathematics and Informatics; 2024 [cited 2024 November 16] Available at: https://urn.nsk.hr/urn:nbn:hr:126:086357

I. Sušac, "Generative Pre-Trained Transformers: Architecture, Pre-Training and Fine-Tuning", Undergraduate thesis, Josip Juraj Strossmayer University of Osijek, School of Applied Mathematics and Informatics, Osijek, 2024. Available at: https://urn.nsk.hr/urn:nbn:hr:126:086357

Metadata
TitleGenerative Pre-Trained Transformers: Architecture, Pre-Training and Fine-Tuning
AuthorIvo Sušac
MentorDomagoj Matijević (mentor)
GranterJosip Juraj Strossmayer University of Osijek
School of Applied Mathematics and Informatics
Osijek
Defense date and country2024-09-23, Croatia
Scientific / art field,
discipline and subdiscipline		NATURAL SCIENCES
Mathematics
Abstract
With an emphasis on GPT models, this thesis explores the design, training, and optimization of large language models (LLMs). It starts by examining preprocessing approaches for text data, such as tokenization techniques (wordbased, character-based, Byte-Pair Encoding) and embeddings. Subsequently, the Transformer block was presented, highlighting the mechanisms that constitute the basis of the model: self-attention and fully-connected layers. The processes involved in optimizing model parameters were highlighted by the examination of the pre-training phase followed by fine-tuning, which demonstrated how models can be adapted efficiently to new tasks, particularly when using parameter-efficient techniques like LoRA and prompt tuning.
Keywords
Languageenglish
URN:NBNurn:nbn:hr:126:086357
Study programmeTitle: Bachelor of Mathematics and Computer Science
Study programme type: university
Study level: undergraduate
Academic / professional title: sveučilišni/a prvostupnik/prvostupnica (baccalaureus/baccalaurea) matematike i računarstva (sveučilišni/a prvostupnik/prvostupnica (baccalaureus/baccalaurea) matematike i računarstva)
Type of resourceText
File originBorn digital
Access conditionsOpen access
Terms of useLicence In copyright icon
Created on2024-09-26 10:43:36