Fine-tuning and LLM

Please work on this assignment in pairs.

If you need help, please ask Nyomi; she developed this assignment.

Overview

For this assignment you will try your hand at fine-tuning a large language model. You will explore existing datasets and pre-trained models and (hopefully) be able to produce a model that executes a specific task.

Model Fine-tuning

Training a large language model to the level of commercially available products such as ChatGPT, Bard, or others requires more computational power than individuals such as ourselves could possibly have. That’s why we’ll be doing fine-tuning! Informally, this essentially means taking a smaller, pretrained version of the model and training on some additional data for the model to be good at what we personally need it to do.

For this, we will be using the LoRA (Low-Rank Adaptation of Large Language Models) method which you might see a lot if you look at fine-tuning tutorials made within the last few months. For our purposes, we will treat this step of the process kind of like a black box, but you can get deeper with the following resources:

Part 1: Let’s get something to work!
We will work with the TinyLlama model to do some fine-tuning. This model is an instance that is theoretically accessible to common users who want to do some smaller scale language tasks. Nyomi has created a Colab notebook to try fine-tuning the model to tell stories which is inspired in combination with these three tutorials:

Please make a copy of the Colab notebook and take a look at the major steps of the process. Try to run it and see if it works! DISCLAIMER: The free version of Colab sometimes will not have GPUs available so sometimes, you’ll have to delay running training-intensive processes to another time. (One trick I’ve used outside of waiting is an incognito tab or a different browser). You can explore other places to host your computation if you need.

Things to note: We are training with 1000 rows from the dataset so that this part of the assignment does not take too long! You might have to adjust it though.

Create a lab manual and add both your names to it. Then provide the following:

  1. Please provide a screenshot of the output from the model’s inference.
  2. Discuss the performance of the model. Does the output make sense? If not, what could make it better?
Part 2: Fine-tune your own model
Now that you’re a little familiar with the fine-tuning process, it’s time to explore some datasets and fine-tune a model for your own purpose. You can use the example from above, inspiration from an online tutorial, or your own method to accomplish this since there are many ways to do this. Please be sure to document the process and successes/difficulties here in the lab notebook.

General Process Notes:

  1. Please provide a link to the dataset that you chose to fine-tune with. You can search HuggingFace, Kaggle, or other sites for ones that already exist. You can also consider using a large language model to generate custom data to fine-tune your large language model if you can’t find anything online.
  2. If you decide to fine-tune a different language model than TinyLLama, or use any new and interesting methods please discuss them here.
  3. What are the limitations of your model? What do you notice about the performance during inference?
  4. Please provide some screenshots of a few prompts and responses from your model.

Submission

Please submit a zipped copy of the following items to the appropriate drop-box on Moodle.
  1. The lab manual.
  2. The code for fine-tuned and improved models.