If you don’t mind me giving you a suggestion. An easy way to get started is by using Unsloth’s Google Colab notebooks. Just by inspecting the code of some of their many notebooks you can get a solid starting point about the fine-tunneling steps, including the dataset formats.

Unsloth - Open source Fine-tuning & RL for LLMs

Unsloth AI - Open Source Fine-tuning & RL for LLMs

Open source fine-tuning & reinforcment learning (RL) for gpt-oss, Llama 4, DeepSeek-R1, Gemma, and Qwen3 LLMs! Beginner friendly.