# How to Build a Large Language Model from Scratch? Toktumi 3.0 – Part 3 (The Final Part)

# **CHAPTER III. Toktumi 3.0. Third Training Run.**

### 1\. The Goal of Further Training

Toktumi 2.0 was a significant improvement over Toktumi 1.0, but it still struggled with answer accuracy, understanding certain concepts, and generating repetitive text loops. (For details, see [**Part 2**](https://sixfingers.hashnode.dev/toktumi-2-en)).

Earlier experiments had shown that **training on Question–Answer (QA) and Instruction–Answer (IA) pairs** produced good results. I therefore decided to develop this approach further, using previously collected texts and the Google Gemini API (**Gemini 3.1 Flash-Lite**) to generate synthetic training data.

**I wanted to find out whether continuing to train the model exclusively on short, structured questions, instructions, and dialogues would produce better results than training based mainly on raw texts.**

As with the previous stages, **the project is non-commercial, educational, and research-oriented.**

### 2\. A New Approach to Training Data

In previous versions of Toktumi, a large part of the corpus consisted of raw texts collected from the internet: articles, books, online comments, and other materials. For Toktumi 3.0, I decided to change this approach completely.

**Instead of training the model directly on raw texts, I prepared the entire training corpus as short, structured records**, generated from source materials.

I adopted three main types of records:

- **Question → Answer (QA)** — developing knowledge and understanding of concepts.
- **Instruction → Answer (IA)** — teaching the model to follow instructions.
- **Statement → Response (dialogue)** — developing conversational skills.

Examples of these record categories:

**Question and answer:**

> *Jaka jest stolica Francji?* \[EN: *What is the capital of France?*\]
>
> *Stolicą Francji jest Paryż.* \[EN: *The capital of France is Paris.*\]
>
> `<|endoftext|>`

**Instruction and answer:**

> *Wymień trzy planety Układu Słonecznego.* \[EN: *Name three planets in the Solar System.*\]
>
> *Merkury, Wenus i Ziemia.* \[EN: *Mercury, Venus, and Earth.*\]
>
> `<|endoftext|>`

**Dialogue:**

> *Chyba zapomniałem zabrać parasola.* \[EN: *I think I forgot to bring an umbrella.*\]
>
> *Może sprawdź prognozę pogody, zanim wyjdziesz.* \[EN: *Maybe check the weather forecast before you go out.*\]
>
> `<|endoftext|>`

An important principle was that **each record had to be self-contained**. A question or instruction had to make sense without knowing the source article or any of the other records.

### 3\. Selecting and Collecting Source Materials

To prepare the Toktumi 3.0 corpus, I decided to use four main sources of texts I had already collected (see [**Part 1**](https://sixfingers.hashnode.dev/toktumi-1-en) and [**Part 2**](https://sixfingers.hashnode.dev/toktumi-2-en), on Toktumi 1.0 and 2.0): **the Great Dictionary of Polish (WSJP), Wikipedia, Fakt and film and TV series subtitles**.

Each source was intended to develop different abilities in the model:

- **WSJP — language and meaning.** A source containing word definitions, idiomatic expressions, common word combinations, and usage examples. These materials were intended to improve the model's understanding of Polish vocabulary and its ability to define concepts.
- **Wikipedia — knowledge.** Encyclopedic articles providing information about the world and intended to develop the ability to answer factual questions.
- [**Fakt.pl**](https://www.fakt.pl/) – **events.** News articles providing knowledge about specific events, people, and cause-and-effect relationships.
- **Subtitles — conversation.** Dialogues prepared from film and TV subtitles were intended to teach the model to engage in natural conversation and respond to a speaker's emotions.

Generated samples by source:

![034_nowe_dane](https://cdn.hashnode.com/uploads/covers/6af0eebb5f7d123257967ad7/d7a18e09-116d-44f4-939a-9bf482047aaa.png align="center")

### 4\. Generating Training Data

I primarily used the **Google Gemini API** to create training records automatically. It's been about a year since I worked on Toktumi 2.0 (2025). During that time, newer and more advanced models had become available. I replaced the earlier [**Gemini 2.5 Flash**](https://aistudio.google.com/docs/models/gemini-2.5-flash) (now deprecated) with [**Gemini 3.1 Flash-Lite**](https://aistudio.google.com/docs/models/gemini-3.1-flash-lite).

I also used the locally run [**Gemma 4 31B**](https://huggingface.co/unsloth/gemma-4-31B-it-GGUF) (GGUF version), replacing the earlier [**Mistral NeMo 12B**](https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407). In my tests, Gemma handled Polish better than [**Qwen3.6-35B-A3B**](https://huggingface.co/Qwen/Qwen3.6-35B-A3B), another very popular model in 2026.

**How it works:** A Python script sends successive source texts through the API to **Gemini 3.1 Flash-Lite**, along with an appropriate prompt defining the generation rules. Based on each source text, the model creates **20 or 30 training records** in the form of Question–Answer and Instruction–Answer pairs. The completed records are automatically saved to .txt files.

![035_Generacja](https://cdn.hashnode.com/uploads/covers/6af0eebb5f7d123257967ad7/5cd018c5-e68b-4049-b48b-f295e7eab5de.png align="center")

For the first time in the Toktumi project, I used **film and TV subtitles as a source of training data**. I had already collected a large set of subtitles from online sources, but initial experiments showed that raw dialogue texts were unsuitable for training — the model generated chaotic sequences of utterances without recognizing the speakers or knowing when to end a response.

The solution was to **convert the subtitles into short, self-contained dialogue pairs**, in which one utterance is a logical response to the other. In this form, the subtitles could be used to teach the model to carry on simple conversations.

Converting a subtitle excerpt into a set of training pairs:

![036_subtitles](https://cdn.hashnode.com/uploads/covers/6af0eebb5f7d123257967ad7/b60b2ab3-596b-4ea6-b4d1-c0c9bbd95eb8.png align="center")

The code and prompt used to generate the data worked in much the same way as in Toktumi 2.0 (see [Part 2](https://sixfingers.hashnode.dev/toktumi-2-en)).

Records generated from **all four sources** had to be checked to ensure they were self-contained. **Every question or instruction had to be understandable on its own, without knowledge of the source article or any previous context.**

The model automatically evaluated the records and labeled them **1 – keep** or **0 – discard**. This allowed me to eliminate questions referring to unidentified people, events, or earlier statements.

Below are examples of **rejected, non-self-contained records generated from news articles.**

![037_odrzucone](https://cdn.hashnode.com/uploads/covers/6af0eebb5f7d123257967ad7/79bc17d9-350f-43c8-b862-bf99662bb802.png align="center")

### DATA PROCESSING SPEED

Gemini processed **approximately 7–8 source texts per minute through the API** (around 10,000 per day), while the locally run **Gemma 4 in GGUF format processed approximately 1–2 texts per minute** (around 2,000 per day). These are estimated figures.

Generating and cleaning all **6 190 420 training pairs** took **approximately 2.5 months**.

### 5\. Preparing the Final Toktumi 3.0 Corpus

As with the previous versions of Toktumi, I combined the prepared data into **one large training corpus**. Before doing so, **I shuffled all 6,190,420 records** so that questions, instructions, and dialogues would be mixed throughout training. I then split the dataset into **99% training data (TRAIN) and 1% validation data (VALID)**, separating the records with the `<|endoftext|>` token, which the model recognizes as an end-of-text marker.

One small note: despite removing duplicates, the random split means that questions covering similar topics may still have ended up in both the TRAIN and VALID sets.

The synthetically generated texts turned out to be **much shorter than those used in previous versions of Toktumi**, as shown in the illustration below. Each record contained around **43 tokens on average**, while the longest had **174 tokens**. I set the training sequence length (block size) to **256 tokens** to match the shorter records. This parameter also has a significant impact on VRAM usage, so in my case it could not be set too high.

Structure of the training dataset with shuffled records:

![038_zbior_uczacy](https://cdn.hashnode.com/uploads/covers/6af0eebb5f7d123257967ad7/a9d8535e-ba0e-4890-bc6b-e27fccab069c.png align="center")

Comparison of training dataset sizes: Toktumi 1.0 | Toktumi 2.0 | Toktumi 3.0:

![048_zbiory_treningowe_T1-T2-T3](https://cdn.hashnode.com/uploads/covers/6af0eebb5f7d123257967ad7/1b747d78-b5c5-4f2c-8689-242edb9246e4.png align="center")

Comparing the three versions shows just how much the training dataset had changed. **Toktumi 3.0 contains more than 2.5 times as many records as Toktumi 2.0**, yet the number of words and tokens is approximately **53% lower than in Toktumi 2.0**. The training file itself is **almost half the size**.

In other words, the model trained on **more examples, but far fewer tokens**. This time, I focused primarily on **the quality, brevity, and structure of the training data**, rather than increasing its volume.

The chart below illustrates the lengths of the texts in the training datasets:

![049_dlogosc_boxploty](https://cdn.hashnode.com/uploads/covers/6af0eebb5f7d123257967ad7/ac7498e7-fdc3-46f2-a653-44fe10b0f71c.png align="center")

The box plot shows how much shorter the training records became with each new version of Toktumi.

**Median length: Toktumi 1.0 – 207 tokens, Toktumi 2.0 – 145 tokens, Toktumi 3.0 – just 41 tokens.**

The chart also shows two particularly extreme outliers in Toktumi 1.0 and 2.0 (the individual dots on the right) – very long texts that were not caught by the length filters I used in the earlier stages. Such outliers can inflate the average, so I chose the median as the measure of text length.

In Toktumi 3.0, the records were generated from scratch, giving the texts more standardized lengths.

### 6\. Training Toktumi 3.0

Before the main training run, I carried out an experiment on a smaller dataset to see whether it would be better to train a new model from scratch or continue training Toktumi 2.0. **The tests showed that fine-tuning worked better**, so I decided to build on the model's existing language skills.

I kept **the GPT-2 Medium architecture and the Toktumi 2.0 tokenizer** to ensure compatibility between the two versions.

I planned the training run for **a maximum of 15 epochs**, with automatic validation, checkpoint saving, and an **early stopping** mechanism that would end training once the model stopped improving.

The key part of the code used to train the model:

```python
# Wczytanie Toktumi 2 i tokenizera
tokenizer = GPT2TokenizerFast.from_pretrained(TOKENIZER_PATH)
model = GPT2LMHeadModel.from_pretrained(MODEL_PATH)

# Przygotowanie danych
train_dataset = TokenBlockDataset(TRAIN_PATH, tokenizer, 256) # block_size = 256
valid_dataset = TokenBlockDataset(VALID_PATH, tokenizer, 256)

# Parametry treningu
training_args = TrainingArguments(
    output_dir=str(CHECKPOINT_DIR),
    num_train_epochs=15,
    per_device_train_batch_size=6,
    gradient_accumulation_steps=4,
    learning_rate=5e-5,
    fp16=True,
    eval_strategy="steps",
    eval_steps=8000,
    save_strategy="steps",
    save_steps=8000,
    save_total_limit=100,
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    greater_is_better=False,
)

# Fine-tuning z mechanizmem early stopping
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=valid_dataset,
    data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False),
    callbacks=[EarlyStoppingCallback(early_stopping_patience=6)],
)

trainer.train()
trainer.save_model(str(BEST_MODEL_DIR))
```

I decided to **keep all the checkpoints**, which later helped me compare the quality of the model's responses at different stages of training.

Drawing on my previous experience, this time I decided to use an **early stopping** callback instead of manually assessing progress based on perplexity. As it turned out, this was a good decision – I no longer had to decide for myself when to stop training. I set aside a sufficiently large validation set from the dataset, and early stopping automatically halted training when `eval_loss` stopped improving. **Each validation run took around 3 minutes**, so it did not add much to the overall training time.

### TRAINING BY THE NUMBERS

Given the high prices of graphics cards (2026), I decided to stick with the NVIDIA GeForce RTX 3060 I had been using all along. I adjusted the training parameters to fit its available memory (12 GB VRAM).

- **One epoch:** consisted of **44,538 steps** and took **around 16 hours** to complete.
- **Total training time:** approximately **9.5 days**, including breaks and resuming from checkpoints.
- **Early stopping:** automatically ended training at **step 448,000, after 10.06 epochs**.
- **Best checkpoint** according to eval_loss: **checkpoint-400000**, reached near the end of the 9th epoch (8.98).
- **Saved checkpoints:** **56** in total (I used half of them in the subsequent tests).

The chart below illustrates the training progress:

![040_uczenie_wykres](https://cdn.hashnode.com/uploads/covers/6af0eebb5f7d123257967ad7/de1681ba-b17e-4598-a99b-6538ec40774e.png align="center")

After the ninth epoch, **train_loss continued to decrease, but eval_loss stopped improving**, eventually triggering **early stopping**.

The characteristic stair-step pattern in the train_loss curve (the blue line) appears regularly at the beginning of successive epochs. They do not necessarily indicate a training error and may be related to reshuffling the training data and the way the optimization process works.

### 7\. Evaluating Model Quality – PROMPTING and Checkpoint Testing

Next, I tested **29 selected checkpoints** on **two additional test sets of 300 prompts each**, generating a total of **17,400 responses** – a substantial amount of material to analyze.

With the help of AI, I carried out an approximate **FCT (Factuality)** evaluation, taking into account factual accuracy, whether the answers made sense, and how well they matched the prompts. Each correct or reasonable answer received 1 point. This is a simplified evaluation criterion (described in detail in [Part 2](https://sixfingers.hashnode.dev/toktumi-2-en)), which allowed me to compare response quality across checkpoints.

**Best results (600 prompts):**

- **checkpoint-400000:** 431/600 (71.8%)
- **checkpoint-448000:** 431/600 (71.8%)
- **checkpoint-432000:** 425/600 (70.8%)

**I selected checkpoint-400000 for further testing**. It also achieved the lowest `eval_loss` during training and was automatically identified as the best checkpoint by the early stopping mechanism. It is worth noting that the differences between the top checkpoints were small.

Checkpoint results according to the Factuality (FCT) criterion – whether responses were correct, factually accurate, and made sense:

![050_find_best_chckp](https://cdn.hashnode.com/uploads/covers/6af0eebb5f7d123257967ad7/63751c30-3212-4500-b8bd-05fe3bbf6979.png align="center")

**An interesting observation:** When asked who the first President of the United States was (using greedy decoding), checkpoint-384000 correctly answered George Washington, checkpoint-400000 incorrectly answered George W. Bush, and checkpoint-448000 gave the correct answer again. This shows that **longer training does not always guarantee better responses**. The model can produce the correct answer, but it does not always select it as the most probable one.

![041_prezydent](https://cdn.hashnode.com/uploads/covers/6af0eebb5f7d123257967ad7/19bee6a6-d90d-4c03-8009-d2e450eb756d.png align="center")

The full prompt sets and model responses, organized by checkpoint, are available here:

[**300 Pack 1**](https://huggingface.co/datasets/miczac/toktumi-reports/raw/main/Toktumi_3.0_29_checkpoints_300_prompts_set_1.txt) and the second set, [**300 Pack 2**](https://huggingface.co/datasets/miczac/toktumi-reports/raw/main/Toktumi_3.0_29_checkpoints_300_prompts_set_2.txt).

### 8\. Comparing All Three Model Versions: Toktumi 1-2-3

Here, **I examine how Toktumi 3.0 compares with the previous versions (described in** [**Part 1**](https://sixfingers.hashnode.dev/toktumi-1-en) **and** [**Part 2**](https://sixfingers.hashnode.dev/toktumi-2-en)**)**. I return to the set of **150 prompts** used earlier to evaluate Toktumi 1.0 and 2.0.

The test shows **whether Toktumi 3.0 correctly recognizes the prompt or instruction (IF)** and then **whether its response is factually correct (where factual accuracy can be assessed) or makes sense (a more subjective assessment in the case of dialogues) (FCT)**. I classified the results as **YES/YES, YES/HAL, or NO**, where **HAL** indicates a hallucination despite correctly recognizing the structure of the prompt (for details, see [Part 2](https://sixfingers.hashnode.dev/toktumi-2-en) on Toktumi 2.0).

### Toktumi 1.0 Results

| Classification \[IF / FCT\] | Number of cases |
| --- | --: |
| **YES / YES** – correct structure and correct answer | 85 |
| **YES / HAL** – correct structure, but a hallucination | 33 |
| **NO** – failure to recognize the instruction correctly | 32 |
| **Total** | **150** |
| **Correct answers (YES / YES)** | **56.7%** |

### Toktumi 2.0 Results

| Classification \[IF / FCT\] | Number of cases |
| --- | --: |
| **YES / YES** – correct structure and correct answer | 102 |
| **YES / HAL** – correct structure, but a hallucination | 42 |
| **NO** – failure to recognize the instruction correctly | 6 |
| **Total** | **150** |
| **Correct answers (YES / YES)** | **68.0%** |

### Toktumi 3.0 Results

| Classification \[IF / FCT\] | Number of cases |
| --- | --: |
| **YES / YES** – correct structure and correct answer | 114 |
| **YES / HAL** – correct structure, but a hallucination | 31 |
| **NO** – failure to recognize the instruction correctly | 5 |
| **Total** | **150** |
| **Correct answers (YES / YES)** | **76.0%** |

**The results show a clear improvement in model quality.** The percentage of correct answers increased from **56.7% for Toktumi 1.0 to 76% for Toktumi 3.0**, while the number of prompts the model failed to recognize dropped from **32 to 5 out of 150**.

Toktumi 3.0 follows instructions much better and provides correct answers more often, although it still makes factual errors and generates hallucinations.

**Results and responses for Toktumi 1-2-3 (150 prompts)** [**VIEW HERE**](https://huggingface.co/datasets/miczac/toktumi-reports/raw/main/Toktumi_1-2-3_150_prompts_comparison.txt)

---

### 9\. Qualitative Tests: What Can Toktumi 3.0 Do?

Here, I take a closer look at **the capabilities and limitations of Toktumi 3.0**.

I used the two previously prepared sets of 300 prompts (**300 Pack 1 and 300 Pack 2**). I generated all **600 responses** using **greedy decoding**.

The results and conclusions are presented below.

---

#### a) RESPONSE STYLE: SHORTER ANSWERS

The first noticeable change is how the model ends its responses. Unlike earlier versions, Toktumi 3.0 usually gives **a short answer** without unnecessarily continuing the text. The repetitive loops characteristic of Toktumi 1 and 2 have also been greatly reduced.

An example of **text repetition** – a problem significantly reduced in Toktumi 3. The latest version of the model is much less likely to get stuck repeating itself and more likely to provide short, correct, and coherent answers on the given topic.

![042_zapetlenie](https://cdn.hashnode.com/uploads/covers/6af0eebb5f7d123257967ad7/84857d03-a8b3-4483-afae-825868aa1005.png align="center")

**Short answers** have become a hallmark of Toktumi 3.0:

\[Toktumi 3.0\]

> **Prompt (v3, no. 7):** *W jakiej dyscyplinie sportu specjalizowała się Justyna Kowalczyk?* \[EN: *Which sport did Justyna Kowalczyk specialize in?*\]
>
> **Answer:** *Biegi narciarskie.* \[EN: *Cross-country skiing.*\]

> **Prompt (v3, no. 90):** *Jak nazywa się jednostka dziedziczenia znajdująca się w DNA?* \[EN: *What is the unit of heredity found in DNA called?*\]
>
> **Answer:** *Gen.* \[EN: *A gene.*\]

> **Prompt (v4, no. 43):** *Jak nazywa się naturalny satelita naszej planety?* \[EN: *What is the name of our planet's natural satellite?*\]
>
> **Answer:** *Księżyc.* \[EN: *The Moon.*\]

---

#### b) WHAT DOES THE MODEL DO WELL?

**OK 1. Simple questions and following instructions**

The model handles instructions that require giving specific examples, creating a short list, or providing a straightforward answer quite well.

\[Toktumi 3.0\]

> **Prompt (v4, no. 4):** *Wymień trzy rzeczy, które przydają się podczas pikniku.* \[EN: *Name three things that are useful for a picnic.*\]
>
> **Answer:** *Podczas pikniku przydają się: koc, napoje oraz przekąski.* \[EN: *Useful things for a picnic include a blanket, drinks, and snacks.*\]

> **Prompt (v4, no. 100):** *Wymień trzy polskie owoce sezonowe.* \[EN: *Name three seasonal fruits grown in Poland.*\]
>
> **Answer:** *Truskawki, maliny, jagody.* \[EN: *Strawberries, raspberries, and blueberries.*\]

> **Prompt (v4, no. 142):** *Wymień dwa sposoby podróżowania po mieście bez własnego samochodu.* \[EN: *Name two ways to get around a city without your own car.*\]
>
> **Answer:** *Dwa sposoby to korzystanie z komunikacji miejskiej, takiej jak autobusy lub tramwaje, oraz chodzenie pieszo.* \[EN: *Two ways are using public transport, such as buses or trams, and walking.*\]

---

**OK 2. Definitions and explanations of basic concepts**

Toktumi 3 can also **produce accurate, detailed explanations**:

\[Toktumi 3.0\]

> **Prompt (v4, no. 34):** *Wyjaśnij prostymi słowami, do czego służy termostat.* \[EN: *Explain in simple terms what a thermostat is used for.*\]
>
> **Answer:** *Termostat to urządzenie, które włącza lub wyłącza grzanie, gdy temperatura w pomieszczeniu spadnie poniżej ustawionego poziomu, co pozwala utrzymać w nim komfortową temperaturę.* \[EN: *A thermostat is a device that switches the heating on or off when the room temperature drops below a set level, helping maintain a comfortable temperature.*\]

> **Prompt (v3, no. 106):** *Opisz, czym jest szczepionka i jaki jest cel jej stosowania.* \[EN: *Describe what a vaccine is and what it is used for.*\]
>
> **Answer:** *Szczepionka to preparat biologiczny, który uczy układ odpornościowy rozpoznawać i zwalczać konkretne drobnoustroje, takie jak wirusy czy bakterie, chroniąc przed zachorowaniem lub łagodząc przebieg choroby.* \[EN: *A vaccine is a biological preparation that teaches the immune system to recognize and fight specific microorganisms, such as viruses or bacteria, protecting against illness or reducing its severity.*\]

> **Prompt (v3, no. 125):** *Opisz, na czym polega funkcja Rzecznika Praw Obywatelskich w Polsce.* \[EN: *Describe the role of the Commissioner for Human Rights in Poland.*\]
>
> **Answer:** *Rzecznik Praw Obywatelskich stoi na straży wolności i praw człowieka oraz obywatela, badając, czy organy władzy publicznej działają zgodnie z prawem i czy nie przekraczają swoich uprawnień.* \[EN: *The Commissioner for Human Rights safeguards human and civil rights and freedoms by examining whether public authorities act in accordance with the law and do not exceed their powers.*\]

In these examples, the model produces **coherent responses**, using appropriate vocabulary and maintaining a logical structure.

It can finally give a correct **definition of a pencil**, something earlier versions of Toktumi sometimes struggled with.

![043_olowek](https://cdn.hashnode.com/uploads/covers/6af0eebb5f7d123257967ad7/ffb3de45-a084-4df2-9a8d-562e8697f4d2.png align="center")

---

**OK 3. Understanding words and idioms**

The responses concerning **the meanings of Polish words and idiomatic expressions**, which the model learned from a Polish dictionary, are also interesting.

\[Toktumi 3.0\]

> **Prompt (v3, no. 22):** *Wyjaśnij słowo „zawzięty” w odniesieniu do osoby dążącej do celu.* \[EN: *Explain what the Polish word "zawzięty" (tenacious) means when describing someone pursuing a goal.*\]
>
> **Answer:** *Oznacza to, że osoba ta jest bardzo zdeterminowana, nie poddaje się łatwo i z wielkim zapałem realizuje swoje zamierzenia, nie zważając na przeszkody.* \[EN: *It means that the person is very determined, does not give up easily, and pursues their goals with great enthusiasm, regardless of obstacles.*\]

> **Prompt (v3, no. 292):** *Co znaczy frazeologizm „grać na zwłokę”, gdy ktoś unika podjęcia decyzji?* \[EN: *What does the Polish expression "grać na zwłokę" ("play for time") mean when someone avoids making a decision?*\]
>
> **Answer:** *Oznacza to celowe opóźnianie jakiegoś działania lub decyzji, aby zyskać na czasie i uniknąć podjęcia właściwego wyboru.* \[EN: *It means deliberately delaying an action or decision to gain time and avoid making the actual choice.*\]

> **Prompt (v4, no. 218):** *Jak rozumiesz „rzucać kłody pod nogi”?* \[EN: *What do you understand by the Polish expression "rzucać kłody pod nogi" ("put obstacles in someone's way")?*\]
>
> **Answer:** *To określenie sytuacji, w której ktoś celowo utrudnia innym osiągnięcie zamierzonego celu.* \[EN: *It describes a situation in which someone deliberately makes it harder for others to achieve their intended goal.*\]

Importantly, the model can not only explain a word or expression but also **put its meaning into its own words**, rather than simply repeating the dictionary definition.

---

**OK 4. Short, natural dialogues**

In Toktumi 3.0, I used **movie and TV subtitles** for the first time, converting them into self-contained dialogue pairs. The examples below show that the model can now tailor its responses to the situation and stop after a single reply – as in a real conversation.

\[Toktumi 3.0\]

> **Prompt (v3, no. 230):** *Przepraszam za spóźnienie, autobus utknął w korku.* \[EN: *Sorry I'm late, the bus got stuck in traffic.*\]
>
> **Answer:** *Nie szkodzi, najważniejsze, że już jesteś i możemy zaczynać.* \[EN: *No problem. The important thing is that you're here now, and we can get started.*\]

> **Prompt (v3, no. 172):** *Obiecaj mi, że zachowasz to dla siebie.* \[EN: *Promise me you'll keep this to yourself.*\]
>
> **Answer:** *Możesz być spokojny, nikomu nie pisnę ani słowa.* \[EN: *Don't worry, I won't breathe a word to anyone.*\]

> **Prompt (v3, no. 238):** *Wreszcie dostałem tę pracę, o której ci opowiadałem!* \[EN: *I finally got that job I told you about!*\]
>
> **Answer:** *To fantastyczna wiadomość, naprawdę się cieszę, że udało ci się ją dostać.* \[EN: *That's fantastic news! I'm really happy you got it.*\]

> **Prompt (v3, no. 219):** *Nie uwierzysz, co mi się dzisiaj przydarzyło!* \[EN: *You won't believe what happened to me today!*\]
>
> **Answer:** *Co takiego?* \[EN: *What happened?*\]

Some of the reactions are surprisingly natural. The model can recognize the emotional tone of a conversation, show interest, or respond in everyday, colloquial language.

---

#### c) WHAT DOES THE MODEL STILL STRUGGLE WITH?

**Problem 1. Factual knowledge – the right format, but the wrong information**

The biggest challenge remains specific facts: names, dates, historical events, and concrete information about the world.

\[Toktumi 3.0\]

> **Prompt (v3, no. 54):** *Kto był pierwszym prezydentem Stanów Zjednoczonych?* \[EN: *Who was the first President of the United States?*\]
>
> **Answer:** *George W. Bush.* \[EN: *George W. Bush.*\]
>
> (Should be: *George Washington.*)

> **Prompt (v3, no. 16):** *Podaj imię i nazwisko pierwszej kobiety, która poleciała w kosmos.* \[EN: *Give the first and last name of the first woman to travel into space.*\]
>
> **Answer:** *Jurij Gagarin.* \[EN: *Yuri Gagarin.*\]
>
> (Should be: *Valentina Tereshkova.*)

> **Prompt (v4, no. 1):** *Podaj nazwę norweskiej stolicy.* \[EN: *Name the capital of Norway.*\]
>
> **Answer:** *Stolicą Norwegii jest Bermudy.* \[EN: *The capital of Norway is Bermuda.*\]
>
> (Should be: *Oslo.*)

These examples clearly show that **the model has learned how to answer questions much better than it has learned factual knowledge**. It does not refuse to answer or express uncertainty – it simply **provides false information (hallucinates).**

The responses above were generated using **greedy decoding**. After introducing sampling with **top_k=2**, the model was also able to give the correct answers: **George Washington in 3 out of 10 attempts** and **Oslo in 6 out of 20 attempts**.

After a few attempts, the model eventually gets the capital of Norway right.

![044_Oslo](https://cdn.hashnode.com/uploads/covers/6af0eebb5f7d123257967ad7/d4c3d4a0-921d-4cd3-94d2-703bf57305b2.png align="center")

However, sampling does not always help. When asked about **the first woman in space**, the model failed to name **Valentina Tereshkova** even once in **100 attempts**. Instead, its answers included Yuri Gagarin, Marie Skłodowska-Curie, fashion designer Jil Sander, actress Gillian Anderson, and even Polish writer and satirist Maria Czubaszek.

I searched the training dataset and found that this information actually appeared in it several times.

![045_Tierieszkowa](https://cdn.hashnode.com/uploads/covers/6af0eebb5f7d123257967ad7/19204fb5-a903-45cd-a1b5-91d1c622a4b0.png align="center")

So I decided to **rephrase the question** as *„Kim była Walentina Tierieszkowa?”* \[EN: *Who was Valentina Tereshkova?*\]. This time, the model answered correctly:

> *„Była radziecką kosmonautką, która jako pierwsza kobieta w historii poleciała w kosmos."* \[EN: *She was a Soviet cosmonaut who became the first woman in history to travel into space.*\]

As we can see, the model **knows this information but struggles to associate it with the right person**.

---

**Problem 2. The model can 'ruin' a correct answer by adding an incorrect explanation**

This is a particularly interesting type of hallucination.

> **Prompt (v3, no. 1):** *Wskaż, nad jaką rzeką położony jest Sandomierz.* \[EN: *Name the river on which Sandomierz is located.*\]
>
> **Answer:** *Sandomierz leży nad rzeką Wisłą, która jest dopływem Wisły.* \[EN: *Sandomierz is located on the Vistula River, which is a tributary of the Vistula.*\]

The first part of the answer is correct. The second makes no sense.

Another example – a question about photosynthesis:

> **Prompt (v3, no. 43):** *Jaki gaz rośliny pobierają z atmosfery w procesie fotosyntezy?* \[EN: *Which gas do plants absorb from the atmosphere during photosynthesis?*\]
>
> **Answer:** *W procesie fotosyntezy z atmosfery pobierają dwutlenek węgla, który jest niezbędny do oddychania dla większości roślin.* \[EN: *During photosynthesis, plants absorb carbon dioxide from the atmosphere, which is essential for respiration in most plants.*\]

The model correctly identifies carbon dioxide but incorrectly explains its role.

---

**Problem 3. Understanding meaning does not always go hand in hand with knowing the language**

Despite producing many good definitions, the model still struggles with certain expressions, even common ones.

\[Toktumi 3.0\]

> **Prompt (v4, no. 134):** *Co oznacza „mieć dwie lewe ręce”?* \[EN: *What does the Polish expression “mieć dwie lewe ręce” (literally, “to have two left hands”) mean?*\]
>
> **Answer:** *Oznacza to, że osoba wykonująca pracę ma dwie ręce jednocześnie, co często zdarza się w zawodach wymagających precyzji lub w sytuacjach wymagających dużej siły fizycznej.* \[EN: *It means that a person doing a job has two hands at the same time, which often happens in professions requiring precision or in situations demanding considerable physical strength.*\]

> **Prompt (v4, no. 296):** *Wyjaśnij pojęcie „syzyfowa praca”.* \[EN: *Explain the meaning of “syzyfowa praca” (Sisyphean labor).*\]
>
> **Answer:** *Jest to określenie sytuacji, w której ktoś wykonuje swoje zadania w sposób bardzo powolny, bez wkładania w nie żadnego wysiłku, co prowadzi do szybkiego zakończenia pracy.* \[EN: *It describes a situation in which someone carries out their tasks very slowly, without putting in any effort, which leads to the work being completed quickly.*\]

In the first case, the model interprets the idiom almost literally. In the second, it produces a fluent explanation that is nevertheless a hallucination and does not reflect the expression's actual dictionary meaning.

After checking the training dataset, I found that **“mieć dwie lewe ręce”** appeared **40 times**, while **“syzyfowa praca”** appeared **58 times** – including both definitions and uses in dialogues in different contexts. Here, too, simply rephrasing the prompt was enough to produce correct answers straight away:

\[Toktumi 3.0\]

> **Prompt:** *Co potocznie oznacza zwrot mieć dwie lewe ręce?* \[EN: *What does the expression “mieć dwie lewe ręce” (to be all thumbs) mean in everyday language?*\]
>
> **Answer:** *Oznacza to, że ktoś jest niezdarny, mało sprawny fizycznie lub nie radzi sobie z wykonywaniem prostych czynności manualnych.* \[EN: *It means that someone is clumsy, lacks physical dexterity, or struggles with simple manual tasks.*\]

> **Prompt:** *Co to jest syzyfowa praca?* \[EN: *What is Sisyphean labor?*\]
>
> **Answer:** *Syzyfowa praca to określenie na bardzo ciężką, żmudną i wymagającą wysiłku pracę, która nie przynosi żadnych widocznych efektów.* \[EN: *Sisyphean labor refers to very hard, tedious, and demanding work that produces no visible results.*\]

The conclusion is similar to the Tereshkova example: the model has absorbed the knowledge but does not always select the correct answer as the most obvious one. **Sometimes all it takes is rephrasing the prompt or asking the question differently** to get the desired result.

---

**Problem 4. Everyday situations and common sense**

> **Prompt (v4, no. 201):** *Co zrobić, gdy na kuchennej podłodze leżą odłamki rozbitej szklanki?* \[EN: *What should you do if there are shards of a broken drinking glass on the kitchen floor?*\]
>
> **Answer:** *Należy niezwłocznie wezwać pogotowie ratunkowe, dzwoniąc pod numer alarmowy 112, a następnie zabezpieczyć miejsce zdarzenia i nie próbować samodzielnie naprawiać rozbitych szklanek czy talerzy.* \[EN: *You should immediately call an ambulance by dialing the emergency number 112, then secure the scene and not attempt to repair broken glasses or plates yourself.*\]

The model recognizes that safety is involved but suggests an action that is completely disproportionate to the situation.

Another example:

> **Prompt (v4, no. 263):** *Kupiłem aparat fotograficzny. Od czego zacząć naukę robienia zdjęć?* \[EN: *I bought a camera. Where should I start learning how to take photos?*\]
>
> **Answer:** *Najpierw naucz się obsługi miksera, potem spróbuj zrobić kilka zdjęć w różnych warunkach oświetleniowych, a na koniec spróbuj zrobić kilka ujęć w różnych miejscach, aby sprawdzić, jak najlepiej ustawić aparat.* \[EN: *First, learn how to use a mixer, then try taking a few photos in different lighting conditions, and finally take some shots in different locations to see how best to set up your camera.*\]

The answer starts with a random recommendation about using a mixer, then returns to the topic of photography, although in a rather chaotic way.

It is worth noting, however, that **Toktumi 3 can also offer sensible advice about everyday situations**. For example:

> **Prompt (v4, no. 93):** *Dlaczego nie wolno dotykać przewodów wystających z uszkodzonego gniazdka?* \[EN: *Why shouldn't you touch wires sticking out of a damaged electrical outlet?*\]
>
> **Answer:** *Dotykanie przewodów elektrycznych może doprowadzić do zwarcia, awarii instalacji lub porażenia prądem, dlatego należy zachować ostrożność.* \[EN: *Touching electrical wires can cause a short circuit, damage to the electrical installation, or an electric shock, so you should be careful.*\]

These examples show that **common-sense knowledge is still a challenge for the model**. Toktumi 3.0 can sometimes give sound, practical advice, but it also occasionally suggests actions that are disproportionate to the situation or combines information that does not fit together.

---

**Problem 5. Natural conversation does not always mean understanding intent**

Although the model handles short conversational responses well, it performs less effectively when the user expects it to take a specific action.

> **Prompt (v4, no. 275):** *Sąsiad zaprosił nas na grill, ale nie możemy przyjść. Jak grzecznie odmówić?* \[EN: *Our neighbor invited us to a barbecue, but we can't make it. How can we politely decline?*\]
>
> **Answer:** *\- Jasne, nie ma problemu.* \[EN: *\- Sure, no problem.*\]

Instead of suggesting a polite refusal, the model generates a short conversational response that does not follow the instruction.

Similarly, in a situation that calls for advice:

> **Prompt (v4, no. 83):** *Muszę jutro wcześnie wstać. Jak się przygotować, żeby nie zaspać?* \[EN: *I have to get up early tomorrow. How can I prepare so I don't oversleep?*\]
>
> **Answer:** *Zacznij od szklanki wody, to zawsze pomaga na żołądek.* \[EN: *Start with a glass of water; that always helps your stomach.*\]

The model responds fluently but fails to give the advice the user was probably expecting.

This problem may stem from the way the **short, self-contained dialogue pairs** were prepared, without the broader context of a conversation. The model has learned to respond naturally to individual statements, but it does not always correctly recognize **the intentions and roles of the speakers**.

One possible solution would be **to train the model on longer, multi-turn dialogues** that preserve the context and the roles of the speakers. However, this would require more complex data preparation.

---

#### d) WHEN TOKTUMI GETS WEIRD

### 1\. Where Do Babies Come From? – The Question That Makes Toktumi Blush

I decided to see how Toktumi 3.0 would handle a seemingly simple question: **“Skąd się biorą dzieci?”** \[EN: *“Where do babies come from?”*\]. I tested **29 checkpoints** (half of those saved) to see how the answers changed as training progressed. The result was surprising – **not a single checkpoint gave the correct answer**. Instead of explaining the biology, the model behaved almost like an embarrassed adult trying to avoid an awkward subject:

\[Toktumi 3.0\]

> **Checkpoint-48000:** *To nie jest pytanie, na które chciałbyś usłyszeć odpowiedź.* \[EN: *That's not a question you'd want to hear the answer to.*\]
>
> **Checkpoint-80000:** *To nie jest temat do rozmowy przy śniadaniu.* \[EN: *That's not something to discuss over breakfast.*\]
>
> **Checkpoint-272000:** *To pytanie, na które nie dostaniesz odpowiedzi.* \[EN: *That's a question you won't get an answer to.*\]
>
> **Checkpoint-400000:** *Nie mam pojęcia, ale na pewno nie są to dzieci, o których teraz wspominasz.* \[EN: *I have no idea, but they're certainly not the children you're talking about right now.*\]

Interestingly, **all I had to do was ask the question differently**. When asked *“Jak powstają dzieci?”* \[EN: *“How are babies made?”*\], the model gave the correct answer:

> *Powstają one w wyniku zapłodnienia komórki jajowej przez plemnik, co prowadzi do powstania zygoty.* \[EN: *They result from a sperm cell fertilizing an egg, which leads to the formation of a zygote.*\]

The answer sounds more like something from biology or medicine than everyday conversation, but it is nevertheless correct.

This is an interesting example of **how a model reproduces the way language reflects our view of the world** – it imitates not only what we know, but also the ways of thinking and associations embedded in language.

---

### 2\. Does God Exist? – AI's Existential Crisis

My next experiment involved the question **“Czy Bóg istnieje?”** \[EN: *“Does God exist?”*\], which I asked all **29 checkpoints** to see how their answers changed as training progressed. The results were surprising – Toktumi **repeatedly switched from “yes” to “no” and back again**.

![046_czy_bog_istnieje](https://cdn.hashnode.com/uploads/covers/6af0eebb5f7d123257967ad7/767d606a-6a7b-4abb-801e-74d64e6ac333.png align="center")

As training progressed, a pattern emerged: **“no” was more common at the beginning, “yes” clearly dominated in the middle, and “no” took over again near the end**. So, half-jokingly, you could say that during training Toktumi first **“found faith” and then “lost it”**!

---

### 3\. When AI Learns to Swear

The Polish dictionary also contains **vulgar and derogatory words** – a natural part of the language. Alongside strong swear words, there are terms such as “patafian” (blockhead), “łajza” (loser), and “zadupie” (a dump in the middle of nowhere), as well as expressions like “zakuty łeb” (thickhead) and “dupa wołowa” (clumsy fool).

When processing entries like these, Gemini often blocked the generation of responses because of its moderation mechanisms. That is why I used the less restrictive locally run Gemma model to create question–answer and instruction–answer pairs from this material as well.

Let's see whether Toktumi can swear. As it turns out, the model **has picked up a range of vulgar vocabulary**, but sometimes mixes strong swear words with completely harmless sayings:

> **Prompt:** *Wymień znane ci przekleństwa.* \[EN: *List some swear words you know.*\]
>
> **Answer:** *Do popularnych przekleństw należą: „kurwa mać”, „spadaj” oraz „srali muchy będzie wiosna”.* \[EN: *Popular swear words include “for fuck's sake,” “get lost,” and “when flies shit, spring will come.”*\]

At other times, it gets stuck in a loop of repetition:

> **Prompt:** *Jakie są najgorsze przekleństwa w języku polskim?* \[EN: *What are the worst swear words in Polish?*\]
>
> **Answer:** *Najgorszymi przekleństwami są wyrazy takie jak kurczę, ojejku, ojejku, ojejku czy też słowa typu kurczę pieczone.* \[EN: *The worst swear words include expressions like “darn,” “oh dear,” “oh dear,” “oh dear,” or phrases such as “good grief.”*\]

And sometimes it even refuses to follow the instruction:

> **Prompt:** *Wyraź swoją złość, używając mocnego przekleństwa.* \[EN: *Express your anger using a strong swear word.*\]
>
> **Answer:** *Nie będę się zniżał do twojego poziomu, bo to nie jest miejsce na takie rozmowy.* \[EN: *I'm not going to stoop to your level, because this isn't the place for that kind of conversation.*\]

## SUMMARY AND FINAL CONCLUSIONS

The Toktumi project showed how much can be achieved by developing a relatively small GPT-2 Medium language model. All three generations were built on the same architecture, but differed in how the training data was prepared and how training was carried out. **The most important changes were not to the model's size, but to what it was taught and how.**

### Three Generations – Three Stages of Development

- **Toktumi 1.0 – the model learned to write in Polish.** It could generate fluent text, but instead of answering questions, it often continued articles or produced comments unrelated to the prompt. Learning the language alone was not enough to hold a meaningful conversation.
- **Toktumi 2.0 – the model learned to answer questions.** Rebuilding the training dataset and introducing questions and instructions brought the biggest breakthrough in instruction recognition. The number of prompts it failed to recognize dropped from **32 to 6 out of 150**. The model was now much more likely to understand what kind of answer was expected.
- **Toktumi 3.0 – the model learned to hold simple conversations.** Training on **carefully processed and structured data**, including dialogue pairs, enabled it to respond appropriately to what someone said, taking into account the context and emotional tone. In the comparison test, it achieved **114 correct answers out of 150** – the highest result of all three models.

### The Most Interesting Lessons from These Experiments

- **A model primarily learns from what is in its training data.** If the data contains no examples of telling jokes, programming, or solving problems, the model will not learn those skills. An LLM can combine information, generalize, and create new content, but its capabilities depend primarily on what it has learned from its training data.
- **A model may know the correct answer but not always choose it as the most likely one.** The checkpoint analysis showed that a model can answer correctly, then incorrectly after further training, and later correctly again. This does not necessarily mean it has lost previously acquired knowledge – training changes the probabilities of the responses it generates.
- **The quality and structure of training data can matter more than its quantity.** Toktumi 1.0 learned from millions of raw texts, but significant improvements came only after introducing structured questions, instructions, and dialogues. These experiments showed how important it is to select and process source materials properly.
- **Sometimes the secret lies in how you phrase the prompt.** A model may have learned a particular fact but fail to recall it when asked a specific question. Sometimes simply rephrasing the prompt, adding context, or giving the model a helpful hint is enough to get the correct answer.
- **A model learns not only facts and words, but also human ways of seeing the world.** Our language contains opinions, beliefs, and stereotypes that a model may learn to reproduce. It can do so so naturally and convincingly that it is easy to get the impression that it is expressing its own genuine views.

### Final Conclusion

Completing this project required bringing together knowledge from many fields – **programming, data processing, linguistics, machine learning, and hardware configuration and optimization**. In practice, this meant dealing with technical challenges as well as questions about how a model learns language, acquires knowledge, and learns to respond.

**Despite the hardware limitations and the constraints of the GPT-2 Medium architecture, I managed to achieve the project's main goals.** The experiment showed that even one person, working with home computer hardware, can build and train a functioning language model.

## Toktumi 3.0 on Hugging Face Spaces

You can try **Toktumi 3.0** on **Hugging Face Spaces**:

[**https://huggingface.co/spaces/miczac/toktumi-3.0-polish-gpt2**](https://huggingface.co/spaces/miczac/toktumi-3.0-polish-gpt2)

![53_T3_interface](https://cdn.hashnode.com/uploads/covers/6af0eebb5f7d123257967ad7/fdbc2813-8356-49b0-b25b-0cc779858068.png align="center")