Project Overview

As a personal project, I decided to build and train an LLM from scratch, designed to operate in Polish. The project is educational, research-oriented, and non-commercial, and its main goal was to gain hands-on experience in building and training language models.

I started the project in 2023. I chose the GPT-2 architecture developed by OpenAI. It was one of the groundbreaking language models that played an important role in the development of generative artificial intelligence. With its code and model weights publicly available, it was a good starting point for experiments at the time.

The idea was to build a similar model from start to finish, although on a smaller scale. This meant independently collecting texts (web scraping), cleaning the data, preparing a large training dataset, developing optimized training code, completing the training process on a home PC, and then finding the most effective prompting methods for the finished model.

I used the Python programming language and the PyTorch and Transformers libraries throughout the project.

The project was ultimately completed in three stages:

Stage 1 - Toktumi 1.0 (2024): I built and trained the first working version of the model from scratch. I used 3 115 002 text samples collected from various online sources and other materials. I prepared my own tokenizer, carried out the training, and performed the first tests of response quality.

Stage 2 - Toktumi 2.0 (2025): I trained a new model from scratch, using an improved training dataset containing 2 336 162 text samples. Based on the experience and lessons learned from the previous stage, I modified both the structure and the content of the training dataset.

Stage 3 - Toktumi 3.0 (2026): I fine-tuned Toktumi 2.0 on an entirely synthetic dataset of 6 190 420 shorter records, consisting exclusively of question–answer pairs, instruction–answer pairs, and simple dialogue pairs. The goal was to improve the model's factual knowledge, expand its language capabilities, and enhance the quality of its generated responses.

I describe each of the three stages in detail in the following parts of this series.

CHAPTER I. Toktumi 1.0. First Training Run.

1. Choosing the Model Size

Of the four available GPT-2 model sizes (XL, Large, Medium, Small), I chose Medium because it offered a reasonable compromise between my hardware resources and the quality of the generated text. Another limiting factor was the amount of VRAM (12 GB) available on my NVIDIA RTX 3060 graphics card. Training GPT-2 Medium with my chosen configuration used approximately 11 GB. I considered this the optimal choice.

Available GPT-2 model sizes

• GPT-2 Small: 124M, or 124 million parameters
• GPT-2 Medium: 345M, or 345 million parameters << selected model
• GPT-2 Large: 762M, or 762 million parameters
• GPT-2 XL: 1.5B, or 1.5 billion parameters

2. Analyzing OpenAI's Approach (GPT-2)

I began by reviewing the available information on GPT-2, published by OpenAI on Hugging Face and GitHub, to learn from the approaches used by the model's creators.

Some background: the original GPT-2 model, introduced by OpenAI in February 2019, was trained on a dataset of approximately 40 GB, called WebText [source: OpenAI], [source: Hugging Face]. This dataset contained approximately 8 million text samples collected from websites. A partial list of those URLs was published on GitHub. However, the company did not release the dataset itself. The available sources still provide some insight into its contents and the methodology used by OpenAI.

I also drew inspiration from Polish-language GPT-2 models available at the time, developed by Sławomir Dadas (polish-gpt2) and Dariusz Kleczek (papuGaPT2).

3. Data Collection

Selected sources:

To train Toktumi 1.0, I used 3 115 002 text samples that I had collected from the following sources: (Wykop, Wikipedia, Fakt, Pudelek, Polsat News, books, and jokes).

My goal was to gather texts representing natural, colloquial language and build a model capable of holding conversations on a wide range of topics in an informal, conversational style.

At this stage, I did not yet know which types of text would work best in the training dataset. My choice of materials was therefore partly experimental — I made many decisions through trial and error, gradually learning about the model's capabilities and possible directions for further development.

a) Wikipedia - I used a data dump of the Polish-language edition of Wikipedia, made available by the Wikimedia Foundation for download under the Creative Commons CC BY-SA 4.0 license. I then converted the XML file into plain text.

I extracted 1 485 976 entries, of which I ultimately used approximately half (718 129), so that Wikipedia's style would not 'dominate' the model's responses.

Wikipedia texts are encyclopedic entries — rich in information, focused on facts, and written in neutral language without an emotional tone or personal opinions.

b) Fakt.pl - To collect texts, I used various web scraping methods for automatically downloading and processing website content. I wrote scripts in Python, using the Scrapy library and the Scrapy-Splash extension running in a Docker container.

The scripts automatically downloaded articles, extracted their content from HTML, and saved each text in a separate TXT file.

In this way, I collected 573 445 text samples, ready for further processing and model training.

Fakt publishes texts on a wide range of topics, including news reports and opinion pieces as well as more tabloid-style content. Its articles are often sensational or emotional and may express opinions. The site offers, among other things, articles, reviews, advice pieces, and horoscopes — a broad range of topics. Texts of this kind help the model develop a more colloquial style and expand its knowledge of the world across different fields.

Later tests and experience showed that this was one of the most valuable sources of text in the entire project, primarily because of its accessible language, variety of genres, and very wide range of topics.

How Scrapy works:

c) PolsatNews.pl - I used the same method (Scrapy, Splash, Docker) to collect texts from this news website.

The challenge was finding a list of article URLs to download, since the website did not provide one. To solve this, I wrote a crawler in Python using the Requests and BeautifulSoup libraries. It automatically found links to articles, which then served as the basis for scraping their content with Scrapy and Splash. In this way, I collected 126 004 text samples.

Polsat News publishes news reports on current events in Poland and abroad. Its news style is formal, factual, and straightforward. The articles contain information from official, verified sources. This type of text helps the model expand its knowledge of the world and learn to use neutral language, but in a style different from Wikipedia's.

d) Pudelek.pl - I used the same web scraping method (Scrapy, Splash, Docker) to collect texts. In this way, I gathered 165 229 text samples.

Pudelek features gossip-oriented content — short articles about celebrities, stars, and show business. They are written in a distinctive style full of irony and snark, often playing on readers' emotions. Texts of this kind help the model imitate lively, informal language, and the typical 'Pudelek style' is immediately recognizable.

Pudelek also publishes distinctive, often biting user comments. These comments were downloaded along with the articles. Ultimately, however, I did not use them, because they appear on the website in a random order. As a result, when the model learned from them, it could not maintain a logical connection between the prompt and the response.

e) Wykop - Wykop is a popular website where users share online "finds," publish posts, and comment on and rate content posted by others. For training the model, I considered it a valuable source of internet-style user comments.

To collect the texts, I used Python and the Selenium library with GeckoDriver, which makes it possible to automate the Firefox browser. In this way, I collected 1 413 999 text samples, including posts and user comments.

Ultimately, however, it turned out that using internet comments, like those from Pudelek, was not a good choice. The comments often contained chaotic, offensive, or off-topic statements that negatively affected the quality of the model's responses. That is why, in later versions of Toktumi, I completely abandoned this data source.

How web scraping with Selenium works:

f) Jokes - I also decided to add jokes collected from more than a dozen publicly accessible websites to the training dataset. This was an experiment to see whether the model could learn to generate coherent humorous texts.

It turned out, however, that teaching a model humor is much harder than I initially expected. It requires an understanding of irony, wordplay, and various linguistic subtleties. Unexpectedly, though, jokes helped the model learn the question–answer (QA) structure — the principle that a question should be followed by an answer. This structure often appears in jokes, whereas the other texts in the training dataset generally did not contain it.

After removing duplicates, I collected 88 897 jokes.

The dataset also required additional cleaning and filtering of offensive or inappropriate content. Jokes are often posted directly by internet users, without adequate moderation. Attempts to remove such content did not always produce satisfactory results. This showed how difficult it is to prepare a valuable training dataset from texts obtained from open online sources.

To download the texts, I used automated browser control (Firefox, GeckoDriver, Selenium, and Python).

g) Books - Finally, I added texts from 285 books found online, divided into 29 299 fragments. These included user manuals, guides, popular science books, reportage, and cookbooks, among others. The aim was to provide additional, more in-depth content.

I found that fiction and imaginative literature in the broad sense were unsuitable for training: plays, poetry, and even movie subtitles. As the tests showed, the model would then start producing language that was overly abstract, taken out of context, or difficult to understand, or it would adopt a novel-like style that made casual conversation difficult. Therefore, when selecting texts in this category, I limited myself to non-fiction and works discussing the real world.

The book texts had to be split into smaller fragments while preserving the broader context. For this purpose, I used an overlapping sliding window method, in which the end of one fragment is repeated at the beginning of the next. This helped maintain continuity and reduce the loss of context.

Splitting data using the overlapping method:

4. Creating the Training File (training set)

I saved each text in a separate .txt file. I chose this format because of its simplicity — text files are easy to browse, edit, and clean, without requiring an additional data structure such as JSON. This made the dataset easier to manage. For example, I could sort the files by size and remove the smallest ones — those that were empty or not useful for training the model.

I split the largest .txt files, exceeding 1024 tokens, into smaller fragments using the overlapping method.

Conversely, I combined shorter texts of the same type (such as jokes or short articles) into larger 'batches' of approximately 1024 tokens, to keep the training dataset well balanced. Each individual text within a batch was separated by a delimiter.

Combining texts of the same type into larger batches:

Each dataset also had to be cleaned. This was a very time-consuming process involving many stages, which I will not describe in detail here. However, it is worth emphasizing that cleaning and preparing the data takes the most time — more than collecting the data and training the model itself.

This shows just how much work this stage requires. At first it may seem less important, but in reality it is one of the most time-consuming and crucial parts of the entire model-training process. It is worth remembering the principle of "Garbage In, Garbage Out" — the quality of the input data has a fundamental impact on the quality of the results.

The next step was reducing the set of characters used in the collected texts. To do this, I converted all texts from UTF-8 encoding to Windows-1250 (up to 256 characters) and back again (to maintain compatibility with the tools used during training). During this conversion and additional cleaning, characters outside that set were replaced or removed.

As a result, the GPT-2 model did not have to process many unusual characters that added little linguistic value. Limiting the number of characters was also intended to improve tokenizer efficiency (in my case, the vocabulary size was 50 000 tokens).

Reducing the character set during conversion from UTF-8 to ANSI:

Next, I combined all the .txt files containing the collected texts into one large training dataset (training set), which was then used to train the model. Each of the more than 3 million text samples was separated by the <|endoftext|> token — the same delimiter used by OpenAI in the original GPT-2 model.

I also set aside a smaller validation set containing data in the same format that did not overlap with the training set. I later used it to evaluate the model's quality (by measuring perplexity).

How the texts were divided into training and validation sets:

TOKTUMI 1.0 TRAINING DATASET SIZE (excluding delimiters):

Number of text samples: 3 115 002

Number of words: 658 078 092

Number of characters: 4 480 020 887

Number of tokens: 978 054 512

File size: 4.46 GB

5. Preparing the Tokenizer

A tokenizer is a set of files and rules that helps a model convert text into tokens — smaller units into which text is divided. The model operates on these tokens, converting them into numerical values and ultimately back into text. Proper tokenizer preparation is essential for data consistency, the quality of generated responses, and the model's overall performance. A poorly prepared tokenizer can significantly harm training and the quality of generated text.

Instead of using a ready-made tokenizer, I decided to build my own, tailored to the Polish language, using a large sample of the texts I had previously collected.

Files commonly included with a tokenizer: vocab.json, merges.txt, tokenizer_config.json, special_tokens_map.json, tokenizer.json.

Contents of the vocab.json file:

Below is the key section of the code used to create the tokenizer.

I chose a vocabulary size of vocab_size=50_000, close to that of the original GPT-2 tokenizer, which contained 50 257 tokens across all model variants.

I used the Tokenizers and Transformers libraries to create the tokenizer.

from tokenizers import ByteLevelBPETokenizer
from transformers import GPT2TokenizerFast

# Utworzenie i trening własnego tokenizera BPE
tokenizer = ByteLevelBPETokenizer()

tokenizer.train(
    files="tokenizer_teksty.txt",
    vocab_size=50_000,
    min_frequency=2,
    special_tokens=[
        "<|endoftext|>",
        "<unk>",
        "[PAD]"
    ]
)

# Zapis słownika i reguł BPE
tokenizer_path = "tokenizer_2025_endoftext"
tokenizer.save_model(tokenizer_path)

# Przygotowanie tokenizera do pracy z Transformers
fast_tokenizer = GPT2TokenizerFast(
    vocab_file=f"{tokenizer_path}/vocab.json",
    merges_file=f"{tokenizer_path}/merges.txt",
    unk_token="<unk>",
    pad_token="[PAD]",
    eos_token="<|endoftext|>"
)

fast_tokenizer.save_pretrained(tokenizer_path)

After creating the tokenizer, I needed to verify that it worked correctly. I tested text encoding and decoding and checked that the special token <|endoftext|> was correctly recognized as a delimiter. All tests passed successfully.

The tokenization process and its verification:

6. Creating an Empty GPT-2 Medium Model

Next, I created an empty GPT-2 model with no pretrained weights. The model configuration (number of layers and attention heads, embedding dimension, context length, and vocabulary size) was matched to the GPT-2 Medium variant and the tokenizer I had prepared. This was the starting point for training GPT-2 from scratch.

from transformers import GPT2LMHeadModel, GPT2Config

# Konfiguracja architektury GPT-2 Medium
config_medium = GPT2Config(
    vocab_size=50_000,  # Rozmiar słownika
    n_positions=1024,   # Maksymalna długość sekwencji
    n_ctx=1024,         # Długość kontekstu
    n_embd=1024,        # Wymiar osadzeń
    n_layer=24,         # Liczba warstw
    n_head=16,          # Liczba głowic atencji
    bos_token_id=0,
    eos_token_id=0,
    pad_token_id=2
)

# Inicjalizacja modelu z losowymi wagami
model_medium = GPT2LMHeadModel(config_medium)

# Zapis modelu i jego konfiguracji
model_medium.save_pretrained("GPT2-MEDIUM-empty_new")

7. Training the Model

I trained the model locally on an NVIDIA GeForce RTX 3060 (12 GB VRAM), using Python, PyTorch, Transformers, and CUDA. One of the first challenges was finding compatible versions of the libraries and drivers — incompatibilities often caused errors or prevented GPU acceleration from working.

After setting up the environment, it is important to check whether PyTorch correctly detects CUDA:

import torch

print("Czy CUDA dostępne =", torch.cuda.is_available())

Using a GPU significantly speeds up model training compared with running computations on the CPU alone.

TRAINING — KEY CHALLENGES:

Below is an excerpt from the training log. The values on the left show the number of completed steps and the total number of steps planned for the entire run (1140000/1771840). This means that 1 140 000 out of 1 771 840 planned steps had been completed. On the right, the value 'epoch': 6.43 indicates that the model is in its sixth epoch. The entire training run was planned for 10 epochs, so overall progress at this point was approximately 64%.

Key concepts for monitoring training progress:

• Steps indicate how many times the model's weights have been updated. A step involves processing one or more batches of data, calculating the loss, and updating the model parameters.
• Epochs indicate how many times the entire training dataset has been processed. One epoch is one complete pass through the training data. Multiple epochs are usually needed for the model to achieve good prediction quality.

Before I developed the final model-training code, I had to conduct many rounds of tests on smaller datasets (text files of approximately 50-100 MB). One of the more serious challenges was deciding: after how many epochs should training stop?

Initially, I tested code with early stopping and two datasets: a training set and a validation set. Training stopped automatically at around the seventh epoch. Based on this, I adopted a safe upper limit of 10 epochs for the final training code. Ultimately, I abandoned early stopping because it proved impractical for training on a large dataset over multiple sessions, when the process had to be broken into smaller stages (and repeatedly paused and resumed).

To determine the best time to stop training, measuring perplexity proved more practical, as I explain in greater detail later.

Below is the most important part of the code used to train Toktumi 1.0, including the hyperparameter configuration.

from transformers import (
    GPT2LMHeadModel, GPT2Tokenizer, TextDataset,
    DataCollatorForLanguageModeling, Trainer, TrainingArguments
)

# Model, tokenizer i dane treningowe
model_path = "GPT2-MEDIUM-empty_new"
tokenizer_path = "tokenizer_2025_endoftext"
train_path = "BIG_UCZ_fin_v2.txt"
output_dir = "GPT2-MEDIUM-Toktumi-1"

model = GPT2LMHeadModel.from_pretrained(model_path)
tokenizer = GPT2Tokenizer.from_pretrained(tokenizer_path)

# Dopasowanie tokenizera do modelu
tokenizer.add_special_tokens({"eos_token": "<|endoftext|>"})
model.resize_token_embeddings(len(tokenizer))

# Przygotowanie danych treningowych
train_dataset = TextDataset(
    tokenizer=tokenizer,
    file_path=train_path,
    block_size=230  # Długość sekwencji dopasowana do pamięci GPU
)

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer,
    mlm=False  # Uczenie przewidywania kolejnego tokenu
)

# Hiperparametry treningu
training_args = TrainingArguments(
    output_dir=output_dir,
    overwrite_output_dir=True,
    num_train_epochs=10,             # Maksymalna liczba epok
    per_device_train_batch_size=8,   # Wielkość batcha
    gradient_accumulation_steps=3,   # Akumulacja gradientów
    learning_rate=5e-5,              # Współczynnik uczenia
    fp16=True,                       # Mieszana precyzja (oszczędność VRAM)
    save_steps=7500,                 # Zapis checkpointu co 7500 kroków
    save_total_limit=300,            # Limit zapisanych checkpointów
    logging_steps=500,               # Logowanie postępu
    save_safetensors=False
)

# Inicjalizacja treningu
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    data_collator=data_collator
)

# Trening od zera
trainer.train()

# Alternatywnie: wznowienie z ostatniego checkpointu
# trainer.train(resume_from_checkpoint=True)

# Zapis wytrenowanego modelu i tokenizera
model.save_pretrained(output_dir)
tokenizer.save_pretrained(output_dir)

During training, the automatic checkpoint-saving feature proved extremely useful (in this case, 'save_steps=7500', meaning a checkpoint was saved every 7500 steps).

Directory containing the checkpoints:

Each checkpoint directory contains the complete model state as well as the necessary temporary files. Checkpoints make it possible to resume training from exactly the point at which a checkpoint was saved. However, storing many checkpoints requires a considerable amount of disk space.

This is particularly important for long training runs lasting many days, when power outages, the need to restart the computer, or other unforeseen events may occur.

END OF TRAINING

Ultimately, the most suitable point at which to stop training turned out to be checkpoint-1140000, corresponding to approximately 6.44 epochs.

Training time: approximately 380 hours (over 15 days).

8. Evaluating Model Quality. Perplexity.

To assess the quality of the trained GPT-2 Medium model, I used perplexity. Perplexity measures how well a model predicts the next words in a sequence — the lower the perplexity, the better the model fits the data.

The model was evaluated using the previously separated validation set — data the model had not 'seen' before. The result is calculated using the trainer.evaluate() function, which returns the loss value.

perplexity = math.exp(eval_results["eval_loss"])

Because of the technical complexity of this task, I relied on existing solutions.

The approach works as follows: the code iterates through the saved checkpoints and calculates perplexity for selected ones. This makes it possible to estimate when further training is no longer worth the cost.

As the chart below shows, at a certain point the decline in perplexity slowed considerably, and further training would have required substantial computing resources for very little improvement. Based on this, I decided to stop training early, before reaching the planned 10 epochs.

9. Evaluating Model Quality. Prompting.

The second stage of model evaluation involved human evaluation — checking how Toktumi responds to specific prompts.

For these tests, I used greedy decoding — a method in which the model selects the most probable next token. As a result, the same prompt always produced the same response, providing a consistent reference point for comparing checkpoints.

This can be achieved by setting do_sample=False or, as in my tests, do_sample=True and top_k=1. The remaining parameters control response length and help prevent repeated passages.

Below is the key section of the code I used to generate and test Toktumi 1.0 responses.

import torch
from transformers import GPT2LMHeadModel, GPT2Tokenizer

# Wczytanie modelu Toktumi 1.0 i tokenizera
model_path = "GPT2-MEDIUM-Toktumi-1"
model = GPT2LMHeadModel.from_pretrained(model_path)
tokenizer = GPT2Tokenizer.from_pretrained(model_path)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device).eval()

# Przykładowy prompt
prompt = "Co to jest grawitacja?"
input_ids = tokenizer.encode(prompt, return_tensors="pt").to(device)

# Parametry generowania odpowiedzi
generate_args = {
    "max_length": 256,            # Maksymalna długość: prompt + odpowiedź
    "min_new_tokens": 2,          # Minimum 2 nowe tokeny
    "no_repeat_ngram_size": 2,    # Blokowanie powtarzających się par tokenów
    "repetition_penalty": 1.1,    # Kara za powtarzanie tokenów
    "do_sample": True,            # Włączony tryb samplingu
    "top_k": 1,                   # Wybór tylko najbardziej prawdopodobnego tokenu
    "pad_token_id": tokenizer.eos_token_id,
    "eos_token_id": tokenizer.eos_token_id
}

# Generowanie i dekodowanie odpowiedzi
with torch.no_grad():
    output = model.generate(input_ids, **generate_args)

generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
print(generated_text)

The model produces the most probable response.

Prompt: Co to jest grawitacja? [EN: What is gravity?]

Answer: - To jest siła, która przyciąga przedmioty. [EN: - It is a force that attracts objects.]

To identify the best checkpoint in terms of response quality, I tested selected checkpoints on a set of 150 prompts with different structures: sentence completions, questions, and instructions:

• Sentence completions, e.g. "Warszawa jest stolicą..." ("Warsaw is the capital of...") and "Ziemia krąży wokół..." ("Earth orbits..."). The model generally recognized this as a structure that required completing a sentence. It also often provided a correct or the most "intuitive" completion. This category produced the fewest errors.
• Questions, ending with a question mark, e.g. "Co to jest planeta?" ("What is a planet?") and "Do czego służy ołówek?" ("What is a pencil used for?"). The model was expected to answer a question, just as a person would. It often recognized the question–answer pattern correctly. However, it also frequently struggled both with understanding the structure (that a question calls for an answer) and with factual knowledge about the world (it responded, but incorrectly).
• Instructions, e.g. "Wymień trzy warzywa" ("Name three vegetables") and "Podaj przykład pojazdu" ("Give an example of a vehicle"). An instruction is not a question, but it still requires an appropriate reaction — in this case, following the instruction or providing an answer. This type of prompt was the most difficult for the model, because it often failed to recognize that a response was required.

The full list of 150 prompts and Toktumi 1.0 responses (using 'greedy decoding') -> VIEW.

10. Ending the Training Run

After analyzing perplexity and the results of the 150-prompt tests, I selected checkpoint-1140000, saved after approximately 6,44 epochs, as the best checkpoint.

Further training was no longer producing enough improvement in response quality, so I decided to end the training run at this point.

The selected checkpoint contains the saved model weights. It forms the basis of the final Toktumi 1.0 model.

11. Final Model Evaluation

The final step was to evaluate the model and determine to what extent it met the objectives established at the beginning of the project — namely, whether it could hold conversations on a wide range of topics using everyday, conversational language.

Here, I will focus on the most important observations. The examples below were generated using greedy decoding.

a) Recognizing the Prompt

FINDINGS: The model often recognizes the structure of a prompt and generates a response that is grammatically and syntactically correct. It also uses an appropriate style. Interestingly, the model learned the Question–Answer structure mainly from the collection of jokes — many jokes take the form of a question followed by an answer (the punchline). This helped the model learn a simple rule: when a question is asked, it should be answered.

Prompt: Co to jest grawitacja? [EN: What is gravity?]

Answer: - To jest siła, która przyciąga przedmioty. [EN: - It is a force that attracts objects.]

Prompt: Co robi kucharz? [EN: What does a cook do?]

Answer: - Smaży kotlety. [EN: - Fries cutlets.]

Prompt: Warzywa są zdrowe, bo [EN: Vegetables are healthy because]

Answer: Warzywa są zdrowe, bo zawierają dużo witamin i minerałów. [EN: Vegetables are healthy because they contain plenty of vitamins and minerals.]

b) Basic Knowledge of the World

When a prompt is relatively simple, concerns commonly known facts, and does not require complex reasoning, the model often gives a correct answer. It therefore has general knowledge of the world and knows basic facts.

It performs best on prompts that require completing a sentence — this is the easiest format for the model to decode.

Prompt: Stolicą Francji jest [EN: The capital of France is]

Answer: - Stolicą Francji jest Paryż. [EN: - The capital of France is Paris.]

Prompt: Najwyższą górą na świecie jest [EN: The highest mountain in the world is]

Answer: Najwyższą górą na świecie jest Mount Everest, który ma wysokość 720 metrów. [EN: The highest mountain in the world is Mount Everest, which is 720 meters tall.]

(although the model gets the height wrong)

Prompt: Najdłuższą rzeką w Polsce jest [EN: The longest river in Poland is]

Answer: Najdłuższą rzeką w Polsce jest rzeka Wisła. [EN: The longest river in Poland is the Vistula River.]

c) Incorrect Answers

Sometimes the model generates a response that has the correct structure but is factually wrong — giving the impression that it is "making things up." This phenomenon is known as hallucination. It occurs when a model presents false or invented information as though it were factual.

As the tests show, hallucinations occur more frequently in less well-trained models, especially during the early stages of training. This is somewhat like a person who does not know the answer but tries to guess it anyway.

Prompt: Do czego służy ołówek? [EN: What is a pencil used for?]

Answer: - Do czyszczenia akwariów. [EN: - For cleaning aquariums.]

Prompt: Z czego robi się mąkę? [EN: What is flour made from?]

Answer: - Z proszku do pieczenia. [EN: - From baking powder.]

Prompt: Najszybszym zwierzęciem na lądzie jest [EN: The fastest land animal is]

Answer: Najszybszym zwierzęciem na lądzie jest koń. [EN: The fastest land animal is a horse.]

d) Lack of Understanding

Sometimes the model misinterprets the structure of a question or provides an internally contradictory answer.

Prompt: Księżyc świeci w [EN: The Moon shines at]

Answer: Księżyc świeci w nocy, a księżyc świeci nocą. [EN: The Moon shines at night, and the Moon shines during the night.]

Prompt: Co to jest ogień? [EN: What is fire?]

Answer: - To jest taki mały, biały, czerwony, świecący, czarny, żółty, zielony, niebieski, pomarańczowy, brązowy, fioletowy. [EN: - It's this little, white, red, glowing, black, yellow, green, blue, orange, brown, purple thing.]

Prompt: Jakie jedzenie przygotowujesz, kiedy masz gości? [EN: What food do you prepare when you have guests?]

Answer: - Na początku tygodnia. [EN: - At the beginning of the week.]

Additionally, when a prompt contains instructions, the model very often fails to recognize the expected type of response at all.

Prompt: Podziel się swoimi planami na weekend. [EN: Share your plans for the weekend.]

Answer: Horoskop dzienny - Lew. Dzisiejszy dzień przyniesie Ci wiele możliwości do rozwoju osobistego i zawodowego... [EN: Daily horoscope - Leo. Today will bring you many opportunities for personal and professional growth...]

(unexpectedly responds in the style of a horoscope)

Prompt: Wymień dwie przyprawy. [EN: Name two spices.]

Answer: Wymień dwie przyprawy. To proste! [EN: Name two spices. It's easy!]

(quotes a passage from an article on Pudelek or Fakt)

e) The 'Komentarz:' ('Comment:') Section

As I mentioned earlier, texts from the 'Wykop.pl' website were included in the Toktumi 1.0 training dataset together with user comments. This was the largest category of texts in the training dataset (1 413 999 samples).

The tests showed that internet comments significantly reduced the quality of the generated responses. They were rarely relevant to the topic and, despite moderation, still contained judgmental and offensive language, reinforced harmful stereotypes, and even included conspiracy theories.

Rather than simply misinterpreting the instruction, the model imitates the style of internet comments — often chaotic, judgmental, and inappropriate for the role of a friendly assistant.

Prompt: Jakie znasz środki transportu? [EN: What means of transportation do you know?]

Answer: (...)

Comment: Nie znam, ale się wypowiem. [EN: I don't know, but I'll give my opinion anyway.]

Prompt: Co sądzisz o podróżach po Europie? [EN: What do you think about traveling around Europe?]

Answer: (...)

Comment: Nie wiem, czy to dobry pomysł. W Polsce jest dużo taniej niż w Niemczech. [EN: I don't know if that's a good idea. It's much cheaper in Poland than in Germany.]

Prompt: Ziemia krąży wokół [EN: The Earth revolves around]

Answer: Ziemia krąży wokół Słońca. [EN: The Earth revolves around the Sun.]

Comment: To jest niesamowite, że w XXI wieku ludzie nadal wierzą w takie bzdury. (...) To tak jakby wierzyć, iż Ziemia krąży dookoła Słońca i jest okrągła. Albo, żeby wierzyć w płaską ziemię. No ale przecież ziemia jest płaska, więc ... [EN: It's incredible that in the 21st century people still believe such nonsense. (...) It's like believing that the Earth revolves around the Sun and is round. Or believing in a flat Earth. But of course the Earth is flat, so ...]

12. Recommendations and Next Steps

Based on these observations, I developed a short set of recommendations for further improving the model. I implemented the first two while developing the next version — Toktumi 2.0.

RECOMMENDATIONS:

1. Remove internet comments from the training dataset.
2. Conduct fine-tuning of the model (or train it again from scratch) on question–answer (QA) and instruction–answer (instruction fine-tuning) datasets.
3. Apply advanced text-cleaning techniques, with more effective filtering of undesirable and harmful content.
4. Consider using a GPT-2 model with more parameters: GPT-2 Large (762M) or GPT-2 XL (1.5B). GPT-2 Medium may have too little capacity and a limited attention mechanism to effectively retain knowledge and generate more complex responses. However, hardware constraints may be a barrier.
5. Use a newer model architecture that offers greater capabilities than GPT-2. However, hardware limitations and the high cost of training more advanced models may be obstacles.

Toktumi 1.0 on Hugging Face Spaces

You can try Toktumi 1.0 on Hugging Face Spaces:

https://huggingface.co/spaces/miczac/toktumi-1.0-polish-gpt2

NEXT PART