LLMs GuruLLMsGuru
Find your AI
AI ModelsAI ToolsLearnGlossaryAI NewsFree ToolsFind your AISpeakRightAstrology
Explore All Tools
← The AI Dictionary
Plain-English definition

Pretraining

In one breath

The long, expensive first phase where a model reads a huge slice of the internet and learns how language works — before anyone teaches it manners.

In more detail

Pretraining is a model’s childhood: the long, costly first phase where it reads an enormous slice of the internet and learns how language works by endlessly predicting missing words. No manners, no helpfulness yet — just raw pattern-learning at staggering scale.

It matters because this is where most of a model’s knowledge, and most of its cost, comes from. Everything afterwards — fine-tuning, RLHF — is finishing school by comparison. When a lab announces a new model generation, months of pretraining are what’s really being announced.

📌 See it in action

During pretraining the model is shown text like ‘The Eiffel Tower is in ____’ and scored on filling the blank. Repeat that trillions of times, across every subject humans write about, and the ability to converse emerges — though at this stage it’s a brilliant savant with no social graces. Those come later.

Goes with
Training Data →RLHF →Fine-Tuning →

Now put the vocabulary to work.

Meet the models →Which AI is mine?