Bedtime Guide to Machine Learning Magic
complete
Topic
Explain the various terms involved in machine learning models (quantization, vectors, parameters, layers, etc) in as entertaining a way as possible for easy listening before bed.
Audio
Generated Text
Imagine, for a while, that we are walking through a very quiet workshop at night. Not a bright laboratory with alarms and polished chrome, but a warm, slightly dusty place with wooden drawers, soft lamps, labeled jars, and long tables where patient machines are being assembled. Outside, perhaps, rain is making a faint sound against the windows. Inside, everything has been set down carefully: little gears, folded diagrams, spools of thread, boxes of tiles, magnifying glasses, notebooks full of numbers. This workshop is a machine learning model. Or, more precisely, it is one way to picture one. Machine learning has gathered around itself a whole little forest of terms: parameters, weights, vectors, embeddings, layers, tokens, inference, training, quantization, gradients, attention. Some of these words sound ordinary. Some sound like they were borrowed from physics. Some sound like they escaped from a mathematics department in the middle of the night. But most of them point to ideas that are not as cold as they first appear. They are ways of describing how a machine stores patterns, how it notices similarities, how it makes guesses, and how it turns a great ocean of examples into something that can respond to a new moment. So let us wander through the workshop slowly. We can begin with a simple question. What is a machine learning model? A model is, in the broadest sense, a shaped thing that helps us make sense of something else. A globe is a model of Earth. It is not Earth. There are no actual clouds drifting over its blue paint, no whales moving through its plastic oceans. But it preserves certain relationships: continents are roughly where they should be, distances make a kind of sense, the poles are at opposite ends. A recipe is a model of a cake. A map is a model of a city. A melody hummed under your breath is a model of a song. A machine learning model is a mathematical object that has been shaped by data. It has looked at many examples and adjusted itself so that, when given something new, it can produce a useful answer, prediction, classification, continuation, translation, recommendation, or image. It is not usually a list of rules written by a person. That is one of the important differences. Older software often works like a cookbook: if this happens, do that; if the button is pressed, open the window; if the number is larger than ten, display a warning. Machine learning is more like teaching by examples. You show the system thousands or millions or billions of cases, and it slowly changes its inner settings until it starts to capture the patterns. Not perfectly. Not magically. But often impressively. Those inner settings are where we meet one of the most common terms: parameters. A parameter is a number inside the model that can be adjusted during training. If we imagine our workshop machine as having a vast wall of tiny dials, each dial set to some value, then the parameters are those dials. One dial might be set to 0.37, another to negative 1.12, another to 5.004, another to something much smaller. A modern language model may have billions of these dials. Each one, by itself, is not very meaningful. You cannot usually point to a single parameter and say, “Ah, this is the one that understands semicolons,” or “This dial contains the memory of sunsets.” The knowledge is spread out, like the flavor of tea diffused through water. This is one of the first gentle surprises. A model’s memory is not like a library shelf where facts sit in alphabetical order. It is more like the shape of a landscape: hills, valleys, paths, slopes. The parameters define that landscape. When information moves through the model, it follows the contours created by those learned numbers. You may also hear parameters called weights. Strictly speaking, weights are a kind of parameter, and in many contexts people use the words almost interchangeably. The word “weight” gives us a useful image. Suppose a model is making a decision and many little signals are whispering suggestions. One says, “This looks like a cat.” Another says, “This looks like a dog.” Another says, “The ears are pointy.” Another says, “The background is a sofa.” A weight decides how much each whisper matters. A large weight means, “Listen closely to this signal.” A small weight means, “This signal is faint.” A negative weight means, in effect, “When this signal appears, lean away from that conclusion.” So parameters are the adjustable parts. Weights are the strengths of connections. Biases, another common term, are also adjustable numbers, and they act a little like a model’s built-in tendency before evidence arrives. If weights are the influence of incoming signals, biases are like the resting tilt of a scale. Imagine an old balance scale on a counter. If it is perfectly balanced, then the next pebble matters cleanly. But if one side is already slightly heavier, the scale has a preference. A bias in a neural network works somewhat like that. It shifts the result before or alongside the input. This is useful because real patterns often do not pass neatly through zero. A model needs not only to know how strongly signals matter, but also where to place its thresholds. Now, what are these signals moving through? They move through layers. A layer is a stage of computation. You can think of layers as rooms in the workshop. Information enters the first room in a raw or simple form. Something happens to it. It is passed to the next room. Something else happens. Then the next room. And so on. In an image model, early layers might notice simple features like edges, corners, patches of brightness, or tiny textures. Later layers combine these into larger patterns: circles, fur, wheels, eyes, handles, leaves. Still later layers may recognize whole objects or scenes. In a language model, the layers do not work in quite such easily visible ways, but the spirit is similar. Earlier layers may handle simpler relationships: local word patterns, grammar-like signals, basic associations. Deeper layers combine information more abstractly: who did what to whom, what tone is being used, what topic is under discussion, what might come next. “Deep learning” gets its name from this depth: many layers stacked together. Not deep in the poetic sense, though it can feel that way, but deep like a tall stack of transformations. Each layer takes in numbers and produces new numbers. This is worth pausing over, because it is both plain and strange. To the model, everything becomes numbers. A sentence, a picture, a sound recording, a chess position, a molecule, a shopping pattern: all of it must be translated into numerical form before the model can work with it. This brings us to vectors. A vector is a list of numbers. That is the simplest definition, and for now it is enough. A vector might look like this: [0.2, -1.7, 3.4, 0.01] In real models, these lists can be hundreds or thousands of numbers long. A vector is like a coordinate in a space. If you have two numbers, you can imagine a point on a flat map: east-west, north-south. If you have three numbers, you can imagine a point in a room: left-right, forward-back, up-down. If you have 768 numbers, or 4096 numbers, you cannot picture it directly, but mathematically it still behaves like a point in a very high-dimensional space. High-dimensional space is a phrase that sounds like it belongs near a telescope. But it is simply a space with many different directions in which something can vary. Let us make it homely. Suppose you were describing a cup of tea. You might rate it on warmth, sweetness, bitterness, strength, floral aroma, milkiness, color, and comfort. That is already eight dimensions. A tea could be a point in tea-space. A strong black tea would sit somewhere different from a pale chamomile, and both would sit far from a fizzy lemonade. Machine learning uses vectors to place things in meaning-spaces. Words, images, users, songs, products, paragraphs — they can all be turned into vectors. And once they are vectors, the model can compare them, combine them, move them around, and discover neighborhoods. This is where embeddings appear. An embedding is a vector representation of something. It is the model’s way of placing an item into a meaningful space. If the word “cat” becomes a vector, that vector is an embedding. If a whole paragraph becomes a vector, that too is an embedding. The important point is that similar things tend to be placed near each other. “Cat” may be near “kitten,” “dog” may be somewhat nearby, “refrigerator” farther away, and “parliamentary procedure” in another district entirely. Although, in a very large and subtle model, even “cat” and “parliamentary procedure” might be connected by some quiet alleyway: perhaps a debate about pet licensing, or a politician photographed with a cat. Meaning is full of side streets. Embeddings are one of the lovelier ideas in machine learning. They suggest that meaning can be arranged like a landscape. Not perfectly, not completely, but usefully. In this landscape, directions can sometimes correspond to concepts. One direction might move from singular to plural. Another from present tense to past tense. Another from formal to informal. Researchers have found amusing examples where vector arithmetic appears to capture analogies, such as king minus man plus woman landing near queen. These examples can be overstated, but they are charming because they show that relationships, not just items, can take shape in the geometry. Now, before a language model can embed words, it has to decide what the pieces are. This leads us to tokens. A token is a chunk of text the model processes as a unit. Tokens can be whole words, parts of words, punctuation marks, spaces, or fragments that seem odd to humans but are practical for the machine. The sentence “The moon is quiet tonight” might be split into tokens something like: “The,” “ moon,” “ is,” “ quiet,” “ tonight.” But a longer word like “unbelievable” might be split into “un,” “believ,” and “able,” depending on the tokenizer. A tokenizer is the little preparatory tool that chops text into tokens. It is like someone in the kitchen slicing vegetables before cooking begins. The model does not usually receive raw text as we see it. It receives token IDs: numbers corresponding to token pieces. “Moon” might be token 11739. A comma might be token 11. A common word might have a small ID; a rare fragment might have another. From there, tokens become embeddings, embeddings move through layers, layers transform them using parameters, and eventually the model produces an output. For a language model, the output is often a prediction of the next token. This simple phrase is almost comically modest: predict the next token. It sounds like a parlor game. Yet if you train a huge model to predict the next token across a vast amount of text, it begins to learn grammar, facts, style, reasoning patterns, dialogue structure, code syntax, jokes, recipes, and how people ask for directions when they are slightly lost. Why? Because predicting the next token well requires understanding a great deal about the world as represented in text. If the sentence is “The bread was placed in the hot oven until it turned…” the next word is more likely to be “golden” than “triangular,” though triangular bread is not forbidden by the universe. To make that guess, the model needs patterns about bread, ovens, cooking, adjectives, and ordinary human phrasing. A language model generates text by repeating this process. It predicts likely next tokens, chooses one, adds it to the text, then predicts the next one, and so on. Like footsteps in fresh snow, each step changes the path for the next. This brings us to probabilities. A model does not usually say, with absolute certainty, “The next token must be this.” Instead, it produces a distribution: many possible next tokens, each with a probability. After “The cat sat on the,” likely continuations include “mat,” “floor,” “sofa,” “windowsill,” and maybe “keyboard,” if the household is realistic. The model assigns scores to possibilities. These raw scores are often called logits. A logit is an unnormalized score before it has been turned into a probability. You might imagine a row of contestants before judges have converted their impressions into final rankings. The logits say, “This one is high, this one is low, this one is possible but unlikely.” Then a mathematical function, often softmax, turns those scores into probabilities that add up to one. Softmax is a wonderfully gentle-sounding word, like a fabric softener for numbers. Its job is to take a collection of scores and make the stronger ones stand out while still leaving room for weaker possibilities. When generating text, there are different ways to choose from the probability distribution. If you always pick the most likely next token, the model may become predictable, even dull. If you sample from the distribution, allowing less likely tokens to appear sometimes, the output can become more varied. This is where temperature enters. Temperature controls randomness in sampling. A low temperature makes the model conservative. It sticks close to the most likely words. A high temperature makes it more adventurous. It gives unusual words more of a chance. The metaphor is surprisingly good. At low temperature, the molecules barely move; the text settles into safe grooves. At high temperature, everything wiggles more; unexpected combinations appear. Too high, and the soup begins to leap out of the pot. Too low, and it may congeal. There are other sampling terms too, like top-k and top-p. Top-k means the model considers only the k most likely next tokens. If k is 50, it looks at the top 50 candidates and ignores the rest. Top-p, also called nucleus sampling, considers the smallest group of tokens whose probabilities add up to some threshold, like 0.9. It is as if the model says, “Let me consider the plausible neighborhood, but not every distant possibility.” Now let us return to training, because training is how the workshop machine gets its shape. Training is the process of adjusting the model’s parameters so it performs better on examples. At the beginning, a model’s parameters may be random. The dials are set without wisdom. It sees an input, produces an output, and that output is likely poor. Then we compare the output to what we wanted. The difference is captured by something called a loss. Loss is a number that measures how wrong the model is. Lower loss is better. If the model predicts the next token poorly, loss is high. If it predicts well, loss is low. Loss is not a moral judgment. The model has not been naughty. It is simply a signal: the current settings are not producing the desired result. The training process uses this signal to adjust the parameters. How does it know which way to adjust them? Here we meet gradients. A gradient tells us the direction in which a function increases most steeply. In training, we usually want to reduce loss, so we move in the opposite direction of the gradient. This process is called gradient descent. Imagine you are standing on a foggy hillside at night. You cannot see the whole landscape, but you can feel the slope under your feet. If your goal is to reach a lower valley, you take a small step downhill. Then you feel the slope again. Another step. Again and again. Gradient descent is like that. The model cannot inspect every possible setting of billions of parameters. That would be impossible. But it can estimate, from examples, how to nudge each parameter so the loss decreases a little. The step size is called the learning rate. If the learning rate is too small, training is very slow, like shuffling downhill in slippers. If it is too large, the model may overshoot the valley and bounce around wildly, never settling. Choosing the learning rate is one of those practical arts that makes machine learning feel less like pure mathematics and more like gardening. Too much water, too little water; too much sun, not enough patience. Backpropagation is the method used to calculate gradients through the layers of a neural network. The name sounds technical, but the image is manageable. The model makes a prediction going forward through its layers. Then the error is traced backward, layer by layer, to determine how each parameter contributed to the mistake. Forward pass: the information flows in, transformations happen, an output appears. Backward pass: the loss sends a message back through the system: “Here is how each part should change, just a little.” This repeats many times over many examples. An epoch is one full pass through the training dataset. If the dataset is a stack of pages, an epoch is reading the whole stack once. In large-scale training, models may not neatly pass through all data many times, but the term remains useful. A batch is a group of examples processed together. Instead of learning from one example at a time, the model looks at a small bundle, averages the signals, and updates itself. A batch is like tasting several spoonfuls from a soup before deciding whether to add salt. The dataset is the collection of examples used for training. It might contain images with labels, audio with transcripts, sentences with translations, or raw text for next-token prediction. The dataset matters deeply. A model learns patterns from what it is shown. If the data is rich, varied, and carefully prepared, the model has a better chance of learning broadly. If the data is narrow or biased or messy, those qualities can appear in the model’s behavior. A label is the desired answer for a supervised training example. If you train a model to recognize handwritten digits, the image of a seven comes with the label “7.” In unsupervised or self-supervised learning, labels may not be provided by humans in the same way. Language models often use self-supervised learning: the text itself provides the training signal. Hide the next token, ask the model to predict it, compare to the actual next token. The book becomes both question and answer key. Now, we have been saying “neural network,” and perhaps we should open that drawer. A neural network is a machine learning model inspired, loosely and historically, by networks of neurons in the brain. But modern artificial neural networks are not little brains in jars. They are mathematical systems made of connected units that transform numbers. A neuron, in this context, is a small computation. It takes inputs, multiplies them by weights, adds them up, applies a bias, and then passes the result through an activation function. The activation function is what gives the network nonlinearity. Without activation functions, a stack of layers would collapse into something much simpler, like folding several transparent sheets and finding they still amount to one sheet. Nonlinearity lets the network model curves, bends, thresholds, and complex relationships. Common activation functions have names like ReLU, sigmoid, tanh, GELU, and Swish. They sound like small creatures from a pond. ReLU, short for Rectified Linear Unit, is especially common. It outputs zero for negative values and leaves positive values alone. A simple gate: below zero, hush; above zero, continue. GELU is smoother, a little more nuanced. The exact details need not trouble us before bed, but the important part is that activation functions shape signals as they pass through, allowing the network to create complex patterns from simple operations. Now we should visit attention, because attention is central to modern language models. Attention is a mechanism that lets a model decide which parts of the input are relevant to each other. In language, words depend on other words. Consider the sentence: “The book that was on the table beside the candles was old.” The word “was” near the end refers back to “book,” not “candles.” Humans handle this easily. A model needs a way to connect distant pieces. Attention lets each token look at other tokens and gather information from them. A common explanation uses three terms: queries, keys, and values. Imagine each token in a sentence as a guest at a quiet dinner party. Each guest has a question, or query: “Who here is relevant to me?” Each guest also wears a little name tag, or key, saying what kind of information they offer. And each guest carries a value, the actual information they can contribute. The attention mechanism compares queries to keys. If a query matches a key well, the model pays more attention to that token’s value. In this way, the word “was” can gather information from “book,” even if several words stand between them. Self-attention means the tokens in a sequence attend to other tokens in the same sequence. Multi-head attention means the model does this in several different ways at once. Each “head” can learn to notice different kinds of relationships. One head might track grammar. Another might connect names to pronouns. Another might notice punctuation. In reality, heads are not always so neatly interpretable, but the image is pleasant: several lanterns shining from different angles, revealing different features of the same garden. A transformer is the architecture that made attention famous. Introduced in a 2017 paper with the memorable title “Attention Is All You Need,” the transformer became the foundation for many modern language models. It uses self-attention and feedforward layers stacked repeatedly. Architecture means the design or arrangement of a model: what kinds of layers it has, how they connect, how information flows. If parameters are the settings inside the machine, architecture is the blueprint of the machine. A feedforward layer is another kind of transformation inside the network. In a transformer, after attention gathers contextual information, feedforward layers process each token’s representation further. You can think of attention as communication among tokens, and feedforward layers as private reflection by each token after the conversation. There is also the context window. The context window is how much text a model can consider at once. If a model has a context window of 8,000 tokens, it can use roughly that many tokens of prompt and conversation when generating the next token. Information outside the window is not directly visible to it, unless it has been summarized or stored in some other system. The context window is like the stretch of path visible in lantern light. Within it, the model can refer back. Beyond it, the darkness begins. Longer context windows let models work with longer documents and conversations, but they also require more computation. A prompt is the input given to a model. It might be a question, an instruction, a paragraph to continue, a document to summarize, or a conversation history. Prompting is the art of setting the initial conditions. A prompt can be brief: “Translate this into French.” Or elaborate: “You are helping a botanist organize field notes from a rainy afternoon.” The model’s response depends on the prompt, its training, its architecture, and its parameters. It is not retrieving a single prewritten answer in most cases. It is generating one token at a time, guided by probabilities shaped during training. The act of using a trained model is called inference. Training is when the model learns. Inference is when it applies what it has learned. Training is like years of apprenticeship in the workshop. Inference is the evening someone brings in a broken clock, and the apprentice, now practiced, takes a look. Inference is much cheaper than training, but for very large models it can still be computationally demanding. This leads us to model size and compute. Compute refers to computational power: the work done by processors, often GPUs or specialized chips. Training large models can require enormous compute, because billions of parameters must be adjusted across vast datasets. Inference also uses compute, especially when many people query the model at once or when responses are long. A GPU, or graphics processing unit, was originally designed to handle graphics, where many simple calculations happen in parallel. It turns out neural networks also involve many parallel numerical operations, especially matrix multiplications. GPUs became the sturdy workhorses of deep learning. Matrix multiplication is one of the basic operations inside neural networks. A matrix is a grid of numbers. When vectors and matrices multiply, they transform the vector — rotating it, stretching it, projecting it into another space. If vectors are points or arrows, matrices are machines for moving them. Much of deep learning is, at heart, an elaborate choreography of matrix multiplications, additions, and nonlinearities. It sounds almost too simple. But a loom is simple too: threads crossing threads. Given enough threads, patterns appear. Now we come to quantization, one of the terms the topic specifically asked for. Quantization is a way of representing a model’s numbers with fewer bits. A bit is the smallest unit of digital information, a 0 or a 1. Computers store numbers using patterns of bits. Many neural network parameters are stored as floating-point numbers, often 32-bit or 16-bit. Floating-point simply means the number can represent decimals across a wide range, like 0.003 or 27.5 or negative 1.2. If a model has billions of parameters, storing each number with many bits uses a lot of memory. Quantization reduces the precision. Instead of storing a weight as a very detailed number like 0.34718293, perhaps we store it approximately as one of a smaller set of allowed values. It is a bit like replacing a high-resolution photograph with a carefully compressed version. Some detail is lost, but the overall image may remain clear enough. Or like measuring ingredients with a teaspoon instead of a laboratory scale. For many recipes, that is fine. Common quantization levels include 8-bit, 4-bit, and sometimes even lower. An 8-bit number has 256 possible values. A 4-bit number has only 16. That sounds drastic, but with clever techniques, models can often remain surprisingly capable while using much less memory and running faster. Quantization matters because it allows larger models to fit on smaller devices or cheaper hardware. It makes inference more practical. The tradeoff is that too much quantization can reduce quality. The model may become slightly less subtle, like a watercolor copied with fewer shades. Related to quantization is precision. Precision describes how detailed the numerical representation is. Float32 is higher precision than float16. There is also bfloat16, used often in machine learning because it balances range and efficiency. These names can look like secret codes, but they are mostly about how many bits are used and how those bits are divided between significant digits and scale. Another term: pruning. Pruning means removing parts of a model that seem less important, such as weights or neurons, to make it smaller or faster. The gardening metaphor is exact. You trim branches so the plant is lighter, perhaps healthier, or at least easier to carry indoors. But prune too much and the plant suffers. Distillation is another compression technique. In knowledge distillation, a smaller model, called the student, is trained to imitate a larger model, called the teacher. The teacher may be slow and powerful; the student is lighter. The student learns not only from hard labels but from the teacher’s softer probability patterns. If the teacher says, “This is probably a cat, but slightly dog-like, and definitely not a toaster,” the student receives richer information than a simple label “cat.” Fine-tuning is when a pretrained model is trained further on a more specific dataset or task. Pretraining gives the model broad general patterns. Fine-tuning specializes it. A language model might be pretrained on broad text, then fine-tuned to follow instructions, write code, answer medical questions, or classify legal documents. Instruction tuning is a kind of fine-tuning where the model learns to respond helpfully to instructions. It sees examples of prompts and desired responses. This changes the model from a raw text completer into something more like an assistant. Reinforcement learning from human feedback, often shortened to RLHF, is another technique used to align models with human preferences. Humans compare model responses or rate them. A reward model learns what kinds of responses people prefer. Then the language model is adjusted to produce responses that score better according to that reward model. Alignment is the broad term for making a model’s behavior match human intentions, values, or rules. It is a complicated area, because humans themselves are varied, context matters, and “helpful” is not a single simple number. But the basic idea is to shape the model not merely to produce plausible text, but to produce text that is useful, safe, and appropriate. Now, since we have mentioned hallucination, perhaps softly, let us define it. In machine learning, especially language models, a hallucination is when a model produces information that sounds plausible but is false or unsupported. The term is imperfect and sometimes debated, but it is common. Why does this happen? Because the model is generating likely text, not consulting reality directly unless connected to tools or retrieval systems. It may have learned patterns of how answers sound without having a reliable internal mechanism for verifying every fact. If asked for a citation, it may produce something citation-shaped. Like a dream inventing a street name that feels familiar. This is why retrieval can matter. Retrieval-augmented generation, or RAG, is a setup where a model retrieves relevant documents from a database or search system and uses them when answering. Instead of relying only on its parameters, it is given fresh reference material. It is like asking someone to answer with an open book beside them rather than from memory alone. The retrieved documents are often found using embeddings. A user’s question is embedded into a vector. Documents are embedded too. The system looks for nearby vectors — texts with similar meaning — and supplies them to the model. This brings us to similarity. In vector spaces, similarity is often measured with cosine similarity. This compares the direction of two vectors rather than their length. If two arrows point in nearly the same direction, they are similar. This works well for embeddings because meaning often lives more in direction than magnitude. There are other distance measures too: Euclidean distance, dot product, Manhattan distance. Each is a way of asking, “How close are these things?” And closeness, in machine learning, often becomes a practical stand-in for relatedness. Now let us glance at classification and regression, two classic task types. Classification means assigning something to a category. Is this email spam or not spam? Is this image a cat, dog, or rabbit? Is this review positive, negative, or neutral? Regression means predicting a continuous number. What will tomorrow’s temperature be? How much will this house sell for? How many minutes will this delivery take? The names are old and a little misleading. Regression sounds like going backward emotionally, but in statistics it has a specific history. Classification sorts into boxes. Regression predicts values along a scale. There is also clustering, where a model groups similar items without being told the categories in advance. Imagine pouring a drawer of buttons onto a table and letting them arrange themselves: shell buttons here, brass buttons there, tiny pearl ones in a little constellation. Clustering looks for natural groupings. Overfitting is another useful term. A model overfits when it learns the training data too specifically and fails to generalize to new examples. It is like a student who memorizes the answers to last year’s test but does not understand the subject. On familiar questions, excellent. On new ones, uncertain. Underfitting is the opposite: the model is too simple or insufficiently trained to capture the pattern. It performs poorly even on the training data. Like trying to describe an entire forest with three words: green, tall, leafy. Not wrong, but not enough. Generalization is the ability to perform well on new data. It is one of the central hopes of machine learning. We do not want a model merely to remember; we want it to learn patterns that carry beyond the examples. A validation set is data held out during training to check how the model performs on examples it is not directly learning from. A test set is another held-out dataset used for final evaluation. These are like inviting the model to a fresh table and seeing whether it can still recognize the dishes. Benchmarks are standardized tests for models. They help compare systems on tasks like reasoning, coding, translation, reading comprehension, image recognition, and so on. Benchmarks are useful but limited. A high score on a benchmark does not mean a model is generally wise. It means it performed well on that benchmark. Models, like people, can be better at exams than at life. Latency is the time it takes for a model to respond. Throughput is how much work it can handle over time, such as tokens per second or requests per minute. Latency is how long you wait for your cup of tea. Throughput is how many cups the teahouse can serve in an hour. A token per second is exactly what it sounds like: how many tokens the model generates each second. Since tokens are pieces of words, this is not identical to words per second, but close enough for a rough feel. A model checkpoint is a saved state of the model’s parameters during or after training. It is like saving your place in a long embroidery project. If something goes wrong, you can return to that state. Different checkpoints may reflect different stages of training or fine-tuning. Open weights means the model’s learned parameters are publicly available. Open source can mean more than that — including code, training details, data, and license — but people sometimes blur the terms. A model can have open weights without being fully open in every sense. A model card is documentation describing a model: what it is, how it was trained, intended uses, limitations, evaluations, and risks. Dataset cards do the same for datasets. These cards are attempts to bring some transparency to systems that can otherwise feel like sealed boxes. Speaking of boxes, black box is a phrase used when a model’s internal decision-making is hard to interpret. Many deep neural networks are black boxes in the sense that, while we can inspect every number, understanding how those numbers produce a particular answer is difficult. It is like having every grain of sand in a dune numbered and still not knowing why the wind shaped it so. Interpretability is the field that tries to understand what models are doing inside. Researchers study neurons, attention heads, circuits, activations, and representations. They ask whether specific internal patterns correspond to concepts or behaviors. Activations are the values produced inside the network as it processes an input. If parameters are the fixed learned dials, activations are the temporary signals flowing through the machine for a particular example. When you ask a question, the model’s parameters stay mostly the same, but its activations bloom and fade as the text passes through. A circuit, in interpretability, is a set of components inside a model that together perform some recognizable function. For example, a circuit might help identify indirect objects in sentences, or match parentheses in code. This is an active area of research, full of careful detective work. Now, one term that often appears in discussions of large language models is emergent abilities. Emergent abilities are capabilities that seem to appear when models reach a certain scale. For example, a small model may struggle with a task, while a larger model suddenly performs much better. There is debate about how abrupt or real these transitions are, and whether some are artifacts of measurement. But the general idea is familiar from nature. Enough water molecules together produce waves. Enough neurons produce thought. Enough small computations may produce surprising behavior. Scale refers to making models larger, training them on more data, or using more compute. Scaling laws are empirical relationships showing how model performance tends to improve as size, data, and compute increase. They are not magic laws of the universe, but practical observations that have guided the development of larger models. Now, let us slow down and imagine the whole journey of a sentence through a language model. You type: “Why do stars seem brighter in winter?” The tokenizer slices this into tokens. Each token becomes an ID number. Each ID is mapped to an embedding, a vector in a high-dimensional space. Positional information is added, because the model needs to know word order. “Dog bites man” is not the same as “Man bites dog,” especially for the man. These token representations enter the first transformer layer. Attention lets each token gather information from other tokens. Feedforward networks process the results. Activations shift. The representations move into the next layer, and the next, and the next. By the final layer, each token’s vector has been transformed many times. The model then produces logits for possible next tokens. Softmax turns them into probabilities. A sampling method chooses the next token, perhaps “Stars.” Then the new token is added to the sequence, and the process repeats. Out comes an answer, one small piece at a time. Perhaps it says something about winter air being clearer and drier, about longer nights, about familiar constellations rising high in the sky. The model has not seen the night sky. It has seen text about the night sky. It has learned the shapes of explanations. There is something humbling in that. The model is powerful, but its power is patterned from examples. It is not a person standing under Orion with cold hands in coat pockets. It is more like an enormous echoing instrument, tuned by human language, able to play new variations when touched by a prompt. Before we finish, let us gather a few terms that often drift nearby. An API is an interface that lets software communicate with a model or service. If the model is a kitchen, the API is the serving window with a menu of acceptable requests. A server is the computer, or group of computers, running the model. The client is the software asking for something. When you use a chatbot, your device may send text to a server, where inference happens, and the response returns. A local model runs on your own machine. A hosted model runs on someone else’s infrastructure. Local models can offer privacy and control, but may require capable hardware. Hosted models can be powerful and convenient, but depend on network access and provider policies. A parameter count tells you how many adjustable numbers a model has. A 7-billion-parameter model has about seven billion learned parameters. More parameters can mean more capacity, but size alone does not determine quality. Data, architecture, training methods, and alignment all matter. Capacity means how much complexity a model can represent. A tiny model may not have enough capacity for a difficult task. A very large model has more room to store patterns, but also needs enough good data and training to use that room well. A latent space is a hidden representation space where the model organizes information. “Latent” means not directly observed. Embedding spaces are often latent spaces. They are like the backstage corridors of meaning. A feature is some measurable pattern or property used by a model. In older machine learning, humans often designed features by hand: number of words in an email, presence of certain phrases, average brightness in an image. Deep learning often learns features automatically through layers. A hyperparameter is a setting chosen before or around training, not learned in the same way as parameters. Learning rate, batch size, number of layers, hidden dimension, dropout rate — these are hyperparameters. If parameters are the dials the model learns, hyperparameters are choices made by the builders about the training setup and architecture. Dropout is a regularization technique where, during training, some neurons are randomly ignored. This encourages the model not to rely too heavily on any one path. It is like rehearsing a song while occasionally asking one instrument to rest, so the ensemble becomes more resilient. Regularization refers broadly to methods that help prevent overfitting. It nudges the model toward simpler or more robust patterns. A loss function is the formula used to calculate loss. Different tasks use different loss functions. Cross-entropy loss is common for classification and language modeling. Mean squared error is common for regression. The loss function defines what “wrong” means for training. Optimization is the process of minimizing loss by adjusting parameters. Optimizers are algorithms that perform this adjustment. Stochastic gradient descent, Adam, AdamW — these are common optimizers. Adam is not a person in the workshop, though one can imagine him there, quietly adjusting dials with a clipboard. Stochastic means involving randomness. Stochastic gradient descent uses random batches of data to estimate gradients. The randomness makes training noisy, but often effectively so. A little noise can help the model avoid certain bad valleys. A neural network’s hidden size or hidden dimension refers to the width of its internal vector representations. Wider layers can carry more information, but require more computation. The vocabulary is the set of tokens the model knows how to use. For a language model, the final prediction is usually over the vocabulary: which token comes next? A sequence is an ordered list of tokens. Sequence length is how many tokens are in it. Many models are designed to handle sequences, whether text, audio frames, DNA bases, or time-series measurements. Modality refers to the type of data: text, image, audio, video, sensor readings. A multimodal model can handle more than one modality, such as text and images. It may look at a picture and answer questions about it, or generate an image from a description. When an image model processes pictures, it may break them into patches, a little like tokens for vision. Each patch becomes a vector. Then transformer-like machinery can operate over those patches. The idea of tokens has quietly spread beyond text. A diffusion model, common in image generation, works differently from a language model. It learns to remove noise from data. During training, images are gradually noised, and the model learns how to reverse the process. During generation, it starts with noise and denoises step by step into an image. It is like watching fog condense into a landscape. A GAN, or generative adversarial network, uses two networks: a generator that creates samples and a discriminator that tries to tell real from fake. They train together in a kind of contest. GANs were once especially prominent for image generation, though diffusion models have become very important. An encoder is a model component that turns input into a representation. A decoder turns representations into output. Some models are encoder-only, useful for understanding tasks. Some are decoder-only, common for modern autoregressive language models. Some are encoder-decoder, often used in translation and summarization. Autoregressive means generating output one step at a time, with each new step depending on previous steps. Language models that predict the next token are autoregressive. Like knitting: each stitch depends on the row already made. Masking is hiding part of the input during training. A masked language model might see “The cat sat on the [MASK]” and learn to predict “mat.” This is different from next-token prediction, but also self-supervised. A corpus is a large collection of text. The plural is corpora, though “corpuses” appears too, wandering around in comfortable shoes. Language models are trained on corpora. Data cleaning is preparing data by removing duplicates, correcting formatting, filtering harmful or low-quality material, and organizing it. It is unglamorous and important. Many machine learning systems owe more to careful data work than their public mythology suggests. Data augmentation means creating modified versions of examples to improve training. For images, you might rotate or crop them. For audio, add slight noise. For text, augmentation is trickier, but paraphrasing or formatting variations may be used. A pipeline is a chain of steps: collect data, clean it, tokenize it, train the model, evaluate it, deploy it, monitor it. The word pipeline makes it sound rigid, but in practice it can be a winding series of experiments and repairs. Deployment is putting a model into actual use. Monitoring means watching how it performs over time. Drift occurs when real-world data changes, so the model’s performance may decline. A model trained on old shopping habits may not understand new ones. A model trained before a new slang term appeared may misread it. And now, as our workshop grows quieter, perhaps we can let the terms settle into one picture. A machine learning model is a shaped mathematical system. Its parameters are learned numbers, like countless tiny dials. Its layers are stages of transformation. Its vectors are lists of numbers that carry meaning through high-dimensional spaces. Its embeddings place words, images, or ideas into those spaces. Its tokens are the pieces of text it can handle. Its attention mechanism lets parts of a sequence look at each other. Its training adjusts weights by following gradients downhill through loss. Its inference uses the trained structure to respond to new inputs. Its quantization makes the numbers smaller and lighter, trading some precision for efficiency. None of these terms needs to be frightening. They are workshop labels. Drawer labels. Names for tools. And the machine itself, for all its scale, is made from repeated simple acts. Multiply numbers. Add numbers. Bend them through activation functions. Compare vectors. Adjust weights. Predict. Measure error. Nudge. Repeat. Out of this repetition, patterns gather. It is easy to make the subject sound futuristic, but there is also something ancient in it. Machine learning is, in one sense, pattern learning. The world leaves traces; the model is shaped by them. Like a path worn into grass by many feet. Like a riverbed carved by water. Like a lullaby remembered because it has been heard many times. The model does not know as we know. It does not rest in a hammock or look up at leaves moving against the evening sky. But it has absorbed, in numerical form, a vast number of human traces: explanations, arguments, poems, manuals, jokes, mistakes, corrections, recipes, letters, fragments of care, fragments of confusion. And when we speak to it, the tokens enter, become vectors, pass through layers, awaken activations, consult attention, and return as words. A small procession through the workshop. A lamp lit in one room, then the next. A set of dials, adjusted long ago, quietly shaping the path.