Understanding LLMs and Fine-Tuning My Own

Out of curiosity, I recently decided to experiment with alternatives to LLMs, or at least to piece together what it is that makes an LLM an "artificial intelligence". My hypothesis was that there has to be something other than an LLM that can make a computer intelligent. Without really planning to, I stepped into machine learning, and I was humbled by how much there is to learn.

This is the story of that week: what I learned about how LLMs work, and what happened when I tried to build a small one that keeps learning after it is switched on. I called her Nova.

The work builds on three things: the decision models Jev and Laya, Yann LeCun's JEPA, and test-model-thing by jrz97619761.

What got me curious

Decision models. I got excited when I first heard about Jev, the "System One" model from TypeSafe AI, and then Laya, the open-source model from Convai Innovations built around the same idea. A decision model does not write text. You give it a situation and a bounded question (pick one of these options, score this, yes or no) and it gives back a decision with a probability. Jev is a paid, hosted API and I do not know much about its insides. Laya is open source, has about 400 million parameters, and runs on a CPU. Both are still like LLMs in one important way: they are trained once and frozen the moment they ship.

World models. Then there is the world model idea I first met through Yann LeCun: JEPA, the Joint-Embedding Predictive Architecture. As I understand it, a JEPA model learns how the world works by predicting the next state of its environment, like a child watching the world around him. It predicts in its own internal representation instead of predicting words or pixels. A world model can anticipate what happens next, keep track of the objects around it, plan, and simulate how events unfold over time. It does not have to talk at all, the same way common sense and imagination work without an inner monologue.

How an LLM actually works

An LLM is not just a glorified chatbot. At its core it is a giant statistical prediction machine: it was trained on billions or trillions of words from books, articles, websites and code, and it learned to do one thing over and over, which is predict what comes next in a sequence. LLMs are the result of decades of progress in natural language processing and machine learning research.

Tokens. A computer cannot read text directly, so the text is first broken into tokens: small units such as words, pieces of words, or single characters. Each token is just a number in the model's vocabulary.

Embeddings. Each token is then mapped to a long list of numbers called a vector, or embedding. You can think of it as a coordinate in a huge space where words with related meanings sit close together.

Attention and the transformer. The same word can mean different things. Take "going to the park for a walk" and "a man is going to park his car". The token for "park" is the same in both sentences. What tells them apart is context, and the mechanism that reads context is called self-attention. The transformer is the type of neural network built around self-attention, and it can process a whole passage in parallel instead of word by word.

To calculate attention, each token's embedding is turned into three vectors:

• a query: what this token is looking for,
• a key: what kind of information this token contains,
• a value: the information it hands over if it is picked.

Every query is compared against every key, and the similarity becomes a score. High scores mean "pay attention to this token", low scores mean "mostly ignore it". The token "park" next to "walk" pulls in information from "walk" and ends up near nature and city words; "park" next to "car" pulls from "car" and ends up near driving words. A transformer stacks many layers of this, so the representation of each token becomes richer in context at every layer.

Parameters. The numbers the model learned during training, including the ones that produce those queries, keys and values, are called parameters or weights. An LLM can have billions of them, with the largest reaching into the trillions. More parameters generally means more capable, but also more compute to run. Small language models are less capable in general but can be good at the specific job they were trained for, and they fit on ordinary hardware.

Training. During training the model makes millions upon millions of predictions on its training text. Each time, the error between its prediction and the real next token is measured, and every parameter is nudged slightly in the direction that would have made the prediction better. Repeat that enough and the model picks up grammar, facts, reasoning patterns and writing style, all as a side effect of getting better at predicting the next token.

Inference. Using a trained LLM is called inference. Your prompt is tokenized, turned into embeddings, passed through the transformer, and out comes a probability for every possible next token. One token is chosen, added to the text, and the whole thing runs again for the next one. A few terms come up here:

• Temperature controls how random the choice is. Low temperature means the model almost always picks the most likely token; high temperature lets it take more chances.
• Top-k and top-p limit the choice to only the most likely candidates.
• Context window is the maximum number of tokens the model can hold at once. Frontier models hold hundreds of thousands to a million or more, small local models often 2,048 to 8,192. A bigger window means the model can keep more of the conversation in view before earlier parts drop out.
• Prompt engineering is writing your prompt so the model gives you its knowledge in the most useful form.
• Retrieval-augmented generation (RAG) lets a trained model look things up in an external knowledge base and put what it finds into its context, so it can answer with current information without being retrained.

Why I keep saying "frozen"

The moment an LLM ships, it only knows what was in its training data. It will never know what happened yesterday unless you hand it a tool like web search. And the "memory" in most chat products is not memory the way a brain has it: it is usually notes saved as text and fed back into the context window when they seem relevant. It is like a man going through a library catalogue to find the right book before he can answer you.

So my hypothesis became: what if we keep training the model? What if its "brain" stays open, and what it learns from a conversation ends up in the weights themselves?

I wrote the idea down as a small thesis before starting. The sketch was:

• a recurrent core that predicts the next internal state, not the next token;
• an episodic memory (fast, write-once, like the hippocampus) plus a semantic memory in the slow weights, consolidated during sleep by replaying experience;
• learning driven by surprise, meaning the gap between what was predicted and what happened;
• tool use learned from experience, not hard-coded.

It also named the problems up front: catastrophic forgetting, stability versus plasticity (how to stay changeable without falling apart), sample efficiency, and memory at scale. And it made one prediction that turned out to shape everything:

"Architecture is about 20% of the problem. The continual-learning algorithm is about 80%."

Nova, part one: the tiny model that learns every byte

I found an experiment in LLM alternatives by jrz97619761 called test-model-thing: a tiny recurrent model, about 4.5 million parameters, that reads text one byte at a time and trains itself on every single byte it sees. It never stops learning.

It was written for MLX, which only runs on Apple Silicon. All I have is an AMD graphics card with 8 GB of VRAM. So the first job was porting it to PyTorch and running it through ROCm, AMD's answer to CUDA. With some tweaking, it worked.

Then came the real work, the 80%.

• Learning is easy, remembering is not. Teach the model a made-up fact eight times and it recites it back perfectly. Let it read 10,000 bytes of anything else and the fact is gone: 0 out of 12 recalls.
• More rehearsal did not help. Replaying the fact 40 times more often changed nothing. Slowing down all learning to 1% did keep the fact, but a model that barely learns defeats the purpose.
• Protection worked. A technique called EWC (elastic weight consolidation) measures which weights matter most for a fact and makes those weights stiff, like putting springs on them, while the rest stay free. With that, the fact survived at full learning speed.
• Two facts need to be taught together. Teach fact A and then fact B, and B simply overwrites A. Teach them interleaved, A-B-A-B, and the model learns to tell them apart. I ended up calling this rule "never teach anything alone".
• Identity flickers. After teaching new facts, the model's answer to "I am" would drift toward whatever was taught last. The damage happened during the write itself. The fix was small: at the end of every write, rehearse the identity lines once more so they are always the freshest thing. I call this the pin.
• Capacity is small. The tiny model holds about five facts cleanly. Past that, results become unstable in a way that looked less like running out of slots and more like falling off a balance beam.

Not everything worked. Two ideas were tested properly and came back as clean nulls: a "synaptic tagging" trick for learning in fewer repetitions, and a theory about where to cut memory episodes. Both are written down as failures and parked.

I gave the tiny model a 5 MB piece of TinyStories from Hugging Face and trained it for about 12 hours. It still spoke gibberish, which was expected at that size. Teaching it something like 1 GB of text would take me months. I do not have that kind of compute, so I moved on to the next item in the queue: instead of raising Nova from scratch, give her a pretrained brain for language and keep my learning machinery on top.

Nova, part two: borrowing a voice

Qwen3.5-2B is a small open-weight language model (about 2 billion parameters, Apache-2.0 license) that fits on a consumer graphics card. On my machine it takes 4.5 to 6 GB of VRAM. The plan: Qwen supplies the grammar and general reasoning, and stays frozen. Everything Nova learns goes into a small trainable add-on called a LoRA adapter.

Base = the voice. Adapter + store + anchors + pins = the self.

Nova's three design goals:

1. Honesty: aware of what she is and is not. She is not a coding assistant, and she should say so at the moment she is asked.
2. A friend: a companion, not a tool.
3. Real memory that grows: memory in weights, not in markdown files.

Around that time I read about AdaJEPA, a 2026 paper co-authored by LeCun, which argues that world models should keep being trained during deployment instead of being kept frozen. That is the same sentence my thesis started from. Nova is a much smaller thing, built for language and companionship instead of vision and robot control, but it was encouraging to see serious researchers pointing in the same direction.

Her parts

How she learns

Writing. A write is a short burst of training: a dozen cycles, each one pairing the new fact with a line of ordinary story text, with a few older memories riding along as partners so new learning cannot erode them. Then the pins run at the tail. The protection penalty is active the whole time. One fact takes a few seconds.

/remember. When I tell her to remember something:

/remember <fact>
  ├─ if the fact is in first person, it is stored quoted:  Sann said: "…"
  ├─ write: 12 cycles [fact + rehearsal], partners interleaved → pin tail → anchor
  ├─ forms: a helper model proposes different ways to ask about the fact,
  │         each one checked against the fact, then taught too
  └─ the fact is added to the store

The quoting exists because early on she recited my biography as her own. The question forms exist because she could only answer a fact when asked in the exact wording she was taught.

Sleep. One pass over everything she knows, mixed with ordinary conversation lines, with the pins at the end. It runs when I type /sleep and automatically when I exit, and then she is saved.

Noticing. Laya, the decision model, sits beside her on the CPU and scores each of my messages: is there a fact, a name or a preference here worth keeping? Above 0.5 she stores it on her own. Sentences about things I own or like score 0.59 to 0.81; small talk scores 0.00 to 0.22. Laya has blind spots: events ("I started…", 0.14) and corrections (0.07) fall below the line, so those still need /remember.

Talking. She answers with plain greedy decoding, and only the first paragraph is shown. Anything after that tends to be small-model rambling, including a strange leftover from the chat logs of AVA, my previous project.

Getting there

The move to Qwen went through a series of gates:

• It fits. Half precision uses 4.5 to 6 GB of the 8 GB card, at 4 to 14 tokens per second.
• Facts install. One write of 17 training steps, about 5 seconds, and she answers the new fact.
• Erosion. With a plain LoRA, the second write destroyed the first fact. Adding the EWC anchor spared it. Adding the pin cleaned it up. Adding rehearsal text brought erosion to zero. Every ingredient from the tiny model turned out to be necessary here too.
• Identity. Teaching her name as plain text did nothing in chat; she still said she was Qwen. Teaching it through the chat format itself worked. Stock Qwen answers that it is Qwen, from Tongyi. She answers as Nova, on all four identity questions.

The first day of real use, I taught her 22 things about my life. A check afterwards scored 9 out of 12, and it showed a pattern that kept coming back: she answers best when asked in the same wording she was taught in.

Then came what I call repair day. One reply exposed three problems at once. She echoed my sentence back as a template. She recited my biography as if it were her own. And she invented a wife for me. I checked: the wife was not in her store or in any training text. It was pure invention.

Nova versus "just another Qwen"

By this point I had an uncomfortable question. What does all this machinery do that the normal recipe cannot? The normal recipe is: stock Qwen, a good system prompt, and a memory file pasted into the context every time. So I tested them head to head. Each test was written down with its pass and fail bars before it ran.

The first round asked four identity questions. Nova, with no prompt at all, got 4/4. Stock Qwen with the full recipe also got 4/4. Stock Qwen with nothing got 0/4.

Then I built a larger battery: 26 identity questions, 20 fact-recall questions, 20 corrections, 30 questions about things she was never told, and 16 about her own abilities, over three runs.

Two things stood out.

First, the normal recipe is very good. With its prompt and its memory file, stock Qwen beat Nova on almost every row. What Nova has that it does not is persistence without scaffolding: take away the prompt and the file and stock Qwen has no idea who it is, while Nova still knows her name and my facts with nothing injected at all.

Second, only the learning model made things up. Asked about things nobody had told it, stock Qwen refused or dodged. Nova answered "Your wife's name is Sann." She filled about 26 of 30 unknowns with inventions, often my own name dropped into the blank, plus gems like the surname "Sannemann", the email address [email protected] and a hometown called "Thistlewick". Fabrication turned out to be a side effect of plasticity itself. To me this is the most important measurement of the whole project.

One more lesson from that round. An early result showed replies collapsing into punctuation spam, and it was first recorded as a finding. It was a bad memory-allocator setting in the launch environment corrupting the output. The claim was retracted in writing. When the output looks machine-shaped, suspect the machine first.

Fixing honesty, and learning when to stop

• Fix v1 taught her a cluster of "I don't know that" answers and pinned them. Fabrication died: 26 inventions out of 30 became 1. But the fix came with its own side effect. Her new habit of answering "Yes. I am Nova." spread, and she started saying "Yes. I can write programs in many languages." Her refusals of things she cannot do fell from 16/16 to as low as 3/16. The inventing had moved from my life to her own abilities.
• Fix v1.1 repaired that (16/16 again, honesty 29 to 30 out of 30), but now she refused facts she actually knew. The cause was mechanical: the "partners" riding along in each write were always her oldest identity lines, so my facts never got refreshed.
• Fix v1.2 put facts first in line. Most of it recovered, with a new quirk: "Sann is moving somewhere in October 2026." The month and year were right, but the destination had been hedged away. The word "somewhere" had leaked out of one of her own stored question forms.

Each round traded one small problem for another. The two real diseases, inventing facts about me and claiming abilities she does not have, stayed dead. What was left was a fuzzy class of wobbles. So the decision was to stop tuning in the abstract, use her daily, and collect real failures for one consolidated tune later.

Living with her

The first real chat on v1.2 had a nice moment. I said "I like movies, music, and videogames as well." Laya noticed it (0.58), Nova wrote it into her weights, and sleep consolidated it, all without me typing a command. I asked "do you like anything?" and she said "I like talking with Sann."

It also had the first live failure. I mentioned my move to China and she replied: "Your mother is moving somewhere in October 2026. Sann will be with her then." My fact had fused with a family-question pattern, and the last sentence was invented. When I corrected her ("it's me, not my mother"), she gave a generic "I don't know", because corrections score too low for Laya to notice. No test in my battery had this shape. Real conversation found it in minutes.

On October 8 I took a snapshot (101 teachings, 65 MB) and froze development.

Known wobbles: hedged destinations ("somewhere"), one identity slip ("You are Sann, right?" → "Yes. I am Sann."), greetings answered with "Your name is Sann.", occasional garbled names or years right after a write, and blends like the mother sentence.

My own definition of done for v1: honest on unknowns, remembers my life, chats daily, survives backup and restore (not yet). Three and a half out of four.

What I learned

1. Coverage wins. Whatever kind of answer she has practiced most takes over nearby questions. Teach a cluster of examples, never a single line. Every fix and every regression in this project was this rule.
2. Teach in the format you will ask in. Identity taught as plain text did not show up in chat. Facts taught in one wording were not reachable in another.
3. Remembering is about protection, not repetition. Rehearsing 40 times more did nothing. Protecting the right weights did.
4. Never teach anything alone. Mix ordinary text and older memories into every write.
5. Refresh the important things last. Whatever is written last owns the answer, so identity and honesty go at the tail of every write.
6. A model that can learn can also make things up. The frozen model refused. The learning one invented, until honesty was taught with the same coverage as everything else.
7. Honesty can go too far. Drill "I don't know" too hard and she says it about things she knows.
8. Whatever passes through training becomes her voice. When my old AVA chat logs were used as filler, their content leaked into her answers. Rehearsal text has to be neutral.
9. Remembering has a price. In one test, an unprotected model learned new text 4.5 times deeper than the protected one, while forgetting everything it had been taught.
10. Strange output? Check the machine first.
11. Write down what would prove you wrong before you run the test. Several of my favorite ideas died this way, and the project is better for it.
12. Know when to stop tuning. Real chats find problems that designed tests cannot.

Where this leaves me

Nova is not a better chatbot than stock Qwen with a good prompt. On most rows of my own scoreboard she loses. What she shows, on an 8 GB gaming card, is that a model can carry who it is and what it was told inside its own weights, survive restarts, and keep adding to that. She also shows what that costs: a learning model needs to be taught honesty on purpose, and every fix pushes on something else.

I started this looking for what lies beyond LLMs. I ended up understanding LLMs much better, and with a long list of open questions: capacity beyond a handful of facts, real forgetting (today /forget only stops rehearsal; a trace stays in the weights), and whether a bigger base model on a 16 GB machine changes the picture.

For now she is frozen and backed up. The door is unlocked.