In Part 1, I fine-tuned SmolLM2-360M-Instruct with normal supervised fine-tuning and LoRA.

In Part 2, I tried a tiny DeepSeek-style GRPO reinforcement-learning experiment. Instead of giving the model the correct answer, I let it generate several attempts, scored them automatically, and trained it from the reward.

That experiment moved the held-out arithmetic result from 8.3% pass@1 to 33.3% pass@1.

After that I wanted to try the missing piece:

What if the tiny model first learns reasoning demonstrations from a stronger open model, and only then gets reinforcement learning?

So this became Part 3: distillation + GRPO.

The basic idea

This time I used two models.

The teacher was:

deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B

The student stayed exactly the same model as before:

HuggingFaceTB/SmolLM2-360M-Instruct

The teacher is around four times larger than the student, but it is still small enough for me to run locally on my RTX 4060 laptop GPU.

The pipeline was:

flowchart TB
    A[DeepSeek-R1-Distill-Qwen-1.5B<br/>teacher] --> B[Generate reasoning demonstrations]
    B --> C[Independent arithmetic verifier]
    C --> D[Keep only verified correct demonstrations]
    D --> E[SmolLM2-360M student]
    E --> F[SFT + LoRA distillation]
    F --> G[Distilled 360M student]
    G --> H[Fresh GRPO LoRA]
    H --> I[Rule-based RL]
    I --> J[Distilled + GRPO student]

This is much closer to the general idea behind modern reasoning pipelines than my Part 2 pure-RL experiment, but it is still a tiny educational reproduction. I am not claiming that a 360M laptop model is a reproduction of DeepSeek-R1.

What does "distillation" mean here?

The word sounds complicated, but the idea is simple.

I let a stronger model solve the training problems first.

For example, the teacher may produce something like:

<think>8 times 9 equals 72.</think>
<answer>8 * 9</answer>

Then the 360M student sees that demonstration during supervised training.

So the student is not discovering everything from random trial and error. It gets a useful starting behavior from a stronger model.

flowchart LR
    T[Stronger teacher] --> D[Correct demonstrations]
    D --> S[Small student]
    S --> K[Student imitates the useful pattern]

This is different from GRPO.

With distillation/SFT, I tell the student:

"Here is a good solution. Learn to produce something like it."

With GRPO, I tell it:

"Try several things. I will score them. Prefer whatever gets more reward."

Part 3 uses both.

I did not trust the teacher blindly

This part ended up being more important than I expected.

At first, every teacher response was rejected.

It looked like the teacher was failing, but when I inspected the raw text, DeepSeek was actually solving the arithmetic correctly. The problem was formatting: it often generated a long reasoning paragraph, sometimes omitted the opening <think> tag, and could hit the generation limit before producing the exact structure my verifier expected.

So I changed the data-generation pipeline.

The teacher is allowed to write in its natural style, but I independently search its output for an arithmetic expression and check it using the same safe AST verifier from Part 2.

Only if the expression:

• uses every supplied number exactly once,
• introduces no extra numbers,
• uses only the allowed arithmetic operators,
• and evaluates exactly to the target,

do I keep the demonstration.

Then I normalize that verified teacher result into the short format used by the student.

flowchart LR
    A[Teacher raw reasoning] --> B[Extract candidate expression]
    B --> C{Verifier says correct?}
    C -->|No| D[Reject / regenerate]
    C -->|Yes| E[Normalize demonstration]
    E --> F[Student training data]

The first pass verified 28 of 40 tasks. A second longer retry pass solved the remaining 12.

Final result:

40 / 40 training demonstrations independently verified.

Teacher generation peaked at about 3.34 GB PyTorch GPU allocation on my laptop.

Stage 1 distilling into the 360M student

I trained a fresh LoRA adapter on the 40 verified DeepSeek demonstrations.

Configuration:

The SFT loss dropped quickly:

The important result, however, is not the training loss. It is the exact same held-out 12-puzzle test set I used in Part 2.

Distillation alone was surprisingly strong

Here are the held-out results after distillation but before the new GRPO stage:

The pass@4 = 100% number means that for every one of the 12 unseen puzzles, at least one of the four sampled answers was correct.

That was a huge jump from the untouched model.

For example, on an unseen test problem:

Numbers: 10, 12
Target: 22

the distilled model generated:

<think>So 10 + 12 is indeed 22.</think>
<answer>10 + 12</answer>

The automatic verifier gave it the full 1.35 reward.

Stage 2 GRPO after distillation

Then I took the distilled student and added a fresh LoRA adapter for GRPO.

One detail mattered here.

I did not want the KL reference to accidentally become the original untouched SmolLM2 model. The reference should be the distilled student because that is the policy I am refining.

So I merged the distilled LoRA into the 360M model in memory, attached a new GRPO LoRA, and used "LoRA disabled" as the distilled reference policy.

flowchart TB
    A[Base SmolLM2-360M] --> B[Merge distilled LoRA]
    B --> C[Distilled student becomes reference]
    C --> D[Attach fresh GRPO LoRA]
    D --> E[Sample 4 attempts per puzzle]
    E --> F[Rule rewards]
    F --> G[GRPO update]

The GRPO stage used:

The reward during those 40 rollouts looked noisy, which is normal for sampled RL:

Some groups were already perfect and produced no relative reward variance, so GRPO correctly skipped the policy update for those groups.

The final comparison

This is the result I really wanted to see.

All four systems were evaluated with the same 12 held-out puzzles and four samples per prompt.

And this is the real result script running in Windows Command Prompt on my laptop:

This result is more interesting than I expected.

Finding #1 a stronger teacher helped the tiny model much more than pure RL alone

Pure GRPO improved pass@1 from 8.3% to 33.3%.

That was already a useful result.

But distillation alone reached 50.0% pass@1 and 100% pass@4.

So for this tiny 360M model and this tiny arithmetic task, giving it verified examples from the stronger reasoning teacher was a much stronger starting point than asking it to discover everything only from sparse reward.

That matches the intuition behind why distillation is attractive for smaller models: the teacher transfers a useful behavior distribution before RL begins.

I would not generalize this from 12 puzzles to "distillation always beats RL." This is one small experiment. But on this setup, the difference is very obvious.

Finding #2 GRPO after distillation improved the first answer

Distillation alone:

pass@1 = 50.0%

After GRPO:

pass@1 = 66.7%

That means the first sampled answer became more likely to be correct.

The mean reward also increased:

1.0677 -> 1.0958

and strict formatting moved from:

95.8% -> 97.9%

So GRPO still had something useful to refine even after the teacher had already given the student a strong supervised starting point.

Finding #3 but pass@4 actually went down

This was the result I did not expect.

Distillation only:

pass@4 = 100%

Distillation + GRPO:

pass@4 = 83.3%

So after RL, the first answer was better on average, but the four sampled answers covered fewer successful alternatives on two of the held-out puzzles.

One possible interpretation is a reliability versus diversity trade-off: GRPO pushes probability mass toward behaviors that earned reward during training, which can make the policy more confident but also less diverse.

I want to be careful with that wording. With only 12 test puzzles and one training seed, I cannot claim that GRPO generally reduces diversity. It is simply what happened in this experiment, and it is exactly the kind of result that deserves a larger multi-seed study later.

This is also why I like measuring both pass@1 and pass@4. If I reported only one number, I would miss half the story.

Finding #4 <think> became easy; correct reasoning was still the hard part

The untouched base model followed the strict reasoning format only 20.8% of the time.

GRPO-only already pushed that to 95.8%.

Distillation + GRPO reached 97.9%.

So formatting is relatively easy to teach.

But formatting is not the same as reasoning.

The model can produce:

<think>something that looks reasonable</think>
<answer>wrong expression</answer>

and still fail the actual arithmetic reward.

That is one of my biggest takeaways from Parts 2 and 3: a model looking like a reasoning model is not evidence that it actually solved the problem correctly.

The external verifier is what makes this experiment meaningful.

Finding #5 tiny local reasoning experiments are surprisingly cheap

Everything here ran on my normal laptop:

• RTX 4060 Laptop GPU with 8 GB VRAM,
• 32 GB system RAM,
• Windows,
• PyTorch + Hugging Face + PEFT.

Observed PyTorch GPU allocation:

Obviously this is nowhere near the scale of training DeepSeek-R1 itself. But for learning how the pipeline works, it is enough to run the whole loop myself:

flowchart LR
    A[Teacher] --> B[Verified data]
    B --> C[Distill]
    C --> D[Student]
    D --> E[RL]
    E --> F[Held-out evaluation]

That is much more satisfying to me than only reading the paper and calling an API.

The three experiments together

At this point my little project has become a nice progression.

flowchart TB
    A[Part 1<br/>SFT + LoRA] --> A1[Teach a small model my own data]
    B[Part 2<br/>GRPO + LoRA] --> B1[Let the model learn from reward]
    C[Part 3<br/>Distillation + GRPO] --> C1[Learn from a stronger teacher<br/>then refine with reward]

The conceptual difference is:

Part 1: "Here is the answer. Imitate it."

Part 2: "Try several answers. I will reward the better ones."

Part 3: "First learn good reasoning examples from a stronger model, then improve the policy using reward."

Limitations

There are plenty.

The test set contains only 12 held-out two-number arithmetic puzzles. That is enough for an educational experiment, but nowhere near enough for a serious claim about general reasoning.

I used one student model, one teacher, and one main random seed. A paper-quality experiment should repeat training across multiple seeds and ideally multiple sub-1B student models.

The teacher demonstrations are automatically normalized after verification. This makes the training data clean and consistent, but it also means I am not trying to preserve every token of DeepSeek's original long reasoning trace.

The tasks themselves are intentionally easy and automatically verifiable. Moving to multi-step mathematics, code, or engineering reasoning would be much harder.

And most importantly: <think> text is observable generated text, not proof that it matches some special hidden internal reasoning process.

Final result

The complete path ended up being:

flowchart TB
    A[SmolLM2-360M base<br/>pass@1 8.3%] --> B[Pure GRPO<br/>pass@1 33.3%]
    A --> C[DeepSeek reasoning distillation<br/>pass@1 50.0%]
    C --> D[GRPO after distillation<br/>pass@1 66.7%]

I started this project just wanting to understand how people "train a reasoning model."

The biggest lesson is that there is no single magic training step.

Pretraining gives the model its language ability. Supervised learning gives it examples of useful behavior. Distillation transfers behavior from a stronger model. Reinforcement learning then reshapes the probability of behaviors according to reward.

And even in this tiny laptop experiment, those stages behaved differently enough that I could actually see the trade-offs myself.

That was the whole point.

References

• DeepSeek-R1: https://arxiv.org/abs/2501.12948
• DeepSeekMath / GRPO: https://arxiv.org/abs/2402.03300
• DeepSeek-R1-Distill-Qwen-1.5B: https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
• Hugging Face Open-R1: https://github.com/huggingface/open-r1