In the previous article, I fine-tuned SmolLM2-360M-Instruct on my own data using Supervised Fine-Tuning (SFT) + LoRA.

That experiment was basically:

Here is the question. Here is the answer I want. Please learn to produce something like this answer.

It worked surprisingly well for narrow personal information. The original model confidently invented that I was a Malaysian politician, while the trained model eventually learned that I am an Indonesian automotive software engineer from Bali working at TMMIN.

But after publishing that article I wanted to try something different.

Instead of always giving the model the correct answer, what if I let the model try several answers, score the attempts, and let it learn from reward?

That led me to the training method used in the DeepSeek reasoning work: GRPO, or Group Relative Policy Optimization.

This is Part 2.

First, this is not me reproducing DeepSeek-R1

I want to make this clear from the beginning.

DeepSeek-R1 was trained at a completely different scale. I have:

• one RTX 4060 Laptop GPU with 8 GB VRAM,
• a 360M parameter SmolLM2 model,
• a tiny generated arithmetic dataset,
• and a few minutes of training time.

So I am not claiming that I reproduced DeepSeek-R1.

What I reproduced is a small educational version of one important idea from the DeepSeek papers: GRPO with automatically verifiable rewards.

DeepSeekMath introduced GRPO as a more memory-efficient alternative to PPO for mathematical reasoning. Later, DeepSeek-R1-Zero used large-scale reinforcement learning with rule-based rewards, including accuracy and output-format rewards.

My experiment uses the same general idea, but shrunk down until it fits on my laptop.

References I used:

• DeepSeekMath: https://arxiv.org/abs/2402.03300
• DeepSeek-R1: https://arxiv.org/abs/2501.12948
• Hugging Face Open-R1: https://github.com/huggingface/open-r1
• TinyZero: https://github.com/Jiayi-Pan/TinyZero

Part 1 vs Part 2

The easiest way to explain the difference is this:

So the old method was basically imitation.

The new method is closer to trial, score, compare, update.

flowchart LR
    subgraph SFT[Part 1 - Supervised Fine-Tuning]
        Q1[Question] --> M1[Model]
        A1[Known correct answer] --> L1[Cross-entropy loss]
        M1 --> L1
        L1 --> U1[Update LoRA]
    end

    subgraph RL[Part 2 - GRPO Reinforcement Learning]
        Q2[Question] --> M2[Model]
        M2 --> G1[Attempt 1]
        M2 --> G2[Attempt 2]
        M2 --> G3[Attempt 3]
        M2 --> G4[Attempt 4]
        G1 --> R[Rule reward]
        G2 --> R
        G3 --> R
        G4 --> R
        R --> C[Compare within group]
        C --> U2[Update LoRA]
    end

There is also an important limitation in this comparison: Part 1 and Part 2 solve different tasks. I should not compare the Part 1 personal-data loss directly with Part 2 arithmetic accuracy and say one algorithm is universally better. They are measuring different things.

What I can compare fairly is the exact same arithmetic test set before and after GRPO.

What exactly is GRPO?

The basic idea is surprisingly simple.

For one question, instead of generating one answer, I generate a group of four answers.

For example:

Numbers: 8, 9

Target: 72

The model may produce four different attempts.

Attempt A: 8 + 9 = 17
Attempt B: 8 * 9 = 72
Attempt C: 9 - 8 = 1
Attempt D: random invalid expression

Then I score every attempt.

If their rewards are different, GRPO asks a relative question:

Which answers were better than the average of this group, and which were worse?

A simplified version of the group-relative advantage is:

advantage = (reward - group mean) / group standard deviation

So if answer B gets the highest reward, training increases the probability of behavior like B relative to the weaker attempts.

There is no separate neural critic/value model in my implementation. That is one reason this is practical on my laptop.

What reward did I use?

The cool part of arithmetic is that the answer can be checked automatically.

I asked the model to respond like this:

<think>
reasoning here
</think>
<answer>
arithmetic expression here
</answer>

The reward checker verifies:

• did it use the required output format?
• did it include an <answer> tag?
• is the arithmetic expression valid?
• did it use every supplied number exactly once?
• did it introduce any illegal extra numbers?
• does the expression actually equal the target?

My reward weights were:

The most important reward is still correctness.

The parser uses Python's AST to inspect the arithmetic expression. It does not execute arbitrary generated Python code.

My tiny R1-style pipeline

flowchart TB
    A[SmolLM2-360M-Instruct] --> B[Attach fresh LoRA policy]
    C[Countdown arithmetic prompt] --> D[Generate 4 completions]
    B --> D

    D --> E[Rule-based verifier]
    E --> F[Accuracy reward]
    E --> G[Format reward]
    E --> H[Expression validity reward]

    F --> I[Group-relative advantage]
    G --> I
    H --> I

    I --> J[GRPO clipped policy objective]

    B --> K[Temporarily disable LoRA]
    K --> L[Original SmolLM2 reference probabilities]
    L --> M[KL penalty]
    M --> J

    J --> N[Backpropagation]
    N --> O[Update only LoRA A/B matrices]
    O --> P[GRPO LoRA checkpoint]

The reference-model trick is useful here. I do not need to load another complete 360M model into VRAM. Because the original base weights remain frozen, I can temporarily disable the LoRA adapter and use the untouched SmolLM2 behavior as the KL reference.

Training configuration

For this experiment I used:

The RL run used the same seven LoRA target projections as Part 1: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj.

One practical difference was obvious immediately: RL is much more generation-heavy than SFT.

In SFT I already know the desired answer. I can run a forward pass, calculate cross-entropy, and update.

In GRPO I first need several complete model generations, then reward them, then calculate policy probabilities again for optimization. Even with the same 360M model, it is a heavier loop.

Before RL

Before changing any weights I evaluated the untouched model on 12 held-out puzzles that were not used as training puzzles.

For every test question I sampled four answers.

The baseline was:

Pass@1 means the first sampled answer was correct.

Pass@4 means at least one of the four attempts was correct.

So the original model could occasionally solve an easy problem, but most of the time it either produced the wrong arithmetic, ignored the required format, or returned just the target number instead of a legal expression.

The first half looked disappointing

This is where the experiment became interesting.

During the first 20 rollout steps, none of the sampled training groups had a verified correct answer.

That meant the model was mostly learning from smaller rewards such as format and expression validity.

At checkpoint 20 I stopped and evaluated it.

That was a weird result.

The model had clearly learned how I wanted the answer formatted, and it was finding a correct answer somewhere among four attempts slightly more often, but the first-attempt accuracy actually got worse.

If I had stopped there and only looked at reward, I could have written a very misleading article saying "RL improved the model!"

It had improved the thing the reward function made easiest to learn first: formatting.

Not necessarily reasoning.

Then the model finally started receiving real correctness reward

I continued the run from the step-20 LoRA checkpoint.

The run had been interrupted, so the second half resumed the adapter weights but restarted the optimizer state. That is another limitation of this small experiment and I do not want to hide it.

Something changed after step 20.

At steps 25, 26, 33, 36, and 38, one of the four generated training attempts was actually verified correct.

For example, several groups had:

correct_rate = 25%

which means one out of the four sampled completions received the full correctness reward.

Now GRPO finally had the signal I really wanted:

this generated solution is genuinely better than the others.

The second-half training history had an average group reward of about 0.2456, with correctness reward appearing in 5 of the 20 resumed rollout groups.

Final result after 40 rollouts

Then I ran the same 12 held-out puzzles again, with the same four-sample evaluation setup.

This was the final result:

So on this tiny held-out arithmetic benchmark:

• Pass@1 went from 8.3% to 33.3%.
• Pass@4 went from 16.7% to 41.7%.
• strict formatting went from 20.8% to 95.8%.
• mean reward went from 0.1115 to 0.3781, around 3.4x the starting value.

That is a real improvement on this test set.

But it is still a tiny test set, so I would not call this proof that my 360M model suddenly became a general reasoning model.

One result I can actually see

I like this held-out example because the difference is obvious.

The puzzle was:

Numbers: 8, 9

Target: 72

Before RL, the first answer was:

<think>Starting from the first number and going through each one, you'll find that 8 + 9 equals 17.</think> <answer>17</answer>

The verifier marked it wrong.

After GRPO, the first answer was:

<think>72 = 8 * 9</think> <answer>8 * 9 = 72</answer>

The verifier marked it correct.

That is much more satisfying to me than only seeing a training-loss number.

Is GRPO better than my old SFT method?

The answer is not simply yes or no.

For my Part 1 goal teaching the model facts about me, TMMIN, my projects, and my work history. SFT is the obvious tool. I already know what answer I want. Giving the model clean examples is much more direct.

For this Part 2 goal where I can automatically judge whether generated behavior is good or bad, reinforcement learning becomes interesting.

I think about them like this:

flowchart TB
    A[Do I know the desired answer?]
    A -->|Yes| B[SFT / LoRA]
    A -->|Not exactly, but I can score behavior| C[RL / GRPO]

    B --> D[Teach style, terminology, known examples, domain responses]
    C --> E[Optimize verifiable behavior, reasoning, code tests, task scores]

So for a company-specific AI, I probably would not choose between SFT and RL as if they were competitors.

I would combine them.

A more realistic company pipeline

For example, imagine an internal automotive assistant.

SFT could teach it:

• company terminology,
• desired report structure,
• engineering writing style,
• how a quality summary should look,
• examples of good diagnostic explanations.

RAG could provide:

• current specifications,
• latest project status,
• test reports,
• current DTC definitions,
• current production or quality data.

And reinforcement learning could optimize things that can be checked automatically, for example:

• does generated code pass unit tests?
• does a diagnostic answer identify the correct DTC from a controlled scenario?
• does a generated test case satisfy required fields?
• does a classification match verified labels?
• does a structured report follow every required rule?
• does an engineering calculation produce the correct result?

flowchart LR
    P[Pretrained local model] --> S[SFT / LoRA<br/>teach domain behavior]
    S --> R[RAG<br/>provide current company truth]
    R --> G[GRPO / RL<br/>optimize verifiable tasks]
    G --> A[Local company assistant]

That makes much more sense to me than trying to store every changing company fact directly inside model weights.

What surprised me most

The biggest surprise was how clearly the reward function shaped behavior.

Formatting became good much earlier than arithmetic accuracy.

That sounds obvious after seeing the result, because formatting is easier. The model only needs to learn patterns like:

<think> ... </think> <answer> ... </answer>

Correct mathematical reasoning is a much harder behavior.

This also shows why reward design matters so much. If I accidentally give too much reward for looking correct, the model can become very good at looking correct without actually solving the task.

That is basically reward hacking in miniature.

My step-20 checkpoint was a nice little warning sign.

Limitations

There are a lot of them.

First, 12 held-out questions is tiny. Going from one correct first answer to four correct first answers produces a big percentage jump. I need hundreds or thousands of test problems before making strong claims.

Second, these are simple two-number countdown-style arithmetic problems. This does not show that the model improved at software engineering, AUTOSAR, natural-language reasoning, or anything else.

Third, generation is stochastic. I fixed the random seed for the before/after evaluation so the comparison is reproducible, but a different sampling seed can change the exact percentages.

Fourth, the run was interrupted at step 20. I resumed the LoRA adapter weights, but the optimizer state was restarted for steps 21–40.

Fifth, the starting model is SmolLM2-360M-Instruct, not a pure pretrained base model. So this is not the same setup as DeepSeek-R1-Zero.

Sixth, I use LoRA. DeepSeek's large training runs update models at a completely different scale. My version optimizes only about 8.68M LoRA parameters, or around 2.34% of the loaded base-plus-adapter parameter count.

And finally, DeepSeek's own R1 work shows that for small models, distilling reasoning traces from a stronger model can be extremely effective. That gives me an obvious Part 3 idea.

The result in one picture

flowchart LR
    A[Untouched SmolLM2-360M<br/>Pass@1 8.3%<br/>Pass@4 16.7%] --> B[GRPO rollouts 1-20]
    B --> C[Step 20<br/>format improves strongly<br/>Pass@1 falls to 0%]
    C --> D[Correct samples finally appear<br/>steps 25,26,33,36,38]
    D --> E[Step 40<br/>Pass@1 33.3%<br/>Pass@4 41.7%<br/>format 95.8%]

The nice part is that the story is not a perfectly smooth line upward.

The model first learned the easy rewarded behavior. Then, once correct samples started appearing in the rollout groups, it finally had useful accuracy reward to learn from.

Final thoughts

Part 1 taught me that a small model can learn very specific information if I give it clean, repeated supervision.

Part 2 taught me something different: I do not always need to show the model the answer. If I can build a trustworthy reward function, I can let the model explore multiple outputs and reinforce the ones that work.

On this tiny experiment, GRPO genuinely improved the held-out arithmetic metrics by the end of 40 rollouts.

But the step-20 result is just as important as the final result. It showed me that an RL model can optimize the easiest part of the reward first and give the illusion of progress.

That makes me much more careful about asking:

What exactly am I rewarding?

instead of only:

Is the reward going up?

And now I understand the DeepSeek-style training idea a lot better than I did from only reading the equation.

For the next experiment, I want to try the other lesson from the R1 work: distillation. Instead of asking a 360M model to discover good reasoning entirely by itself, I can generate higher-quality reasoning examples from a much stronger model, use those for SFT, and then apply GRPO afterward.

That would give me a nice progression:

flowchart LR
    A[Part 1<br/>SFT + LoRA] --> B[Part 2<br/>GRPO + reward]
    B --> C[Part 3?<br/>Reasoning distillation]
    C --> D[SFT + GRPO<br/>combined pipeline]

And that is probably where this small local AI experiment goes next.