Blog 2: Marshmallows, Patience & Dopamine Loops

Scene 1: The Famous Experiment

It's the late 1960s at Stanford University. A child sits in a quiet room, staring at a single marshmallow placed on the table.

The researcher says:

"You can eat this marshmallow right now. But if you wait for 15 minutes without eating it, I'll give you two marshmallows instead."

The researcher leaves. Now it's just the child, the marshmallow, and the ticking clock.

Some kids grab it immediately. Some twist, squirm, cover their eyes, sing songs, anything to distract themselves. Some hold out and eventually win the second marshmallow.

This simple test became one of the most iconic psychology experiments ever - and a perfect real-world example of Markov Decision Processes in action.

Scene 2: Decisions in the Marshmallow Room

In MDP terms, here's what's happening:

• States (S):
  • Waiting with 1 marshmallow on the table
  • Rewarded with 2 marshmallows
  • No marshmallow left (ate it too soon)
• Actions (A):
  • Eat now
  • Wait
• Rewards (R):
  • Eat now → reward = +1 marshmallow (instant).
  • Wait → reward = +2 marshmallows (delayed).
• Transition Probabilities (P):
  • If you wait, there's uncertainty - can you actually resist temptation? (say only 60% kids succeed).
  • If you eat now, the transition is 100%: no marshmallow later.

Mathematically:

$$\text{Expected Reward (Eat Now)} = 1 \times 1 = 1$$ $$\text{Expected Reward (Wait)} = 0.6 \times 2 + 0.4 \times 0 = 1.2$$

So logically, waiting is the better choice. But still, many kids couldn't. Why?

Scene 3: Enter the Brain's Reward System

The answer lies in how the brain handles time and dopamine.

• Dopamine is like the brain's "reward signal."
• Every time you imagine eating the marshmallow now, your brain gives you a small dopamine surge.
• Future rewards, however, get discounted - your brain treats them as less valuable the farther away they are in time.

This is where the discount factor (\(t\)) comes in. In reinforcement learning, the value of a reward at time \(t\) is discounted as:

$$\text{Value} = R \times \gamma^t$$

• If \(\gamma\) is close to 1 → you care about future rewards almost as much as present ones.
• If \(\gamma\) is small → you're impulsive, heavily biased toward instant gratification.

So in our marshmallow test:

• Patient kids had a higher \(\gamma\). They valued the "two later" enough to resist.
• Impulsive kids had a lower \(\gamma\). They gave in to the "one now."

Scene 4: Beyond the Marshmallow Room

The brilliance of this experiment is that it didn't just test candy preference - it revealed deep psychological traits.

• Long-term Studies: Kids who resisted tended to do better later in life - higher SAT scores, healthier lifestyles, even stronger social skills.
• Why? Because they were better at delaying gratification, essentially using a high discount factor in their life decisions.

Now think about adults:

• Choosing savings over impulse shopping.
• Choosing gym over Netflix.
• Choosing career growth over short-term comfort.

Every one of these is a marshmallow test in disguise.

Scene 5: The Bellman Equation in the Background

In reinforcement learning, the decision process is captured by the Bellman Equation, which recursively defines the value of a state:

$$V(s) = R(s,a) + \gamma \sum_{s'} P(s'|s,a) V(s')$$

This looks heavy, but in plain words it means:

• The value of your current choice = reward now + (discounted value of future outcomes).

So when a child waits for two marshmallows, their brain is (perhaps unconsciously) computing something very similar to this equation!

Scene 6: Wrapping Up - From Candy to Life

The marshmallow test isn't just about sweets. It's about how humans weigh present vs future.

Sometimes we fail - we scroll Instagram instead of studying. Sometimes we succeed - we save money instead of spending it recklessly.

The lesson is simple: our minds are notoriously bad at valuing the future, but with awareness and training, we can shift our internal discount factor to favor long-term gains.

In the next blog, we'll explore what happens when this balance breaks completely - when the brain gets trapped in dopamine shortcuts. We'll step into the world of addiction, and see how MDP explains why people knowingly choose harm for the sake of instant reward.

Takeaway: Every marshmallow moment in life is a test of patience vs impulse. The math is simple, but the psychology is profound.