Equationpedia

Bellman Optimality Equation

Finite-MDP optimal state-value form

The Bellman optimality equation expresses the best achievable value of a state in terms of immediate reward and future value. For each available action, it averages over possible outcomes; it then chooses the action with the greatest expected return. The same value function appears on both sides, making the equation a recursive consistency condition.

Optimal state value
Best action
Transition expectation
Immediate reward
Discounted future value
V∗(s)=max⁡a∑s′,rp(s′,r∣s,a)[r+γV∗(s′)]\htmlData{region=optimal-state-value}{\htmlData{term=optimal-value}{V^*}(\htmlData{term=current-state}{s})} = \htmlData{region=best-action}{\max_{\htmlData{term=action}{a}}} \htmlData{region=transition-expectation}{\sum_{\htmlData{term=next-state}{s'},\htmlData{term=reward}{r}} \htmlData{term=transition-model}{p}\left(\htmlData{term=next-state}{s'},\htmlData{term=reward}{r}\mid \htmlData{term=current-state}{s},\htmlData{term=action}{a}\right)}\left[\htmlData{region=immediate-reward}{\htmlData{term=reward}{r}} + \htmlData{region=discounted-future-value}{\htmlData{term=discount-factor}{\gamma} \htmlData{term=optimal-value}{V^*}(\htmlData{term=next-state}{s'})}\right]

Notes

This is the optimal state-value equation for a finite Markov decision process with discrete, bounded rewards and discount factor 0≤γ<10 \leq \gamma < 1. The joint model p(s′,r∣s,a)p(s', r \mid s, a) specifies the probability of next state s′s' and immediate reward rr given current state ss and action aa, and the maximization runs over actions available in ss.

The prime in s′s' labels a successor state rather than a derivative, and the star marks optimality rather than multiplication or complex conjugation. This equation defines what the optimal value must satisfy; it is not by itself an algorithm for solving an unknown environment.

Sources