Bellman Optimality Equation
Finite-MDP optimal state-value form
The Bellman optimality equation expresses the best achievable value of a state in terms of immediate reward and future value. For each available action, it averages over possible outcomes; it then chooses the action with the greatest expected return. The same value function appears on both sides, making the equation a recursive consistency condition.
Notes
This is the optimal state-value equation for a finite Markov decision process with discrete, bounded rewards and discount factor . The joint model specifies the probability of next state and immediate reward given current state and action , and the maximization runs over actions available in .
The prime in labels a successor state rather than a derivative, and the star marks optimality rather than multiplication or complex conjugation. This equation defines what the optimal value must satisfy; it is not by itself an algorithm for solving an unknown environment.
Sources
- Reinforcement Learning: An Introduction, Chapter 3 — Sutton and BartoSutton & Barto Chapter 3.