Grid problem in Reinforcement Learning explained using Bellman Optimality Equation (Stochastic case) скачать в хорошем качестве

Grid problem in Reinforcement Learning explained using Bellman Optimality Equation (Stochastic case)

Understanding Value Functions in Reinforcement Learning: In this video, I demonstrate how to calculate the value function in a 3x3 gridworld where the agent receives a reward only at the terminal state. I compute the value function over two iterations in two scenarios: When all directions have equal probability. When movement probabilities differ by direction. While explaining the second case (different probabilities), I made a small oversight in the first iteration: I only showed the maximum value instead of displaying the value updates for all directions. In the second iteration, however, I correctly presented the values for all directions and then took the maximum. I decided to keep this in the video to highlight that even small slips are part of the learning process—and the key takeaway remains intact. This exercise highlights why value functions are fundamental in actor-critic methods. In actor-critic, we estimate the advantage: A=r+γV(s′)−V(s) The critic minimizes: Loss(critic)== A^2 The actor updates its policy with: Loss(actor)=−(log⁡P(a∣s))⋅A Convergence is reached when the value function stabilizes across iterations (or equivalently when A≈0). By working through this toy example, my goal is to show how value functions serve as the backbone for more advanced reinforcement learning algorithms and why they matter in practice. #ai #reinforcementlearning