Q-learning Mathematical Background

Prerequisites: Q-Learning.

In the following derivations, the symbols defined as in the prerequisite article will be used.
The Q-learning technique is based on the Bellman Equation.

where,
E : Expectation
t+1 : next state
: discount factor

Rephrasing the above equation in the form of Q-Value:-

The optimal Q-value is given by

Policy Iteration: It is the process of determining the optimal policy for the model and consists of the following two steps:-

1. Policy Evaluation: This process estimates the value of the long-term reward function with the greedy policy obtained from the last Policy Improvement step.
2. Policy Improvement: This process updates the policy with the action that maximizes V for each of the state. This process is repeated until convergence is achieved.

Steps Involved:-

• Initialization:

  = any real random number
  = any A(s) arbitrarily chosen

• Policy Evaluation:

  
  while()
  {
      for each s in S
      {    
          
          
          
      }
  }
  
  
  
  

• Policy Improvement:

  
  while(true)
      for each s in S
      {
          
          
          if()
              
          if()
              break from both loops
      }
  return V,
  

• Value Iteration: This process updates the function V according to the Optimal Bellman Equation.

  

Working Steps:

• Initialization: Initialize array V by any random real number.
• Computing the optimal value:

  
  while()
  {
      for each s in S
      {
          
          
          
      }
  }
  
  
  return