Overview

For imitation learning, it is easy for us to have a fit on the expert behavior, However when we encounter a new situation, we can’t solve the problem. So we imagine that there is an expert strategy in the real world, and this expert strategy will give a reward function to guide our agents to learn. AIRL is based on the past GAIL. In GAIL, we directly use GAN to fit the behavior of the agent to the expert behavior, and do not directly give the reward function. This behavior will lead to our learning is not clear and not good. Migration, so we will record the function of the discriminator in GAN as D, and use D to represent our reward function. Below I will write in detail, how this is derived.

the goal of the IRL

Firstly, we define a distribution $p_{\theta }(\tau )$. in another word, it is the expectation for the accumulated over the expert trajectory. so it can be wrote in this form. And D is the demonstrations of the expert.

$p_{\theta }(\tau )\propto p(s_0){\prod }_{t=0}^{T-1}p(s_{t+1}|s_t,a_t)e^{r_{\theta }(s_t,a_t)}$

the goal of IRL can been seen as training a generative model over trajectories as:

$\underset{\theta }{\max }\{J(\theta )\}=\underset{\theta }{\max }{E}_{\tau \sim D}[\log p_{\theta }(\tau )]$

Then we calculate the gradient with respect to $\theta$(where $p_{\theta ,t}={\int }_{s_{t^{\prime }}\ne t,a_{t^{\prime }}\ne t}{p}_{\theta (\tau )}$ denote the state-action marginal at time t):
$\frac{\partial }{\partial \theta }J(\theta )={\sum }_{t=0}^T{E}_{\mathcal{D}}\left[\frac{\partial }{\partial \theta }{r}_{\theta }\left(s_t,a_t\right)\right]-E_{p_{\theta ,t}}\left[\frac{\partial }{\partial \theta }{r}_{\theta }\left(s_t,a_t\right)\right]$

then we do importance sampling. here we instead train a separate importance sampling distribution $\mu (\tau )$ . And the $\mu (a|s)=\frac{1}{2}\pi (a|s)+\frac{1}{2}\widehat{p(a|s)}$.$\pi (a|s)$ is our agent policy $\hat{p}(a|s)$ is a rough density estimate trained on the demonstrations.

So, the gradient can be convert to the form below:
$\frac{\partial }{\partial \theta }J(\theta )={\sum }_{t=0}^T{E}_{\mathcal{D}}\left[\frac{\partial }{\partial \theta }{r}_{\theta }\left(s_t,a_t\right)\right]-E_{{\mu }_t}\left[\frac{p_{\theta ,t}\left(s_t,a_t\right)}{{\mu }_t\left(s_t,a_t\right)}\frac{\partial }{\partial \theta }{r}_{\theta }\left(s_t,a_t\right)\right]$

Additionally, we wish to adapt the importance sampler $\pi$ to reduce variance, by minimizing the KL divergency between $\pi$ and $p_{\theta }$, and the KL divergency can be calculate as follow.

$\because D_{KL}(p\Vert q)=E[\log p(x)-\log q(x)]$
$\because p_{\theta }(\tau )=p(s_0){\prod }_{t=0}^{T-1}p(s_{t+1}|s_t,a_t)e^{r_{\theta }(s_t,a_t)}$
$\because \pi (\tau )=p\left(s_0\right){\prod }_{t=0}^{T-1}p\left(s_{t+1}|s_t,a_t\right)\pi \left(a_t|s_t\right)$
$\therefore D_{KL}(\pi (\tau )\Vert p_{\theta }(\tau ))=log(\frac{p\left(s_0\right){\prod }_{t=0}^{T-1}p\left(s_{t+1}|s_t,a_t\right)\pi \left(a_t|s_t\right)}{p(s_0){\prod }_{t=0}^{T-1}p(s_{t+1}|s_t,a_t)e^{r_{\theta }(s_t,a_t)}})$
$\therefore D_{KL}(\pi (\tau )\Vert p_{\theta }(\tau ))=log\pi (a_t|s_t)-{\sum }_{t=0}^T{r}_{\theta }(s_t,a_t)$

in a another word minimize the KL divergency is same to maximize the entropy
regularized policy objective:

${\max }_{\pi }{E}_{\pi }\left[{\sum }_{t=0}^T{r}_{\theta }\left(s_t,a_t\right)-\log \pi \left(a_t|s_t\right)\right)]$

For AIRL, we calculate the D function from the loss of cross-entropy between expert demonstrations and generated samples(equal to the objective of the discriminator):

$\mathcal{L}(\theta )={\sum }_{t=0}^T-E_{\mathcal{D}}\left[\log D_{\theta }\left(s_t,a_t\right)\right]-E_{{\pi }_t}\left[\log \left(1-D_{\theta }\left(s_t,a_t\right)\right)\right]$

Ask the gradient of the loss function equal to zero.

$-\frac{p_{\theta }}{D_{\theta }(s,a)}+\frac{\pi (a|s)}{1-D_{\theta }(s,a)}=0$

$\therefore D_{\theta }(s,a)=\frac{p_{\theta }}{p_{\theta }+\pi (a|s)}$
Here we regard the $p_{\theta }=e^{f_{\theta (s,a)}}$, so the D function represent as follow:
$D_{\theta }(s,a)=\frac{\exp \left\{f_{\theta }(s,a)\right\}}{\exp \left\{f_{\theta }(s,a)\right\}+\pi (a|s)}$

In the end, back to the entropy regularized policy objective, we re-write it in to the new expectation of reward as follow:

$\begin{array}{rl}{\hat{r}}_t(s,a)& =f_{\theta }(s,a)-\log \pi (a|s)\\ & =\log \frac{e^{f_{\theta }(s,a)}}{e^{f_{\theta }(s,a)}+\pi (a|s)}-\log \frac{\pi (a|s)}{e^{f_{\theta }(s,a)}+\pi (a|s)}\\ & =\log \left(D_{\theta }(s,a)\right)-\log \left(1-D_{\theta }(s,a)\right)\end{array}$

End of derivation