IQN

Definition

The Implicit Quantile Network (IQN) algorithm is a Distributional Reinforcement Learning algorithm that implicitly estimates the quantiles of a return distribution by learning a function that maps a quantile fraction to the corresponding quantile value.

Architecture

IQN estimates the quantile value $Z(s,a,\tau )=F_Z^{-1}(\tau )$ for a given state $s$, action $a$, and a quantile fraction $\tau \in [0,1]$. The output is the estimated quantile value corresponding to that fraction $\tau$.

The input state $s$ processed by the encoder layers of neural network to produce a state embedding vector $\varphi (s)$.

The quantile fraction $\tau$ is embedded into a higher-dimensional vector $\psi (\tau )$ using a set of basis functions, where whose dimension is the same as the one of the state-feature $\varphi (s)$. ${\psi }_i(\tau )=\cos (i\pi \tau )$ The two embeddings: state embedding and quantile-embedding, are combined using Hadamard Product $\varphi (s)\odot \psi (\tau )$.

The combined embedding is fed into the further layers $f_{\theta }$ to predict the quantile value $Z(s,a,\tau )$

In summary, the quantile value for a given state is estimated by $Z(s,a,\tau ;\theta )\approx f_{\theta }(\varphi (s),\psi (\tau ))$

By sampling different quantile fractions $\tau$, IQN can approximate the entire return distribution. The expectation of the return can be approximated by averaging the estimated quantile values over the sampled quantile fractions $Q(s,a;\theta )\approx \frac{1}{N}\sum _{i=1}^{N}Z(s,a,{\tau }_i;\theta )$ where $\{{\tau }_i{\}}_{i=1}^N$ are sampled quantile fractions.

Loss Function

IQN uses a quantile Huber loss function, similar to QR-DQN. Given a transition $(s,a,r,s^{\prime })$, the loss is defined as $L(\theta )=\sum _{i=1}^N{E}_{{\tau }^{\prime }}[{\rho }_{{\hat{\tau }}_i}^{\kappa }(\mathcal{T}{Z}_{{\tau }_j^{\prime }}-Z(s,a,{\tau }_i;\theta ))]$ Where:

• $\{{\hat{\tau }}_i{\}}_{i=1}^N$ and $\{{\tau }_j^{\prime }{\}}_{j=1}^{N^{\prime }}$ are quantile fractions for the current state-action pair and next state, respectively.
• $\mathcal{T}{Z}_{{\tau }_j^{\prime }}=r+\gamma Z(s^{\prime },a^{\ast },{\tau }_j^{\prime };\hat{\theta })$ and $a^{\ast }=\underset{a}{\mathrm{argmax}}\frac{1}{N^{\prime }}\sum _{j=1}^{N^{\prime }}Z(s^{\prime },a,{\tau }_j^{\prime };\hat{\theta })\approx \underset{a}{\mathrm{argmax}}Q(s^{\prime },a;\hat{\theta })$.
• ${\rho }_{\tau }^{\kappa }(x)=L_{\kappa }(x)(\tau -1(x<0))$ where $L_{\kappa }(x)$ is Huber Loss.

Risk-Sensitive Policy

$\beta :[0,1]\to [0,1]$ The IQN has information about the entire distribution by estimating the distribution of returns. This property makes it suitable for implementing risk-sensitive policies, allowing for decision-making considering risk. Risk-sensitive policies focus on not just the expected return, but the variability or uncertainty in the returns. IQN makes it feasible by providing estimated quantiles of the return distribution.

The function $\beta :[0,1]\to [0,1]$ is used to define a distortion risk measure that focuses on specific parts of the return distribution. By applying $\beta$ to the quantile fraction $\tau$, we can re-wright each outcome or change the sampling distribution.

These are examples of distortion measures:

• Cumulative Probability Weighting Parametrization (CPW): $\mathrm{CPW}(\eta ,\tau )=\frac{{\tau }^{\eta }}{({\tau }^{\eta }+(1-\tau {)}^{\eta }{)}^{1/\eta }}$
• Wang: $\mathrm{Wang}(\eta ,\tau )=\Phi ({\Phi }^{-1}(\tau )+\eta )$ where $\Phi$ is the CDF for standard normal distribution.
• Power Formula: $\mathrm{Pow}(\eta ,\tau )=\{\begin{array}{ll}{\tau }^{1/(1+|\eta |)}& \eta \ge 0\\ 1-(1-\tau {)}^{1/(1+|\eta |)}& \eta <0\end{array}$
• Conditional Value at Risk (CVaR): $\mathrm{CVaR}(\eta ,\tau )=\eta \tau$

Algorithm

1. Initialize behavior quantile network $Z$ and target quantile network $\hat{Z}$ with random weights $\theta$, the sample sizes $K,N,N^{\prime }$, and a distortion measure $\beta$.
2. Repeat for each episode:
   1. Initialize sequence $s_0$.
   2. Repeat for each step of an episode until terminal, $t=0,1,\dots ,T-1$ :
      1. Sample $K$ quantile fractions $\{{\tilde{\tau }}_k\sim \beta (\cdot ){\}}_{k=1}^K$ under the distortion measure $\beta$, | and select a greedy action $a_t=\underset{a}{\mathrm{argmax}}\frac{1}{K}\sum _{k=1}^{K}Z(s_t,a,{\tilde{\tau }}_k;\theta )\approx \underset{a}{\mathrm{argmax}}Q(s_t,a;\theta )$.
      2. Take the action $a_t$ and observe a reward $r_{t+1}$ and a next state $s_{t+1}$.
      3. Store transition $(s_t,a_t,r_{t+1},s_{t+1})$ in the replay buffer $\mathcal{R}$.
      4. Sample a random transition $(s_t,a_t,r_{t+1},s_{t+1})$ from $\mathcal{R}$.
      5. Sample $N^{\prime }$ quantile fractions $\{{\tau }_j^{\prime }\sim U(0,1){\}}_{j=1}^{N^{\prime }}$ for the next state and select a greedy action $a^{\ast }=\underset{a}{\mathrm{argmax}}\frac{1}{N^{\prime }}\sum _{j=1}^{N^{\prime }}Z(s_{t+1},a,{\tau }_j^{\prime };\hat{\theta })\approx \underset{a}{\mathrm{argmax}}Q(s^{\prime },a;\hat{\theta })$.
      6. Compute the target quantile values $\mathcal{T}{Z}_{{\tau }_j^{\prime }}=r_{t+1}+\gamma Z(s_{t+1},a^{\ast },{\tau }_j^{\prime };\hat{\theta })$.
      7. Perform Gradient Descent on the loss $L(\theta )=\sum _{i=1}^N\sum _{j=1}^{N^{\prime }}{\rho }_{{\hat{\tau }}_i}^{\kappa }(\mathcal{T}{Z}_{{\tau }_j^{\prime }}-Z(s,a,{\hat{\tau }}_i;\theta ))$.
      8. Update the target network $\hat{Z}$ parameter $\hat{\theta }\leftarrow \theta$ every $C$ steps.
      9. Update $s_t\leftarrow s_{t+1}$.