Graph Attention Network

Definition

Graph attention network (GAT) applied Attention mechanism to the aggregation stage of GraphSAGE model. Instead of assigning same weights to all neighbors, GAT uses attention coefficient between nodes.

Architecture

Attention

The attention values are calculated as $a_{ij}=\underset{v_j\in \mathcal{N}(v_i)}{\mathrm{softmax}}(a^{⊺}\cdot \mathrm{concat}(W{h}_i,W{h}_j))$ where $a$ and $W$ are learnable parameters shared across all nodes.

Aggregation

The feature of node $v_i$ at $k$-th stage is calculated with attention $h_i^{(k)}=\sigma \left(\sum _{v_j\in \mathcal{N}(v_i)}{a}_{ij}{W}^{(k)}{h}_i^{(k-1)}\right)$ where $\sigma$ is a non-linear Activation Function

Multi-Head Attention

To stabilize the learning process, GAT employs multi-head attention. The independent attentions are executed with the same input, and the output of the output features are concatenated or averaged. $h_i^{(k)}=\mathrm{AGG}(\{h_i^{(k)}[h]|h=1,\dots ,H\})$ where $h_i^{(k)}[h]=\sigma \left(\sum _{v_j\in \mathcal{N}(v_i)}{a}_{ij}^h{W}^{(k)}{h}_i^{(k-1)}\right)$, and $H$ is the number of attention heads.