Neural Network

Definition

Neural network can be thought as a non-linear generalization of linear model.

The derived features $Z_m$ are constructed by an Activation Function and linear combinations of the inputs. $Z_m=\sigma ({\alpha }_{0m}+{\alpha }_m^{⊺}X),m=1,\dots ,M$ where $\sigma$ is an Activation Function

Output nodes are the linear combinations of $Z$ $T_k={\beta }_{0k}+{\beta }_k^{⊺}Z,k=1,\dots ,K$ And the output is modeled by a function of a linear combinations of $Z_m$ $f_k(X)=g_k(T),k=1,\dots ,K$ where $g_k(T)$ is called an output function.

Facts

The output function $g_k(T)$ varies by the problem. For regression $g_k$ is Identity Function, and for $k$-class classification Softmax Function $\sigma$ is used as the $g_k$.

For regression problem, Sum of Squared Errors Loss is used as Loss Function. For classification problem, we use Cross-Entropy Loss

With the softmax activation function and the Cross-Entropy Loss, the neural network model is exactly a linear Logistic Regression model in the hidden units.

The parameters of a neural network are estimated by Backpropagation.

Neural network is especially effective in problem with a high signal-to-noise ratio.