1.激活函数

sigmoid

$$f(x)=\frac{1}{1+e^{-x}}$$ $$f^{\prime }(x)=f(x)(1-f(x))$$

tanh

$$f(x)=\frac{e^x-e^{-x}}{e^x+e^{-x}}$$

ReLU

$$f(x)=\{\begin{array}{l}x,x>0\\ 0,x\le 0\end{array}$$

Leaky ReLU

$$f(x)=\{\begin{array}{l}x,x>0\\ az,x\le 0\end{array}$$

softmax

$$\mathrm{softmax}(x_i)=\frac{x_i}{{\sum }_j{x}_j}$$