Activation Function
An activation function is the non-linear function a neuron applies to its weighted input, which lets neural networks model complex, non-linear relationships.
What is Activation Function?
Without activation functions, any stack of layers would collapse into a single linear calculation. Common choices include ReLU, sigmoid, tanh and GELU, the last being popular in transformers.
How it works
After summing its weighted inputs, a neuron passes the result through the activation function; ReLU, for example, outputs the value if positive and zero otherwise.
← All Artificial Intelligence & Superintelligence definitions · ← Back to the A–Z definitions