activation functions, loss functions, validation metrics in deep learning : the key differences

understanding activation functions, loss functions, and validation metrics is essential for anyone working with deep learning. these three components determine how a neural network learns, adapts, and is evaluated. yet, many confuse their roles and where they fit in the training process. in this guide, we break down their differences and show how they connect to build an efficient deep learning model.

why does this only apply to deep learning and not traditional machine learning?

in traditional machine learning (e.g., decision trees, support vector machines, linear regression), models do not rely on activation functions or backpropagation. instead, they use predefined mathematical transformations and optimization techniques that do not require deep layers of neurons.

• activation functions are unique to neural networks because they introduce non-linearity, allowing deep learning models to capture complex patterns. traditional ml models, like decision trees or linear regression, do not need them since they are either inherently non-linear (e.g., decision trees) or explicitly linear (e.g., linear regression).
• loss functions exist in both ml and dl, but in ml, they are used mainly for optimization techniques like gradient boosting or svm margin maximization. in deep learning, loss functions drive the learning process via backpropagation.
• validation metrics apply to both, but in ml, they are often simpler because traditional models are less prone to overfitting compared to deep neural networks. deep learning requires more careful evaluation due to its complexity and sensitivity to data distribution.

in short, activation functions are exclusive to deep learning, while loss functions and validation metrics are more general but take on a unique role in deep learning due to the iterative weight updates through backpropagation.

a neural network works in several steps

1️. input: we give it data (e.g., an image or text).
2️. processing: multiple layers of neurons analyze the data.
3️. output: the network makes a prediction.
4️. comparison with ground truth: we check if the prediction is correct.
5️. learning (backpropagation): we adjust the network’s weights to improve.

this is where loss functions, validation metrics, and activation functions come into play.

1) activation functions

activation functions introduce non-linearity into neural networks, allowing them to learn complex relationships in data. without activation functions, a neural network would just be a simple linear combination of its inputs, making it useless for complex tasks.

✅ where?

in each layer of the neural network. applied after each layer to transform values (except sometimes in the final layer).

✅ role?

introduce non-linearity to learn complex relationships.

✅ popular examples:

• relu (most commonly used in deep learning)
• leaky relu, prelu (improved versions of relu)
• swish, mish (help with convergence)
• sigmoid, softmax (often used in output layers)
• gelu (used in transformers and nlp)

2) loss functions

loss functions measure the model’s error to adjust weights. they indicate whether the model is making errors, and this value is what the optimization algorithm (ex gradient descent) tries to minimize during training.

✅ where?

right after the model’s prediction, during training.

✅ role?

quantifies the error to guide learning.

✅ popular examples:

classification:

• cross-entropy loss (for multi-class classification)
• binary cross-entropy (for binary classification)
• focal loss (for imbalanced classes)

regression:

• mean squared error (mse)
• mean absolute error (mae)
• huber loss (less sensitive to outliers)

other contexts:

• wasserstein loss (used in gans)
• dice loss / iou loss (image segmentation)

3) validation metrics

validation metrics evaluate model performance but are not directly optimized during training. some metrics are not differentiable and cannot be used as loss functions, but they are essential for comparing models and assessing real-world performance. without validation metrics, we wouldn’t know if a model is good or overfitting.

✅ where?

after training, on test or validation data.

✅ role?

they provide insights into model performance that loss functions alone do not capture.

✅ popular examples:

classification:

• accuracy
• f1 score
• auc-roc / auc-pr
• log loss

regression:

• r² score
• rmse / mae / msle

image segmentation:

• iou
• dice coefficient

conclusion: how do they connect?

• the neural network processes data using activation functions.
• it makes a prediction and compares it to the ground truth with a loss function.
• it adjusts its weights based on this error and learns.
• finally, we measure its performance with validation metrics.

Sirine Amrane