DeployBot: [deploy-bot] Deploy from CI (8c92aeb)

2026-04-24T07:08:58Z

[deploy-bot] Deploy from CI (8c92aeb)

← Older revision		Revision as of 07:08, 24 April 2026
Line 107:		Line 107:
	[[Category:Machine Learning]]		[[Category:Machine Learning]]
	[[Category:Intermediate]]		[[Category:Intermediate]]
	~~<!--v1.2.0 cache-bust-->~~
	~~<!-- pass 2 -->~~

DeployBot: Pass 2 force re-parse

2026-04-24T07:00:32Z

Pass 2 force re-parse

← Older revision		Revision as of 07:00, 24 April 2026
Line 108:		Line 108:
	[[Category:Intermediate]]		[[Category:Intermediate]]
	<!--v1.2.0 cache-bust-->		<!--v1.2.0 cache-bust-->
			<!-- pass 2 -->

DeployBot: Force re-parse after Math source-mode rollout (v1.2.0)

2026-04-24T06:57:56Z

Force re-parse after Math source-mode rollout (v1.2.0)

← Older revision		Revision as of 06:57, 24 April 2026
Line 107:		Line 107:
	[[Category:Machine Learning]]		[[Category:Machine Learning]]
	[[Category:Intermediate]]		[[Category:Intermediate]]
			<!--v1.2.0 cache-bust-->

DeployBot: [deploy-bot] Deploy from CI (775ba6e)

2026-04-24T04:01:41Z

[deploy-bot] Deploy from CI (775ba6e)

New page

{{LanguageBar | page = Cross-Entropy Loss}}
{{ArticleInfobox | topic_area = Machine Learning | difficulty = Intermediate | prerequisites = [[Loss Functions]], [[Softmax Function]]}}
{{ContentMeta | generated_by = claude-opus | model_used = claude-opus-4-6 | generated_date = 2026-03-13}}

'''Cross-entropy loss''' (also called '''log loss''') is the most widely used loss function for classification tasks in machine learning. Rooted in information theory, it measures the dissimilarity between the true label distribution and the model's predicted probability distribution, providing a smooth, differentiable objective that drives probabilistic classifiers toward confident, correct predictions.

== Information-Theoretic Foundations ==

=== Entropy ===

The '''entropy''' of a discrete probability distribution <math>p</math> quantifies its uncertainty:

:<math>H(p) = -\sum_{k=1}^{K} p_k \log p_k</math>

For a deterministic distribution (one-hot label), <math>H(p) = 0</math>. Entropy is maximized when all outcomes are equally likely.

=== KL Divergence ===

The '''Kullback-Leibler divergence''' measures how one distribution <math>q</math> differs from a reference distribution <math>p</math>:

:<math>D_{\mathrm{KL}}(p \,\|\, q) = \sum_{k=1}^{K} p_k \log \frac{p_k}{q_k}</math>

KL divergence is non-negative and equals zero if and only if <math>p = q</math>.

=== Cross-Entropy ===

The '''cross-entropy''' between distributions <math>p</math> (true) and <math>q</math> (predicted) is:

:<math>H(p, q) = -\sum_{k=1}^{K} p_k \log q_k = H(p) + D_{\mathrm{KL}}(p \,\|\, q)</math>

Since <math>H(p)</math> is constant with respect to model parameters, minimizing cross-entropy is equivalent to minimizing KL divergence — i.e., making the predicted distribution <math>q</math> as close to the true distribution <math>p</math> as possible.

== Binary Cross-Entropy ==

For binary classification with true label <math>y \in \{0, 1\}</math> and predicted probability <math>\hat{y} = \sigma(z)</math> (where <math>\sigma</math> is the [[Softmax Function|sigmoid function]]):

:<math>\mathcal{L}_{\mathrm{BCE}} = -\bigl[y \log \hat{y} + (1 - y) \log(1 - \hat{y})\bigr]</math>

Over a dataset of <math>N</math> samples:

:<math>\mathcal{L} = -\frac{1}{N} \sum_{i=1}^{N} \bigl[y_i \log \hat{y}_i + (1 - y_i) \log(1 - \hat{y}_i)\bigr]</math>

The gradient with respect to the logit <math>z</math> takes the elegantly simple form <math>\hat{y} - y</math>, which is both intuitive and computationally efficient.

== Categorical Cross-Entropy ==

For multi-class classification with <math>K</math> classes, the true label is typically a one-hot vector <math>\mathbf{y}</math> with <math>y_c = 1</math> for the correct class <math>c</math>. The predicted probabilities <math>\hat{\mathbf{y}}</math> are obtained via the [[Softmax Function]]:

:<math>\mathcal{L}_{\mathrm{CE}} = -\sum_{k=1}^{K} y_k \log \hat{y}_k = -\log \hat{y}_c</math>

This reduces to the negative log-probability of the correct class, which is why categorical cross-entropy is also called '''negative log-likelihood''' in this context.

== Numerical Stability ==

=== The Log-Sum-Exp Trick ===

Naively computing <math>\log(\mathrm{softmax}(z_k))</math> involves exponentiating potentially large logits, causing overflow. The '''log-sum-exp''' trick avoids this:

:<math>\log \hat{y}_k = z_k - \log \sum_{j=1}^{K} e^{z_j} = z_k - \left(m + \log \sum_{j=1}^{K} e^{z_j - m}\right)</math>

where <math>m = \max_j z_j</math>. Subtracting the maximum logit ensures the largest exponent is zero, preventing overflow. All major deep learning frameworks implement this fused operation (e.g., PyTorch's <code>CrossEntropyLoss</code> accepts raw logits).

=== Clamping ===

Predicted probabilities should be clamped away from exactly 0 and 1 to avoid <math>\log(0) = -\infty</math>. A small epsilon (e.g., <math>10^{-7}</math>) is typically used.

== Label Smoothing ==

'''Label smoothing''' (Szegedy et al., 2016) replaces the hard one-hot target with a soft distribution:

:<math>y_k^{\mathrm{smooth}} = (1 - \alpha)\, y_k + \frac{\alpha}{K}</math>

where <math>\alpha</math> is a small constant (commonly 0.1). This prevents the model from becoming overconfident, improves calibration, and often yields better generalization. It is standard practice in training large image classifiers and Transformer models.

== Comparison with Other Losses ==

{| class="wikitable"
|-
! Loss !! Formula !! Typical use
|-
| '''Cross-entropy''' || <math>-\sum y_k \log \hat{y}_k</math> || Classification
|-
| '''Mean squared error''' || <math>\frac{1}{K}\sum(y_k - \hat{y}_k)^2</math> || Regression (poor for classification)
|-
| '''Hinge loss''' || <math>\max(0, 1 - y \cdot z)</math> || SVM-style classification
|-
| '''Focal loss''' || <math>-(1-\hat{y}_c)^\gamma \log \hat{y}_c</math> || Imbalanced classification
|}

Cross-entropy has steeper gradients than MSE when the prediction is confidently wrong, leading to faster correction of large errors.

== See also ==

* [[Loss Functions]]
* [[Softmax Function]]
* [[Logistic regression]]
* [[Information theory]]
* [[Neural Networks]]

== References ==

* Shannon, C. E. (1948). "A Mathematical Theory of Communication". ''Bell System Technical Journal''.
* Goodfellow, I., Bengio, Y. and Courville, A. (2016). ''Deep Learning''. MIT Press, Chapter 6.
* Szegedy, C. et al. (2016). "Rethinking the Inception Architecture for Computer Vision". ''CVPR''.
* Lin, T.-Y. et al. (2017). "Focal Loss for Dense Object Detection". ''ICCV''.

[[Category:Machine Learning]]
[[Category:Intermediate]]

Cross-Entropy Loss - Revision history

DeployBot: [deploy-bot] Deploy from CI (8c92aeb)

DeployBot: Pass 2 force re-parse

DeployBot: Force re-parse after Math source-mode rollout (v1.2.0)

DeployBot: [deploy-bot] Deploy from CI (775ba6e)