Batch Normalization Accelerating Deep Network Training - Revision history

DeployBot: [deploy-bot] Drop {{LanguageBar}} (v1.4.1)

2026-04-27T02:35:51Z

[deploy-bot] Drop {{LanguageBar}} (v1.4.1)

← Older revision		Revision as of 02:35, 27 April 2026
Line 1:		Line 1:
	<languages />		<languages />
	~~{{LanguageBar \| page = Batch Normalization Accelerating Deep Network Training}}~~

	<translate>		<translate>
Line 19:		Line 18:
	'''Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift''' is a 2015 paper by Ioffe and Szegedy from Google that introduced '''batch normalization''' (BatchNorm), a technique for normalizing layer inputs during neural network training. By reducing what the authors termed '''internal covariate shift''' — the change in the distribution of network activations as parameters are updated — batch normalization allowed the use of much higher learning rates, reduced sensitivity to initialization, and in some cases acted as a regularizer, eliminating the need for dropout.		'''Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift''' is a 2015 paper by Ioffe and Szegedy from Google that introduced '''batch normalization''' (BatchNorm), a technique for normalizing layer inputs during neural network training. By reducing what the authors termed '''internal covariate shift''' — the change in the distribution of network activations as parameters are updated — batch normalization allowed the use of much higher learning rates, reduced sensitivity to initialization, and in some cases acted as a regularizer, eliminating the need for dropout.

	~~== Overview ==~~ <!--T:3-->		<!--T:3-->
			== Overview ==

	<!--T:4-->		<!--T:4-->
Line 27:		Line 27:
	Batch normalization addresses this by normalizing the inputs to each layer using statistics computed over the current mini-batch. This ensures that each layer receives inputs with a stable mean and variance, regardless of changes in preceding layers. The technique is applied as a differentiable transformation inserted into the network architecture, making it compatible with standard backpropagation and stochastic gradient descent.		Batch normalization addresses this by normalizing the inputs to each layer using statistics computed over the current mini-batch. This ensures that each layer receives inputs with a stable mean and variance, regardless of changes in preceding layers. The technique is applied as a differentiable transformation inserted into the network architecture, making it compatible with standard backpropagation and stochastic gradient descent.

	~~== Key Contributions ==~~ <!--T:6-->		<!--T:6-->
			== Key Contributions ==

	<!--T:7-->		<!--T:7-->
Line 35:		Line 36:
	* '''Inception-BN architecture''': A batch-normalized version of GoogLeNet (Inception) that exceeded the original's accuracy and approached human-level performance on ImageNet.		* '''Inception-BN architecture''': A batch-normalized version of GoogLeNet (Inception) that exceeded the original's accuracy and approached human-level performance on ImageNet.

	~~== Methods ==~~ <!--T:8-->		<!--T:8-->
			== Methods ==

	<!--T:9-->		<!--T:9-->
Line 64:		Line 66:
	The authors also observed that batch normalization reduces the dependence on precise initialization, permits higher learning rates without divergence, and provides a mild regularization effect because each sample's normalized value depends on the other samples in its mini-batch, introducing stochastic noise.		The authors also observed that batch normalization reduces the dependence on precise initialization, permits higher learning rates without divergence, and provides a mild regularization effect because each sample's normalized value depends on the other samples in its mini-batch, introducing stochastic noise.

	~~== Results ==~~ <!--T:18-->		<!--T:18-->
			== Results ==

	<!--T:19-->		<!--T:19-->
Line 78:		Line 81:
	Ablation experiments showed that the combination of batch normalization with higher learning rates and the removal of dropout produced the best results.		Ablation experiments showed that the combination of batch normalization with higher learning rates and the removal of dropout produced the best results.

	~~== Impact ==~~ <!--T:22-->		<!--T:22-->
			== Impact ==

	<!--T:23-->		<!--T:23-->
Line 89:		Line 93:
	Batch normalization also influenced how practitioners think about network design. By stabilizing the training dynamics, it made hyperparameter search more forgiving and encouraged the development of deeper and wider architectures. The technique's interaction with other components — learning rate, weight initialization, and regularization — remains an active area of study.		Batch normalization also influenced how practitioners think about network design. By stabilizing the training dynamics, it made hyperparameter search more forgiving and encouraged the development of deeper and wider architectures. The technique's interaction with other components — learning rate, weight initialization, and regularization — remains an active area of study.

	~~== See also ==~~ <!--T:26-->		<!--T:26-->
			== See also ==

	<!--T:27-->		<!--T:27-->
Line 96:		Line 101:
	* [[Dropout A Simple Way to Prevent Overfitting]]		* [[Dropout A Simple Way to Prevent Overfitting]]

	~~== References ==~~ <!--T:28-->		<!--T:28-->
			== References ==

	<!--T:29-->		<!--T:29-->

DeployBot: Marked this version for translation

2026-04-27T00:31:27Z

Marked this version for translation

← Older revision		Revision as of 00:31, 27 April 2026
Line 19:		Line 19:
	'''Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift''' is a 2015 paper by Ioffe and Szegedy from Google that introduced '''batch normalization''' (BatchNorm), a technique for normalizing layer inputs during neural network training. By reducing what the authors termed '''internal covariate shift''' — the change in the distribution of network activations as parameters are updated — batch normalization allowed the use of much higher learning rates, reduced sensitivity to initialization, and in some cases acted as a regularizer, eliminating the need for dropout.		'''Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift''' is a 2015 paper by Ioffe and Szegedy from Google that introduced '''batch normalization''' (BatchNorm), a technique for normalizing layer inputs during neural network training. By reducing what the authors termed '''internal covariate shift''' — the change in the distribution of network activations as parameters are updated — batch normalization allowed the use of much higher learning rates, reduced sensitivity to initialization, and in some cases acted as a regularizer, eliminating the need for dropout.

	<!--T:3-->		== Overview == <!--T:3-->
	~~== Overview ==~~

	<!--T:4-->		<!--T:4-->
Line 28:		Line 27:
	Batch normalization addresses this by normalizing the inputs to each layer using statistics computed over the current mini-batch. This ensures that each layer receives inputs with a stable mean and variance, regardless of changes in preceding layers. The technique is applied as a differentiable transformation inserted into the network architecture, making it compatible with standard backpropagation and stochastic gradient descent.		Batch normalization addresses this by normalizing the inputs to each layer using statistics computed over the current mini-batch. This ensures that each layer receives inputs with a stable mean and variance, regardless of changes in preceding layers. The technique is applied as a differentiable transformation inserted into the network architecture, making it compatible with standard backpropagation and stochastic gradient descent.

	<!--T:6-->		== Key Contributions == <!--T:6-->
	~~== Key Contributions ==~~

	<!--T:7-->		<!--T:7-->
Line 37:		Line 35:
	* '''Inception-BN architecture''': A batch-normalized version of GoogLeNet (Inception) that exceeded the original's accuracy and approached human-level performance on ImageNet.		* '''Inception-BN architecture''': A batch-normalized version of GoogLeNet (Inception) that exceeded the original's accuracy and approached human-level performance on ImageNet.

	<!--T:8-->		== Methods == <!--T:8-->
	~~== Methods ==~~

	<!--T:9-->		<!--T:9-->
Line 67:		Line 64:
	The authors also observed that batch normalization reduces the dependence on precise initialization, permits higher learning rates without divergence, and provides a mild regularization effect because each sample's normalized value depends on the other samples in its mini-batch, introducing stochastic noise.		The authors also observed that batch normalization reduces the dependence on precise initialization, permits higher learning rates without divergence, and provides a mild regularization effect because each sample's normalized value depends on the other samples in its mini-batch, introducing stochastic noise.

	<!--T:18-->		== Results == <!--T:18-->
	~~== Results ==~~

	<!--T:19-->		<!--T:19-->
Line 82:		Line 78:
	Ablation experiments showed that the combination of batch normalization with higher learning rates and the removal of dropout produced the best results.		Ablation experiments showed that the combination of batch normalization with higher learning rates and the removal of dropout produced the best results.

	<!--T:22-->		== Impact == <!--T:22-->
	~~== Impact ==~~

	<!--T:23-->		<!--T:23-->
Line 94:		Line 89:
	Batch normalization also influenced how practitioners think about network design. By stabilizing the training dynamics, it made hyperparameter search more forgiving and encouraged the development of deeper and wider architectures. The technique's interaction with other components — learning rate, weight initialization, and regularization — remains an active area of study.		Batch normalization also influenced how practitioners think about network design. By stabilizing the training dynamics, it made hyperparameter search more forgiving and encouraged the development of deeper and wider architectures. The technique's interaction with other components — learning rate, weight initialization, and regularization — remains an active area of study.

	<!--T:26-->		== See also == <!--T:26-->
	~~== See also ==~~

	<!--T:27-->		<!--T:27-->
Line 102:		Line 96:
	* [[Dropout A Simple Way to Prevent Overfitting]]		* [[Dropout A Simple Way to Prevent Overfitting]]

	<!--T:28-->		== References == <!--T:28-->
	~~== References ==~~

	<!--T:29-->		<!--T:29-->

DeployBot: [deploy-bot] Convert Batch Normalization Accelerating Deep Network Training to Translate-extension page

2026-04-27T00:31:25Z

[deploy-bot] Convert Batch Normalization Accelerating Deep Network Training to Translate-extension page

New page

<languages />
{{LanguageBar | page = Batch Normalization Accelerating Deep Network Training}}

<translate>

{{PaperInfobox
| topic_area = Deep Learning
| difficulty = Research
| authors = Sergey Ioffe; Christian Szegedy
| year = 2015
| venue = ICML
| arxiv_id = 1502.03167
| source_url = https://arxiv.org/abs/1502.03167
| pdf_url = https://arxiv.org/pdf/1502.03167
}}
{{ContentMeta | generated_by = claude-opus | model_used = claude-opus-4-6 | generated_date = 2026-03-13}}


'''Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift''' is a 2015 paper by Ioffe and Szegedy from Google that introduced '''batch normalization''' (BatchNorm), a technique for normalizing layer inputs during neural network training. By reducing what the authors termed '''internal covariate shift''' — the change in the distribution of network activations as parameters are updated — batch normalization allowed the use of much higher learning rates, reduced sensitivity to initialization, and in some cases acted as a regularizer, eliminating the need for dropout.


== Overview ==


Training deep neural networks is complicated by the fact that each layer's input distribution changes during training as the parameters of all preceding layers are updated. This phenomenon, which the authors called internal covariate shift, forces the use of lower learning rates and careful parameter initialization, slowing training considerably.


Batch normalization addresses this by normalizing the inputs to each layer using statistics computed over the current mini-batch. This ensures that each layer receives inputs with a stable mean and variance, regardless of changes in preceding layers. The technique is applied as a differentiable transformation inserted into the network architecture, making it compatible with standard backpropagation and stochastic gradient descent.


== Key Contributions ==


* '''Batch normalization''': A method that normalizes each scalar feature independently over the mini-batch, using learnable scale and shift parameters to preserve representational capacity.
* '''Internal covariate shift hypothesis''': Identification and formalization of the problem of shifting input distributions during training as a contributor to optimization difficulty.
* '''Training acceleration''': Demonstration that batch normalization enables 14x faster convergence to the same accuracy level on ImageNet, and allows the use of much higher learning rates.
* '''Inception-BN architecture''': A batch-normalized version of GoogLeNet (Inception) that exceeded the original's accuracy and approached human-level performance on ImageNet.


== Methods ==


For a mini-batch <math>\mathcal{B} = \{x_1, x_2, \ldots, x_m\}</math> of activations at a given layer, the batch normalization transform computes:


<math>\mu_{\mathcal{B}} = \frac{1}{m} \sum_{i=1}^{m} x_i</math>


<math>\sigma_{\mathcal{B}}^2 = \frac{1}{m} \sum_{i=1}^{m} (x_i - \mu_{\mathcal{B}})^2</math>


<math>\hat{x}_i = \frac{x_i - \mu_{\mathcal{B}}}{\sqrt{\sigma_{\mathcal{B}}^2 + \epsilon}}</math>


<math>y_i = \gamma \hat{x}_i + \beta</math>


where <math>\epsilon</math> is a small constant for numerical stability, and <math>\gamma</math> and <math>\beta</math> are learnable parameters that allow the network to undo the normalization if it is not beneficial. These scale and shift parameters are critical: without them, normalization would constrain the representation to have zero mean and unit variance, potentially reducing the model's expressiveness.


During training, the mean and variance are computed per mini-batch. During inference, batch statistics are replaced with '''population statistics''' — running averages accumulated during training — so that the output for a single sample is deterministic and does not depend on other samples in the batch.


Batch normalization is typically applied before the activation function, after the linear or convolutional transformation. When used with convolutional layers, the normalization is performed per feature map (channel) rather than per individual activation, sharing statistics across all spatial locations within a feature map.


The authors also observed that batch normalization reduces the dependence on precise initialization, permits higher learning rates without divergence, and provides a mild regularization effect because each sample's normalized value depends on the other samples in its mini-batch, introducing stochastic noise.


== Results ==


On the ImageNet classification task:


* A batch-normalized network matched the accuracy of the original Inception model in only '''7% of the training steps''' (14x acceleration).
* '''BN-Inception''' (with batch normalization and other modifications) achieved a top-5 validation error of 4.82%, exceeding the accuracy of the original GoogLeNet (6.67%) and approaching human performance.
* Using batch normalization allowed training with a learning rate 10x higher than the baseline without divergence.
* On some configurations, batch normalization eliminated the need for dropout without accuracy loss, simplifying the architecture and reducing training time further.


Ablation experiments showed that the combination of batch normalization with higher learning rates and the removal of dropout produced the best results.


== Impact ==


Batch normalization became one of the most ubiquitous components in deep learning architectures. It was adopted almost universally in convolutional networks throughout the late 2010s and remains standard in many architectures. The technique's success inspired a family of normalization methods, including layer normalization (preferred in Transformers and recurrent networks), instance normalization (used in style transfer), and group normalization (useful for small batch sizes).


While the original internal covariate shift explanation has been debated — subsequent work by Santurkar et al. (2018) argued that the primary benefit comes from smoothing the optimization landscape rather than reducing distributional shift — the practical effectiveness of batch normalization is undisputed. It was a key enabler of training the deep networks that drove progress in computer vision throughout the 2010s.


Batch normalization also influenced how practitioners think about network design. By stabilizing the training dynamics, it made hyperparameter search more forgiving and encouraged the development of deeper and wider architectures. The technique's interaction with other components — learning rate, weight initialization, and regularization — remains an active area of study.


== See also ==


* [[Deep Residual Learning for Image Recognition]]
* [[ImageNet Classification with Deep CNNs]]
* [[Dropout A Simple Way to Prevent Overfitting]]


== References ==


* Ioffe, S. & Szegedy, C. (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. ''Proceedings of ICML 2015''. [https://arxiv.org/abs/1502.03167 arXiv:1502.03167]
* Szegedy, C., Liu, W., Jia, Y., et al. (2015). Going Deeper with Convolutions. ''CVPR 2015''.
* Santurkar, S., Tsipras, D., Ilyas, A., & Madry, A. (2018). How Does Batch Normalization Help Optimization? ''NeurIPS 2018''.


[[Category:Deep Learning]] [[Category:Research]] [[Category:Research Papers]]
</translate>