View source for Decoupled Weight Decay Regularization/paper

<languages />
{{PaperTabs}}
{{PaperInfobox
 | topic_area  = Machine Learning
 | difficulty  = Research
 | authors     = Ilya Loshchilov; Frank Hutter
 | year        = 2017
 | arxiv_id    = 1711.05101
 | source_url  = https://arxiv.org/abs/1711.05101
 | pdf_url     = https://arxiv.org/pdf/1711.05101.pdf
}}
{{ContentMeta
 | generated_by   = arxiv-importer-raw
 | model_used     = strategy:html5
 | generated_date = 2026-04-27
}}

<translate>
<!--T:1-->
<span id="decoupled-weight-decay-regularization"></span>

<!--T:2-->
= Decoupled Weight Decay Regularization =

<!--T:3-->
Ilya Loshchilov &amp; Frank Hutter<br />
University of Freiburg<br />
Freiburg, Germany,<br />
<code>{ilya,fh}@cs.uni-freiburg.de</code>

<!--T:4-->
<span id="abstract"></span>

<!--T:5-->
== Abstract ==

<!--T:6-->
L<sub>2</sub> regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is ''not'' the case for adaptive gradient algorithms, such as Adam. While common implementations of these algorithms employ L<sub>2</sub> regularization (often calling it “weight decay” in what may be misleading due to the inequivalence we expose), we propose a simple modification to recover the original formulation of weight decay regularization by ''decoupling'' the weight decay from the optimization steps taken w.r.t. the loss function. We provide empirical evidence that our proposed modification (i) decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam and (ii) substantially improves Adam’s generalization performance, allowing it to compete with SGD with momentum on image classification datasets (on which it was previously typically outperformed by the latter). Our proposed decoupled weight decay has already been adopted by many researchers, and the community has implemented it in TensorFlow and PyTorch; the complete source code for our experiments is available at https://github.com/loshchil/AdamW-and-SGDW

<!--T:7-->
<div id="S1" class="section">

<!--T:8-->
<span id="introduction"></span>

<!--T:9-->
== 1 Introduction ==

<!--T:10-->
Adaptive gradient methods, such as AdaGrad (Duchi et al., [[#bib.bib5|2011]]), RMSProp (Tieleman &amp; Hinton, [[#bib.bib19|2012]]), Adam (Kingma &amp; Ba, [[#bib.bib10|2014]]) and most recently AMSGrad  (Reddi et al., [[#bib.bib17|2018]]) have become a default method of choice for training feed-forward and recurrent neural networks (Xu et al., [[#bib.bib24|2015]]; Radford et al., [[#bib.bib15|2015]]). Nevertheless, state-of-the-art results for popular image classification datasets, such as CIFAR-10 and CIFAR-100 Krizhevsky ([[#bib.bib11|2009]]), are still obtained by applying SGD with momentum (Gastaldi, [[#bib.bib6|2017]]; Cubuk et al., [[#bib.bib3|2018]]). Furthermore, Wilson et al. ([[#bib.bib23|2017]]) suggested that adaptive gradient methods do not generalize as well as SGD with momentum when tested on a diverse set of deep learning tasks, such as image classification, character-level language modeling and constituency parsing. Different hypotheses about the origins of this worse generalization have been investigated, such as the presence of sharp local minima (Keskar et al., [[#bib.bib9|2016]]; Dinh et al., [[#bib.bib4|2017]]) and inherent problems of adaptive gradient methods (Wilson et al., [[#bib.bib23|2017]]). In this paper, we investigate whether it is better to use L<sub>2</sub> regularization or weight decay regularization to train deep neural networks with SGD and Adam. We show that a major factor of the poor generalization of the most popular adaptive gradient method, Adam, is due to the fact that L<sub>2</sub> regularization is not nearly as effective for it as for SGD. Specifically, our analysis of Adam leads to the following observations:

<!--T:11-->
; 
: L<sub>2</sub> regularization and weight decay are not identical. The two techniques can be made equivalent for SGD by a reparameterization of the weight decay factor based on the learning rate; however, as is often overlooked, this is not the case for Adam. In particular, when combined with adaptive gradients, L<sub>2</sub> regularization leads to weights with large historic parameter and/or gradient amplitudes being regularized less than they would be when using weight decay.
; 
: L<sub>2</sub> regularization is not effective in Adam. One possible explanation why Adam and other adaptive gradient methods might be outperformed by SGD with momentum is that common deep learning libraries only implement L<sub>2</sub> regularization, not the original weight decay. Therefore, on tasks/datasets where the use of L<sub>2</sub> regularization is beneficial for SGD (e.g., on many popular image classification datasets), Adam leads to worse results than SGD with momentum (for which L<sub>2</sub> regularization behaves as expected).
; 
: Weight decay is equally effective in both SGD and Adam. For SGD, it is equivalent to L<sub>2</sub> regularization, while for Adam it is not.
; 
: Optimal weight decay depends on the total number of batch passes/weight updates. Our empirical analysis of SGD and Adam suggests that the larger the runtime/number of batch passes to be performed, the smaller the optimal weight decay.
; 
: Adam can substantially benefit from a scheduled learning rate multiplier. The fact that Adam is an adaptive gradient algorithm and as such adapts the learning rate for each parameter does ''not'' rule out the possibility to substantially improve its performance by using a global learning rate multiplier, scheduled, e.g., by cosine annealing.

<!--T:12-->
The main contribution of this paper is to ''improve regularization in Adam by decoupling the weight decay from the gradient-based update''. In a comprehensive analysis, we show that Adam generalizes substantially better with decoupled weight decay than with L<sub>2</sub> regularization, achieving 15% relative improvement in test error (see Figures [[#S4.F2|2]] and [[#S4.F3|3]]); this holds true for various image recognition datasets (CIFAR-10 and ImageNet32x32), training budgets (ranging from 100 to 1800 epochs), and learning rate schedules (fixed, drop-step, and cosine annealing; see Figure [[#S3.F1|1]]). We also demonstrate that our decoupled weight decay renders the optimal settings of the learning rate and the weight decay factor much more independent, thereby easing hyperparameter optimization (see Figure [[#S4.F2|2]]).

<!--T:13-->
The main motivation of this paper is to improve Adam to make it competitive w.r.t. SGD with momentum even for those problems where it did not use to be competitive. We hope that as a result, practitioners do not need to switch between Adam and SGD anymore, which in turn should reduce the common issue of selecting dataset/task-specific training algorithms and their hyperparameters.

<!--T:14-->
<div id="S2" class="section">

<!--T:15-->
<span id="decoupling-the-weight-decay-from-the-gradient-based-update"></span>

<!--T:16-->
== 2 Decoupling the Weight Decay from the Gradient-based Update ==

<!--T:17-->
In the weight decay described by Hanson &amp; Pratt ([[#bib.bib7|1988]]), the weights <math display="inline">\mathbf{θ}</math> decay exponentially as

<!--T:18-->
{| id="A4.EGx1" class="wikitable"
|-
|
| <math display="inline">{{\mathbf{θ}}_{t + 1} = {{{({1 - \lambda})}\hspace{0pt}{\mathbf{θ}}_{t}} - {\alpha\hspace{0pt}{\nabla f_{t}}\hspace{0pt}{({\mathbf{θ}}_{t})}}}},</math>
|
| (1)
|}

<!--T:19-->
where <math display="inline">\lambda</math> defines the rate of the weight decay per step and <math display="inline">{\nabla f_{t}}\hspace{0pt}{({\mathbf{θ}}_{t})}</math> is the <math display="inline">t</math>-th batch gradient to be multiplied by a learning rate <math display="inline">\alpha</math>. For standard SGD, it is equivalent to standard L<sub>2</sub> regularization:

<!--T:20-->
<span id="proposition-1weight-decay-l2-reg-for-standard-sgd."></span>

<!--T:21-->
== Proposition 1(Weight decay = L2 reg for standard SGD). ==

<!--T:22-->
Standard SGD with base learning rate <math display="inline">\alpha</math> executes the same steps on batch loss functions <math display="inline">f_{t}\hspace{0pt}{({\mathbf{θ}})}</math> with weight decay <math display="inline">\lambda</math> (defined in Equation [[#S2.E1|1]]) as it executes without weight decay on <math display="inline">{f_{t}^{\text{reg}}\hspace{0pt}{({\mathbf{θ}})}} = {{f_{t}\hspace{0pt}{({\mathbf{θ}})}} + {\frac{\lambda^{\prime}}{2}\hspace{0pt}\left. \parallel{\mathbf{θ}}\parallel \right._{2}^{2}}}</math>, with <math display="inline">\lambda^{\prime} = \frac{\lambda}{\alpha}</math>.

<!--T:23-->
The proofs of this well-known fact, as well as our other propositions, are given in Appendix [[#A1|A]].

<!--T:24-->
Due to this equivalence, L<sub>2</sub> regularization is very frequently referred to as weight decay, including in popular deep learning libraries. However, as we will demonstrate later in this section, this equivalence does ''not'' hold for adaptive gradient methods. One fact that is often overlooked already for the simple case of SGD is that in order for the equivalence to hold, the L<sub>2</sub> regularizer <math display="inline">\lambda^{\prime}</math> has to be set to <math display="inline">\frac{\lambda}{\alpha}</math>, i.e., if there is an overall best weight decay value <math display="inline">\lambda</math>, the best value of <math display="inline">\lambda^{\prime}</math> is tightly coupled with the learning rate <math display="inline">\alpha</math>. In order to decouple the effects of these two hyperparameters, we advocate to decouple the weight decay step as proposed by Hanson &amp; Pratt ([[#bib.bib7|1988]]) (Equation [[#S2.E1|1]]).

<!--T:25-->
<div id="alg1" class="figure">

<!--T:26-->
1:  given initial learning rate <math display="inline">\alpha \in {IR}</math>, momentum factor <math display="inline">\beta_{1} \in {IR}</math>, weight decay/L<sub>2</sub> regularization factor <math display="inline">\lambda \in {IR}</math> 2:  initialize time step <math display="inline">t\leftarrow 0</math>, parameter vector <math display="inline">{\mathbf{θ}}_{t = 0} \in {IR}^{n}</math>, first moment vector <math display="inline">\text{m}_{t = 0}\leftarrow\text{0}</math>, schedule multiplier <math display="inline">\eta_{t = 0} \in {IR}</math> 3:  repeat 4:     <math display="inline">t\leftarrow{t + 1}</math> 5:     <math display="inline">{{\nabla f_{t}}\hspace{0pt}{({\mathbf{θ}}_{t - 1})}}\leftarrow{\text{SelectBatch}\hspace{0pt}{({\mathbf{θ}}_{t - 1})}}</math> <math display="inline">\rhd</math>  select batch and return the corresponding gradient 6:     <math display="inline">\text{g}_{t}\leftarrow{{\nabla f_{t}}\hspace{0pt}{({\mathbf{θ}}_{t - 1})}}</math> <math display="inline">+ {\lambda\hspace{0pt}{\mathbf{θ}}_{t - 1}}</math> 7:     <math display="inline">\eta_{t}\leftarrow{\text{SetScheduleMultiplier}\hspace{0pt}{(t)}}</math> <math display="inline">\rhd</math>  can be fixed, decay, be used for warm restarts 8:     <math display="inline">\text{m}_{t}\leftarrow{{\beta_{1}\hspace{0pt}\text{m}_{t - 1}} + {\eta_{t}\hspace{0pt}\alpha\hspace{0pt}\text{g}_{t}}}</math> 9:     <math display="inline">{\mathbf{θ}}_{t}\leftarrow{{\mathbf{θ}}_{t - 1} - \text{m}_{t}}</math> <math display="inline">- {\eta_{t}\hspace{0pt}\lambda\hspace{0pt}{\mathbf{θ}}_{t - 1}}</math> 10:  until  stopping criterion is met 11:  return  optimized parameters <math display="inline">{\mathbf{θ}}_{t}</math>

<!--T:27-->
<div id="alg2" class="figure">

<!--T:28-->
1:  given <math display="inline">{\alpha = 0.001},{{\beta_{1} = 0.9},{{\beta_{2} = 0.999},{{\epsilon = 10^{- 8}},{\lambda \in {IR}}}}}</math> 2:  initialize time step <math display="inline">t\leftarrow 0</math>, parameter vector <math display="inline">{\mathbf{θ}}_{t = 0} \in {IR}^{n}</math>, first moment vector <math display="inline">\text{m}_{t = 0}\leftarrow\text{0}</math>, second moment vector <math display="inline">\text{v}_{t = 0}\leftarrow\text{0}</math>, schedule multiplier <math display="inline">\eta_{t = 0} \in {IR}</math> 3:  repeat 4:     <math display="inline">t\leftarrow{t + 1}</math> 5:     <math display="inline">{{\nabla f_{t}}\hspace{0pt}{({\mathbf{θ}}_{t - 1})}}\leftarrow{\text{SelectBatch}\hspace{0pt}{({\mathbf{θ}}_{t - 1})}}</math> <math display="inline">\rhd</math>  select batch and return the corresponding gradient 6:     <math display="inline">\text{g}_{t}\leftarrow{{\nabla f_{t}}\hspace{0pt}{({\mathbf{θ}}_{t - 1})}}</math> <math display="inline">+ {\lambda\hspace{0pt}{\mathbf{θ}}_{t - 1}}</math> 7:     <math display="inline">\text{m}_{t}\leftarrow{{\beta_{1}\hspace{0pt}\text{m}_{t - 1}} + {{({1 - \beta_{1}})}\hspace{0pt}\text{g}_{t}}}</math> <math display="inline">\rhd</math>  here and below all operations are element-wise 8:     <math display="inline">\text{v}_{t}\leftarrow{{\beta_{2}\hspace{0pt}\text{v}_{t - 1}} + {{({1 - \beta_{2}})}\hspace{0pt}\text{g}_{t}^{2}}}</math> 9:     <math display="inline">{\hat{\text{m}}}_{t}\leftarrow{\text{m}_{t}/{({1 - \beta_{1}^{t}})}}</math> <math display="inline">\rhd</math>  <math display="inline">\beta_{1}</math> is taken to the power of <math display="inline">t</math> 10:     <math display="inline">{\hat{\text{v}}}_{t}\leftarrow{\text{v}_{t}/{({1 - \beta_{2}^{t}})}}</math> <math display="inline">\rhd</math>  <math display="inline">\beta_{2}</math> is taken to the power of <math display="inline">t</math> 11:     <math display="inline">\eta_{t}\leftarrow{\text{SetScheduleMultiplier}\hspace{0pt}{(t)}}</math> <math display="inline">\rhd</math>  can be fixed, decay, or also be used for warm restarts 12:     <math display="inline">{\mathbf{θ}}_{t}\leftarrow{{\mathbf{θ}}_{t - 1} - {\eta_{t}\hspace{0pt}\left( {{{\alpha\hspace{0pt}{\hat{\text{m}}}_{t}}/{({\sqrt{{\hat{\text{v}}}_{t}} + \epsilon})}}\hspace{0pt}{+ {\lambda\hspace{0pt}{\mathbf{θ}}_{t - 1}}}} \right)}}</math> 13:  until  stopping criterion is met 14:  return  optimized parameters <math display="inline">{\mathbf{θ}}_{t}</math>

<!--T:29-->
Looking first at the case of SGD, we propose to decay the weights simultaneously with the update of <math display="inline">{\mathbf{θ}}_{t}</math> based on gradient information in Line 9 of Algorithm 1. This yields our proposed variant of SGD with momentum using decoupled weight decay (SGDW). This simple modification explicitly decouples <math display="inline">\lambda</math> and <math display="inline">\alpha</math> (although some problem-dependent implicit coupling may of course remain as for any two hyperparameters). In order to account for a possible scheduling of both <math display="inline">\alpha</math> and <math display="inline">\lambda</math>, we introduce a scaling factor <math display="inline">\eta_{t}</math> delivered by a user-defined procedure <math display="inline">S\hspace{0pt}e\hspace{0pt}t\hspace{0pt}S\hspace{0pt}c\hspace{0pt}h\hspace{0pt}e\hspace{0pt}d\hspace{0pt}u\hspace{0pt}l\hspace{0pt}e\hspace{0pt}M\hspace{0pt}u\hspace{0pt}l\hspace{0pt}t\hspace{0pt}i\hspace{0pt}p\hspace{0pt}l\hspace{0pt}i\hspace{0pt}e\hspace{0pt}r\hspace{0pt}{(t)}</math>.

<!--T:30-->
Now, let’s turn to adaptive gradient algorithms like the popular optimizer Adam Kingma &amp; Ba ([[#bib.bib10|2014]]), which scale gradients by their historic magnitudes. Intuitively, when Adam is run on a loss function <math display="inline">f</math> plus L<sub>2</sub> regularization, weights that tend to have large gradients in <math display="inline">f</math> do not get regularized as much as they would with decoupled weight decay, since the gradient of the regularizer gets scaled along with the gradient of <math display="inline">f</math>. This leads to an inequivalence of L<sub>2</sub> and decoupled weight decay regularization for adaptive gradient algorithms:

<!--T:31-->
<span id="proposition-2weight-decay-neq-l2-reg-for-adaptive-gradients."></span>

<!--T:32-->
== Proposition 2(Weight decay \neq L2 reg for adaptive gradients). ==

<!--T:33-->
Let <math display="inline">O</math> denote an optimizer that has iterates <math display="inline">{\mathbf{θ}}_{t + 1}\leftarrow{{\mathbf{θ}}_{t} - {\alpha\hspace{0pt}\mathbf{M}_{t}\hspace{0pt}{\nabla f_{t}}\hspace{0pt}{({\mathbf{θ}}_{t})}}}</math> when run on batch loss function <math display="inline">f_{t}\hspace{0pt}{({\mathbf{θ}})}</math> ''without'' weight decay, and <math display="inline">{\mathbf{θ}}_{t + 1}\leftarrow{{{({1 - \lambda})}\hspace{0pt}{\mathbf{θ}}_{t}} - {\alpha\hspace{0pt}\mathbf{M}_{t}\hspace{0pt}{\nabla f_{t}}\hspace{0pt}{({\mathbf{θ}}_{t})}}}</math> when run on <math display="inline">f_{t}\hspace{0pt}{({\mathbf{θ}})}</math> ''with'' weight decay, respectively, with <math display="inline">\mathbf{M}_{t} \neq {k\hspace{0pt}\mathbf{I}}</math> (where <math display="inline">k \in {\mathbb{R}}</math>). Then, for <math display="inline">O</math> there exists no L<sub>2</sub> coefficient <math display="inline">\lambda^{\prime}</math> such that running <math display="inline">O</math> on batch loss <math display="inline">{f_{t}^{\text{reg}}\hspace{0pt}{({\mathbf{θ}})}} = {{f_{t}\hspace{0pt}{({\mathbf{θ}})}} + {\frac{\lambda^{\prime}}{2}\hspace{0pt}\left. \parallel{\mathbf{θ}}\parallel \right._{2}^{2}}}</math> without weight decay is equivalent to running <math display="inline">O</math> on <math display="inline">f_{t}\hspace{0pt}{({\mathbf{θ}})}</math> with decay <math display="inline">\lambda \in {\mathbb{R}}^{+}</math>.

<!--T:34-->
We decouple weight decay and loss-based gradient updates in Adam as shown in line 12 of Algorithm 2; this gives rise to our variant of Adam with decoupled weight decay (AdamW).

<!--T:35-->
Having shown that L<sub>2</sub> regularization and weight decay regularization differ for adaptive gradient algorithms raises the question of how they differ and how to interpret their effects. Their equivalence for standard SGD remains very helpful for intuition: both mechanisms push weights closer to zero, at the same rate. However, for adaptive gradient algorithms they differ: with L<sub>2</sub> regularization, the sums of the gradient of the loss function and the gradient of the regularizer (i.e., the L<sub>2</sub> norm of the weights) are adapted, whereas with decoupled weight decay, only the gradients of the loss function are adapted (with the weight decay step separated from the adaptive gradient mechanism). With L<sub>2</sub> regularization both types of gradients are normalized by their typical (summed) magnitudes, and therefore weights <math display="inline">x</math> with large typical gradient magnitude <math display="inline">s</math> are regularized by a smaller relative amount than other weights. In contrast, decoupled weight decay regularizes all weights with the same rate <math display="inline">\lambda</math>, effectively regularizing weights <math display="inline">x</math> with large <math display="inline">s</math> more than standard L<sub>2</sub> regularization does. We demonstrate this formally for a simple special case of adaptive gradient algorithm with a fixed preconditioner:

<!--T:36-->
<span id="proposition-3weight-decay-scale-adjusted-l_2-reg-for-adaptive-gradient-algorithm-with-fixed-preconditioner."></span>

<!--T:37-->
== Proposition 3(Weight decay = scale-adjusted L_{2} reg for adaptive gradient algorithm with fixed preconditioner). ==

<!--T:38-->
Let <math display="inline">O</math> denote an algorithm with the same characteristics as in Proposition 2, and using a fixed preconditioner matrix <math display="inline">\text{M}_{t} = {\text{diag}\hspace{0pt}{(\text{s})}^{- 1}}</math> (with <math display="inline">s_{i} > 0</math> for all <math display="inline">i</math>). Then, <math display="inline">O</math> with base learning rate <math display="inline">\alpha</math> executes the same steps on batch loss functions <math display="inline">f_{t}\hspace{0pt}{({\mathbf{θ}})}</math> with weight decay <math display="inline">\lambda</math> as it executes without weight decay on the scale-adjusted regularized batch loss

<!--T:39-->
{| id="S2.E2" class="wikitable"
|-
|
| <math display="block">{{f_{t}^{\text{sreg}}\hspace{0pt}{({\mathbf{θ}})}} = {{f_{t}\hspace{0pt}{({\mathbf{θ}})}} + {\frac{\lambda^{\prime}}{2\hspace{0pt}\alpha}\hspace{0pt}\left. \parallel{{\mathbf{θ}} \odot \sqrt{\text{s}}}\parallel \right._{2}^{2}}}},</math>
|
| (2)
|}

<!--T:40-->
where <math display="inline">\odot</math> and <math display="inline">\sqrt{\cdot}</math> denote element-wise multiplication and square root, respectively, and <math display="inline">\lambda^{\prime} = \frac{\lambda}{\alpha}</math>.

<!--T:41-->
We note that this proposition does ''not'' directly apply to practical adaptive gradient algorithms, since these change the preconditioner matrix at every step. Nevertheless, it can still provide intuition about the equivalent loss function being optimized in each step: parameters <math display="inline">\theta_{i}</math> with a large inverse preconditioner <math display="inline">s_{i}</math> (which in practice would be caused by historically large gradients in dimension <math display="inline">i</math>) are regularized relatively more than they would be with L<sub>2</sub> regularization; specifically, the regularization is proportional to <math display="inline">\sqrt{s_{i}}</math>.

<!--T:42-->
<div id="S3" class="section">

<!--T:43-->
<span id="justification-of-decoupled-weight-decay-via-a-view-of-adaptive-gradient-methods-as-bayesian-filtering"></span>

<!--T:44-->
== 3 Justification of Decoupled Weight Decay via a View of Adaptive Gradient Methods as Bayesian Filtering ==

<!--T:45-->
We now discuss a justification of decoupled weight decay in the framework of Bayesian filtering for a unified theory of adaptive gradient algorithms due to Aitchison ([[#bib.bib1|2018]]). After we posted a preliminary version of our current paper on arXiv, Aitchison noted that his theory “gives us a theoretical framework in which we can understand the superiority of this weight decay over <math display="inline">L_{2}</math> regularization, because it is weight decay, rather than <math display="inline">L_{2}</math> regularization that emerges through the straightforward application of Bayesian filtering.”(Aitchison, [[#bib.bib1|2018]]). While full credit for this theory goes to Aitchison, we summarize it here to shed some light on why weight decay may be favored over <math display="inline">L_{2}</math> regularization.

<!--T:46-->
Aitchison ([[#bib.bib1|2018]]) views stochastic optimization of <math display="inline">n</math> parameters <math display="inline">\theta_{1},\ldots,\theta_{n}</math> as a Bayesian filtering problem with the goal of inferring a distribution over the optimal values of each of the parameters <math display="inline">\theta_{i}</math> given the current values of the other parameters <math display="inline">{\mathbf{θ}}_{- i}\hspace{0pt}{(t)}</math> at time step <math display="inline">t</math>. When the other parameters do not change this is an optimization problem, but when they do change it becomes one of “tracking” the optimizer using Bayesian filtering as follows. One is given a probability distribution <math display="inline">P\hspace{0pt}{({{\mathbf{θ}}_{t} \mid {\mathbf{y}}_{\mathbf{1}:{\mathbf{t}}}})}</math> of the optimizer at time step <math display="inline">t</math> that takes into account the data <math display="inline">{\mathbf{y}}_{\mathbf{1}:{\mathbf{t}}}</math> from the first <math display="inline">t</math> mini batches, a state transition prior <math display="inline">P\hspace{0pt}{({{\mathbf{θ}}_{t + 1} \mid {\mathbf{θ}}_{t}})}</math> reflecting a (small) data-independent change in this distribution from one step to the next, and a likelihood <math display="inline">P\hspace{0pt}{({{\mathbf{y}}_{t + 1} \mid {\mathbf{θ}}_{t + 1}})}</math> derived from the mini batch at step <math display="inline">t + 1</math>. The posterior distribution <math display="inline">P\hspace{0pt}{({{\mathbf{θ}}_{t + 1} \mid {\mathbf{y}}_{\mathbf{1}:{{\mathbf{t}} + \mathbf{1}}}})}</math> of the optimizer at time step <math display="inline">t + 1</math> can then be computed (as usual in Bayesian filtering) by marginalizing over <math display="inline">{\mathbf{θ}}_{t}</math> to obtain the one-step ahead predictions <math display="inline">P\hspace{0pt}{({{\mathbf{θ}}_{t + 1} \mid {\mathbf{y}}_{\mathbf{1}:{\mathbf{t}}}})}</math> and then applying Bayes’ rule to incorporate the likelihood <math display="inline">P\hspace{0pt}{({{\mathbf{y}}_{t + 1} \mid {\mathbf{θ}}_{t + 1}})}</math>. Aitchison ([[#bib.bib1|2018]]) assumes a Gaussian state transition distribution <math display="inline">P\hspace{0pt}{({{\mathbf{θ}}_{t + 1} \mid {\mathbf{θ}}_{t}})}</math> and an approximate conjugate likelihood <math display="inline">P\hspace{0pt}{({{\mathbf{y}}_{t + 1} \mid {\mathbf{θ}}_{t + 1}})}</math>, leading to the following closed-form update of the filtering distribution’s mean:

<!--T:47-->
{| id="S3.E3" class="wikitable"
|-
|
| <math display="block">{{\mathbf{μ}}_{p\hspace{0pt}o\hspace{0pt}s\hspace{0pt}t} = {{\mathbf{μ}}_{p\hspace{0pt}r\hspace{0pt}i\hspace{0pt}o\hspace{0pt}r} + {\mathbf{\Sigma}_{p\hspace{0pt}o\hspace{0pt}s\hspace{0pt}t} \times {\mathbf{g}}}}},</math>
|
| (3)
|}

<!--T:48-->
where <math display="inline">\mathbf{g}</math> is the gradient of the log likelihood of the mini batch at time <math display="inline">t</math>. This result implies a preconditioner of the gradients that is given by the posterior uncertainty <math display="inline">\mathbf{\Sigma}_{p\hspace{0pt}o\hspace{0pt}s\hspace{0pt}t}</math> of the filtering distribution: updates are larger for parameters we are more uncertain about and smaller for parameters we are more certain about. Aitchison ([[#bib.bib1|2018]]) goes on to show that popular adaptive gradient methods, such as Adam and RMSprop, as well as Kronecker-factorized methods are special cases of this framework.

<!--T:49-->
Decoupled weight decay very naturally fits into this unified framework as part of the state-transition distribution: Aitchison ([[#bib.bib1|2018]]) assumes a slow change of the optimizer according to the following Gaussian:

<!--T:50-->
{| id="S3.E4" class="wikitable"
|-
|
| <math display="block">{{P\hspace{0pt}{({{\mathbf{θ}}_{t + 1} \mid {\mathbf{θ}}_{t}})}} = {\mathcal{N}\hspace{0pt}{({{({{\mathbf{I}} - {\mathbf{A}}})}\hspace{0pt}{\mathbf{θ}}_{t}},{\mathbf{Q}})}}},</math>
|
| (4)
|}

<!--T:51-->
where <math display="inline">\mathbf{Q}</math> is the covariance of Gaussian perturbations of the weights, and <math display="inline">\mathbf{A}</math> is a regularizer to avoid values growing unboundedly over time. When instantiated as <math display="inline">{\mathbf{A}} = {\lambda \times {\mathbf{I}}}</math>, this regularizer <math display="inline">\mathbf{A}</math> plays exactly the role of decoupled weight decay as described in Equation [[#S2.E1|1]], since this leads to multiplying the current mean estimate <math display="inline">{\mathbf{θ}}_{t}</math> by <math display="inline">({1 - \lambda})</math> at each step. Notably, this regularization is also directly applied to the prior and does not depend on the uncertainty in each of the parameters (which would be required for <math display="inline">L_{2}</math> regularization).

<!--T:52-->
<div id="S3.F1" class="figure">

<!--T:53-->
[[File:Arxiv_1711_05101_x1.png|114x117px|Refer to caption]] [[File:Arxiv_1711_05101_x2.png|114x117px|Refer to caption]] [[File:Arxiv_1711_05101_x3.png|114x117px|Refer to caption]] [[File:Arxiv_1711_05101_x4.png|114x117px|Refer to caption]] [[File:Arxiv_1711_05101_x5.png|114x117px|Refer to caption]] [[File:Arxiv_1711_05101_x6.png|114x117px|Refer to caption]]

<!--T:54-->
<div id="S4" class="section">

<!--T:55-->
<span id="experimental-validation"></span>

<!--T:56-->
== 4 Experimental Validation ==

<!--T:57-->
We now evaluate the performance of decoupled weight decay under various training budgets and learning rate schedules. Our experimental setup follows that of Gastaldi ([[#bib.bib6|2017]]), who proposed, in addition to L<sub>2</sub> regularization, to apply the new Shake-Shake regularization to a 3-branch residual DNN that allowed to achieve new state-of-the-art results of 2.86% on the CIFAR-10 dataset (Krizhevsky, [[#bib.bib11|2009]]). We used the same model/source code based on fb.resnet.torch <sup>11</sup>1https://github.com/xgastaldi/shake-shake. We always used a batch size of 128 and applied the regular data augmentation procedure for the CIFAR datasets. The base networks are a 26 2x64d ResNet (i.e. the network has a depth of 26, 2 residual branches and the first residual block has a width of 64) and a 26 2x96d ResNet with 11.6M and 25.6M parameters, respectively. For a detailed description of the network and the Shake-Shake method, we refer the interested reader to Gastaldi ([[#bib.bib6|2017]]). We also perform experiments on the ImageNet32x32 dataset (Chrabaszcz et al., [[#bib.bib2|2017]]), a downsampled version of the original ImageNet dataset with 1.2 million 32<math display="inline">\times</math>32 pixels images.

<!--T:58-->
<div id="S4.SS1" class="section">

<!--T:59-->
<span id="evaluating-decoupled-weight-decay-with-different-learning-rate-schedules"></span>

<!--T:60-->
=== 4.1 Evaluating Decoupled Weight Decay With Different Learning Rate Schedules ===

<!--T:61-->
In our first experiment, we compare Adam with <math display="inline">L_{2}</math> regularization to Adam with decoupled weight decay (AdamW), using three different learning rate schedules: a fixed learning rate, a drop-step schedule, and a cosine annealing schedule (Loshchilov &amp; Hutter, [[#bib.bib13|2016]]). Since Adam already adapts its parameterwise learning rates it is not as common to use a learning rate multiplier schedule with it as it is with SGD, but as our results show such schedules can substantially improve Adam’s performance, and we advocate not to overlook their use for adaptive gradient algorithms.

<!--T:62-->
For each learning rate schedule and weight decay variant, we trained a 2x64d ResNet for 100 epochs, using different settings of the initial learning rate <math display="inline">\alpha</math> and the weight decay factor <math display="inline">\lambda</math>. Figure [[#S3.F1|1]] shows that decoupled weight decay outperforms <math display="inline">L_{2}</math> regularization for all learning rate schedules, with larger differences for better learning rate schedules. We also note that decoupled weight decay leads to a more separable hyperparameter search space, especially when a learning rate schedule, such as step-drop and cosine annealing is applied. The figure also shows that cosine annealing clearly outperforms the other learning rate schedules; we thus used cosine annealing for the remainder of the experiments.

<!--T:63-->
<div id="S4.F2" class="figure">

<!--T:64-->
[[File:Arxiv_1711_05101_x7.png|147x122px|Refer to caption]]
          [[File:Arxiv_1711_05101_x8.png|147x122px|Refer to caption]]

<!--T:65-->
[[File:Arxiv_1711_05101_x9.png|147x122px|Refer to caption]]
          [[File:Arxiv_1711_05101_x10.png|147x122px|Refer to caption]]

<!--T:66-->
<div id="S4.F3" class="figure">

<!--T:67-->
[[File:Arxiv_1711_05101_x11.png|148x120px|Refer to caption]]
              [[File:Arxiv_1711_05101_x12.png|148x121px|Refer to caption]]

<!--T:68-->
[[File:Arxiv_1711_05101_x13.png|148x120px|Refer to caption]]
              [[File:Arxiv_1711_05101_x14.png|148x121px|Refer to caption]]

<!--T:69-->
<div id="S4.SS2" class="section">

<!--T:70-->
<span id="decoupling-the-weight-decay-and-initial-learning-rate-parameters"></span>

<!--T:71-->
=== 4.2 Decoupling the Weight Decay and Initial Learning Rate Parameters ===

<!--T:72-->
In order to verify our hypothesis about the coupling of <math display="inline">\alpha</math> and <math display="inline">\lambda</math>, in Figure [[#S4.F2|2]] we compare the performance of L<sub>2</sub> regularization vs. decoupled weight decay in SGD (SGD vs. SGDW, top row) and in Adam (Adam vs. AdamW, bottom row). In SGD (Figure [[#S4.F2|2]], top left), L<sub>2</sub> regularization is not decoupled from the learning rate (the common way as described in Algorithm 1), and the figure clearly shows that the basin of best hyperparameter settings (depicted by color and top-10 hyperparameter settings by black circles) is not aligned with the x-axis or y-axis but lies on the diagonal. This suggests that the two hyperparameters are interdependent and need to be changed simultaneously, while only changing one of them might substantially worsen results. Consider, e.g., the setting at the top left black circle (<math display="inline">\alpha = {1/2}</math>, <math display="inline">\lambda = {{1/8} \ast 0.001}</math>); only changing either <math display="inline">\alpha</math> or <math display="inline">\lambda</math> by itself would worsen results, while changing both of them could still yield clear improvements. We note that this coupling of initial learning rate and L<sub>2</sub> regularization factor might have contributed to SGD’s reputation of being very sensitive to its hyperparameter settings.

<!--T:73-->
In contrast, the results for SGD with decoupled weight decay (SGDW) in Figure [[#S4.F2|2]] (top right) show that weight decay and initial learning rate are decoupled. The proposed approach renders the two hyperparameters more separable: even if the learning rate is not well tuned yet (e.g., consider the value of 1/1024 in Figure [[#S4.F2|2]], top right), leaving it fixed and only optimizing the weight decay factor would yield a good value (of 1/4*0.001). This is not the case for SGD with L<sub>2</sub> regularization (see Figure [[#S4.F2|2]], top left).

<!--T:74-->
The results for Adam with L<sub>2</sub> regularization are given in Figure [[#S4.F2|2]] (bottom left). Adam’s best hyperparameter settings performed clearly worse than SGD’s best ones (compare Figure [[#S4.F2|2]], top left). While both methods used L<sub>2</sub> regularization, Adam did not benefit from it at all: its best results obtained for non-zero L<sub>2</sub> regularization factors were comparable to the best ones obtained without the L<sub>2</sub> regularization, i.e., when <math display="inline">\lambda = 0</math>. Similarly to the original SGD, the shape of the hyperparameter landscape suggests that the two hyperparameters are coupled.

<!--T:75-->
In contrast, the results for our new variant of Adam with decoupled weight decay (AdamW) in Figure [[#S4.F2|2]] (bottom right) show that AdamW largely decouples weight decay and learning rate. The results for the best hyperparameter settings were substantially better than the best ones of Adam with L<sub>2</sub> regularization and rivaled those of SGD and SGDW.

<!--T:76-->
In summary, the results in Figure [[#S4.F2|2]] support our hypothesis that the weight decay and learning rate hyperparameters can be decoupled, and that this in turn simplifies the problem of hyperparameter tuning in SGD and improves Adam’s performance to be competitive w.r.t. SGD with momentum.

<!--T:77-->
<div id="S4.SS3" class="section">

<!--T:78-->
<span id="better-generalization-of-adamw"></span>

<!--T:79-->
=== 4.3 Better Generalization of AdamW ===

<!--T:80-->
While the previous experiment suggested that the basin of optimal hyperparameters of AdamW is broader and deeper than the one of Adam, we next investigated the results for much longer runs of 1800 epochs to compare the generalization capabilities of AdamW and Adam.

<!--T:81-->
We fixed the initial learning rate to 0.001 which represents both the default learning rate for Adam and the one which showed reasonably good results in our experiments. Figure [[#S4.F3|3]] shows the results for 12 settings of the L<sub>2</sub> regularization of Adam and 7 settings of the normalized weight decay of AdamW (the normalized weight decay represents a rescaling formally defined in Appendix [[#A2.SS1|B.1]]; it amounts to a multiplicative factor which depends on the number of batch passes). Interestingly, while the dynamics of the learning curves of Adam and AdamW often coincided for the first half of the training run, AdamW often led to lower training loss and test errors (see Figure [[#S4.F3|3]] top left and top right, respectively). Importantly, the use of L<sub>2</sub> weight decay in Adam did not yield as good results as decoupled weight decay in AdamW (see also Figure [[#S4.F3|3]], bottom left). Next, we investigated whether AdamW’s better results were only due to better convergence or due to better generalization. ''The results in Figure [[#S4.F3|3]] (bottom right) for the best settings of Adam and AdamW suggest that AdamW did not only yield better training loss but also yielded better generalization performance for similar training loss values''. The results on ImageNet32x32 (see SuppFigure 4 in the Appendix) yield the same conclusion of substantially improved generalization performance.

<!--T:82-->
<div id="S4.F4" class="figure">

<!--T:83-->
[[File:Arxiv_1711_05101_x15.png|168x140px|Refer to caption]]
              [[File:Arxiv_1711_05101_x16.png|169x138px|Refer to caption]]

<!--T:84-->
<div id="S4.SS4" class="section">

<!--T:85-->
<span id="adamwr-with-warm-restarts-for-better-anytime-performance"></span>

<!--T:86-->
=== 4.4 AdamWR with Warm Restarts for Better Anytime Performance ===

<!--T:87-->
In order to improve the anytime performance of SGDW and AdamW we extended them with the warm restarts we introduced in Loshchilov &amp; Hutter ([[#bib.bib13|2016]]), to obtain SGDWR and AdamWR, respectively (see Section [[#A2.SS2|B.2]] in the Appendix). As Figure [[#S4.F4|4]] shows, AdamWR greatly sped up AdamW on CIFAR-10 and ImageNet32x32, up to a factor of 10 (see the results at the first restart). For the default learning rate of 0.001, ''AdamW achieved 15% relative improvement in test error compared to Adam both on CIFAR-10'' (also see SuppFigure [[#A4.F5|5]]) ''and ImageNet32x32'' (also see SuppFigure [[#A4.F6|6]]).

<!--T:88-->
''AdamWR achieved the same improved results but with a much better anytime performance.'' These improvements closed most of the gap between Adam and SGDWR on CIFAR-10 and yielded comparable performance on ImageNet32x32.

<!--T:89-->
<div id="S4.SS5" class="section">

<!--T:90-->
<span id="use-of-adamw-on-other-datasets-and-architectures"></span>

<!--T:91-->
=== 4.5 Use of AdamW on other datasets and architectures ===

<!--T:92-->
Several other research groups have already successfully applied AdamW in citable works. For example, Wang et al. ([[#bib.bib22|2018]]) used AdamW to train a novel architecture for face detection on the standard WIDER FACE dataset (Yang et al., [[#bib.bib25|2016]]), obtaining almost 10x faster predictions than the previous state of the art algorithms while achieving comparable performance. Völker et al. ([[#bib.bib21|2018]]) employed AdamW with cosine annealing to train convolutional neural networks to classify and characterize error-related brain signals measured from intracranial electroencephalography (EEG) recordings. While their paper does not provide a comparison to Adam, they kindly provided us with a direct comparison of the two on their best-performing problem-specific network architecture Deep4Net and a variant of ResNet. AdamW with the same hyperparameter setting as Adam yielded higher test set accuracy on Deep4Net (73.68% versus 71.37%) and statistically significantly higher test set accuracy on ResNet (72.04% versus 61.34%). Radford et al. ([[#bib.bib16|2018]]) employed AdamW to train Transformer (Vaswani et al., [[#bib.bib20|2017]]) architectures to obtain new state-of-the-art results on a wide range of benchmarks for natural language understanding. Zhang et al. ([[#bib.bib26|2018]]) compared L<sub>2</sub> regularization vs. weight decay for SGD, Adam and the Kronecker-Factored Approximate Curvature (K-FAC) optimizer (Martens &amp; Grosse, [[#bib.bib14|2015]]) on the CIFAR datasets with ResNet and VGG architectures, reporting that decoupled weight decay consistently outperformed L<sub>2</sub> regularization in cases where they differ.

<!--T:93-->
<div id="S5" class="section">

<!--T:94-->
<span id="conclusion-and-future-work"></span>

<!--T:95-->
== 5 Conclusion and Future Work ==

<!--T:96-->
Following suggestions that adaptive gradient methods such as Adam might lead to worse generalization than SGD with momentum (Wilson et al., [[#bib.bib23|2017]]), we identified and exposed the inequivalence of L<sub>2</sub> regularization and weight decay for Adam. We empirically showed that our version of Adam with decoupled weight decay yields substantially better generalization performance than the common implementation of Adam with L<sub>2</sub> regularization. We also proposed to use warm restarts for Adam to improve its anytime performance.

<!--T:97-->
Our results obtained on image classification datasets must be verified on a wider range of tasks, especially ones where the use of regularization is expected to be important. It would be interesting to integrate our findings on weight decay into other methods which attempt to improve Adam, e.g, normalized direction-preserving Adam (Zhang et al., [[#bib.bib27|2017]]). While we focused our experimental analysis on Adam, we believe that similar results also hold for other adaptive gradient methods, such as AdaGrad (Duchi et al., [[#bib.bib5|2011]]) and AMSGrad  (Reddi et al., [[#bib.bib17|2018]]).

<!--T:98-->
<div id="S6" class="section">

<!--T:99-->
<span id="acknowledgments"></span>

<!--T:100-->
== 6 Acknowledgments ==

<!--T:101-->
We thank Patryk Chrabaszcz for help with running experiments with ImageNet32x32; Matthias Feurer and Robin Schirrmeister for providing valuable feedback on this paper in several iterations; and Martin Völker, Robin Schirrmeister, and Tonio Ball for providing us with a comparison of AdamW and Adam on their EEG data. We also thank the following members of the deep learning community for implementing decoupled weight decay in various deep learning libraries:

<!--T:102-->
<ul>
<li><p><span id="S6.I1.i1">•</span></p>
<p>Jingwei Zhang, Lei Tai, Robin Schirrmeister, and Kashif Rasul for their implementations in PyTorch (see https://github.com/pytorch/pytorch/pull/4429)</p></li>
<li><p><span id="S6.I1.i2">•</span></p>
<p>Phil Jund for his implementation in TensorFlow described at<br />
https://www.tensorflow.org/api_docs/python/tf/contrib/opt/DecoupledWeightDecayExtension</p></li>
<li><p><span id="S6.I1.i3">•</span></p>
<p>Sylvain Gugger, Anand Saha, Jeremy Howard and other members of fast.ai for their implementation available at https://github.com/sgugger/Adam-experiments</p></li>
<li><p><span id="S6.I1.i4">•</span></p>
<p>Guillaume Lambard for his implementation in Keras available at https://github.com/GLambard/AdamW_Keras</p></li>
<li><p><span id="S6.I1.i5">•</span></p>
<p>Yagami Lin for his implementation in Caffe available at https://github.com/Yagami123/Caffe-AdamW-AdamWR</p></li></ul>

<!--T:103-->
This work was supported by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme under grant no. 716721, by the German Research Foundation (DFG) under the BrainLinksBrainTools Cluster of Excellence (grant number EXC 1086) and through grant no. INST 37/935-1 FUGG, and by the German state of Baden-Württemberg through bwHPC.

<!--T:104-->
<div id="bib" class="section">

<!--T:105-->
<span id="references"></span>

<!--T:106-->
== References ==

<!--T:107-->
* <span id="bib.bib1">Aitchison (2018) Laurence Aitchison. A unified theory of adaptive stochastic gradient descent as Bayesian filtering. ''arXiv:1507.02030'', 2018.</span>
* <span id="bib.bib2">Chrabaszcz et al. (2017) Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter. A downsampled variant of ImageNet as an alternative to the CIFAR datasets. ''arXiv:1707.08819'', 2017.</span>
* <span id="bib.bib3">Cubuk et al. (2018) Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. ''arXiv preprint arXiv:1805.09501'', 2018.</span>
* <span id="bib.bib4">Dinh et al. (2017) Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. ''arXiv:1703.04933'', 2017.</span>
* <span id="bib.bib5">Duchi et al. (2011) John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. ''The Journal of Machine Learning Research'', 12:2121–2159, 2011.</span>
* <span id="bib.bib6">Gastaldi (2017) Xavier Gastaldi. Shake-Shake regularization. ''arXiv preprint arXiv:1705.07485'', 2017.</span>
* <span id="bib.bib7">Hanson &amp; Pratt (1988) Stephen José Hanson and Lorien Y Pratt. Comparing biases for minimal network construction with back-propagation. In ''Proceedings of the 1st International Conference on Neural Information Processing Systems'', pp.  177–185, 1988.</span>
* <span id="bib.bib8">Huang et al. (2017) Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E Hopcroft, and Kilian Q Weinberger. Snapshot ensembles: Train 1, get m for free. ''arXiv:1704.00109'', 2017.</span>
* <span id="bib.bib9">Keskar et al. (2016) Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. ''arXiv:1609.04836'', 2016.</span>
* <span id="bib.bib10">Kingma &amp; Ba (2014) Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ''arXiv:1412.6980'', 2014.</span>
* <span id="bib.bib11">Krizhevsky (2009) Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.</span>
* <span id="bib.bib12">Li et al. (2017) Hao Li, Zheng Xu, Gavin Taylor, and Tom Goldstein. Visualizing the loss landscape of neural nets. ''arXiv preprint arXiv:1712.09913'', 2017.</span>
* <span id="bib.bib13">Loshchilov &amp; Hutter (2016) Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. ''arXiv:1608.03983'', 2016.</span>
* <span id="bib.bib14">Martens &amp; Grosse (2015) James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In ''International conference on machine learning'', pp. 2408–2417, 2015.</span>
* <span id="bib.bib15">Radford et al. (2015) Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. ''arXiv:1511.06434'', 2015.</span>
* <span id="bib.bib16">Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. ''URL https://s3-us-west-2. amazonaws. com/openai-assets/research-covers/language-unsupervised/language_ understanding_paper. pdf'', 2018.</span>
* <span id="bib.bib17">Reddi et al. (2018) Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. ''International Conference on Learning Representations'', 2018.</span>
* <span id="bib.bib18">Smith (2016) Leslie N Smith. Cyclical learning rates for training neural networks. ''arXiv:1506.01186v3'', 2016.</span>
* <span id="bib.bib19">Tieleman &amp; Hinton (2012) Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. ''COURSERA: Neural networks for machine learning'', 4(2):26–31, 2012.</span>
* <span id="bib.bib20">Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In ''Advances in Neural Information Processing Systems'', pp. 5998–6008, 2017.</span>
* <span id="bib.bib21">Völker et al. (2018) Martin Völker, Jiří Hammer, Robin T Schirrmeister, Joos Behncke, Lukas DJ Fiederer, Andreas Schulze-Bonhage, Petr Marusič, Wolfram Burgard, and Tonio Ball. Intracranial error detection via deep learning. ''arXiv preprint arXiv:1805.01667'', 2018.</span>
* <span id="bib.bib22">Wang et al. (2018) Jianfeng Wang, Ye Yuan, Gang Yu, and Sun Jian. Sface: An efficient network for face detection in large scale variations. ''arXiv preprint arXiv:1804.06559'', 2018.</span>
* <span id="bib.bib23">Wilson et al. (2017) Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. ''arXiv:1705.08292'', 2017.</span>
* <span id="bib.bib24">Xu et al. (2015) Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ''International Conference on Machine Learning'', pp. 2048–2057, 2015.</span>
* <span id="bib.bib25">Yang et al. (2016) Shuo Yang, Ping Luo, Chen-Change Loy, and Xiaoou Tang. Wider face: A face detection benchmark. In ''Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition'', pp.  5525–5533, 2016.</span>
* <span id="bib.bib26">Zhang et al. (2018) Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization. ''arXiv preprint arXiv:1810.12281'', 2018.</span>
* <span id="bib.bib27">Zhang et al. (2017) Zijun Zhang, Lin Ma, Zongpeng Li, and Chuan Wu. Normalized direction-preserving adam. ''arXiv:1709.04546'', 2017.</span>
* <span id="bib.bib28">Zoph et al. (2017) Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transferable architectures for scalable image recognition. In ''arXiv:1707.07012 [cs.CV]'', 2017.</span>

<!--T:108-->
Appendix

<!--T:109-->
<div id="A1" class="section">

<!--T:110-->
<span id="appendix-a-formal-analysis-of-weight-decay-vs-l2-regularization"></span>

<!--T:111-->
== Appendix A Formal Analysis of Weight Decay vs L2 Regularization ==

<!--T:112-->
Proof of Proposition 1<br />
The proof for this well-known fact is straight-forward. SGD without weight decay has the following iterates on <math display="inline">{f_{t}^{\text{reg}}\hspace{0pt}{({\mathbf{θ}})}} = {{f_{t}\hspace{0pt}{({\mathbf{θ}})}} + {\frac{\lambda^{\prime}}{2}\hspace{0pt}\left. \parallel{\mathbf{θ}}\parallel \right._{2}^{2}}}</math>:

<!--T:113-->
{| id="A1.E5" class="wikitable"
|-
|
| <math display="block">{{\mathbf{θ}}_{t + 1}\leftarrow{{\mathbf{θ}}_{t} - {\alpha\hspace{0pt}{\nabla f_{t}^{\text{reg}}}\hspace{0pt}{({\mathbf{θ}}_{t})}}} = {{\mathbf{θ}}_{t} - {\alpha\hspace{0pt}{\nabla f_{t}}\hspace{0pt}{({\mathbf{θ}}_{t})}} - {\alpha\hspace{0pt}\lambda^{\prime}\hspace{0pt}{\mathbf{θ}}_{t}}}}.</math>
|
| (5)
|}

<!--T:114-->
SGD with weight decay has the following iterates on <math display="inline">f_{t}\hspace{0pt}{({\mathbf{θ}})}</math>:

<!--T:115-->
{| id="A1.E6" class="wikitable"
|-
|
| <math display="block">{{\mathbf{θ}}_{t + 1}\leftarrow{{{({1 - \lambda})}\hspace{0pt}{\mathbf{θ}}_{t}} - {\alpha\hspace{0pt}{\nabla f_{t}}\hspace{0pt}{({\mathbf{θ}}_{t})}}}}.</math>
|
| (6)
|}

<!--T:116-->
These iterates are identical since <math display="inline">\lambda^{\prime} = \frac{\lambda}{\alpha}</math>. ∎

<!--T:117-->
Proof of Proposition 2<br />
Similarly to the proof of Proposition 1, the iterates of <math display="inline">O</math> without weight decay on <math display="inline">{f_{t}^{\text{reg}}\hspace{0pt}{({\mathbf{θ}})}} = {{f_{t}\hspace{0pt}{({\mathbf{θ}})}} + {\frac{1}{2}\hspace{0pt}\lambda^{\prime}\hspace{0pt}\left. \parallel{\mathbf{θ}}\parallel \right._{2}^{2}}}</math> and <math display="inline">O</math> with weight decay <math display="inline">\lambda</math> on <math display="inline">f_{t}</math> are, respectively:

<!--T:118-->
{| id="A4.EGx2" class="wikitable"
|-
|
| <math display="inline">{\mathbf{θ}}_{t + 1}</math>
| <math display="inline">\leftarrow</math>
| <math display="inline">{{\mathbf{θ}}_{t} - {\alpha\hspace{0pt}\lambda^{\prime}\hspace{0pt}\mathbf{M}_{t}\hspace{0pt}{\mathbf{θ}}_{t}} - {\alpha\hspace{0pt}\mathbf{M}_{t}\hspace{0pt}{\nabla f_{t}}\hspace{0pt}{({\mathbf{θ}}_{t})}}}.</math>
|
| (7)
|-
|
| <math display="inline">{\mathbf{θ}}_{t + 1}</math>
| <math display="inline">\leftarrow</math>
| <math display="inline">{{{({1 - \lambda})}\hspace{0pt}{\mathbf{θ}}_{t}} - {\alpha\hspace{0pt}\mathbf{M}_{t}\hspace{0pt}{\nabla f_{t}}\hspace{0pt}{({\mathbf{θ}}_{t})}}}.</math>
|
| (8)
|}

<!--T:119-->
The equality of these iterates for all <math display="inline">{\mathbf{θ}}_{t}</math> would imply <math display="inline">{\lambda\hspace{0pt}{\mathbf{θ}}_{t}} = {\alpha\hspace{0pt}\lambda^{\prime}\hspace{0pt}\mathbf{M}_{t}\hspace{0pt}{\mathbf{θ}}_{t}}</math>. This can only hold for all <math display="inline">{\mathbf{θ}}_{t}</math> if <math display="inline">\mathbf{M}_{t} = {k\hspace{0pt}\mathbf{I}}</math>, with <math display="inline">k \in {\mathbb{R}}</math>, which is not the case for <math display="inline">O</math>. Therefore, no L<sub>2</sub> regularizer <math display="inline">\lambda^{\prime}\hspace{0pt}\left. \parallel{\mathbf{θ}}\parallel \right._{2}^{2}</math> exists that makes the iterates equivalent. ∎

<!--T:120-->
Proof of Proposition 3<br />
<math display="inline">O</math> without weight decay has the following iterates on <math display="inline">{f_{t}^{\text{sreg}}\hspace{0pt}{({\mathbf{θ}})}} = {{f_{t}\hspace{0pt}{({\mathbf{θ}})}} + {\frac{\lambda^{\prime}}{2}\hspace{0pt}\left. \parallel{{\mathbf{θ}} \odot \sqrt{\text{s}}}\parallel \right._{2}^{2}}}</math>:

<!--T:121-->
{| id="A4.EGx3" class="wikitable"
|-
|
| <math display="inline">{\mathbf{θ}}_{t + 1}</math>
| <math display="inline">\leftarrow</math>
| <math display="inline">{\mathbf{θ}}_{t} - {{\alpha\hspace{0pt}{\nabla f_{t}^{\text{sreg}}}\hspace{0pt}{({\mathbf{θ}}_{t})}}/\text{s}}</math>
|
| (9)
|-
|
|
| <math display="inline">=</math>
| <math display="inline">{\mathbf{θ}}_{t} - {{\alpha\hspace{0pt}{\nabla f_{t}}\hspace{0pt}{({\mathbf{θ}}_{t})}}/\text{s}} - {{{\alpha\hspace{0pt}\lambda^{\prime}\hspace{0pt}{\mathbf{θ}}_{t}} \odot \text{s}}/\text{s}}</math>
|
| (10)
|-
|
|
| <math display="inline">=</math>
| <math display="inline">{{\mathbf{θ}}_{t} - {{\alpha\hspace{0pt}{\nabla f_{t}}\hspace{0pt}{({\mathbf{θ}}_{t})}}/\text{s}} - {\alpha\hspace{0pt}\lambda^{\prime}\hspace{0pt}{\mathbf{θ}}_{t}}},</math>
|
| (11)
|}

<!--T:122-->
where the division by s is element-wise. <math display="inline">O</math> with weight decay has the following iterates on <math display="inline">f_{t}\hspace{0pt}{({\mathbf{θ}})}</math>:

<!--T:123-->
{| id="A4.EGx4" class="wikitable"
|-
|
| <math display="inline">{\mathbf{θ}}_{t + 1}</math>
| <math display="inline">\leftarrow</math>
| <math display="inline">{{({1 - \lambda})}\hspace{0pt}{\mathbf{θ}}_{t}} - {{\alpha\hspace{0pt}{\nabla f}\hspace{0pt}{({\mathbf{θ}}_{t})}}/\text{s}}</math>
|
| (12)
|-
|
|
| <math display="inline">=</math>
| <math display="inline">{{\mathbf{θ}}_{t} - {{\alpha\hspace{0pt}{\nabla f}\hspace{0pt}{({\mathbf{θ}}_{t})}}/\text{s}} - {\lambda\hspace{0pt}{\mathbf{θ}}_{t}}},</math>
|
| (13)
|}

<!--T:124-->
These iterates are identical since <math display="inline">\lambda^{\prime} = \frac{\lambda}{\alpha}</math>. ∎

<!--T:125-->
<div id="A2" class="section">

<!--T:126-->
<span id="appendix-b-additional-practical-improvements-of-adam"></span>

<!--T:127-->
== Appendix B Additional Practical Improvements of Adam ==

<!--T:128-->
Having discussed decoupled weight decay for improving Adam’s generalization, in this section we introduce two additional components to improve Adam’s performance in practice.

<!--T:129-->
<div id="A2.SS1" class="section">

<!--T:130-->
<span id="b.1-normalized-weight-decay"></span>

<!--T:131-->
=== B.1 Normalized Weight Decay ===

<!--T:132-->
Our preliminary experiments showed that different weight decay factors are optimal for different computational budgets (defined in terms of the number of batch passes). Relatedly, Li et al. ([[#bib.bib12|2017]]) demonstrated that a smaller batch size (for the same total number of epochs) leads to the shrinking effect of weight decay being more pronounced. Here, we propose to reduce this dependence by normalizing the values of weight decay. Specifically, we replace the hyperparameter <math display="inline">\lambda</math> by a new (more robust) normalized weight decay hyperparameter <math display="inline">\lambda_{n\hspace{0pt}o\hspace{0pt}r\hspace{0pt}m}</math>, and use this to set <math display="inline">\lambda</math> as <math display="inline">\lambda = {\lambda_{n\hspace{0pt}o\hspace{0pt}r\hspace{0pt}m}\hspace{0pt}\sqrt{\frac{b}{B\hspace{0pt}T}}}</math>, where <math display="inline">b</math> is the batch size, <math display="inline">B</math> is the total number of training points and <math display="inline">T</math> is the total number of epochs.<sup>22</sup>2In the context of our AdamWR variant discussed in Section [[#A2.SS2|B.2]], <math display="inline">T</math> is the total number of epochs in the current restart. Thus, <math display="inline">\lambda_{n\hspace{0pt}o\hspace{0pt}r\hspace{0pt}m}</math> can be interpreted as the weight decay used if only one batch pass is allowed. We emphasize that our choice of normalization is merely one possibility informed by few experiments; a more lasting conclusion we draw is that using ''some'' normalization can substantially improve results.

<!--T:133-->
<div id="A2.SS2" class="section">

<!--T:134-->
<span id="b.2-adam-with-cosine-annealing-and-warm-restarts"></span>

<!--T:135-->
=== B.2 Adam with Cosine Annealing and Warm Restarts ===

<!--T:136-->
We now apply cosine annealing and warm restarts to Adam, following our recent work (Loshchilov &amp; Hutter, [[#bib.bib13|2016]]). There, we proposed Stochastic Gradient Descent with Warm Restarts (SGDR) to improve the anytime performance of SGD by quickly cooling down the learning rate according to a cosine schedule and periodically increasing it. SGDR has been successfully adopted to lead to new state-of-the-art results for popular image classification benchmarks (Huang et al., [[#bib.bib8|2017]]; Gastaldi, [[#bib.bib6|2017]]; Zoph et al., [[#bib.bib28|2017]]), and we therefore already tried extending it to Adam shortly after proposing it. However, while our initial version of Adam with warm restarts had better anytime performance than Adam, it was not competitive with SGD with warm restarts, precisely because L<sub>2</sub> regularization was not working as well as in SGD. Now, having fixed this issue by means of the original weight decay regularization (Section [[#S2|2]]) and also having introduced normalized weight decay (Section [[#A2.SS1|B.1]]), our original work on cosine annealing and warm restarts directly carries over to Adam.

<!--T:137-->
In the interest of keeping the presentation self-contained, we briefly describe how SGDR schedules the change of the effective learning rate in order to accelerate the training of DNNs. Here, we decouple the initial learning rate <math display="inline">\alpha</math> and its multiplier <math display="inline">\eta_{t}</math> used to obtain the actual learning rate at iteration <math display="inline">t</math> (see, e.g., line [[#alg1.l8|8]] in Algorithm 1). In SGDR, we simulate a new warm-started run/restart of SGD once <math display="inline">T_{i}</math> epochs are performed, where <math display="inline">i</math> is the index of the run. Importantly, the restarts are not performed from scratch but emulated by increasing <math display="inline">\eta_{t}</math> while the old value of <math display="inline">{\mathbf{θ}}_{t}</math> is used as an initial solution. The amount by which <math display="inline">\eta_{t}</math> is increased controls to which extent the previously acquired information (e.g., momentum) is used. Within the <math display="inline">i</math>-th run, the value of <math display="inline">\eta_{t}</math> decays according to a cosine annealing (Loshchilov &amp; Hutter, [[#bib.bib13|2016]]) learning rate for each batch as follows:

<!--T:138-->
{| id="A4.EGx5" class="wikitable"
|-
|
| <math display="inline">{\eta_{t} = {\eta_{m\hspace{0pt}i\hspace{0pt}n}^{(i)} + {0.5\hspace{0pt}{({\eta_{m\hspace{0pt}a\hspace{0pt}x}^{(i)} - \eta_{m\hspace{0pt}i\hspace{0pt}n}^{(i)}})}\hspace{0pt}{({1 + {\cos{({{\pi\hspace{0pt}T_{c\hspace{0pt}u\hspace{0pt}r}}/T_{i}})}}})}}}},</math>
|
| (14)
|}

<!--T:139-->
where <math display="inline">\eta_{m\hspace{0pt}i\hspace{0pt}n}^{(i)}</math> and <math display="inline">\eta_{m\hspace{0pt}a\hspace{0pt}x}^{(i)}</math> are ranges for the multiplier and <math display="inline">T_{c\hspace{0pt}u\hspace{0pt}r}</math> accounts for how many epochs have been performed since the last restart. <math display="inline">T_{c\hspace{0pt}u\hspace{0pt}r}</math> is updated at each batch iteration <math display="inline">t</math> and is thus not constrained to integer values. Adjusting (e.g., decreasing) <math display="inline">\eta_{m\hspace{0pt}i\hspace{0pt}n}^{(i)}</math> and <math display="inline">\eta_{m\hspace{0pt}a\hspace{0pt}x}^{(i)}</math> at every <math display="inline">i</math>-th restart (see also Smith ([[#bib.bib18|2016]])) could potentially improve performance, but we do not consider that option here because it would involve additional hyperparameters. For <math display="inline">\eta_{m\hspace{0pt}a\hspace{0pt}x}^{(i)} = 1</math> and <math display="inline">\eta_{m\hspace{0pt}i\hspace{0pt}n}^{(i)} = 0</math>, one can simplify Eq. ([[#A2.E14|14]]) to

<!--T:140-->
{| id="A4.EGx6" class="wikitable"
|-
|
| <math display="inline">{\eta_{t} = {0.5 + {0.5\hspace{0pt}{\cos{({{\pi\hspace{0pt}T_{c\hspace{0pt}u\hspace{0pt}r}}/T_{i}})}}}}}.</math>
|
| (15)
|}

<!--T:141-->
In order to achieve good anytime performance, one can start with an initially small <math display="inline">T_{i}</math> (e.g., from 1% to 10% of the expected total budget) and multiply it by a factor of <math display="inline">T_{m\hspace{0pt}u\hspace{0pt}l\hspace{0pt}t}</math> (e.g., <math display="inline">T_{m\hspace{0pt}u\hspace{0pt}l\hspace{0pt}t} = 2</math>) at every restart. The <math display="inline">({i + 1})</math>-th restart is triggered when <math display="inline">T_{c\hspace{0pt}u\hspace{0pt}r} = T_{i}</math> by setting <math display="inline">T_{c\hspace{0pt}u\hspace{0pt}r}</math> to 0. An example setting of the schedule multiplier is given in [[#A3|C]].

<!--T:142-->
Our proposed AdamWR algorithm represents AdamW (see Algorithm 2) with <math display="inline">\eta_{t}</math> following Eq. ([[#A2.E15|15]]) and <math display="inline">\lambda</math> computed at each iteration using normalized weight decay described in Section [[#A2.SS1|B.1]]. We note that normalized weight decay allowed us to use a constant parameter setting across short and long runs performed within AdamWR and SGDWR (SGDW with warm restarts).

<!--T:143-->
<div id="A3" class="section">

<!--T:144-->
<span id="appendix-c-an-example-setting-of-the-schedule-multiplier"></span>

<!--T:145-->
== Appendix C An Example Setting of the Schedule Multiplier ==

<!--T:146-->
An example schedule of the schedule multiplier <math display="inline">\eta_{t}</math> is given in SuppFigure [[#A3.F1|1]] for <math display="inline">T_{i = 0} = 100</math> and <math display="inline">T_{m\hspace{0pt}u\hspace{0pt}l\hspace{0pt}t} = 2</math>. After the initial 100 epochs the learning rate will reach 0 because <math display="inline">\eta_{t = 100} = 0</math>. Then, since <math display="inline">T_{c\hspace{0pt}u\hspace{0pt}r} = T_{i = 0}</math>, we restart by resetting <math display="inline">T_{c\hspace{0pt}u\hspace{0pt}r} = 0</math>, causing the multiplier <math display="inline">\eta_{t}</math> to be reset to 1 due to Eq. ([[#A2.E15|15]]). This multiplier will then decrease again from 1 to 0, but now over the course of 200 epochs because <math display="inline">T_{i = 1} = {T_{i = 0}\hspace{0pt}T_{m\hspace{0pt}u\hspace{0pt}l\hspace{0pt}t}} = 200</math>. Solutions obtained right before the restarts, when <math display="inline">\eta_{t} = 0</math> (e.g., at epoch indexes 100, 300, 700 and 1500 as shown in SuppFigure [[#A3.F1|1]]) are recommended by the optimizer as the solutions, with more recent solutions prioritized.

<!--T:147-->
<div id="A3.F1" class="figure">

<!--T:148-->
[[File:Arxiv_1711_05101_x17.png|211x165px|Refer to caption]]

<!--T:149-->
<div id="A4" class="section">

<!--T:150-->
<span id="appendix-d-additional-results"></span>

<!--T:151-->
== Appendix D Additional Results ==

<!--T:152-->
We investigated whether the use of much longer runs (1800 epochs) of “standard Adam” (Adam with L<sub>2</sub> regularization and a fixed learning rate) makes the use of cosine annealing unnecessary. SuppFigure [[#A4.F2|2]] shows the results of standard Adam for a 4 by 4 logarithmic grid of hyperparameter settings (the coarseness of the grid is due to the high computational expense of runs for 1800 epochs). Even after taking the low resolution of the grid into account, the results appear to be at best comparable to the ones obtained with AdamW with 18 times less epochs and a smaller network (see SuppFigure [[#A4.F3|3]], top row, middle). These results are not very surprising given Figure 1 in the main paper (which demonstrates both the improvements possible by using some learning rate schedule, such as cosine annealing, and the effectiveness of decoupled weight decay).

<!--T:153-->
Our experimental results with Adam and SGD suggest that the total runtime in terms of the number of epochs affect the basin of optimal hyperparameters (see SuppFigure [[#A4.F3|3]]). More specifically, the greater the total number of epochs the smaller the values of the weight decay should be. SuppFigure 4 shows that our remedy for this problem, the normalized weight decay defined in Eq. ([[#A2.E15|15]]), simplifies hyperparameter selection because the optimal values observed for short runs are similar to the ones for much longer runs. We used our initial experiments on CIFAR-10 to suggest the square root normalization we proposed in Eq. ([[#A2.E15|15]]) and double-checked that this is not a coincidence on the ImageNet32x32 dataset (Chrabaszcz et al., [[#bib.bib2|2017]]), a downsampled version of the original ImageNet dataset with 1.2 million 32<math display="inline">\times</math>32 pixels images, where an epoch is 24 times longer than on CIFAR-10. This experiment also supported the square root scaling: the best values of the normalized weight decay observed on CIFAR-10 represented nearly optimal values for ImageNet32x32 (see SuppFigure [[#A4.F3|3]]). In contrast, had we used the same raw weight decay values <math display="inline">\lambda</math> for ImageNet32x32 as for CIFAR-10 and for the same number of epochs, ''without the proposed normalization, <math display="inline">\lambda</math> would have been roughly 5 times too large for ImageNet32x32, leading to much worse performance''. The optimal normalized weight decay values were also very similar (e.g., <math display="inline">\lambda_{n\hspace{0pt}o\hspace{0pt}r\hspace{0pt}m} = 0.025</math> and <math display="inline">\lambda_{n\hspace{0pt}o\hspace{0pt}r\hspace{0pt}m} = 0.05</math>) across SGDW and AdamW. These results clearly show that normalizing weight decay can substantially improve performance; while square root scaling performed very well in our experiments we emphasize that these experiments were not very comprehensive and that even better scaling rules are likely to exist.

<!--T:154-->
SuppFigure [[#A4.F4|4]] is the equivalent of Figure 3 in the main paper, but for ImageNet32x32 instead of for CIFAR-10. The qualitative results are identical: weight decay leads to better training loss (cross-entropy) than L<sub>2</sub> regularization, and to an even greater improvement of test error.

<!--T:155-->
SuppFigure [[#A4.F5|5]] and SuppFigure [[#A4.F6|6]] are the equivalents of Figure 4 in the main paper but supplemented with training loss curves in its bottom row. The results show that Adam and its variants with decoupled weight decay converge faster (in terms of training loss) on CIFAR-10 than the corresponding SGD variants (the difference for ImageNet32x32 is small). As is discussed in the main paper, when the same values of training loss are considered, AdamW demonstrates better values of test error than Adam. Interestingly, SuppFigure [[#A4.F5|5]] and SuppFigure [[#A4.F6|6]] show that the restart variants AdamWR and SGDWR also demonstrate better generalization than AdamW and SGDW, respectively.

<!--T:156-->
<div id="A4.F2" class="figure">

<!--T:157-->
[[File:Arxiv_1711_05101_x18.png|198x203px|Refer to caption]]

<!--T:158-->
<div id="A4.F3" class="figure">

<!--T:159-->
[[File:Arxiv_1711_05101_x19.png|127x136px|Refer to caption]] [[File:Arxiv_1711_05101_x20.png|127x132px|Refer to caption]] [[File:Arxiv_1711_05101_x21.png|127x132px|Refer to caption]] [[File:Arxiv_1711_05101_x22.png|127x137px|Refer to caption]] [[File:Arxiv_1711_05101_x23.png|126x133px|Refer to caption]] [[File:Arxiv_1711_05101_x24.png|126x133px|Refer to caption]] [[File:Arxiv_1711_05101_x25.png|126x131px|Refer to caption]] [[File:Arxiv_1711_05101_x26.png|127x133px|Refer to caption]] [[File:Arxiv_1711_05101_x27.png|127x133px|Refer to caption]] [[File:Arxiv_1711_05101_x28.png|126x131px|Refer to caption]] [[File:Arxiv_1711_05101_x29.png|126x133px|Refer to caption]] [[File:Arxiv_1711_05101_x30.png|126x133px|Refer to caption]]

<!--T:160-->
<div id="A4.F4" class="figure">

<!--T:161-->
[[File:Arxiv_1711_05101_x31.png|198x158px|Refer to caption]]
   [[File:Arxiv_1711_05101_x32.png|198x160px|Refer to caption]]<br />
  <br />
[[File:Arxiv_1711_05101_x33.png|198x157px|Refer to caption]]    [[File:Arxiv_1711_05101_x34.png|198x161px|Refer to caption]]

<!--T:162-->
<div id="A4.F5" class="figure">

<!--T:163-->
[[File:Arxiv_1711_05101_x35.png|380x315px|Refer to caption]] [[File:Arxiv_1711_05101_x36.png|380x308px|Refer to caption]]

<!--T:164-->
<div id="A4.F6" class="figure">

<!--T:165-->
[[File:Arxiv_1711_05101_x37.png|380x312px|Refer to caption]] [[File:Arxiv_1711_05101_x38.png|380x311px|Refer to caption]]
</translate>

[[Category:Machine Learning]]
[[Category:Research]]
[[Category:Research Papers]]