View source for Searching for Activation Functions/en

<languages />
{{PaperTabs}}
{{PaperInfobox
 | topic_area  = Machine Learning
 | difficulty  = Research
 | authors     = Prajit Ramachandran; Barret Zoph; Quoc V. Le
 | year        = 2017
 | arxiv_id    = 1710.05941
 | source_url  = https://arxiv.org/abs/1710.05941
 | pdf_url     = https://arxiv.org/pdf/1710.05941.pdf
}}
{{ContentMeta
 | generated_by   = claude-code-direct
 | model_used     = claude-opus-4-7
 | generated_date = 2026-04-27
}}

'''Searching for Activation Functions''' is a 2017 paper by Prajit Ramachandran, Barret Zoph, and Quoc V. Le of Google Brain that uses automated search to discover scalar activation functions for deep neural networks. The search yields a family of simple, non-monotonic functions, of which the authors highlight one — '''Swish''', defined as <math>f(x) = x \cdot \sigma(\beta x)</math> — and show that it consistently matches or outperforms ReLU on deep models across image classification and machine translation benchmarks. The paper was presented at the [[International Conference on Learning Representations|ICLR]] 2018 workshop track.

== Overview ==

Activation functions sit at the heart of every deep network and have a large effect on optimization and generalization. Despite a long line of hand-designed alternatives — Leaky ReLU, PReLU, ELU, SELU, GELU, Softplus — the [[Rectified linear unit|ReLU]] <math>f(x) = \max(x, 0)</math> remained the de facto default because competing functions tended to give inconsistent gains across models and datasets.

Rather than designing yet another activation function by hand, the authors apply [[Neural architecture search|automated search]] over a compositional space of unary and binary primitives. The best discovered function, which they name Swish, is structurally close to ReLU but smooth and non-monotonic. The authors show that simply replacing ReLUs with Swish improves accuracy on a wide variety of pretrained-style architectures with minimal hyperparameter tuning, and that the result is robust enough to generalize from the small CIFAR-10 child networks used during search to large ImageNet- and translation-scale models.

== Key Contributions ==

* A compositional search space for scalar activation functions, built from a small library of unary functions (e.g. <math>x</math>, <math>x^2</math>, <math>\sigma(x)</math>, <math>\tanh(x)</math>, <math>\sin(x)</math>) and binary functions (e.g. <math>x_1 + x_2</math>, <math>x_1 \cdot x_2</math>, <math>\max</math>, <math>\sigma(x_1)\cdot x_2</math>).
* A search procedure combining exhaustive enumeration for small spaces with an [[Reinforcement learning|RL]]-trained RNN controller (using [[Proximal policy optimization|PPO]]) for spaces too large to enumerate.
* The discovery and detailed analysis of Swish, <math>f(x) = x \cdot \sigma(\beta x)</math>, where <math>\beta</math> is a constant or a per-channel trainable parameter.
* Extensive empirical comparison against seven baseline activation functions (ReLU, LReLU, PReLU, Softplus, ELU, SELU, GELU) on CIFAR-10/100, [[ImageNet]], and WMT 2014 English→German translation.

== Methods ==

The search space treats activation functions as repeated applications of a "core unit" of the form <math>b(u_1(x_1), u_2(x_2))</math>, where <math>u_1, u_2</math> are unary functions, <math>b</math> is a binary function, and the inputs <math>x_1, x_2</math> are either the layer preactivation <math>x</math> or the output of an earlier core unit. Different search spaces are constructed by varying the number of core units and the available primitives.

Candidate activation functions are evaluated by training a small "child network" — a [[Residual neural network|ResNet]]-20 on [[CIFAR-10]] for 10K steps — and reporting validation accuracy. For small spaces the authors enumerate exhaustively; for spaces of order <math>10^{12}</math> they train an RNN controller with reinforcement learning to maximize validation accuracy, using an exponential moving average of rewards as a baseline. Search is parallelized across worker machines that pull candidate activation functions from a queue, train a child network, and report the final validation accuracy back to the search algorithm.

After the search, the top candidates are stress-tested on three larger CIFAR architectures — preactivation ResNet-164, Wide ResNet 28-10, and DenseNet 100-12 — to filter out functions that overfit to the small child-network setting. Six of the eight top novel functions transfer; two of them, <math>x \cdot \sigma(\beta x)</math> and <math>\max(x, \sigma(x))</math>, match or beat ReLU on every model. The authors then commit to evaluating <math>x \cdot \sigma(\beta x)</math> at scale, partly because early experiments suggested better generalization.

The search uncovers several recurring patterns: simple functions outperform complex ones (1–2 core units suffice), the top functions tend to use the raw preactivation as one input to the final binary operation (mirroring ReLU's structure), and division-based functions rarely work because their outputs explode near zero. Among the top candidates, the authors single out

:<math>f(x) = x \cdot \sigma(\beta x), \qquad \sigma(z) = (1 + \exp(-z))^{-1}</math>

which they call Swish. Setting <math>\beta = 1</math> recovers the Sigmoid-weighted Linear Unit (SiL) of Elfwing et al.; setting <math>\beta \to \infty</math> recovers ReLU; setting <math>\beta = 0</math> gives the linear function <math>x/2</math>. Swish can therefore be viewed as a smooth interpolant between linear and ReLU behaviour, with <math>\beta</math> controlling the degree of nonlinearity.

The first derivative of Swish is

:<math>f'(x) = \sigma(\beta x) + \beta x \cdot \sigma(\beta x)\bigl(1 - \sigma(\beta x)\bigr) = \beta f(x) + \sigma(\beta x)\bigl(1 - \beta f(x)\bigr)</math>

so Swish is smooth everywhere, unbounded above, bounded below, and non-monotonic — it dips below zero in a small "bump" for <math>x</math> roughly between <math>-5</math> and <math>0</math>, before approaching zero from below as <math>x \to -\infty</math>. The authors show empirically that a large fraction of preactivations land inside this bump and argue it is an essential part of the function's behaviour. The shape of the bump is controlled by <math>\beta</math>: when <math>\beta</math> is treated as a per-channel trainable parameter, fitted values on Mobile NASNet-A spread between 0 and 1.5 with a peak near 1, suggesting models do exploit the extra flexibility.

Implementation is a single-line change in modern frameworks (e.g. <code>x * tf.sigmoid(beta * x)</code>). The authors note that BatchNorm's scale parameter must remain enabled (some libraries disable it by default for ReLU) and that learning rates often need to be slightly lower than those tuned for ReLU.

== Results ==

On CIFAR-10 and CIFAR-100, Swish and Swish-1 match or outperform ReLU on every model considered (preactivation ResNet-164, Wide ResNet 28-10, DenseNet 100-12). The "best baseline" varies by model — Softplus, GELU, and PReLU each lead on different rows — but Swish is the only function consistently at or near the top.

On [[ImageNet]] classification, replacing ReLU with Swish gives:

* '''Mobile NASNet-A''': +1.4% top-1 accuracy on average over three runs (<math>73.5 \to 74.9</math>%).
* '''Inception-ResNet-v2''': +0.5–0.6% top-1 (<math>79.6 \to 80.2</math>%).
* '''MobileNet''': +2.2% top-1 (<math>72.0 \to 74.2</math>%).
* '''Inception-v3''' and '''Inception-v4''': roughly +0.1% top-1, within noise.

For context, the authors note that an entire year of architectural tuning between Inception-v3 and Inception-ResNet-v2 yielded a 1.3% improvement, so the gains from a one-line activation swap are economically meaningful. On a 12-layer "Base [[Transformer (machine learning model)|Transformer]]" trained on WMT 2014 English→German, Swish-1 also matches or exceeds every baseline across four newstest sets, with the largest gain on newstest2016 (+0.6 BLEU over the next-best).

A summary sign test against each baseline (counting wins, ties, and losses across nine models) shows Swish strictly winning more often than losing against all seven of ReLU, LReLU, PReLU, Softplus, ELU, SELU, and GELU.

Swish's gains are largest on mobile-sized convolutional architectures (Mobile NASNet-A, MobileNet) and on the Transformer, while on Inception-v4 the gap narrows to within noise. Softplus, the next most consistent baseline, is competitive on large image classifiers but collapses on machine translation (3+ BLEU below ReLU on the WMT newstest sets), illustrating the cross-domain inconsistency the paper sets out to overcome.

== Impact ==

The Swish paper had outsized practical influence relative to its theoretical novelty. The function had in fact been independently proposed under the name Sigmoid-weighted Linear Unit (SiL) by Elfwing, Uchibe, and Doya in a [[reinforcement learning]] context, and the closely related [[Gaussian Error Linear Unit|GELU]] (Hendrycks and Gimpel, 2016) shares the same smooth, non-monotonic shape. The contribution here is the first systematic empirical demonstration that such functions improve accuracy on large-scale image and language models, together with the recipe — searching over a compositional space using a child network as a fast proxy — that produced it.

After release, Swish was added to mainstream frameworks (e.g. <code>tf.nn.swish</code>) and adopted in production architectures such as [[EfficientNet]]. The variant '''Hard Swish''' — a piecewise-linear approximation defined as <math>x \cdot \mathrm{ReLU6}(x + 3)/6</math> — was introduced in MobileNetV3 to recover Swish's accuracy gains while being cheap on mobile hardware. GELU itself was later popularized by [[BERT (language model)|BERT]] and the GPT family, where it became the default activation in [[Transformer (machine learning model)|Transformer]] feed-forward blocks, vindicating the broader category that Swish helped make mainstream.

The paper also helped legitimize the use of automated search for low-level neural-network components, complementing the Google Brain team's parallel work on architecture search (NASNet) and optimizer search. The authors explicitly tie their findings to the "architectural improvements lessen the need for individual components to preserve gradients" argument that emerged after [[Residual neural network|residual connections]] and Transformer-style attention removed many of the obstacles ReLU was originally designed to mitigate.

A common misreading of the paper is that "Swish beats ReLU everywhere"; the actual experimental record is more nuanced. On large image classifiers the gap is small and architecture-dependent — Inception-v4 is essentially a tie — and any retraining of these networks ought to retune learning rates from scratch rather than reusing ReLU-tuned schedules. The robust takeaway is the comparative one: across nine architectures and three domains, Swish is the least-bad default, and the search procedure can plausibly be re-run to find activations specialized to a new architecture.

== See also ==

* [[Rectified linear unit]]
* [[Gaussian Error Linear Unit]]
* [[Neural architecture search]]
* [[Transformer (machine learning model)]]
* [[ImageNet]]

== References ==

* Ramachandran, P., Zoph, B., & Le, Q. V. (2017). ''Searching for Activation Functions''. arXiv:1710.05941.
* Elfwing, S., Uchibe, E., & Doya, K. (2017). ''Sigmoid-weighted linear units for neural network function approximation in reinforcement learning''. arXiv:1702.03118.
* Hendrycks, D., & Gimpel, K. (2016). ''Gaussian Error Linear Units (GELUs)''. arXiv:1606.08415.
* Zoph, B., & Le, Q. V. (2017). ''Neural Architecture Search with Reinforcement Learning''. ICLR.
* Bello, I., Zoph, B., Vasudevan, V., & Le, Q. V. (2017). ''Neural Optimizer Search with Reinforcement Learning''. ICML.
* Howard, A., et al. (2019). ''Searching for MobileNetV3''. ICCV — introduces Hard Swish.

[[Category:Machine Learning]]
[[Category:Research]]
[[Category:Research Papers]]