https://marovi.ai/index.php?action=history&feed=atom&title=Searching_for_Activation_Functions%2Fpaper%2Fzh
Searching for Activation Functions/paper/zh - Revision history
2026-04-27T17:01:08Z
Revision history for this page on the wiki
MediaWiki 1.39.1
https://marovi.ai/index.php?title=Searching_for_Activation_Functions/paper/zh&diff=12923&oldid=prev
DeployBot: Batch translate Searching for Activation Functions/paper unit 22 → zh
2026-04-27T08:05:17Z
<p>Batch translate Searching for Activation Functions/paper unit 22 → zh</p>
<table style="background-color: #fff; color: #202122;" data-mw="interface">
<col class="diff-marker" />
<col class="diff-content" />
<col class="diff-marker" />
<col class="diff-content" />
<tr class="diff-title" lang="en">
<td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td>
<td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 08:05, 27 April 2026</td>
</tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l276">Line 276:</td>
<td colspan="2" class="diff-lineno">Line 276:</td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>[[File:Arxiv_1710_05941_x7.png|423x276px|见图说明]]</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>[[File:Arxiv_1710_05941_x7.png|423x276px|见图说明]]</div></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><del style="font-weight: bold; text-decoration: none;"><div class="mw-translate-fuzzy"></del></div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>Swish <ins style="font-weight: bold; text-decoration: none;">和 </ins>ReLU <ins style="font-weight: bold; text-decoration: none;">之间最显著的差异是 </ins>Swish 在 <math display="inline">x < 0</math> 时的非单调"<ins style="font-weight: bold; text-decoration: none;">凸起</ins>"。如图 [[#S4.F7|7]] <ins style="font-weight: bold; text-decoration: none;">所示，很大比例的预激活值落在该凸起的范围内 (</ins><math display="inline">- 5 \leq x \leq 0)</math><ins style="font-weight: bold; text-decoration: none;">，这表明非单调凸起是 </ins>Swish <ins style="font-weight: bold; text-decoration: none;">的一个重要方面。可以通过改变 </ins><math display="inline">\beta</math> <ins style="font-weight: bold; text-decoration: none;">参数来控制凸起的形状。虽然固定 </ins><math display="inline">\beta = 1</math> <ins style="font-weight: bold; text-decoration: none;">在实践中是有效的，但实验部分表明训练 </ins><math display="inline">\beta</math> <ins style="font-weight: bold; text-decoration: none;">可以在某些模型上进一步提高性能。图 </ins>[[#S4.F7|7]] <ins style="font-weight: bold; text-decoration: none;">绘制了来自 </ins>Mobile NASNet-A 模型 <ins style="font-weight: bold; text-decoration: none;">(Zoph 等</ins>, [[#bib.bib53|2017]]<ins style="font-weight: bold; text-decoration: none;">) 的训练后 </ins><math display="inline">\beta</math> <ins style="font-weight: bold; text-decoration: none;">值的分布。训练后的 </ins><math display="inline">\beta</math> 值分布在 <math display="inline">0</math> <ins style="font-weight: bold; text-decoration: none;">和 </ins><math display="inline">1.5</math> 之间，并在 <math display="inline">\beta \approx 1</math> <ins style="font-weight: bold; text-decoration: none;">处有一个峰值，这表明该模型利用了可训练 </ins><math display="inline">\beta</math> <ins style="font-weight: bold; text-decoration: none;">参数所提供的额外灵活性。</ins></div></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>Swish <del style="font-weight: bold; text-decoration: none;">与 </del>ReLU <del style="font-weight: bold; text-decoration: none;">最显著的差别是 </del>Swish 在 <math display="inline">x < 0</math> 时的非单调"<del style="font-weight: bold; text-decoration: none;">凸包</del>"。如图 [[#S4.F7|7]] <del style="font-weight: bold; text-decoration: none;">所示，大量预激活落入凸包的范围（</del><math display="inline">- 5 \leq x \leq 0)</math><del style="font-weight: bold; text-decoration: none;">，表明非单调凸包是 </del>Swish <del style="font-weight: bold; text-decoration: none;">的一个重要特征。凸包的形状可以通过改变 </del><math display="inline">\beta</math> <del style="font-weight: bold; text-decoration: none;">参数来控制。虽然在实践中固定 </del><math display="inline">\beta = 1</math> <del style="font-weight: bold; text-decoration: none;">已经有效，但实验部分表明对某些模型而言训练 </del><math display="inline">\beta</math> <del style="font-weight: bold; text-decoration: none;">可进一步提高性能。图 </del>[[#S4.F7|7]] <del style="font-weight: bold; text-decoration: none;">绘制了 </del>Mobile NASNet-A 模型 <del style="font-weight: bold; text-decoration: none;">（Zoph et al.</del>, [[#bib.bib53|2017]]<del style="font-weight: bold; text-decoration: none;">）训练得到的 </del><math display="inline">\beta</math> <del style="font-weight: bold; text-decoration: none;">值分布。训练得到的 </del><math display="inline">\beta</math> 值分布在 <math display="inline">0</math> <del style="font-weight: bold; text-decoration: none;">与 </del><math display="inline">1.5</math> 之间，并在 <math display="inline">\beta \approx 1</math> <del style="font-weight: bold; text-decoration: none;">处出现峰值，表明模型利用了可训练 </del><math display="inline">\beta</math> <del style="font-weight: bold; text-decoration: none;">参数的额外灵活性。</del></div></td><td colspan="2" class="diff-side-added"></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><del style="font-weight: bold; text-decoration: none;"></div></del></div></td><td colspan="2" class="diff-side-added"></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>在实践中，在大多数深度学习库中只需修改一行代码即可实现 Swish，例如在 TensorFlow（Abadi et al., [[#bib.bib1|2016]]）中使用 <code>x * tf.sigmoid(beta * x)</code>，或在本工作提交之后发布的 TensorFlow 版本中使用 <code>tf.nn.swish(x)</code>。需要注意的是，如果使用 BatchNorm（Ioffe &amp; Szegedy, [[#bib.bib21|2015]]），应当设置其 scale 参数。一些高层库由于 ReLU 是分段线性函数而默认关闭 scale 参数，但该设置对 Swish 并不正确。对于训练 Swish 网络，我们发现略微降低用于训练 ReLU 网络的学习率效果很好。</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>在实践中，在大多数深度学习库中只需修改一行代码即可实现 Swish，例如在 TensorFlow（Abadi et al., [[#bib.bib1|2016]]）中使用 <code>x * tf.sigmoid(beta * x)</code>，或在本工作提交之后发布的 TensorFlow 版本中使用 <code>tf.nn.swish(x)</code>。需要注意的是，如果使用 BatchNorm（Ioffe &amp; Szegedy, [[#bib.bib21|2015]]），应当设置其 scale 参数。一些高层库由于 ReLU 是分段线性函数而默认关闭 scale 参数，但该设置对 Swish 并不正确。对于训练 Swish 网络，我们发现略微降低用于训练 ReLU 网络的学习率效果很好。</div></td></tr>
</table>
DeployBot
https://marovi.ai/index.php?title=Searching_for_Activation_Functions/paper/zh&diff=12892&oldid=prev
DeployBot: Clear fuzzy flag
2026-04-27T07:36:36Z
<p>Clear fuzzy flag</p>
<a href="https://marovi.ai/index.php?title=Searching_for_Activation_Functions/paper/zh&diff=12892">Show changes</a>
DeployBot