View source for Language Modeling with Gated Convolutional Networks/paper

<languages />
{{PaperTabs}}
{{PaperInfobox
 | topic_area  = NLP
 | difficulty  = Research
 | authors     = Yann N. Dauphin; Angela Fan; Michael Auli; David Grangier
 | year        = 2016
 | arxiv_id    = 1612.08083
 | source_url  = https://arxiv.org/abs/1612.08083
 | pdf_url     = https://arxiv.org/pdf/1612.08083.pdf
}}
{{ContentMeta
 | generated_by   = arxiv-importer-raw
 | model_used     = strategy:html5
 | generated_date = 2026-04-27
}}

<translate>
<!--T:1-->
<span id="language-modeling-with-gated-convolutional-networks"></span>

<!--T:2-->
= Language Modeling with Gated Convolutional Networks =

<!--T:3-->
Yann N. Dauphin    Angela Fan    Michael Auli    David Grangier

<!--T:4-->
<span id="language-modeling-with-gated-convolutional-networks-1"></span>

<!--T:5-->
= Language Modeling with Gated Convolutional Networks =

<!--T:6-->
Yann N. Dauphin    Angela Fan    Michael Auli    David Grangier

<!--T:7-->
<span id="abstract"></span>

<!--T:8-->
== Abstract ==

<!--T:9-->
The pre-dominant approach to language modeling to date is based on recurrent neural networks. Their success on this task is often linked to their ability to capture unbounded context. In this paper we develop a finite context approach through stacked convolutions, which can be more efficient since they allow parallelization over sequential tokens. We propose a novel simplified gating mechanism that outperforms Oord et al. ([[#bib.bib27|2016b]]) and investigate the impact of key architectural decisions. The proposed approach achieves state-of-the-art on the WikiText-103 benchmark, even though it features long-term dependencies, as well as competitive results on the Google Billion Words benchmark. Our model reduces the latency to score a sentence by an order of magnitude compared to a recurrent baseline. To our knowledge, this is the first time a non-recurrent approach is competitive with strong recurrent models on these large scale language tasks.

<!--T:10-->
convolutional neural nets, recurrent nets, language modeling<br />

<!--T:11-->
<div id="S1" class="section">

<!--T:12-->
<span id="introduction"></span>

<!--T:13-->
== 1 Introduction ==

<!--T:14-->
Statistical language models estimate the probability distribution of a sequence of words by modeling the probability of the next word given preceding words, i.e.

<!--T:15-->
{| id="Sx1.EGx1" class="wikitable"
|-
|
| <math display="inline">{{P\hspace{0pt}{(w_{0},\ldots,w_{N})}} = {P\hspace{0pt}{(w_{0})}\hspace{0pt}{\prod\limits_{i = 1}^{N}{P\hspace{0pt}{(\left. w_{i} \middle| {w_{0},\ldots,w_{i - 1}} \right.)}}}}},</math>
|
|}

<!--T:16-->
where <math display="inline">w_{i}</math> are discrete word indices in a vocabulary. Language models are a critical part of systems for speech recognition (Yu &amp; Deng, [[#bib.bib35|2014]]) and machine translation (Koehn, [[#bib.bib18|2010]]).

<!--T:17-->
Recently, neural networks (Bengio et al., [[#bib.bib1|2003]]; Mikolov et al., [[#bib.bib23|2010]]; Jozefowicz et al., [[#bib.bib15|2016]]) have been shown to outperform classical ''n-gram'' language models (Kneser &amp; Ney, [[#bib.bib17|1995]]; Chen &amp; Goodman, [[#bib.bib3|1996]]). These classical models suffer from data sparsity, which makes it difficult to represent large contexts and thus, long-range dependencies. Neural language models tackle this issue by embedding words in continuous space over which a neural network is applied. The current state of the art for language modeling is based on long short term memory networks (LSTM; Hochreiter et al., 1997) which can theoretically model arbitrarily long dependencies.

<!--T:18-->
In this paper, we introduce new gated convolutional networks and apply them to language modeling. Convolutional networks can be stacked to represent large context sizes and extract hierarchical features over larger and larger contexts with more abstractive features (LeCun &amp; Bengio, [[#bib.bib20|1995]]). This allows them to model long-term dependencies by applying <math display="inline">\mathcal{O}\hspace{0pt}{(\frac{N}{k})}</math> operations over a context of size <math display="inline">N</math> and kernel width <math display="inline">k</math>. In contrast, recurrent networks view the input as a chain structure and therefore require a linear number <math display="inline">\mathcal{O}\hspace{0pt}{(N)}</math> of operations.

<!--T:19-->
Analyzing the input hierarchically bears resemblance to classical grammar formalisms which build syntactic tree structures of increasing granuality, e.g., sentences consist of noun phrases and verb phrases each comprising further internal structure (Manning &amp; Schütze, [[#bib.bib21|1999]]; Steedman, [[#bib.bib32|2002]]). Hierarchical structure also eases learning since the number of non-linearities for a given context size is reduced compared to a chain structure, thereby mitigating the vanishing gradient problem (Glorot &amp; Bengio, [[#bib.bib7|2010]]).

<!--T:20-->
Modern hardware is well suited to models that are highly parallelizable. In recurrent networks, the next output depends on the previous hidden state which does not enable parallelization over the elements of a sequence. Convolutional networks, however, are very amenable to this computing paradigm since the computation of all input words can be performed simultaneously (§[[#S2|2]]).

<!--T:21-->
Gating has been shown to be essential for recurrent neural networks to reach state-of-the-art performance (Jozefowicz et al., [[#bib.bib15|2016]]). Our gated linear units reduce the vanishing gradient problem for deep architectures by providing a linear path for the gradients while retaining non-linear capabilities (§[[#S5.SS2|5.2]]).

<!--T:22-->
We show that gated convolutional networks outperform other recently published language models such as LSTMs trained in a similar setting on the Google Billion Word Benchmark (Chelba et al., [[#bib.bib2|2013]]). We also evaluate the ability of our models to deal with long-range dependencies on the WikiText-103 benchmark for which the model is conditioned on an entire paragraph rather than a single sentence and we achieve a new state-of-the-art on this dataset (Merity et al., [[#bib.bib22|2016]]). Finally, we show that gated linear units achieve higher accuracy and converge faster than the LSTM-style gating of Oord et al. (2016; §[[#S4|4]], §[[#S5|5]]).

<!--T:23-->
<div id="S2" class="section">

<!--T:24-->
<span id="approach"></span>

<!--T:25-->
== 2 Approach ==

<!--T:26-->
In this paper we introduce a new neural language model that replaces the recurrent connections typically used in recurrent networks with gated temporal convolutions. Neural language models (Bengio et al., [[#bib.bib1|2003]]) produce a representation <math display="inline">\mathbf{H} = {\lbrack\mathbf{h}_{0},\ldots,\mathbf{h}_{N}\rbrack}</math> of the context for each word <math display="inline">w_{0},\ldots,w_{N}</math> to predict the next word <math display="inline">P\hspace{0pt}{(\left. w_{i} \middle| \mathbf{h}_{i} \right.)}</math>. Recurrent neural networks <math display="inline">f</math> compute <math display="inline">\mathbf{H}</math> through a recurrent function <math display="inline">\mathbf{h}_{i} = {f\hspace{0pt}{(\mathbf{h}_{i - 1},w_{i - 1})}}</math> which is an inherently sequential process that cannot be parallelized over <math display="inline">i</math>.<sup>11</sup>1Parallelization is usually done over multiple sequences instead.

<!--T:27-->
Our proposed approach convolves the inputs with a function <math display="inline">f</math> to obtain <math display="inline">\mathbf{H} = {f \ast w}</math> and therefore has no temporal dependencies, so it is easier to parallelize over the individual words of a sentence. This process will compute each context as a function of a number of preceding words. Compared to recurrent networks, the context size is finite but we will demonstrate both that infinite contexts are not necessary and our models can represent large enough contexts to perform well in practice (§[[#S5|5]]).

<!--T:28-->
<div id="S2.F1" class="figure">

<!--T:29-->
[[File:Arxiv_1612_08083_x1.png|230x391px|Refer to caption]]

<!--T:30-->
Figure [[#S2.F1|1]] illustrates the model architecture. Words are represented by a vector embedding stored in a lookup table <math display="inline">\mathbf{D}^{{|\mathcal{V}|} \times e}</math> where <math display="inline">|\mathcal{V}|</math> is the number of words in the vocabulary and <math display="inline">e</math> is the embedding size. The input to our model is a sequence of words <math display="inline">w_{0},\ldots,w_{N}</math> which are represented by word embeddings <math display="inline">\mathbf{E} = {\lbrack\mathbf{D}_{w_{0}},\ldots,\mathbf{D}_{w_{N}}\rbrack}</math>. We compute the hidden layers <math display="inline">h_{0},\ldots,h_{L}</math> as

<!--T:31-->
{| id="Sx1.EGx2" class="wikitable"
|-
|
| <math display="inline">{h_{l}\hspace{0pt}{(\mathbf{X})}} = {{{({{\mathbf{X} \ast \mathbf{W}} + \mathbf{b}})} \otimes \sigma}\hspace{0pt}{({{\mathbf{X} \ast \mathbf{V}} + \mathbf{c}})}}</math>
|
| (1)
|}

<!--T:32-->
where <math display="inline">m,n</math> are respectively the number of input and output feature maps and <math display="inline">k</math> is the patch size, <math display="inline">\mathbf{X} \in {\mathbb{R}}^{N \times m}</math> is the input of layer <math display="inline">h_{l}</math> (either word embeddings or the outputs of previous layers), <math display="inline">\mathbf{W} \in {\mathbb{R}}^{k \times m \times n}</math>, <math display="inline">\mathbf{b} \in {\mathbb{R}}^{n}</math>, <math display="inline">\mathbf{V} \in {\mathbb{R}}^{k \times m \times n}</math>, <math display="inline">\mathbf{c} \in {\mathbb{R}}^{n}</math> are learned parameters, <math display="inline">\sigma</math> is the sigmoid function and <math display="inline">\otimes</math> is the element-wise product between matrices.

<!--T:33-->
When convolving inputs, we take care that <math display="inline">\mathbf{h}_{i}</math> does not contain information from future words. We address this by shifting the convolutional inputs to prevent the kernels from seeing future context (Oord et al., [[#bib.bib26|2016a]]). Specifically, we zero-pad the beginning of the sequence with <math display="inline">k - 1</math> elements, assuming the first input element is the beginning of sequence marker which we do not predict and <math display="inline">k</math> is the width of the kernel.

<!--T:34-->
The output of each layer is a linear projection <math display="inline">{\mathbf{X} \ast \mathbf{W}} + \mathbf{b}</math> modulated by the gates <math display="inline">\sigma\hspace{0pt}{({{\mathbf{X} \ast \mathbf{V}} + \mathbf{c}})}</math>. Similar to LSTMs, these gates multiply each element of the matrix <math display="inline">{\mathbf{X} \ast \mathbf{W}} + \mathbf{b}</math> and control the information passed on in the hierarchy. We dub this gating mechanism Gated Linear Units (GLU). Stacking multiple layers on top of the input <math display="inline">\mathbf{E}</math> gives a representation of the context for each word <math display="inline">\mathbf{H} = {{h_{L} \circ \ldots \circ h_{0}}\hspace{0pt}{(\mathbf{E})}}</math>. We wrap the convolution and the gated linear unit in a pre-activation residual block that adds the input of the block to the output (He et al., [[#bib.bib11|2015a]]). The blocks have a bottleneck structure for computational efficiency and each block has up to 5 layers.

<!--T:35-->
The simplest choice to obtain model predictions is to use a softmax layer, but this choice is often computationally inefficient for large vocabularies and approximations such as noise contrastive estimation ([[#bib.bib10|Gutmann &amp; Hyvärinen,]] ) or hierarchical softmax (Morin &amp; Bengio, [[#bib.bib25|2005]]) are preferred. We choose an improvement of the latter known as ''adaptive softmax'' which assigns higher capacity to very frequent words and lower capacity to rare words (Grave et al., [[#bib.bib8|2016a]]). This results in lower memory requirements as well as faster computation at both training and test time.

<!--T:36-->
<div id="S3" class="section">

<!--T:37-->
<span id="gating-mechanisms"></span>

<!--T:38-->
== 3 Gating Mechanisms ==

<!--T:39-->
Gating mechanisms control the path through which information flows in the network and have proven to be useful for recurrent neural networks (Hochreiter &amp; Schmidhuber, [[#bib.bib13|1997]]). LSTMs enable long-term memory via a separate cell controlled by ''input'' and ''forget'' gates. This allows information to flow unimpeded through potentially many timesteps. Without these gates, information could easily vanish through the transformations of each timestep. In contrast, convolutional networks do not suffer from the same kind of vanishing gradient and we find experimentally that they do not require forget gates.

<!--T:40-->
Therefore, we consider models possessing solely output gates, which allow the network to control what information should be propagated through the hierarchy of layers. We show this mechanism to be useful for language modeling as it allows the model to select which words or features are relevant for predicting the next word. Parallel to our work, Oord et al. ([[#bib.bib27|2016b]]) have shown the effectiveness of an LSTM-style mechanism of the form <math display="inline">{{\text{tanh}\hspace{0pt}{({{\mathbf{X} \ast \mathbf{W}} + \mathbf{b}})}} \otimes \sigma}\hspace{0pt}{({{\mathbf{X} \ast \mathbf{V}} + \mathbf{c}})}</math> for the convolutional modeling of images. Later, Kalchbrenner et al. ([[#bib.bib16|2016]]) extended this mechanism with additional gates for use in translation and character-level language modeling.

<!--T:41-->
Gated linear units are a simplified gating mechanism based on the work of Dauphin &amp; Grangier ([[#bib.bib6|2015]]) for non-deterministic gates that reduce the vanishing gradient problem by having linear units coupled to the gates. This retains the non-linear capabilities of the layer while allowing the gradient to propagate through the linear unit without scaling. The gradient of the LSTM-style gating of which we dub gated tanh unit (GTU) is

<!--T:42-->
{| id="Sx1.EGx3" class="wikitable"
|-
|
| <math display="inline">{\nabla{\lbrack{{{\text{tanh}\hspace{0pt}{(\mathbf{X})}} \otimes \sigma}\hspace{0pt}{(\mathbf{X})}}\rbrack}} = {\text{tanh}^{\prime}\hspace{0pt}{(\mathbf{X})}\hspace{0pt}{\nabla{\mathbf{X} \otimes \sigma}}\hspace{0pt}{(\mathbf{X})}}</math>
|
|
|-
|
| <math display="inline">{+ {\sigma^{\prime}\hspace{0pt}{(\mathbf{X})}\hspace{0pt}{\nabla{\mathbf{X} \otimes \text{tanh}}}\hspace{0pt}{(\mathbf{X})}}}.</math>
|
| (2)
|}

<!--T:43-->
Notice that it gradually vanishes as we stack layers because of the downscaling factors <math display="inline">\text{tanh}^{\prime}\hspace{0pt}{(\mathbf{X})}</math> and <math display="inline">\sigma^{\prime}\hspace{0pt}{(\mathbf{X})}</math>. In contrast, the gradient of the gated linear unit

<!--T:44-->
{| id="Sx1.EGx4" class="wikitable"
|-
|
| <math display="inline">{\nabla{\lbrack{{\mathbf{X} \otimes \sigma}\hspace{0pt}{(\mathbf{X})}}\rbrack}} = {{{\nabla{\mathbf{X} \otimes \sigma}}\hspace{0pt}{(\mathbf{X})}} + {{\mathbf{X} \otimes \sigma^{\prime}}\hspace{0pt}{(\mathbf{X})}\hspace{0pt}{\nabla\mathbf{X}}}}</math>
|
| (3)
|}

<!--T:45-->
has a path <math display="inline">{\nabla{\mathbf{X} \otimes \sigma}}\hspace{0pt}{(\mathbf{X})}</math> without downscaling for the activated gating units in <math display="inline">\sigma\hspace{0pt}{(\mathbf{X})}</math>. This can be thought of as a multiplicative skip connection which helps gradients flow through the layers. We compare the different gating schemes experimentally in Section §[[#S5.SS2|5.2]] and we find gated linear units allow for faster convergence to better perplexities.

<!--T:46-->
<div id="S4" class="section">

<!--T:47-->
<span id="experimental-setup"></span>

<!--T:48-->
== 4 Experimental Setup ==

<!--T:49-->
<div id="S4.SS1" class="section">

<!--T:50-->
<span id="datasets"></span>

<!--T:51-->
=== 4.1 Datasets ===

<!--T:52-->
We report results on two public large-scale language modeling datasets. First, the Google Billion Word dataset (Chelba et al., [[#bib.bib2|2013]]) is considered one of the largest language modeling datasets with almost one billion tokens and a vocabulary of over 800K words. In this dataset, words appearing less than 3 times are replaced with a special unknown symbol. The data is based on an English corpus of <math display="inline">30,301,028</math> sentences whose order has been shuffled. Second, WikiText-103 is a smaller dataset of over 100M tokens with a vocabulary of about 200K words (Merity et al., [[#bib.bib22|2016]]). Different from GBW, the sentences are consecutive which allows models to condition on larger contexts rather than single sentences. For both datasets, we add a beginning of sequence marker &lt;S &gt; at the start of each line and an end of sequence marker &lt;/S&gt; at the end of each line. On the Google Billion Word corpus each sequence is a single sentence, while on WikiText-103 a sequence is an entire paragraph. The model sees &lt;S&gt; and &lt;/S &gt; as input but only predicts the end of sequence marker &lt;/S&gt;. We evaluate models by computing the perplexity <math display="inline">\text{e}^{{\frac{1}{N}\hspace{0pt}\sum_{i}^{N}} - {{\log p}\hspace{0pt}{({w_{i}|{\ldots,w_{i - 1}}})}}}</math> on the standard held out test portion of each dataset.

<!--T:53-->
<div id="S4.T1" class="figure">

<!--T:54-->
{| id="S4.T1.22" class="wikitable"
|- id="S4.T1.22.23.1"
! id="S4.T1.22.23.1.1" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| Name
| id="S4.T1.22.23.1.2" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| GCNN-13
| id="S4.T1.22.23.1.3" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| GCNN-14B
| id="S4.T1.22.23.1.4" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| GCNN-9
| id="S4.T1.22.23.1.5" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| GCNN-8B
| id="S4.T1.22.23.1.6" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| GCNN-8
| id="S4.T1.22.23.1.7" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| GCNN-14
|- id="S4.T1.22.24.2"
! id="S4.T1.22.24.2.1" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| Dataset
| id="S4.T1.22.24.2.2" colspan="4" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| Google Billion Word
| id="S4.T1.22.24.2.3" colspan="2" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| wikitext-103
|- id="S4.T1.22.25.3"
! id="S4.T1.22.25.3.1" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| Lookup
| id="S4.T1.22.25.3.2" colspan="4" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| 128
| id="S4.T1.22.25.3.3" colspan="2" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| 280
|- id="S4.T1.6.6"
! id="S4.T1.6.6.7" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| Conv1
| id="S4.T1.1.1.1" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| <math display="inline">\lbrack 4,1268\rbrack \times 1</math>
| id="S4.T1.2.2.2" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| <math display="inline">\lbrack 5,512\rbrack \times 1</math>
| id="S4.T1.3.3.3" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| <math display="inline">\lbrack 4,807\rbrack \times 1</math>
| id="S4.T1.4.4.4" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| <math display="inline">\lbrack 1,512\rbrack \times 1</math>
| id="S4.T1.5.5.5" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| <math display="inline">\lbrack 4,900\rbrack \times 1</math>
| id="S4.T1.6.6.6" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| <math display="inline">\lbrack 6,850\rbrack \times 3</math>
|- id="S4.T1.12.12"
! id="S4.T1.12.12.7" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| Conv2.x
| id="S4.T1.7.7.1" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| <math display="inline">\begin{bmatrix}
{4,1268} \\
{4,1268}
\end{bmatrix} \times 12</math>
| id="S4.T1.8.8.2" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| <math display="inline">\begin{bmatrix}
{1,128} \\
{5,128} \\
{1,512}
\end{bmatrix} \times 3</math>
| id="S4.T1.9.9.3" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| <math display="inline">\begin{bmatrix}
{4,807} \\
{4,807}
\end{bmatrix} \times 4</math>
| id="S4.T1.10.10.4" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| <math display="inline">\begin{bmatrix}
{1,128} \\
{5,128} \\
{1,512}
\end{bmatrix} \times 3</math>
| id="S4.T1.11.11.5" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| <math display="inline">\lbrack 4,900\rbrack \times 7</math>
| id="S4.T1.12.12.6" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| <math display="inline">\lbrack 1,850\rbrack \times 1</math>
|- id="S4.T1.15.15"
! id="S4.T1.15.15.4" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| Conv3.x
| id="S4.T1.15.15.5" style="padding-top: 0.9pt; padding-bottom: 0.9pt"|
| id="S4.T1.13.13.1" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| <math display="inline">\begin{bmatrix}
{1,512} \\
{5,512} \\
{1,1024}
\end{bmatrix} \times 3</math>
| id="S4.T1.15.15.6" style="padding-top: 0.9pt; padding-bottom: 0.9pt"|
| id="S4.T1.14.14.2" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| <math display="inline">\begin{bmatrix}
{1,256} \\
{5,256} \\
{1,512}
\end{bmatrix} \times 3</math>
| id="S4.T1.15.15.7" style="padding-top: 0.9pt; padding-bottom: 0.9pt"|
| id="S4.T1.15.15.3" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| <math display="inline">\lbrack 5,850\rbrack \times 4</math>
|- id="S4.T1.18.18"
! id="S4.T1.18.18.4" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| Conv4.x
| id="S4.T1.18.18.5" style="padding-top: 0.9pt; padding-bottom: 0.9pt"|
| id="S4.T1.16.16.1" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| <math display="inline">\begin{bmatrix}
{1,1024} \\
{5,1024} \\
{1,2048}
\end{bmatrix} \times 6</math>
| id="S4.T1.18.18.6" style="padding-top: 0.9pt; padding-bottom: 0.9pt"|
| id="S4.T1.17.17.2" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| <math display="inline">\begin{bmatrix}
{1,1024} \\
{1,1024} \\
{1,2048}
\end{bmatrix} \times 1</math>
| id="S4.T1.18.18.7" style="padding-top: 0.9pt; padding-bottom: 0.9pt"|
| id="S4.T1.18.18.3" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| <math display="inline">\lbrack 1,850\rbrack \times 1</math>
|- id="S4.T1.20.20"
! id="S4.T1.20.20.3" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| Conv5.x
| id="S4.T1.20.20.4" style="padding-top: 0.9pt; padding-bottom: 0.9pt"|
| id="S4.T1.19.19.1" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| <math display="inline">\begin{bmatrix}
{1,1024} \\
{5,1024} \\
{1,4096}
\end{bmatrix} \times 1</math>
| id="S4.T1.20.20.5" style="padding-top: 0.9pt; padding-bottom: 0.9pt"|
| id="S4.T1.20.20.6" style="padding-top: 0.9pt; padding-bottom: 0.9pt"|
| id="S4.T1.20.20.7" style="padding-top: 0.9pt; padding-bottom: 0.9pt"|
| id="S4.T1.20.20.2" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| <math display="inline">\lbrack 4,850\rbrack \times 3</math>
|- id="S4.T1.21.21"
! id="S4.T1.21.21.2" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| Conv6.x
| id="S4.T1.21.21.3" style="padding-top: 0.9pt; padding-bottom: 0.9pt"|
| id="S4.T1.21.21.4" style="padding-top: 0.9pt; padding-bottom: 0.9pt"|
| id="S4.T1.21.21.5" style="padding-top: 0.9pt; padding-bottom: 0.9pt"|
| id="S4.T1.21.21.6" style="padding-top: 0.9pt; padding-bottom: 0.9pt"|
| id="S4.T1.21.21.7" style="padding-top: 0.9pt; padding-bottom: 0.9pt"|
| id="S4.T1.21.21.1" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| <math display="inline">\lbrack 4,1024\rbrack \times 1</math>
|- id="S4.T1.22.22"
! id="S4.T1.22.22.2" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| Conv7.x
| id="S4.T1.22.22.3" style="padding-top: 0.9pt; padding-bottom: 0.9pt"|
| id="S4.T1.22.22.4" style="padding-top: 0.9pt; padding-bottom: 0.9pt"|
| id="S4.T1.22.22.5" style="padding-top: 0.9pt; padding-bottom: 0.9pt"|
| id="S4.T1.22.22.6" style="padding-top: 0.9pt; padding-bottom: 0.9pt"|
| id="S4.T1.22.22.7" style="padding-top: 0.9pt; padding-bottom: 0.9pt"|
| id="S4.T1.22.22.1" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| <math display="inline">\lbrack 4,2048\rbrack \times 1</math>
|- id="S4.T1.22.26.4"
! id="S4.T1.22.26.4.1" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| AdaSoftmax
| id="S4.T1.22.26.4.2" colspan="2" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| 10k,40k,200k
| id="S4.T1.22.26.4.3" colspan="2" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| 4k,40k,200k
| id="S4.T1.22.26.4.4" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| 2k,10k,50k
| id="S4.T1.22.26.4.5" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| 10k,20k,200k
|}

<!--T:55-->
<div id="S4.T2" class="figure">

<!--T:56-->
{| id="S4.T2.1" class="wikitable"
|- id="S4.T2.1.1.1"
| id="S4.T2.1.1.1.1" style="padding-top: 1pt; padding-bottom: 1pt"| Model
| id="S4.T2.1.1.1.2" style="padding-top: 1pt; padding-bottom: 1pt"| Test PPL
| id="S4.T2.1.1.1.3" style="padding-top: 1pt; padding-bottom: 1pt"| Hardware
|- id="S4.T2.1.2.2"
| id="S4.T2.1.2.2.1" style="padding-top: 1pt; padding-bottom: 1pt"| Sigmoid-RNN-2048 (Ji et al., [[#bib.bib14|2015]])
| id="S4.T2.1.2.2.2" style="padding-top: 1pt; padding-bottom: 1pt"| 68.3
| id="S4.T2.1.2.2.3" style="padding-top: 1pt; padding-bottom: 1pt"| 1 CPU
|- id="S4.T2.1.3.3"
| id="S4.T2.1.3.3.1" style="padding-top: 1pt; padding-bottom: 1pt"| Interpolated KN 5-Gram (Chelba et al., [[#bib.bib2|2013]])
| id="S4.T2.1.3.3.2" style="padding-top: 1pt; padding-bottom: 1pt"| 67.6
| id="S4.T2.1.3.3.3" style="padding-top: 1pt; padding-bottom: 1pt"| 100 CPUs
|- id="S4.T2.1.4.4"
| id="S4.T2.1.4.4.1" style="padding-top: 1pt; padding-bottom: 1pt"| Sparse Non-Negative Matrix LM (Shazeer et al., [[#bib.bib30|2014]])
| id="S4.T2.1.4.4.2" style="padding-top: 1pt; padding-bottom: 1pt"| 52.9
| id="S4.T2.1.4.4.3" style="padding-top: 1pt; padding-bottom: 1pt"| -
|- id="S4.T2.1.5.5"
| id="S4.T2.1.5.5.1" style="padding-top: 1pt; padding-bottom: 1pt"| RNN-1024 + MaxEnt 9 Gram Features (Chelba et al., [[#bib.bib2|2013]])
| id="S4.T2.1.5.5.2" style="padding-top: 1pt; padding-bottom: 1pt"| 51.3
| id="S4.T2.1.5.5.3" style="padding-top: 1pt; padding-bottom: 1pt"| 24 GPUs
|- id="S4.T2.1.6.6"
| id="S4.T2.1.6.6.1" style="padding-top: 1pt; padding-bottom: 1pt"| LSTM-2048-512 (Jozefowicz et al., [[#bib.bib15|2016]])
| id="S4.T2.1.6.6.2" style="padding-top: 1pt; padding-bottom: 1pt"| 43.7
| id="S4.T2.1.6.6.3" style="padding-top: 1pt; padding-bottom: 1pt"| 32 GPUs
|- id="S4.T2.1.7.7"
| id="S4.T2.1.7.7.1" style="padding-top: 1pt; padding-bottom: 1pt"| 2-layer LSTM-8192-1024 (Jozefowicz et al., [[#bib.bib15|2016]])
| id="S4.T2.1.7.7.2" style="padding-top: 1pt; padding-bottom: 1pt"| 30.6
| id="S4.T2.1.7.7.3" style="padding-top: 1pt; padding-bottom: 1pt"| 32 GPUs
|- id="S4.T2.1.8.8"
| id="S4.T2.1.8.8.1" style="padding-top: 1pt; padding-bottom: 1pt"| BIG GLSTM-G4 (Kuchaiev &amp; Ginsburg, [[#bib.bib19|2017]])
| id="S4.T2.1.8.8.2" style="padding-top: 1pt; padding-bottom: 1pt"| 23.3<sup>11</sup>footnotemark: 1
| id="S4.T2.1.8.8.3" style="padding-top: 1pt; padding-bottom: 1pt"| 8 GPUs
|- id="S4.T2.1.9.9"
| id="S4.T2.1.9.9.1" style="padding-top: 1pt; padding-bottom: 1pt"| LSTM-2048 (Grave et al., [[#bib.bib8|2016a]])
| id="S4.T2.1.9.9.2" style="padding-top: 1pt; padding-bottom: 1pt"| 43.9
| id="S4.T2.1.9.9.3" style="padding-top: 1pt; padding-bottom: 1pt"| 1 GPU
|- id="S4.T2.1.10.10"
| id="S4.T2.1.10.10.1" style="padding-top: 1pt; padding-bottom: 1pt"| 2-layer LSTM-2048 (Grave et al., [[#bib.bib8|2016a]])
| id="S4.T2.1.10.10.2" style="padding-top: 1pt; padding-bottom: 1pt"| 39.8
| id="S4.T2.1.10.10.3" style="padding-top: 1pt; padding-bottom: 1pt"| 1 GPU
|- id="S4.T2.1.11.11"
| id="S4.T2.1.11.11.1" style="padding-top: 1pt; padding-bottom: 1pt"| GCNN-13
| id="S4.T2.1.11.11.2" style="padding-top: 1pt; padding-bottom: 1pt"| 38.1
| id="S4.T2.1.11.11.3" style="padding-top: 1pt; padding-bottom: 1pt"| 1 GPU
|- id="S4.T2.1.12.12"
| id="S4.T2.1.12.12.1" style="padding-top: 1pt; padding-bottom: 1pt"| GCNN-14 Bottleneck
| id="S4.T2.1.12.12.2" style="padding-top: 1pt; padding-bottom: 1pt"| 31.9
| id="S4.T2.1.12.12.3" style="padding-top: 1pt; padding-bottom: 1pt"| 8 GPUs
|}

<!--T:57-->
<div id="S4.SS2" class="section">

<!--T:58-->
<span id="training"></span>

<!--T:59-->
=== 4.2 Training ===

<!--T:60-->
We implement our models in Torch (Collobert et al., [[#bib.bib5|2011]]) and train on Tesla M40 GPUs. The majority of our models are trained on single GPU, as we focused on identifying compact architectures with good generalization and efficient computation at test time. We trained larger models with an 8-GPU setup by copying the model onto each GPU and dividing the batch such that each worker computes 1/8th of the gradients. The gradients are then summed using Nvidia NCCL. The multi-GPU setup allowed us to train models with larger hidden units.

<!--T:61-->
We train using Nesterov’s momentum (Sutskever et al., [[#bib.bib33|2013]]). While the cost in terms of memory is storing another vector of the size of the parameters, it increases the speed of convergence significantly with minimal additional computation compared to standard stochastic gradient descent. The speed of convergence was further increased with gradient clipping (Pascanu et al., [[#bib.bib28|2013]]) and weight normalization (Salimans &amp; Kingma, [[#bib.bib29|2016]]).

<!--T:62-->
Pascanu et al. ([[#bib.bib28|2013]]) argue for gradient clipping because it prevents the gradient explosion problem that characterizes RNNs. However, gradient clipping is not tied to RNNs, as it can be derived from the general concept of trust region methods. Gradient clipping is found using a spherical trust region

<!--T:63-->
{| id="Sx1.EGx5" class="wikitable"
|-
|
| <math display="inline">\Delta\hspace{0pt}\theta^{\ast}</math>
| <math display="inline">= {{{\operatorname{argmin}\limits_{\substack{{\text{s. t.}\hspace{0pt}{\|{\Delta\hspace{0pt}\theta}\|}} \leq \epsilon}}f}\hspace{0pt}{(\theta)}} + {\nabla{f^{T}\hspace{0pt}\Delta\hspace{0pt}\theta}}}</math>
|
|
|-
|
|
| <math display="inline">{= {- {{\max{({\|{\nabla f}\|},\epsilon)}}\hspace{0pt}\frac{\nabla f}{\|{\nabla f}\|}}}}.</math>
|
| (4)
|}

<!--T:64-->
Empirically, our experiments converge significantly faster with the use of gradient clipping even though we do not use a recurrent architecture.

<!--T:65-->
In combination, these methods led to stable and fast convergence with comparatively large learning rates such as <math display="inline">1</math>.

<!--T:66-->
<div id="S4.SS3" class="section">

<!--T:67-->
<span id="hyper-parameters"></span>

<!--T:68-->
=== 4.3 Hyper-parameters ===

<!--T:69-->
We found good hyper-parameter configurations by cross-validating with random search on a validation set. For model architecture, we select the number of residual blocks between <math display="inline">\{ 1,\ldots,10\}</math>, the size of the embeddings with <math display="inline">\{ 128,\ldots,256\}</math>, the number of units between <math display="inline">\{ 128,\ldots,2048\}</math>, and the kernel width between <math display="inline">\{ 3,\ldots,5\}</math>. In general, finding a good architecture was simple and the rule of thumb is that the larger the model, the better the performance. In terms of optimization, we initialize the layers of the model with the Kaiming initialization (He et al., [[#bib.bib12|2015b]]), with the learning rate sampled uniformly in the interval <math display="inline">\lbrack 1.,2.\rbrack</math>, the momentum set to <math display="inline">0.99</math>, and clipping set to <math display="inline">0.1</math>. Good hyper-parameters for the optimizer are quite straightforward to find and the optimal values do not change much between datasets.

<!--T:70-->
<div id="S5" class="section">

<!--T:71-->
<span id="results"></span>

<!--T:72-->
== 5 Results ==

<!--T:73-->
LSTMs and recurrent networks are able to capture long term dependencies and are fast becoming cornerstones in natural language processing. In this section, we compare strong LSTM and RNN models from the literature to our gated convolutional approach on two datasets.

<!--T:74-->
<div id="S5.F2" class="figure">

<!--T:75-->
[[File:Arxiv_1612_08083_x2.png|230x153px|Refer to caption]]

<!--T:76-->
We find the GCNN outperforms the comparable LSTM results on Google billion words. To accurately compare these approaches, we control for the same number of GPUs and the adaptive softmax output model (Grave et al., [[#bib.bib8|2016a]]), as these variables have a significant influence on performance. In this setting, the GCNN reaches <math display="inline">38.1</math> test perplexity while the comparable LSTM has <math display="inline">39.8</math> perplexity (Table [[#S4.T2|2]]).

<!--T:77-->
Further, the GCNN obtains strong performance with much greater computational efficiency. Figure [[#S5.F2|2]] shows that our approach closes the previously significant gap between models that use the full softmax and models with the usually less accurate hierarchical softmax. Thanks to the adaptive softmax, the GCNN only requires a fraction of the operations to reach the same perplexity values. The GCNN outperforms other single model state-of-the-art approaches except the much larger LSTM of Jozefowicz et al. ([[#bib.bib15|2016]]), a model which requires more GPUs and the much more computationally expensive full softmax. In comparison, the largest model we have trained reaches <math display="inline">31.9</math> test perplexity compared to the <math display="inline">30.6</math> of that approach, but only requires training for 2 weeks on 8 GPUs compared to 3 weeks of training on 32 GPUs for the LSTM. Note that these results can be improved by either using mixtures of experts (Shazeer et al., [[#bib.bib31|2017]]) or ensembles of these models.

<!--T:78-->
<div id="S5.T3" class="figure">

<!--T:79-->
{| id="S5.T3.1" class="wikitable"
|- id="S5.T3.1.1.1"
! id="S5.T3.1.1.1.1" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| Model
! id="S5.T3.1.1.1.2" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| Test PPL
! id="S5.T3.1.1.1.3" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| Hardware
|- id="S5.T3.1.2.1"
| id="S5.T3.1.2.1.1" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| LSTM-1024 (Grave et al., [[#bib.bib9|2016b]])
| id="S5.T3.1.2.1.2" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| 48.7
| id="S5.T3.1.2.1.3" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| 1 GPU
|- id="S5.T3.1.3.2"
| id="S5.T3.1.3.2.1" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| GCNN-8
| id="S5.T3.1.3.2.2" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| 44.9
| id="S5.T3.1.3.2.3" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| 1 GPU
|- id="S5.T3.1.4.3"
| id="S5.T3.1.4.3.1" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| GCNN-14
| id="S5.T3.1.4.3.2" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| 37.2
| id="S5.T3.1.4.3.3" style="padding-top: 0.9pt; padding-bottom: 0.9pt"| 4 GPUs
|}

<!--T:80-->
<sup>**</sup>footnotetext: appeared after submission
Another relevant concern is if the GCNN’s fixed context size can thoroughly model long sequences. On Google Billion Word, the average sentence length is quite short — only 20 words. We evaluate on WikiText-103 to determine if the model can perform well on a dataset where much larger contexts are available. On WikiText-103, an input sequence is an entire Wikipedia article instead of an individual sentence - increasing the average length to 4000 words. However, the GCNN outperforms LSTMs on this problem as well (Table [[#S5.T3|3]]). The GCNN-8 model has 8 layers with <math display="inline">800</math> units each and the LSTM has <math display="inline">1024</math> units. These results show that GCNNs can model enough context to achieve strong results.

<!--T:81-->
We evaluated on the Gigaword dataset following Chen et al. ([[#bib.bib4|2016]]) to compare with fully connected models. We found that the fully connected and convolutional network reach respectively 55.6 and 29.4 perplexity. We also ran preliminary experiments on the much smaller Penn tree bank dataset. When we score the sentences independently, the GCNN and LSTM have comparable test perplexity with 108.7 and 109.3 respectively. However, it is possible to achieve better results by conditioning on previous sentences. Unlike the LSTM, we found that the GCNN overfits on this quite small dataset and so we note the model is better suited to larger scale problems.

<!--T:82-->
<div id="S5.SS1" class="section">

<!--T:83-->
<span id="computational-efficiency"></span>

<!--T:84-->
=== 5.1 Computational Efficiency ===

<!--T:85-->
<div id="S5.T4" class="figure">

<!--T:86-->
{| id="S5.T4.1" class="wikitable"
|- id="S5.T4.1.1.1"
! id="S5.T4.1.1.1.1" style="padding-top: 1pt; padding-bottom: 1pt"|
! id="S5.T4.1.1.1.2" colspan="2" style="padding-top: 1pt; padding-bottom: 1pt"| Throughput
! id="S5.T4.1.1.1.3" style="padding-top: 1pt; padding-bottom: 1pt"| Responsiveness
|- id="S5.T4.1.2.2"
! id="S5.T4.1.2.2.1" style="padding-top: 1pt; padding-bottom: 1pt"|
! id="S5.T4.1.2.2.2" style="padding-top: 1pt; padding-bottom: 1pt"| (CPU)
! id="S5.T4.1.2.2.3" style="padding-top: 1pt; padding-bottom: 1pt"| (GPU)
! id="S5.T4.1.2.2.4" style="padding-top: 1pt; padding-bottom: 1pt"| (GPU)
|- id="S5.T4.1.3.1"
! id="S5.T4.1.3.1.1" style="padding-top: 1pt; padding-bottom: 1pt"| LSTM-2048
! id="S5.T4.1.3.1.2" style="padding-top: 1pt; padding-bottom: 1pt"| 169
| id="S5.T4.1.3.1.3" style="padding-top: 1pt; padding-bottom: 1pt"| 45,622
| id="S5.T4.1.3.1.4" style="padding-top: 1pt; padding-bottom: 1pt"| 2,282
|- id="S5.T4.1.4.2"
! id="S5.T4.1.4.2.1" style="padding-top: 1pt; padding-bottom: 1pt"| GCNN-9
! id="S5.T4.1.4.2.2" style="padding-top: 1pt; padding-bottom: 1pt"| 121
| id="S5.T4.1.4.2.3" style="padding-top: 1pt; padding-bottom: 1pt"| 29,116
| id="S5.T4.1.4.2.4" style="padding-top: 1pt; padding-bottom: 1pt"| 29,116
|- id="S5.T4.1.5.3"
! id="S5.T4.1.5.3.1" style="padding-top: 1pt; padding-bottom: 1pt"| GCNN-8 Bottleneck
! id="S5.T4.1.5.3.2" style="padding-top: 1pt; padding-bottom: 1pt"| 179
| id="S5.T4.1.5.3.3" style="padding-top: 1pt; padding-bottom: 1pt"| 45,878
| id="S5.T4.1.5.3.4" style="padding-top: 1pt; padding-bottom: 1pt"| 45,878
|}

<!--T:87-->
<div id="S5.F3" class="figure">

<!--T:88-->
[[File:Arxiv_1612_08083_x3.png|207x138px|Refer to caption]] [[File:Arxiv_1612_08083_x4.png|207x138px|Refer to caption]]

<!--T:89-->
Computational cost is an important consideration for language models. Depending on the application, there are a number of metrics to consider. We measure the ''throughput'' of a model as the number of tokens that can be processed per second. Throughput can be maximized by processing many sentences in parallel to amortize sequential operations. In contrast, ''responsiveness'' is the speed of processing the input sequentially, one token at a time. Throughput is important because it indicates the time required to process a corpus of text and responsiveness is an indicator of the time to finish processing a sentence. A model can have low responsiveness but high throughput by evaluating many sentences simultaneously through batching. In this case, such a model is slow in finishing processing individual sentences, but can process many sentences at a good rate.

<!--T:90-->
We evaluate the throughput and responsiveness for models that reach approximately <math display="inline">43.9</math> perplexity on the Google Billion Word benchmark. We consider the LSTM with <math display="inline">2048</math> units in Table [[#S4.T2|2]], a GCNN-8Bottleneck with 7 Resnet blocks that have a bottleneck structure as described by (He et al., [[#bib.bib11|2015a]]) and a GCNN-8 without bottlenecks. A bottleneck block wedges a <math display="inline">k > 1</math> convolution between two <math display="inline">k = 1</math> layers. This designs reduces computational cost by reducing and increasing dimensionality with the <math display="inline">k = 1</math> layers so that the convolution operates in a lower dimensional space. Our results show that the use of bottleneck blocks is important to maintaining computational efficiency.

<!--T:91-->
The throughput of the LSTM is measured by using a large batch of <math display="inline">750</math> sequences of length <math display="inline">20</math>, resulting in <math display="inline">15,000</math> tokens per batch. The responsiveness is the average speed to process a sequence of <math display="inline">15,000</math> contiguous tokens. Table [[#S5.T4|4]] shows that the throughput for the LSTM and the GCNN are similar. The LSTM performs very well on GPU because the large batch size of <math display="inline">750</math> enables high parallelization over different sentences. This is because the LSTM implementation has been thoroughly optimized and uses cuDNN, whereas the cuDNN implementation of convolutions is not been optimized for the 1-D convolutions we use in our model. We believe much better performance can be achieved by a more efficient 1-D cuDNN convolution. Unlike the LSTM, the GCNN can be parallelized both over sequences as well as across the tokens of each sequence, allowing the GCNN to have 20x higher responsiveness.

<!--T:92-->
<div id="S5.F4" class="figure">

<!--T:93-->
[[File:Arxiv_1612_08083_x5.png|207x138px|Refer to caption]] [[File:Arxiv_1612_08083_x6.png|207x138px|Refer to caption]]

<!--T:94-->
<div id="S5.SS2" class="section">

<!--T:95-->
<span id="gating-mechanisms-1"></span>

<!--T:96-->
=== 5.2 Gating Mechanisms ===

<!--T:97-->
In this section, we compare the gated linear unit with other mechanisms as well as to models without gating. We consider the LSTM-style gating mechanism (GTU) <math display="inline">{{\text{tanh}\hspace{0pt}{({{\mathbf{X} \ast \mathbf{W}} + \mathbf{b}})}} \otimes \sigma}\hspace{0pt}{({{\mathbf{X} \ast \mathbf{V}} + \mathbf{c}})}</math> of (Oord et al., [[#bib.bib27|2016b]]) and networks that use regular ReLU or Tanh activations. Gating units add parameters, so for fair comparison, we carefully cross-validate models with a comparable number of parameters. Figure [[#S5.F3|3]] (left) shows that GLU networks converge to a lower perplexity than the other approaches on WikiText-103. Similar to gated linear units, the ReLU has a linear path that lets the gradients easily pass through the active units. This translates to much faster convergence for both the ReLU and the GLU. On the other hand, neither Tanh nor GTU have this linear path, and thus suffer from the vanishing gradient problem. In the GTU, both the inputs as well as the gating units can cut the gradient when the units saturate.

<!--T:98-->
Comparing the GTU and Tanh models allows us to measure the effect of gating since the Tanh model can be thought of as a GTU network with the sigmoid gating units removed. The results (Figure [[#S5.F3|3]], left) show that the gating units make a vast difference and provide useful modeling capabilities, as there is a large difference in the performance between GTU and Tanh units. Similarly, while ReLU unit is not an exact ablation of the gating units in the GLU, it can be seen as a simplification <math display="inline">{\text{ReLU}\hspace{0pt}{(\mathbf{X})}} = {\mathbf{X} \otimes {({\mathbf{X} > 0})}}</math> where the gates become active depending on the sign of the input. Also in this case, GLU units lead to lower perplexity.

<!--T:99-->
In Figure [[#S5.F3|3]] (right) we repeat the same experiment on the larger Google Billion Words dataset. We consider a fixed time budget of <math display="inline">100</math> hours because of the considerable training time required for this task. Similar to WikiText-103, the gated linear units achieve the best results on this problem. There is a gap of about 5 perplexity points between the GLU and ReLU which is similar to the difference between the LSTM and RNN models measured by (Jozefowicz et al., [[#bib.bib15|2016]]) on the same dataset.

<!--T:100-->
<div id="S5.SS3" class="section">

<!--T:101-->
<span id="non-linear-modeling"></span>

<!--T:102-->
=== 5.3 Non-linear Modeling ===

<!--T:103-->
The experiments so far have shown that the gated linear unit benefits from the linear path the unit provides compared to other non-linearities. Next, we compare networks with GLUs to purely linear networks and networks with bilinear layers in order to measure the impact of the non-linear path provided by the gates of the GLU. One motivation for this experiment is the success of linear models on many natural language processing tasks (Manning &amp; Schütze, [[#bib.bib21|1999]]). We consider deep ''linear'' convolutional networks where the layers lack the gating units of the GLU and take the form <math display="inline">{h_{l}\hspace{0pt}{(\mathbf{X})}} = {{\mathbf{X} \ast \mathbf{W}} + \mathbf{b}}</math>. Stacking several layers on top of each other is simply a factorization of the model which remains linear up to the softmax, at which point it becomes log-linear. Another variation of GLUs are bilinear layers (Mnih &amp; Hinton, [[#bib.bib24|2007]]) which take the form <math display="inline">{h_{l}\hspace{0pt}{(\mathbf{X})}} = {{({{\mathbf{X} \ast \mathbf{W}} + \mathbf{b}})} \otimes {({{\mathbf{X} \ast \mathbf{V}} + \mathbf{c}})}}</math>.

<!--T:104-->
<div id="S5.F5" class="figure">

<!--T:105-->
[[File:Arxiv_1612_08083_x7.png|230x153px|Refer to caption]]

<!--T:106-->
Figure [[#S5.F5|5]] shows that GLUs perform best, followed by bilinear layers and then linear layers. Bilinear layers improve over linear ones by more than <math display="inline">40</math> perplexity points, and the GLU improves another <math display="inline">20</math> perplexity points over the bilinear model. The linear model performs very poorly at perplexity <math display="inline">115</math> even compared to <math display="inline">67.6</math> of a Kneser-Ney 5-gram model, even though the former has access to more context. Surprisingly, the introduction of the bilinear units is enough to reach <math display="inline">61</math> perplexity on Google Billion Word, which surpasses both Kneser-Ney 5-gram models and the non-linear neural model of (Ji et al., [[#bib.bib14|2015]]).

<!--T:107-->
<div id="S5.SS4" class="section">

<!--T:108-->
<span id="context-size"></span>

<!--T:109-->
=== 5.4 Context Size ===

<!--T:110-->
Figure [[#S5.F4|4]] shows the impact of context size for the gated CNN. We tried different combinations of network depth and kernel widths for each context size and chose the best performing one for each size. Generally, larger contexts improve accuracy but returns drastically diminish with windows larger than 40 words, even for WikiText-103 where we may condition on an entire Wikipedia article. This means that the unlimited context offered by recurrent models is not strictly necessary for language modeling. Furthermore, this finding is also congruent with the fact that good performance with recurrent networks can be obtained by truncating gradients after only 40 timesteps using truncated back propagation through time. Figure [[#S5.F4|4]] also shows that WikiText-103 benefits much more from larger context size than Google Billion Word as the performance degrades more sharply with smaller contexts. WikiText-103 provides much more context than Google Billion Word where the average sentence size is 20. However, while the average size of the documents is close to 4000 tokens, we find that strong performance can be achieved with a context size as low as 30 tokens.

<!--T:111-->
<div id="S5.SS5" class="section">

<!--T:112-->
<span id="training-1"></span>

<!--T:113-->
=== 5.5 Training ===

<!--T:114-->
In this section, we perform an ablation study of the impact of weight normalization and gradient clipping. We separately cross-validate the hyper-parameters of each configuration to make the comparison fair. Due to the high cost of each of these experiments, we only consider a single iteration over the training data. Figure [[#S5.F6|6]] shows that both methods significantly speed up convergence. Weight normalization in particular improves the speed by over two times. This speedup is partly due to the ability to use much larger learning rates (<math display="inline">1</math> instead of <math display="inline">0.01</math>) than would otherwise be possible. Both clipping and weight normalization add computational overhead, but it is minor compared to the large gains in convergence speed.

<!--T:115-->
<div id="S5.F6" class="figure">

<!--T:116-->
[[File:Arxiv_1612_08083_x8.png|230x153px|Refer to caption]]

<!--T:117-->
<div id="S6" class="section">

<!--T:118-->
<span id="conclusion"></span>

<!--T:119-->
== 6 Conclusion ==

<!--T:120-->
We introduce a convolutional neural network for language modeling with a novel gating mechanism. Compared to recurrent neural networks, our approach builds a hierarchical representation of the input words that makes it easier to capture long-range dependencies, similar in spirit to the tree-structured analysis of linguistic grammar formalisms. The same property eases learning since features are passed through a fixed number of layers and non-linearities, unlike for recurrent networks where the number of processing steps differs depending on the position of the word in the input. The results show that our gated convolutional network achieves a new state of the art on WikiText-103. On the Google Billion Word benchmark, we show competitive results can be achieved with significantly fewer resources.

<!--T:121-->
<div id="Sx1" class="section">

<!--T:122-->
<span id="acknowledgments"></span>

<!--T:123-->
== Acknowledgments ==

<!--T:124-->
We would like to thank Ben Graham, Jonas Gehring, Edouard Grave, Armand Joulin and Ronan Collobert for helpful discussions.

<!--T:125-->
<div id="bib" class="section">

<!--T:126-->
<span id="references"></span>

<!--T:127-->
== References ==

<!--T:128-->
* <span id="bib.bib1">Bengio et al. (2003) Bengio, Yoshua, Ducharme, Réjean, Vincent, Pascal, and Jauvin, Christian. A neural probabilistic language model. ''journal of machine learning research'', 3(Feb):1137–1155, 2003.</span>
* <span id="bib.bib2">Chelba et al. (2013) Chelba, Ciprian, Mikolov, Tomas, Schuster, Mike, Ge, Qi, Brants, Thorsten, Koehn, Phillipp, and Robinson, Tony. One billion word benchmark for measuring progress in statistical language modeling. ''arXiv preprint arXiv:1312.3005'', 2013.</span>
* <span id="bib.bib3">Chen &amp; Goodman (1996) Chen, Stanley F and Goodman, Joshua. An empirical study of smoothing techniques for language modeling. In ''Proceedings of the 34th annual meeting on Association for Computational Linguistics'', pp.  310–318. Association for Computational Linguistics, 1996.</span>
* <span id="bib.bib4">Chen et al. (2016) Chen, Wenlin, Grangier, David, and Auli, Michael. Strategies for training large vocabulary neural language models. ''CoRR'', abs/1512.04906, 2016.</span>
* <span id="bib.bib5">Collobert et al. (2011) Collobert, Ronan, Kavukcuoglu, Koray, and Farabet, Clement. Torch7: A Matlab-like Environment for Machine Learning. In ''BigLearn, NIPS Workshop'', 2011. URL http://torch.ch.</span>
* <span id="bib.bib6">Dauphin &amp; Grangier (2015) Dauphin, Yann N and Grangier, David. Predicting distributions with linearizing belief networks. ''arXiv preprint arXiv:1511.05622'', 2015.</span>
* <span id="bib.bib7">Glorot &amp; Bengio (2010) Glorot, Xavier and Bengio, Yoshua. Understanding the difficulty of training deep feedforward neural networks. ''The handbook of brain theory and neural networks'', 2010.</span>
* <span id="bib.bib8">Grave et al. (2016a) Grave, E., Joulin, A., Cissé, M., Grangier, D., and Jégou, H. Efficient softmax approximation for GPUs. ''ArXiv e-prints'', September 2016a.</span>
* <span id="bib.bib9">Grave et al. (2016b) Grave, E., Joulin, A., and Usunier, N. Improving Neural Language Models with a Continuous Cache. ''ArXiv e-prints'', December 2016b.</span>
* <span id="bib.bib10">(10) Gutmann, Michael and Hyvärinen, Aapo. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.</span>
* <span id="bib.bib11">He et al. (2015a) He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Deep residual learning for image recognition. ''arXiv preprint arXiv:1512.03385'', 2015a.</span>
* <span id="bib.bib12">He et al. (2015b) He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ''Proceedings of the IEEE International Conference on Computer Vision'', pp.  1026–1034, 2015b.</span>
* <span id="bib.bib13">Hochreiter &amp; Schmidhuber (1997) Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. ''Neural computation'', 9(8):1735–1780, 1997.</span>
* <span id="bib.bib14">Ji et al. (2015) Ji, Shihao, Vishwanathan, SVN, Satish, Nadathur, Anderson, Michael J, and Dubey, Pradeep. Blackout: Speeding up recurrent neural network language models with very large vocabularies. ''arXiv preprint arXiv:1511.06909'', 2015.</span>
* <span id="bib.bib15">Jozefowicz et al. (2016) Jozefowicz, Rafal, Vinyals, Oriol, Schuster, Mike, Shazeer, Noam, and Wu, Yonghui. Exploring the limits of language modeling. ''arXiv preprint arXiv:1602.02410'', 2016.</span>
* <span id="bib.bib16">Kalchbrenner et al. (2016) Kalchbrenner, Nal, Espeholt, Lasse, Simonyan, Karen, van den Oord, Aaron, Graves, Alex, and Kavukcuoglu, Koray. Neural Machine Translation in Linear Time. ''arXiv'', 2016.</span>
* <span id="bib.bib17">Kneser &amp; Ney (1995) Kneser, Reinhard and Ney, Hermann. Improved backing-off for m-gram language modeling. In ''Acoustics, Speech, and Signal Processing, 1995. ICASSP-95., 1995 International Conference on'', volume 1, pp.  181–184. IEEE, 1995.</span>
* <span id="bib.bib18">Koehn (2010) Koehn, Philipp. ''Statistical Machine Translation''. Cambridge University Press, New York, NY, USA, 1st edition, 2010. ISBN 0521874157, 9780521874151.</span>
* <span id="bib.bib19">Kuchaiev &amp; Ginsburg (2017) Kuchaiev, Oleksii and Ginsburg, Boris. Factorization tricks for LSTM networks. ''CoRR'', abs/1703.10722, 2017. URL http://arxiv.org/abs/1703.10722.</span>
* <span id="bib.bib20">LeCun &amp; Bengio (1995) LeCun, Yann and Bengio, Yoshua. Convolutional networks for images, speech, and time series. ''The handbook of brain theory and neural networks'', 3361(10):1995, 1995.</span>
* <span id="bib.bib21">Manning &amp; Schütze (1999) Manning, Christopher D and Schütze, Hinrich. Foundations of statistical natural language processing, 1999.</span>
* <span id="bib.bib22">Merity et al. (2016) Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer Sentinel Mixture Models. ''ArXiv e-prints'', September 2016.</span>
* <span id="bib.bib23">Mikolov et al. (2010) Mikolov, Tomáš, Martin, Karafiát, Burget, Lukáš, Cernocký, Jan, and Khudanpur, Sanjeev. Recurrent Neural Network based Language Model. In ''Proc. of INTERSPEECH'', pp.  1045–1048, 2010.</span>
* <span id="bib.bib24">Mnih &amp; Hinton (2007) Mnih, Andriy and Hinton, Geoffrey. Three new graphical models for statistical language modelling. In ''Proceedings of the 24th international conference on Machine learning'', pp.  641–648. ACM, 2007.</span>
* <span id="bib.bib25">Morin &amp; Bengio (2005) Morin, Frederic and Bengio, Yoshua. Hierarchical probabilistic neural network language model. In ''Aistats'', volume 5, pp.  246–252. Citeseer, 2005.</span>
* <span id="bib.bib26">Oord et al. (2016a) Oord, Aaron van den, Kalchbrenner, Nal, and Kavukcuoglu, Koray. Pixel recurrent neural networks. ''arXiv preprint arXiv:1601.06759'', 2016a.</span>
* <span id="bib.bib27">Oord et al. (2016b) Oord, Aaron van den, Kalchbrenner, Nal, Vinyals, Oriol, Espeholt, Lasse, Graves, Alex, and Kavukcuoglu, Koray. Conditional image generation with pixelcnn decoders. ''arXiv preprint arXiv:1606.05328'', 2016b.</span>
* <span id="bib.bib28">Pascanu et al. (2013) Pascanu, Razvan, Mikolov, Tomas, and Bengio, Yoshua. On the difficulty of training recurrent neural networks. In ''Proceedings of The 30th International Conference on Machine Learning'', pp.  1310–1318, 2013.</span>
* <span id="bib.bib29">Salimans &amp; Kingma (2016) Salimans, Tim and Kingma, Diederik P. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. ''arXiv preprint arXiv:1602.07868'', 2016.</span>
* <span id="bib.bib30">Shazeer et al. (2014) Shazeer, Noam, Pelemans, Joris, and Chelba, Ciprian. Skip-gram language modeling using sparse non-negative matrix probability estimation. ''arXiv preprint arXiv:1412.1454'', 2014.</span>
* <span id="bib.bib31">Shazeer et al. (2017) Shazeer, Noam, Mirhoseini, Azalia, Maziarz, Krzysztof, Davis, Andy, Le, Quoc V., Hinton, Geoffrey E., and Dean, Jeff. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. ''CoRR'', abs/1701.06538, 2017. URL http://arxiv.org/abs/1701.06538.</span>
* <span id="bib.bib32">Steedman (2002) Steedman, Mark. The syntactic process. 2002.</span>
* <span id="bib.bib33">Sutskever et al. (2013) Sutskever, Ilya, Martens, James, Dahl, George E, and Hinton, Geoffrey E. On the importance of initialization and momentum in deep learning. 2013.</span>
* <span id="bib.bib34">Wang et al. (2015) Wang, Mingxuan, Lu, Zhengdong, Li, Hang, Jiang, Wenbin, and Liu, Qun. ''gen''cnn: A convolutional architecture for word sequence prediction. ''CoRR'', abs/1503.05034, 2015. URL http://arxiv.org/abs/1503.05034.</span>
* <span id="bib.bib35">Yu &amp; Deng (2014) Yu, Dong and Deng, Li. ''Automatic Speech Recognition: A Deep Learning Approach''. Springer Publishing Company, Incorporated, 2014. ISBN 1447157788, 9781447157786.</span>
</translate>

[[Category:NLP]]
[[Category:Research]]
[[Category:Research Papers]]