View source for Deep & Cross Network for Ad Click Predictions

<languages />
{{PaperTabs}}
{{PaperInfobox
 | topic_area  = Machine Learning
 | difficulty  = Research
 | authors     = Ruoxi Wang; Bin Fu; Gang Fu; Mingliang Wang
 | year        = 2017
 | arxiv_id    = 1708.05123
 | source_url  = https://arxiv.org/abs/1708.05123
 | pdf_url     = https://arxiv.org/pdf/1708.05123.pdf
}}
{{ContentMeta
 | generated_by   = claude-code-direct
 | model_used     = claude-opus-4-7
 | generated_date = 2026-04-27
}}

<translate>
<!--T:1-->
'''Deep & Cross Network for Ad Click Predictions''' (DCN) is a 2017 neural network architecture proposed by Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang of Google and Stanford for click-through rate (CTR) prediction. It augments a standard deep neural network with a parallel ''cross network'' that explicitly composes feature interactions of bounded degree at each layer, learning all polynomial cross terms up to a user-specified order with a parameter count that grows only linearly in the input dimension.

== Overview == <!--T:2-->

<!--T:3-->
CTR prediction underpins billions of dollars of online advertising revenue but operates over feature spaces that are massive, sparse, and overwhelmingly categorical. Linear models scale well and are interpretable but cannot capture the cross-feature signal that drives accuracy; pure deep neural networks (DNNs) can in principle learn arbitrary functions but represent feature crosses only implicitly through stacked nonlinearities, often inefficiently.

<!--T:4-->
DCN sits between these two regimes. After embedding sparse categorical inputs into low-dimensional dense vectors and stacking them with normalized continuous features, the model splits into two parallel branches: a ''cross network'' that applies an explicit, residual-style feature-crossing operation at every layer, and a standard ''deep network'' of fully connected ReLU layers. Their outputs are concatenated and passed through a logistic head trained with log loss. The cross network adds only <math>O(d \cdot L_c)</math> parameters on top of the DNN, where <math>d</math> is the embedded input dimension and <math>L_c</math> is the number of cross layers, yet captures all cross terms up to degree <math>L_c + 1</math>.

== Key Contributions == <!--T:5-->

<!--T:6-->
* A novel '''cross network''' that applies explicit feature crossing at every layer, with the highest polynomial degree of the represented interactions provably equal to the layer depth plus one.
* A '''joint architecture''' that trains the cross network in parallel with a DNN, combining bounded-degree explicit crosses with deep implicit nonlinearities under one log-loss objective.
* A '''theoretical analysis''' showing that the cross network reproduces all multinomial cross terms of bounded degree, generalizes factorization machines (FMs) from a single shallow interaction to a stack of high-order ones, and projects the implicit <math>d^2</math> pairwise interactions back to dimension <math>d</math> in linear time and memory.
* '''Empirical gains on Criteo Display Ads''' — the standard public CTR benchmark — together with strong results on the UCI ''forest covertype'' and ''Higgs'' datasets, showing that DCN matches or beats deep baselines while using substantially less memory.

== Methods == <!--T:7-->

<!--T:8-->
The DCN model is composed of four stages: an embedding and stacking layer, the cross network, the deep network, and a combination layer.

<!--T:9-->
'''Embedding and stacking.''' Each sparse categorical input <math>\mathbf{x}_i</math> is mapped through a learned matrix <math>W_{\text{embed},i} \in \mathbb{R}^{n_e \times n_v}</math> to a dense vector. Embedded categorical features are concatenated with normalized dense features <math>\mathbf{x}_{\text{dense}}</math> into a single vector <math>\mathbf{x}_0</math> that feeds both branches.

<!--T:10-->
'''Cross network.''' Let <math>\mathbf{x}_l \in \mathbb{R}^d</math> denote the output of cross layer <math>l</math>. Each layer applies

<!--T:11-->
:<math>\mathbf{x}_{l+1} = \mathbf{x}_0 \mathbf{x}_l^{T} \mathbf{w}_l + \mathbf{b}_l + \mathbf{x}_l,</math>

<!--T:12-->
where <math>\mathbf{w}_l, \mathbf{b}_l \in \mathbb{R}^d</math>. The outer-product term <math>\mathbf{x}_0 \mathbf{x}_l^{T}</math> creates pairwise interactions between the original input and the current state; the residual connection preserves lower-order signal. A theorem in the paper establishes that an <math>l</math>-layer cross network contains every cross term <math>x_1^{\alpha_1} x_2^{\alpha_2} \cdots x_d^{\alpha_d}</math> of degree <math>1 \le |\boldsymbol{\alpha}| \le l + 1</math>, each with a distinct coefficient determined by the weights <math>\{\mathbf{w}_k\}</math>.

<!--T:13-->
'''Deep network.''' A standard fully connected feed-forward stack with ReLU activations:

<!--T:14-->
:<math>\mathbf{h}_{l+1} = f(W_l \mathbf{h}_l + \mathbf{b}_l).</math>

<!--T:15-->
'''Combination layer.''' The final cross-network output <math>\mathbf{x}_{L_1}</math> and deep-network output <math>\mathbf{h}_{L_2}</math> are concatenated and passed through a logistic head:

<!--T:16-->
:<math>p = \sigma\!\left(\mathbf{w}_{\text{logits}}^{T} [\mathbf{x}_{L_1};\, \mathbf{h}_{L_2}]\right),\qquad \sigma(x) = \frac{1}{1 + e^{-x}}.</math>

<!--T:17-->
The training loss is the regularized log loss

<!--T:18-->
:<math>\mathcal{L} = -\frac{1}{N}\sum_{i=1}^{N} \big[ y_i \log p_i + (1 - y_i) \log(1 - p_i) \big] + \lambda \|\mathbf{w}\|^2.</math>

<!--T:19-->
'''Efficient projection.''' A direct construction of all <math>d^2</math> pairwise interactions would be cubic in <math>d</math>; the cross-layer formula collapses this to <math>O(d)</math> work and parameters per layer because <math>\mathbf{x}_0 \mathbf{x}_l^{T} \mathbf{w}_l</math> can be computed as <math>\mathbf{x}_0 (\mathbf{x}_l^{T} \mathbf{w}_l)</math> — a vector–scalar product.

<!--T:20-->
'''Connection to FMs.''' In an FM, each feature <math>x_i</math> carries a vector <math>\mathbf{v}_i</math> and the weight of <math>x_i x_j</math> is <math>\langle \mathbf{v}_i, \mathbf{v}_j \rangle</math>. In DCN the analogous parameters are scalars <math>\{w_k^{(i)}\}_{k=1}^{l}</math>, and the weight of <math>x_i x_j</math> is a product across cross layers. DCN therefore extends FM's parameter sharing from a single second-order interaction to arbitrary-degree interactions across multiple layers.

== Results == <!--T:21-->

<!--T:22-->
'''Criteo Display Ads.''' On the public Criteo CTR challenge (about 41 million records, 13 integer and 26 categorical features), DCN achieved a test log loss of '''0.4422 ± 9 × 10⁻⁵''', compared with 0.4430 ± 3.7 × 10⁻⁴ for a tuned DNN, 0.4430 ± 4.3 × 10⁻⁴ for Deep Crossing (DC), and weaker results for logistic regression, FMs, and Wide & Deep. The optimal DCN used 6 cross layers and 2 deep layers of size 1024; that the deepest cross configuration won supports the claim that higher-order explicit interactions are valuable. In follow-up sweeps over memory budget and loss tolerance, DCN matched DNN accuracy while using roughly 40% fewer parameters, and matched the best DNN log loss with about an order-of-magnitude smaller deep stack.

<!--T:23-->
'''Non-CTR datasets.''' On UCI ''forest covertype'' (581k samples, 54 features), DCN reached 0.9740 test accuracy versus 0.9737 for DNN and DC, with the smallest memory footprint. On ''Higgs'' (11M samples, 28 features), DCN obtained log loss 0.4494 against 0.4506 for DNN, while using roughly half the parameters.

== Impact == <!--T:24-->

<!--T:25-->
DCN became one of the canonical baselines for deep CTR and recommender models, alongside Wide & Deep and DeepFM. Its core idea — a parameter-efficient module that performs explicit, higher-order feature crossing alongside a DNN — was widely adopted in industry feature-interaction modeling, and the original cross-layer formulation was later refined into '''DCN-V2''' (Wang et al., 2021) using a full weight matrix per cross layer for greater expressiveness at production scale at Google. Beyond advertising, the architecture's strong showing on dense classification tasks helped popularize parallel "explicit + implicit" feature-interaction designs in tabular deep learning.

== See also == <!--T:26-->

<!--T:27-->
* [[Wide & Deep Learning for Recommender Systems]]
* [[Factorization Machines]]
* [[Deep Crossing]]
* [[Batch Normalization: Accelerating Deep Network Training]]
* [[Adam: A Method for Stochastic Optimization]]
* [[Click-through rate]]

== References == <!--T:28-->

<!--T:29-->
* Wang, R., Fu, B., Fu, G., & Wang, M. (2017). Deep & Cross Network for Ad Click Predictions. ''Proceedings of the ADKDD'17''. arXiv:1708.05123.
* Cheng, H.-T. ''et al.'' (2016). Wide & Deep Learning for Recommender Systems. ''DLRS''.
* Rendle, S. (2010). Factorization Machines. ''ICDM''.
* Shan, Y. ''et al.'' (2016). Deep Crossing: Web-Scale Modeling without Manually Crafted Combinatorial Features. ''KDD''.
* Ioffe, S., & Szegedy, C. (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. ''ICML''.
* Kingma, D. P., & Ba, J. (2014). Adam: A Method for Stochastic Optimization. ''ICLR''.
* Wang, R. ''et al.'' (2021). DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems. ''WWW''.
</translate>

[[Category:Machine Learning]]
[[Category:Research]]
[[Category:Research Papers]]