View source for MTGR: Industrial-Scale Generative Recommendation Framework in Meituan/paper

<languages />
{{PaperTabs}}
{{PaperInfobox
 | topic_area  = NLP
 | difficulty  = Research
 | authors     = Ruidong Han; Bin Yin; Shangyu Chen; He Jiang; Fei Jiang; Xiang Li; Chi Ma; Mincong Huang; Xiaoguang Li; Chunzhen Jing; Yueming Han; Menglei Zhou; Lei Yu; Chuan Liu; Wei Lin
 | year        = 2025
 | arxiv_id    = 2505.18654
 | source_url  = https://arxiv.org/abs/2505.18654
 | pdf_url     = https://arxiv.org/pdf/2505.18654.pdf
}}
{{ContentMeta
 | generated_by   = arxiv-importer-raw
 | model_used     = strategy:html5
 | generated_date = 2026-04-27
}}

<translate>
<!--T:1-->
<span id="mtgr-industrial-scale-generative-recommendation-framework-in-meituan"></span>

<!--T:2-->
= MTGR: Industrial-Scale Generative Recommendation Framework in Meituan =

<!--T:3-->
Ruidong Han MeituanBeijingChina [mailto:hanruidong@meituan.com hanruidong@meituan.com] ,  Bin Yin MeituanBeijingChina [mailto:yinbin05@meituan.com%20 yinbin05@meituan.com] ,  Shangyu Chen MeituanBeijingChina [mailto:chenshangyu03@meituan.com chenshangyu03@meituan.com] ,  He Jiang MeituanBeijingChina [mailto:jianghe06@meituan.com jianghe06@meituan.com] ,  Fei Jiang MeituanBeijingChina [mailto:jiangfei05@meituan.com jiangfei05@meituan.com] ,  Xiang Li MeituanBeijingChina [mailto:lixiang245@meituan.com lixiang245@meituan.com] ,  Chi Ma MeituanBeijingChina [mailto:machi04@meituan.com machi04@meituan.com] ,  Mincong Huang MeituanBeijingChina [mailto:huangmincong@meituan.com huangmincong@meituan.com] ,  Xiaoguang Li MeituanBeijingChina [mailto:lixiaoguang12@meituan.com lixiaoguang12@meituan.com] ,  Chunzhen Jing MeituanBeijingChina [mailto:jingchunzhen@meituan.com jingchunzhen@meituan.com] ,  Yueming Han MeituanBeijingChina [mailto:hanyueming02@meituan.com hanyueming02@meituan.com] ,  Menglei Zhou MeituanBeijingChina [mailto:zhoumenglei@meituan.com zhoumenglei@meituan.com] ,  Lei Yu MeituanBeijingChina [mailto:yulei37@meituan.com yulei37@meituan.com] ,  Chuan Liu MeituanBeijingChina [mailto:liuchuan11@meituan.com liuchuan11@meituan.com]  and  Wei Lin MeituanBeijingChina [mailto:linwei31@meituan.com linwei31@meituan.com]

<!--T:4-->
<span id="abstract."></span>

<!--T:5-->
=== Abstract. ===

<!--T:6-->
Scaling law has been extensively validated in many domains such as natural language processing and computer vision. In the recommendation system, recent work has adopted generative recommendations to achieve scalability, but their generative approaches require abandoning the carefully constructed cross features of traditional recommendation models. We found that this approach significantly degrades model performance, and scaling up cannot compensate for it at all. In this paper, we propose MTGR (Meituan Generative Recommendation) to address this issue. MTGR is modeling based on the HSTU (Zhai et al., [[#bib.bib23|2024]]) architecture and can retain the original deep learning recommendation model (DLRM) features, including cross features. Additionally, MTGR achieves training and inference acceleration through user-level compression to ensure efficient scaling. We also propose Group-Layer Normalization (GLN) to enhance the performance of encoding within different semantic spaces and the dynamic masking strategy to avoid information leakage. We further optimize the training frameworks, enabling support for our models with 10 to 100 times computational complexity compared to the DLRM, without significant cost increases. MTGR achieved <math display="inline">65</math>x FLOPs for single-sample forward inference compared to the DLRM model, resulting in the largest gain in nearly two years both offline and online. This breakthrough was successfully deployed on Meituan, the world’s largest food delivery platform, where it has been handling the main traffic.

<!--T:7-->
Scaling Law; Generative Recommendation <sup>††</sup>ccs: Information systems Recommendation systems

<!--T:8-->
<div id="S1" class="section">

<!--T:9-->
<span id="introduction"></span>

<!--T:10-->
== 1. Introduction ==

<!--T:11-->
Scaling law have been proven to apply to most deep learning tasks, including language models(Kaplan et al., [[#bib.bib10|2020]]), computer vision(Zhai et al., [[#bib.bib24|2022]]; Peebles and Xie, [[#bib.bib15|2023]]), and information retrieval(Fang et al., [[#bib.bib5|2024]]). Under the requirements of high QPS (Queries Per Second) and low latency in industrial recommendation systems, the scaling of the model is usually limited by both the training cost and inference time. Currently, the study on scaling ranking models can be divided into two types: Deep Learning Recommendation Model (DLRM) and Generative Recommendation Model (GRM). DLRM models individual user-item pairs to learn the probability of interest for ranking and scales up by developing more complex mappings. GRM organizes data by token like natural language and performs next token prediction through a transformer architecture.

<!--T:12-->
In industrial recommendation systems, DLRM has been used for nearly a decade, typically with inputs that include a large number of well-designed handcrafted features such as cross features <sup>11</sup>1Cross features measure the interactions of multiple raw features, like the user’s historical click-through rate for the target candidate to improve model performance. However, DLRM has two significant drawbacks when it comes to scaling: 1) with exponential growth of user behavior, traditional DLRM cannot efficiently process entire user behaviors, often resorting to sequence retrieval or designing low-complexity modules for learning, which limits the model’s learning capability (Pi et al., [[#bib.bib16|2020]]; Chen et al., [[#bib.bib2|2021]]). 2) scaling based on DLRM incurs approximately linear costs in training and inference with the number of candidates, making the expenses unbearably high.

<!--T:13-->
For GRM, recent studies have pointed out its excellent scalability (Zhai et al., [[#bib.bib23|2024]]; Deng et al., [[#bib.bib4|2025]]). We identify two key factors: 1) GRM directly models the complete chain of user behavior, which compresses multiple samples of exposure under the same user into one. This significantly reduces computational redundancy, while allowing end-to-end encoding of longer sequence compared to DLRM; 2) GRM adopts a transformer architecture with efficient attention computation(Zhai et al., [[#bib.bib23|2024]]; Dao et al., [[#bib.bib3|2022]]), enabling the model’s training and inference to meet the requirements of industrial recommendation systems. However, GRM heavily relies on next token prediction to model a complete user behavior sequence, which requires removing cross features between candidates and the user. We found that excluding cross features severely damages the model’s performance, and this degeneration cannot be compensated by scaling up at all.

<!--T:14-->
How can we build a ranking model that utilizes cross features to ensure effectiveness while also possessing the scalability of GRM? To address this problem, we propose the Meituan Generative Recommendation (MTGR). Compared to traditional DLRM and GRM, MTGR utilizes the advantages and discards disadvantage of the methods. MTGR maintains inputs feature consistent with traditional DLRM, including cross features. Specially, MTGR reorganizes features by converting user and candidate features into different tokens, leading to a token sequence for efficient model scaling. Then MTGR incorporates the cross feature in the candidate tokens and learns with a discriminative loss.

<!--T:15-->
MTGR employs the similar HSTU (Hierarchical Sequential Transduction Units) architecture as used in (Zhai et al., [[#bib.bib23|2024]]) for modeling. In HSTU, we propose Group-layer Normalization(GLN) to normalize different types of token separately, enabling better modeling of multiple heterogeneous information simultaneously. In addition, we propose a dynamic masking strategy, where full-attention, auto-regressive and visibility only to itself are used to ensure performance and avoid information leakage.

<!--T:16-->
Unlike the commonly used TensorFlow in the industry, MTGR training framework is built based on TorchRec(Ivchenko et al., [[#bib.bib9|2022]]) and optimized for computational efficiency. Specifically, to handle the real-time insert/delete of sparse embedding entries, we employ dynamic hash tables to replace static tables. To improve efficiency, we conduct dynamic sequence balancing to address the computation load imbalances among GPUs and adopt embedding ID de-duplication alongside automatic table merging to accelerate embedding lookup. We also incorporate implementation optimization including mixed precision training and operator fusion. Compared to TorchRec, our optimized framework improves training throughput by <math display="inline">1.6</math>x – <math display="inline">2.4</math>x while achieving good scalability when running over <math display="inline">100</math> GPUs.

<!--T:17-->
We validate the scalability of MTGR on a small-scale dataset. Then, we design three models of varying sizes, which are trained using following over six months of data, to verify the scaling law for both offline and online performance. The large version, compared to the DLRM baseline optimized over years, demonstrates <math display="inline">65</math>x FLOPs per sample for forward computation and increases the conversion volumes by <math display="inline">1.22\%</math> and the CTR (Click-Through Rate) by <math display="inline">1.31\%</math> in our scenarios. Meanwhile, the training cost remains unchanged and the inference cost is reduced by 12%. MTGR-large has been deployed in the Meituan take-away recommendation system, serving hundreds of millions of users.

<!--T:18-->
In summary, our contributions are summarized as follows:

<!--T:19-->
<ul>
<li><p><span id="S1.I1.i1">•</span></p>
<p>MTGR combines the advantages of DLRM and GRM, retaining all the features of DLRM, including the cross feature, while demonstrating excellent scalability as GRM.</p></li>
<li><p><span id="S1.I1.i2">•</span></p>
<p>We propose Group-Layer Normalization and dynamic masking strategies to achieve better performance.</p></li>
<li><p><span id="S1.I1.i3">•</span></p>
<p>We perform systematic optimizations on the TorchRec-based MTGR training framework to enhance training performance.</p></li>
<li><p><span id="S1.I1.i4">•</span></p>
<p>Both offline and online experiments were conducted to demonstrate the power-law relationship between the performance of MTGR and computational complexity, and its superiority compared to DLRM.</p></li></ul>

<!--T:20-->
<div id="S2" class="section">

<!--T:21-->
<span id="related-works"></span>

<!--T:22-->
== 2. Related Works ==

<!--T:23-->
<div id="S2.SS1" class="section">

<!--T:24-->
<span id="deep-learning-recommendation-model"></span>

<!--T:25-->
=== 2.1. Deep Learning Recommendation Model ===

<!--T:26-->
A classic DLRM structure typically includes many inputs such as context (e.g. time, location), user profile (e.g. gender, age), user behavior sequences, and target item with many cross features. Two particularly important modules in ranking model are behavior sequence processing and feature interactions learning. The behavior sequence module(Zhou et al., [[#bib.bib27|2018]]; Pi et al., [[#bib.bib16|2020]]; Si et al., [[#bib.bib18|2024]]) usually employs target attention mechanisms to capture the similarity between the historical behavior of the users and the item to be estimated. The feature interactions module(Lian et al., [[#bib.bib13|2018]]; Tang et al., [[#bib.bib19|2020]]; Wang et al., [[#bib.bib20|2021]], [[#bib.bib21|2024]]) is designed to capture interactions among different features including user and item to produce the final prediction.

<!--T:27-->
<div id="S2.SS2" class="section">

<!--T:28-->
<span id="scaling-up-recommendation-model"></span>

<!--T:29-->
=== 2.2. Scaling up Recommendation Model ===

<!--T:30-->
Based on the different scaling modules in DLRM, there are two distinct approaches. One approach is the scaling cross module, i.e. scaling up the feature interactions module that integrates user and item information. (Zhang et al., [[#bib.bib25|2024b]]) introduces a stackable Wukong layer for scaling up. (Guo et al., [[#bib.bib7|2023]]) employ a multi-embedding strategy to address the embedding collapse, thereby enhancing the model’s scalability. The other approach is the scaling user module, where only the user part is scaled up, making this approach more inference-friendly. (Zhang et al., [[#bib.bib26|2024a]]; Han et al., [[#bib.bib8|2024]]) reduce online inference costs by scaling only user representations and caching or broadcasting them to different items that will be estimated. (Shin et al., [[#bib.bib17|2023]]; Yan et al., [[#bib.bib22|2025]]) design a pre-training method for user representations, demonstrating scalability in downstream tasks.

<!--T:31-->
The counterpart to DLRM is GRM. (Zhai et al., [[#bib.bib23|2024]]) validate the scaling law through HSTU with scaling up to trillion-level parameters. (Deng et al., [[#bib.bib4|2025]]) use semantic coding to replace traditional ID representations, combining DPO optimization with HSTU to replace the cascaded learning framework with a unified generative model.

<!--T:32-->
<div id="S3" class="section">

<!--T:33-->
<span id="preliminary"></span>

<!--T:34-->
== 3. Preliminary ==

<!--T:35-->
<div id="S3.SS1" class="section">

<!--T:36-->
<span id="data-arrangement"></span>

<!--T:37-->
=== 3.1. Data Arrangement ===

<!--T:38-->
Traditionally, for a user and the corresponding <math display="inline">K</math> candidates, the <math display="inline">i</math>-th sample for the user and <math display="inline">i</math>-th candidate pair can be represented as <math display="inline">{\mathbb{D}}_{i} = {\lbrack\mathbf{U},\overset{\rightarrow}{\mathbf{S}},\overset{\rightarrow}{\mathbf{R}},\mathbf{C}{}_{}^{}\mathbf{I}{}_{}^{}}</math>. Specifically, <math display="inline">\mathbf{U} = {\lbrack\mathbf{U}{}_{}^{}\ldots,\mathbf{U}{}_{}^{N\mathbf{U}}}</math> represents the user’s profile feature (<math display="inline">\mathbf{U}^{i}</math>), such as age, gender, etc. Each feature <math display="inline">\mathbf{U}^{i}</math> is a scalar, and <math display="inline">N_{\mathbf{U}}</math> represents the number of features used. <math display="inline">\overset{\rightarrow}{\mathbf{S}} = {\lbrack\mathbf{S}{}_{}^{}\ldots,\mathbf{S}{}_{}^{N\mathbf{S}}}</math> contains the sequence of items that have historically been interacted with by the user. Each element in <math display="inline">\mathbf{S}{}_{}^{}{\lbrack\mathbf{s}{}_{}^{}\ldots,\mathbf{s}{}_{}^{N\mathbf{s}}}</math> represents an item, it is comprised by selected features (<math display="inline">\mathbf{s}^{i}</math>) such as the item’s ID, tag, average CTR on the item and etc. Similar to <math display="inline">\overset{\rightarrow}{\mathbf{S}}</math>, <math display="inline">\overset{\rightarrow}{\mathbf{R}}</math> records the closet interaction to current request within a few hours or a day. <math display="inline">\overset{\rightarrow}{\mathbf{R}}</math> represents user’s real-time actions and preference. It shares the same features with <math display="inline">\overset{\rightarrow}{\mathbf{S}}</math>. <math display="inline">\mathbf{C} = {\lbrack\mathbf{C}{}_{}^{}\ldots,\mathbf{C}{}_{}^{N\mathbf{C}}}</math> comprises the cross features between the user and the candidates. <math display="inline">\mathbf{I} = {\lbrack\mathbf{I}{}_{}^{}\ldots,\mathbf{I}{}_{}^{N\mathbf{I}}}</math> contains the features used for candidates, such as item’s ID, tag, and brand. <math display="inline">\mathbf{I}</math> is candidate-dependent and shared for different users.

<!--T:39-->
<div id="S3.SS2" class="section">

<!--T:40-->
<span id="ranking-model-in-recommendation-systems"></span>

<!--T:41-->
=== 3.2. Ranking Model in Recommendation Systems ===

<!--T:42-->
<div id="S3.F1" class="figure">

<!--T:43-->
[[File:Arxiv_2505_18654_x1.png|830x846px|Refer to caption]]

<!--T:44-->
With input samples <math display="inline">\mathbb{D}</math>, traditional recommendation system processes the samples independently. Specially, it firstly embeds the features in <math display="inline">\mathbb{D}</math> and converts the samples to a dense representation. Formally, the features in <math display="inline">\mathbf{U}</math>, <math display="inline">\mathbf{C}</math>, <math display="inline">\mathbf{I}</math> are embedded and concatenated to <math display="inline">{\mathbf{E}\mathbf{m}\mathbf{b}}{}_{}^{}{\mathbb{R}}^{K \times d_{\mathbf{U}}}</math>, <math display="inline">{\mathbf{E}\mathbf{m}\mathbf{b}}{}_{}^{}{\mathbb{R}}^{K \times d_{\mathbf{C}}}</math> and <math display="inline">{\mathbf{E}\mathbf{m}\mathbf{b}}{}_{}^{}{\mathbb{R}}^{K \times d_{\mathbf{I}}}</math>, respectively. For features in <math display="inline">\overset{\rightarrow}{\mathbf{S}}</math> and <math display="inline">\overset{\rightarrow}{\mathbf{R}}</math> <sup>22</sup>2In the following description, since <math display="inline">\overset{\rightarrow}{\mathbf{S}}</math> and <math display="inline">\overset{\rightarrow}{\mathbf{R}}</math> are processed in a similar way, only <math display="inline">\overset{\rightarrow}{\mathbf{S}}</math>’s modeling is described for clarity., each item(<math display="inline">\mathbf{S}^{i}</math>)’s feature are similarly embedded and concatenated to <math display="inline">{\mathbf{E}\mathbf{m}\mathbf{b}}_{\mathbf{S}^{i}} \in {\mathbb{R}}^{d_{\mathbf{s}}}</math>, items in <math display="inline">\overset{\rightarrow}{\mathbf{S}}</math> are concatenated in other dimension, leading to <math display="inline">{\mathbf{E}\mathbf{m}\mathbf{b}}_{\overset{\rightarrow}{\mathbf{S}}} \in {\mathbb{R}}^{N_{\overset{\rightarrow}{\mathbf{S}}} \times d_{\mathbf{s}}}</math> <sup>33</sup>3<math display="inline">N_{\ast}</math> represents the sequence length of <math display="inline">\ast</math>.

<!--T:45-->
To extract user interest between the historical interacted items and candidates, target attention normally be used with target as query and sequence feature as key/value. Formally,

<!--T:46-->
{| id="S7.EGx1" class="wikitable"
|-
| (1)
|
| <math display="inline">{\mathbf{F}\mathbf{e}\mathbf{a}\mathbf{t}}{}_{\left. \mathbf{S}\rightarrow \right.}^{}\text{Attention}{({\mathbf{E}\mathbf{m}\mathbf{b}}{}_{}^{}{\mathbf{E}\mathbf{m}\mathbf{b}}_{\overset{\rightarrow}{\mathbf{S}}},{\mathbf{E}\mathbf{m}\mathbf{b}}_{\overset{\rightarrow}{\mathbf{S}}})} \in {\mathbb{R}}^{K \times d_{\mathbf{S}}}</math>
|
|}

<!--T:47-->
Eq.[[#S3.E1|1]] aggregates <math display="inline">\overset{\rightarrow}{\mathbf{S}}</math> according to <math display="inline">\mathbf{I}</math>. Finally, embedded and processed features from <math display="inline">\mathbb{D}</math> are concatenated and represented as:

<!--T:48-->
{| id="S7.EGx2" class="wikitable"
|-
| (2)
|
| <math display="inline">{\mathbf{F}\mathbf{e}\mathbf{a}\mathbf{t}}{}_{}^{}{\lbrack{\mathbf{E}\mathbf{m}\mathbf{b}}{}_{}^{}{\mathbf{F}\mathbf{e}\mathbf{a}\mathbf{t}}{}_{\left. \mathbf{S}\rightarrow \right.}^{}{\mathbf{F}\mathbf{e}\mathbf{a}\mathbf{t}}{}_{\left. \mathbf{R}\rightarrow \right.}^{}{\mathbf{E}\mathbf{m}\mathbf{b}}{}_{}^{}{\mathbf{E}\mathbf{m}\mathbf{b}}_{\mathbf{s}}\rbrack} \in {\mathbb{R}}^{K \times {({d_{\mathbf{U}} + d_{\mathbf{S}} + d_{\mathbf{C}} + d_{\mathbf{I}}})}}</math>
|
|}

<!--T:49-->
<math display="inline">{\mathbf{F}\mathbf{e}\mathbf{a}\mathbf{t}}_{\mathbb{D}}</math> is further fed to a multiple layer perceptron (MLP) to output logit for each sample. The logit is used for learning in training and for ranking when inference.

<!--T:50-->
Fig.[[#S3.F1|1]] shows a simplified data arrangement and workflow under the traditional ranking model. These features are firstly embedded, the leading embeddings are processed in different methods. Finally, the processed features are concatenated and processed by MLP for feature interaction. The final logit is generate for each candidates.

<!--T:51-->
<div id="S3.SS3" class="section">

<!--T:52-->
<span id="scaling-dilemma-in-recommendation-systems"></span>

<!--T:53-->
=== 3.3. Scaling Dilemma in Recommendation Systems ===

<!--T:54-->
Model scaling has been a common method for performance improvement in ranking. Generally, model scaling aims at scaling parameters in user module and cross module. The user module processes user feature including sequence features and generates user-dependent representation. Scaling user module brings in better representation for user. Besides, since the user is shared and inferred once for all candidates, a large inference cost in the user module does not lead to a burdensome system overload. However, scaling user module alone does not contribute to feature interaction in user and item directly.

<!--T:55-->
On the contrary, another trend of method aims at scaling cross module, i.e., the feature interaction MLP after feature concatenation. These methods enhance ranking ability by paying more attention to interaction between user and candidates. However, since cross module is inferred for each candidates, computation increasing is scaled linearly with number of candidates. Cross module scaling brings in unacceptable system latency.

<!--T:56-->
The scaling dilemma in traditional recommendation system necessitates a new scaling method, leading to efficient feature interaction between user and candidates, while at the same time brings sub-linear inference cost with number of candidates. MTGR innovates the scaling approach in recommendation system by data rearrangement and corresponding architecture optimization.

<!--T:57-->
<div id="S4" class="section">

<!--T:58-->
<span id="data-rearrangement-and-architecture-of-mtgr"></span>

<!--T:59-->
== 4. Data Rearrangement and Architecture of MTGR ==

<!--T:60-->
<div id="S4.SS1" class="section">

<!--T:61-->
<span id="user-sample-aggregation-for-training-and-inference-efficiency"></span>

<!--T:62-->
=== 4.1. User Sample Aggregation for Training and Inference Efficiency ===

<!--T:63-->
<div id="S4.F2" class="figure">

<!--T:64-->
[[File:Arxiv_2505_18654_x2.png|830x1163px|Refer to caption]]

<!--T:65-->
Compared with feature categorization in Sec.[[#S3.SS1|3.1]], for <math display="inline">i</math>-th sample in candidates, MTGR organizes features into <math display="inline">{\mathbb{D}}_{i} = {\lbrack\mathbf{U},\overset{\rightarrow}{\mathbf{S}},\overset{\rightarrow}{\mathbf{R}},{\lbrack\mathbf{C}{}_{}^{}\mathbf{I}{}_{}^{}}\rbrack}</math>. Specially, the cross feature <math display="inline">\mathbf{C}</math> is arranged as part of the item feature of candidates. In MTGR, candidates are aggregated by user in a specific window for training and by request for inference. Since the aggregation is performed by the same user, the aggregated sample can use the same user representation (<math display="inline">\mathbf{U}</math>, <math display="inline">\overset{\rightarrow}{\mathbf{S}}</math>, <math display="inline">\overset{\rightarrow}{\mathbf{R}}</math>). In particular, <math display="inline">\overset{\rightarrow}{\mathbf{R}}</math> arranges all the user’s real-time interaction items within another specific window in chronological order of interaction time.

<!--T:66-->
Fig.[[#S4.F2|2]](a) demonstrates the aggregation: Compared with Fig.[[#S3.F1|1]] where only one candidate is predicted, Fig.[[#S4.F2|2]](a) aggregates three items in one sample, reusing the same user representation. Formally, it forms a feature representation for the same user:

<!--T:67-->
{| id="S7.EGx3" class="wikitable"
|-
| (3)
|
| <math display="inline">{\mathbb{D}} = {\lbrack\mathbf{U},\overset{\rightarrow}{\mathbf{S}},\overset{\rightarrow}{\mathbf{R}},{\lbrack\mathbf{C},\mathbf{I}\rbrack}_{1},\ldots,{\lbrack\mathbf{C},\mathbf{I}\rbrack}_{K}\rbrack}</math>
|
|}

<!--T:68-->
By aggregating candidates into one sample, MTGR reduces a tremendous amount of resource by performing only a single computation and producing scores for all candidates. Specially, the user aggregation procedure greatly reduce the training samples from the sizes of all candidates to all users. For inference, the candidates in a request are grouped as mentioned above, MTGR only perform one-time inference for all the candidate ranking, instead of inference of candidates’ number. The aggregation circumvents the dependency of candidate’ volume in inference cost, leaving possibility and potential for model scaling.

<!--T:69-->
Eq.[[#S4.E3|3]] is a combination of scalar and sequence features. To unify the input format, MTGR converts features and sequence into tokens. Specially, for scalar features in <math display="inline">\mathbf{U}</math>, each feature is naturally converted to individual token with dimension <math display="inline">{\mathbf{F}\mathbf{e}\mathbf{a}\mathbf{t}}{}_{}^{}{\mathbb{R}}^{N_{\mathbf{U}} \times d_{\text{model}}}</math>. <math display="inline">d_{\text{model}}</math> is a set unified dimension for all tokens. For sequence feature of <math display="inline">\overset{\rightarrow}{\mathbf{S}}</math> and <math display="inline">\overset{\rightarrow}{\mathbf{R}}</math>, each item <math display="inline">\mathbf{S}</math> is regarded as a token. Features in <math display="inline">\mathbf{S}</math> is firstly embedded and concatenated, then a MLP module is adopted for dimension unification. Formally, <math display="inline">{\mathbf{F}\mathbf{e}\mathbf{a}\mathbf{t}}{}_{\mathbf{S}_{i}}^{}\text{MLP}{(\text{Concat}{({\mathbf{E}\mathbf{m}\mathbf{b}}{}_{}^{}})} \in {\mathbb{R}}^{d_{\text{model}}}</math>. The features of <math display="inline">\mathbf{S}</math> in the sequence is concatenated along another dimension, leading to <math display="inline">{\mathbf{F}\mathbf{e}\mathbf{a}\mathbf{t}}{}_{\left. \mathbf{S}\rightarrow \right.}^{}{\mathbb{R}}^{N_{\overset{\rightarrow}{\mathbf{S}}} \times d_{\text{model}}}</math>.

<!--T:70-->
Similar, each item <math display="inline">\mathbf{I}</math> in candidates is regarded as a token. The features in candidates are embedded and concatenated, converted to the unified dimension by another MLP. Candidates are concatenated as a series of tokens: <math display="inline">{\mathbf{F}\mathbf{e}\mathbf{a}\mathbf{t}}{}_{}^{}\text{Concat}{(\text{MLP}{(\text{Concat}{({\mathbf{E}\mathbf{m}\mathbf{b}}{}_{\mathbf{C}_{i}}^{}{\mathbf{E}\mathbf{m}\mathbf{b}}{}_{\mathbf{I}_{i}}^{}})})} \in {\mathbb{R}}^{N_{\mathbf{I}}}</math> Finally, the constructed tokens from <math display="inline">\mathbf{U}</math>, <math display="inline">\overset{\rightarrow}{\mathbf{S}}</math>, <math display="inline">\lbrack\mathbf{C},\mathbf{s}\rbrack</math> are concatenated, leading a long sequence of tokens:

<!--T:71-->
{| id="S7.EGx4" class="wikitable"
|-
|
| <math display="inline">{\mathbf{F}\mathbf{e}\mathbf{a}\mathbf{t}}_{\mathbb{D}}</math>
| <math display="inline">=</math>
| <math display="inline">\text{Concat}{({\lbrack{\mathbf{F}\mathbf{e}\mathbf{a}\mathbf{t}}{}_{}^{}{\mathbf{F}\mathbf{e}\mathbf{a}\mathbf{t}}{}_{\left. \mathbf{S}\rightarrow \right.}^{}{\mathbf{F}\mathbf{e}\mathbf{a}\mathbf{t}}{}_{\left. \mathbf{R}\rightarrow \right.}^{}{\mathbf{F}\mathbf{e}\mathbf{a}\mathbf{t}}{}_{}^{}})}</math>
|
|-
|
|
| <math display="inline">\in</math>
| <math display="inline">{\mathbb{R}}^{{({N_{\mathbf{U}} + N_{\overset{\rightarrow}{\mathbf{S}}} + N_{\overset{\rightarrow}{\mathbf{R}}} + N_{\mathbf{I}}})} \times d_{\text{model}}}</math>
|
|}

<!--T:72-->
<div id="S4.SS2" class="section">

<!--T:73-->
<span id="unified-hstu-encoder"></span>

<!--T:74-->
=== 4.2. Unified HSTU Encoder ===

<!--T:75-->
Samples from a user are aggregated as a sequence of tokens, which is naturally suitable for processing using self-attention. Inspired by HSTU (Zhai et al., [[#bib.bib23|2024]]), MTGR uses stacks of self-attention layer and encoder-only architecture for modeling.

<!--T:76-->
Similar in LLM, the input token sequences are processed layer-wisely. As shown in Fig.[[#S4.F2|2]], in the self-attention block, input sequence <math display="inline">\mathbf{X}</math> is firstly normalized by group layer norm. Features of the same domain (for example, <math display="inline">\mathbf{U}</math>) form a group. Then, the group layer norm ensures that tokens from different domains share a similar distribution before self-attention and aligns different semantic spaces of different domains <math display="inline">\overset{\sim}{X} = {\text{GroupLN}\hspace{0pt}{(\mathbf{X})}}</math>. The normalized input is then projected to 4 different representation: <math display="inline">{\mathbf{K},\mathbf{Q},\mathbf{V},\mathbf{U}} = {\text{MLP}_{\mathbf{K}/\mathbf{Q}/\mathbf{V}/\mathbf{U}}\hspace{0pt}{(\overset{\sim}{\mathbf{X}})}}</math>. <math display="inline">\mathbf{Q}</math>, <math display="inline">\mathbf{K}</math> is utilized for multi-head attention computation with silu non-linear activation. The leading attention is divided by total length of input features as an average factor. Next, a customized mask (<math display="inline">\mathbf{M}</math>) is imposed on the attention score and the projected <math display="inline">\mathbf{V}</math> is applied for value update:

<!--T:77-->
{| id="S7.EGx5" class="wikitable"
|-
| (5)
|
| <math display="inline">\overset{\sim}{\mathbf{V}}</math>
| <math display="inline">=</math>
| <math display="inline">\frac{\text{silu}\hspace{0pt}{({\mathbf{K}^{T}\hspace{0pt}\mathbf{Q}})}}{({N_{\mathbf{U}} + N_{\overset{\rightarrow}{\mathbf{S}}} + N_{\overset{\rightarrow}{\mathbf{R}}} + N_{\mathbf{I}}})}\hspace{0pt}{\mathbf{M}\mathbf{V}}</math>
|
|}

<!--T:78-->
The projected <math display="inline">\mathbf{U}</math> and the updated <math display="inline">\overset{\sim}{\mathbf{V}}</math> is dot-producted. Then, another group layer norm is applied. Finally, we add a residual module and place another MLP above it:

<!--T:79-->
{| id="S7.EGx6" class="wikitable"
|-
| (6)
|
| <math display="inline">\mathbf{X}</math>
| <math display="inline">=</math>
| <math display="inline">{\text{MLP}\hspace{0pt}{({\text{GroupLN}\hspace{0pt}{({\overset{\sim}{\mathbf{V}} \odot \mathbf{U}})}})}} + \mathbf{X}</math>
|
|}

<!--T:80-->
Dynamic Masking (Zhai et al., [[#bib.bib23|2024]]) utilizes the causal mask for sequence modeling. However, such a implementation does not bring in significant improvement in MTGR. Besides, since <math display="inline">\overset{\rightarrow}{\mathbf{R}}</math> records the user’s most recent interactions which timing might coincide with the sample aggregation window. Using a simple causal mask in MTGR could result in information leakage. For example, interactions in the evening should not be exposed to candidates in the afternoon, yet this information may be aggregated into one sample. This dilemma requires a flexible and efficient masking.

<!--T:81-->
In MTGR, <math display="inline">\mathbf{U}</math>, <math display="inline">\overset{\rightarrow}{\mathbf{S}}</math> is regarded as static (in the following, we denote <math display="inline">\mathbf{U}</math>, <math display="inline">\overset{\rightarrow}{\mathbf{S}}</math> as ‘static sequence’) because its information comes from before the aggregation window, thus it does not cause causality errors. <math display="inline">\overset{\rightarrow}{\mathbf{R}}</math> is dynamic as it progressively include user’s interaction (‘dynamic sequence’ for <math display="inline">\overset{\rightarrow}{\mathbf{S}}</math>) in real-time. MTGR applies full attention for static sequence, auto-regressive with dynamic masking for <math display="inline">\overset{\rightarrow}{\mathbf{R}}</math> and diagonal masking for inter-candidates. Specially, 3 rules are set for MTGR’s masking:

<!--T:82-->
<ul>
<li><p><span id="S4.I1.i1">•</span></p>
<p>The static sequence is visible to all tokens.</p></li>
<li><p><span id="S4.I1.i2">•</span></p>
<p>The visibility of the dynamic sequence adheres to causality, where each token is only visible to tokens that occur afterward, which include candidate tokens.</p></li>
<li><p><span id="S4.I1.i3">•</span></p>
<p>Candidate tokens (<math display="inline">\mathbf{C}</math>, <math display="inline">\mathbf{I}</math>) is visible to itself only.</p></li></ul>

<!--T:83-->
Fig.[[#S4.F2|2]] (c) demonstrates an example for the dynamic masking: ‘age’, ‘ctr’ represent feature token from <math display="inline">\mathbf{U}</math>; ‘seq1’, ‘seq2’ for <math display="inline">\overset{\rightarrow}{\mathbf{S}}</math>; ‘rt1’, ‘rt2’ for <math display="inline">\overset{\rightarrow}{\mathbf{R}}</math> and ‘target1’ - ‘target3’ for candidates. White block in row represents that the certain token is able to use the information from others while column represents the token is visible to others or not. Full attention is utilized for <math display="inline">\mathbf{U}</math> and <math display="inline">\mathbf{S}</math>, leading to a white square from ‘age’ to ‘seq2’. For ‘rt1’ to ‘rt2’, we assume that ‘rt1’ appear latter than ‘rt2’, Therefore, a tiny square from ‘rt1’ to ‘rt2’ is constructed with white blocks in upper triangle, meaning that ‘rt1’ is able to use information from ‘rt2’ while ‘rt1’ is not visible to ‘rt2’. Besides, ‘target2’ and ‘target3’ is assumed to appear earlier than ‘rt1’, leading to ‘rt1’ is not visible to them. ‘rt2’ appear earlier than all ‘target1’ and ‘target2’ while latter than ‘target3’. Therefore, ‘rt2’ is not visible to ‘target3’, which appears earlier than all ‘rt’s, leading to its unable to use information from ‘rt’s.

<!--T:84-->
<div id="S5" class="section">

<!--T:85-->
<span id="training-system"></span>

<!--T:86-->
== 5. Training System ==

<!--T:87-->
To facilitate the design and development of MTGR model structure and to conveniently incorporate more features from the fast-growing LLM community, we decided not to continue with our previous TensorFlow-based training framework. Instead, we opted to reconstruct our training framework within the PyTorch ecosystem. Specifically, we expanded and optimized TorchRec’s functionality, made enhancements specifically catering to MTGR model’s characteristics, and ultimately accomplished efficient training of MTGR model. Compared to TorchRec, our optimized framework improves training throughput by 1.6x – 2.4x while achieving good scalability when running over 100 GPUs. Compared to the DLRM baseline, we achieved 65x FLOPs per sample for forward computation, while the training cost remained nearly unchanged. We provide some of our key work as follows.

<!--T:88-->
Dynamic Hash Table. TorchRec use a fixed-size table to process sparse embeddings, which is not suitable for large-scale industrial streaming training scenarios. Firstly, additional embeddings cannot be allocated to new users and items at real-time once the static table hits its preset capacity. Although some default embeddings may be used in this case, or some eviction strategies can be used to eliminate ‘old’ IDs, model accuracy will be degraded. Secondly, static tables typically require pre-allocation of capacity exceeding actual requirements to prevent ID overflow, inevitably leading to inefficient memory utilization. To tackle the challenges, we implement a hash-based and high performance embedding table, dynamically allocating space for sparse IDs during the training process. We employ a decoupled architecture for hash table, which separates key and value storage into distinct structures. The key structure maintains a lightweight mapping table containing keys and corresponding pointers to embedding vectors, while the value structure stores both the embedding vectors and auxiliary metadata (e.g., counters and timestamps) required for eviction policies. This dual-structure design achieves two critical objectives: (1) enabling dynamic capacity expansion through replication of only the compact key structure rather than huge embedding data, and (2) optimizing key scanning efficiency by storing keys in compact memory layouts while accommodating potentially sparse key distributions.

<!--T:89-->
Embedding Lookup. The embedding lookup process employs All-to-all communication for cross-device embedding exchange. We adopt two-stage ID unique operation to reduce redundant IDs before and after ID communication to avoid repetitive embedding transfers among the devices. We also design a feature configuration interface to implement automatic table merging, which can reduce the number of embedding lookup operators and thus speed up the process.

<!--T:90-->
Load balance. In recommendation systems, user behavior sequences often exhibit a long-tail distribution where only a few users have long sequences, while most have short ones. This leads to significant computational load imbalance when training with a fixed batch size (abbreviated as BS). A common solution is to use sequence packing techniques(Krell et al., [[#bib.bib11|2021]]), where multiple short sequences are merged into a longer one. However, this approach requires careful mask adjustments to prevent different sequences from interfering with each other during the attention calculation, which can be quite costly to implement. Our straightforward solution is to introduce dynamic BS. Each GPU’s local BS is adjusted based on the actual sequence length of the input data, ensuring a similar computational load. Additionally, we’ve tweaked the gradient aggregation strategy to weight each GPU’s gradients according to its BS, maintaining computational logic consistency with fixed BS.

<!--T:91-->
Other Optimizations. To further enhance training efficiency, we maximize parallelism through pipeline technology using three streams: copy, dispatch, and compute. Specifically, the copy stream loads input data from CPU to GPU, the dispatch stream performs table lookups based on IDs, and the compute stream handles forward computation and backward updates. For instance, while the compute stream executes forward and backward passes for batch <math display="inline">\mathbf{T}</math>, the copy stream concurrently loads batch <math display="inline">\mathbf{T} + 1</math> to mask I/O latency. Upon completing backward updates for batch <math display="inline">\mathbf{T}</math>, the dispatch stream immediately initiates table lookups and communication for batch <math display="inline">\mathbf{T} + 1</math>. Moreover, we employ bf16 mixed-precision training and develop a well-crafted cutlass-based attention kernel which is similar to FlashAttention to speed up the training process.

<!--T:92-->
<div id="S6" class="section">

<!--T:93-->
<span id="experiments"></span>

<!--T:94-->
== 6. Experiments ==

<!--T:95-->
<div id="S6.SS1" class="section">

<!--T:96-->
<span id="experiment-setup"></span>

<!--T:97-->
=== 6.1. Experiment setup ===

<!--T:98-->
Datasets. Public datasets widely use independent ID and attribute features, where cross features are seldom introduced. However, cross features demonstrate importance in real-world application. Cross features are an important category of features in our scenario. They are often meticulously hand-crafted, including interactions like user-item, user and higher-level categories, item and spatio-temporal information. To compensate the absence of cross feature in public datasets, we construct a training dataset based on logs from the real industrial-scale recommendation system in Meituan. Unlike public datasets, our real dataset contains a richer cross features set and longer user behavior sequences. In addition, the volume of our dataset is large, allowing complex models to achieve more adequate convergence during training. For the offline experiments, we collect data over a <math display="inline">10</math>-day period. The statistics of the dataset are shown in Table  [[#S6.T1|1]]. For the online experiments, in order to compare with the DLRM baseline that has been trained for over <math display="inline">2</math> years, we constructed a longer-term dataset for the experiments, using data spanning more than <math display="inline">6</math> months.

<!--T:99-->
<div id="S6.T1" class="figure">

<!--T:100-->
{| class="wikitable"
|-
! Dataset
! #Users
! #Items
! #Exposure
! #Click
! #Purchases
|-
! Train
| 0.21 billion
| 4,302,391
| 23.74 billion
| 1.08 billion
| 0.18 billion
|-
! Test
| 3,021,198
| 3,141,997
| 76,855,608
| 4,545,386
| 769,534
|}

<!--T:101-->
Baseline. For DLRM, we compare two methods in sequence modeling: SIM based on sequence retrieval and End2End modeling for original long sequences (E2E). In terms of scaling, we compared DNN, MoE(Ma et al., [[#bib.bib14|2018]]), Wukong(Zhang et al., [[#bib.bib25|2024b]]), MultiEmbed(Guo et al., [[#bib.bib7|2023]]), and UserTower.

<!--T:102-->
MoE uses <math display="inline">4</math> experts, with each expert containing a network of the same complexity as the base DNN. Wukong and MultiEmbed are configured to the same computational complexity as MoE. UserTower uses a set of learnable queries and inserts a qFormer(Li et al., [[#bib.bib12|2023]]) layer with another MoE (16 experts) module on user behavior. UserTower’s computational complexity is tripled than MoE method, but it can share this computation for multiple predicted items for the same user during inference, thereby reducing inference costs. It has achieved good results in our scenario.

<!--T:103-->
MTGR employs E2E to handle all sequence information. Additionally, as shown in Table [[#S6.T2|2]], we have set up three different scales to verify the scalability of MTGR.

<!--T:104-->
<div id="S6.T2" class="figure">

<!--T:105-->
{| class="wikitable"
|-
! Model
! Setting
! Learning rate
! GFLOPs/example
|-
| UserTower-SIM
| -
| <math display="inline">8 \times 10^{- 4}</math>
| 0.86
|-
| MTGR-small
| <math display="inline">n_{\text{layer}} = 3</math>, <math display="inline">d_{\text{model}} = 512</math>, <math display="inline">n_{\text{heads}} = 2</math>
| <math display="inline">3 \times 10^{- 4}</math>
| 5.47
|-
| MTGR-medium
| <math display="inline">n_{\text{layer}} = 5</math>, <math display="inline">d_{\text{model}} = 768</math>, <math display="inline">n_{\text{heads}} = 3</math>
| <math display="inline">3 \times 10^{- 4}</math>
| 18.59
|-
| MTGR-large
| <math display="inline">n_{\text{layer}} = 15</math>, <math display="inline">d_{\text{model}} = 768</math>, <math display="inline">n_{\text{heads}} = 3</math>
| <math display="inline">1 \times 10^{- 4}</math>
| 55.76
|}

<!--T:106-->
Evaluation Metrics. In offline, we focus on learning of two tasks: CTR and CTCVR (Click-Through Conversion Rate) and use AUC(Ferri et al., [[#bib.bib6|2011]]) and GAUC (Group AUC) for evaluation. GAUC is averaged on the the AUCs under users. Compared to AUC, GAUC pays more attention to the model’s ranking ability for the same user. For online evaluation, we focus on two metrics: PV_CTR (CTR per page view) and UV_CTCVR (CTCVR per user view), where UV_CTCVR is the most crucial metric for evaluating the growth of business.

<!--T:107-->
Parameter Setting. Our model is trained using the Adam optimizer. For DLRM, each GPU processes a batch size of 2400, with 8 NVIDIA A100 GPUs for training. In the case of MTGR, the batch size is set to 96, utilizing 16 NVIDIA A100 GPUs for training. As shown in Table [[#S6.T2|2]], the learning rate decreases as the complexity of the model increases. Moreover, as the model size grows, we proportionally increase the size of the sparse parameters. Assuming that a token consists of <math display="inline">k</math> features, the embedding dimension for each feature is generally set to an integer close to <math display="inline">d_{\text{model}}/k</math>. Finally, the maximum length of <math display="inline">\overset{\rightarrow}{\mathbf{S}}</math> is set to <math display="inline">1000</math>, while <math display="inline">\overset{\rightarrow}{\mathbf{R}}</math> is set to <math display="inline">100</math>.

<!--T:108-->
<div id="S6.SS2" class="section">

<!--T:109-->
<span id="overall-performance-comparison"></span>

<!--T:110-->
=== 6.2. Overall Performance Comparison ===

<!--T:111-->
We evaluate the performance of MTGR and other baseline methods using our 10-day dataset. Table [[#S6.T3|3]] shows the performance of different models. The differences among various models across different offline metrics are quite consistent. Based on previous experience, an increase <math display="inline">0.001</math> in our offline metric is considered significant. Among the various versions of DLRM, Wukong-SIM and MultiEmbed-SIM achieve a better result than MoE-SIM. UserTower-SIM performs the best and UserTower-E2E shows a slight decline in performance compared to UserTower-SIM. We speculate that under the DLRM paradigm, the model complexity is insufficient to model all sequence information, leading to underfitting. Our proposed MTGR, even the smallest version, exceeds the strongest DLRM model. Additionally, models of three different sizes exhibit scalability, with their performance smoothly increasing as the complexity of the model increases.

<!--T:112-->
<div id="S6.T3" class="figure">

<!--T:113-->
{| class="wikitable"
|-
! rowspan="2"| Model
| colspan="2"| CTR
| colspan="2"| CTCVR
|-
| AUC
| GAUC
| AUC
| GAUC
|-
! DNN-SIM
| 0.7432
| 0.6679
| 0.8737
| 0.6504
|-
! MoE-SIM
| 0.7484
| 0.6698
| 0.8750
| 0.6519
|-
! MultiEmbed-SIM
| 0.7501
| 0.6715
| 0.8766
| 0.6525
|-
! Wukong-SIM
| 0.7568
| 0.6759
| 0.8800
| 0.6530
|-
! UserTower-SIM
| 0.7593
| 0.6792
| 0.8815
| 0.6550
|-
! UserTower-E2E
| 0.7576
| 0.6787
| 0.8818
| 0.6548
|-
! MTGR-small
| 0.7631
| 0.6826
| 0.8840
| 0.6603
|-
! MTGR-medium
| 0.7645
| 0.6843
| 0.8849
| 0.6625
|-
! MTGR-large
| 0.7661
| 0.6865
| 0.8862
| 0.6646
|-
! Impr.%
| 0.8956
| 1.0748
| 0.4990
| 1.4656
|}

<!--T:114-->
<div id="S6.T4" class="figure">

<!--T:115-->
{| class="wikitable"
|-
! rowspan="2"| Model
! colspan="2"| CTR
! colspan="2"| CTCVR
|-
! AUC
! GAUC
! AUC
! GAUC
|-
! MTGR-small
| 0.7631
| 0.6826
| 0.8840
| 0.6603
|-
! w/o cross features
| 0.7495
| 0.6689
| 0.8736
| 0.6514
|-
! w/o GLN
| 0.7606
| 0.6809
| 0.8826
| 0.6585
|-
! w/o dynamic mask
| 0.7620
| 0.6810
| 0.8828
| 0.6587
|}

<!--T:116-->
<div id="S6.SS3" class="section">

<!--T:117-->
<span id="ablation-study"></span>

<!--T:118-->
=== 6.3. Ablation Study ===

<!--T:119-->
We perform ablation studies on two components of MTGR: Dynamic Masking and group layer norm (GLN) based on our small one. The ablation results are shown in Fig [[#S6.T4|4]]. Eliminating any of these from MTGR will result in a significant decline in performance, with the degree of decline being comparable to the increase from MTGR-small to MTGR-medium. This indicates the importance of Dynamic Masking and GLN for MTGR. In addition, we conducted extra experiments on the importance of cross features to MTGR. After removing the cross features, the performance metrics show a significant drop, which even erases the gain of MTGR-large over DLRM, highlighting the vital role of cross features in the real recommendation system.

<!--T:120-->
<div id="S6.SS4" class="section">

<!--T:121-->
<span id="scalability"></span>

<!--T:122-->
=== 6.4. Scalability ===

<!--T:123-->
Fig. [[#S6.F3|3]] illustrates the scalability of our MTGR. We perform tests based on MTGR-small for three different hyperparameters: the number of HSTU blocks, the <math display="inline">d_{\text{model}}</math>, and the input sequence length. As can be observed, MTGR demonstrates good scalability across different hyperparameters. Furthermore, Fig. [[#S6.F3|3]](d) presents a power-law relationship between performance and computational complexity. The vertical axis indicates the gain in the CTCVR GAUC metric relative to our best DLRM model, UserTower-SIM, while the horizontal axis reflects the logarithmic multiple of computational complexity compared to UserTower-SIM.

<!--T:124-->
<div id="S6.F3" class="figure">

<!--T:125-->
[[File:Arxiv_2505_18654_x3.png|813x685px|Refer to caption]]

<!--T:126-->
<div id="S6.SS5" class="section">

<!--T:127-->
<span id="online-experiments"></span>

<!--T:128-->
=== 6.5. Online Experiments ===

<!--T:129-->
To further validate the effectiveness of MTGR, we deployed MTGR in the Meituan Take-away platform, conducting AB testing with <math display="inline">2\%</math> of the traffic. The volume of experimental traffic covers millions of exposures per day, demonstrating the confidence of the experiment. The comparison baseline is the most advanced online DLRM model (UserTower-SIM), and it has been continuously learning for <math display="inline">2</math> years. We use data from the last <math display="inline">6</math> months to train the MTGR model, which is then deployed online for comparison.

<!--T:130-->
Although the volume of training data is significantly lower than the counterpart of DLRM model, the offline and online metrics still greatly exceed the DLRM base. As shown in Table  [[#S6.T5|5]], both the offline and online metrics demonstrate scalability. We also found that as the number of training tokens increases, the benefits compared to DLRM continue to amplify. Eventually, in terms of CTCVR GAUC, our large version even exceeds the cumulative increase of all optimizations over the past year.

<!--T:131-->
The model has already been fully deployed in our scenario, with training costs equal to DLRM, and inference costs reduced by <math display="inline">12\%</math>. For DLRM, its inference cost is approximately linear with the number of candidates. However, MTGR uses user aggregation for all candidates in a request, leading to sub-linear inference cost scaling with the number of candidates. This helps us reduce the overhead of online inference.

<!--T:132-->
<div id="S6.T5" class="figure">

<!--T:133-->
{| class="wikitable"
|-
!
! colspan="2"| Offline Metric diff
! colspan="2"| Online Metric diff
|-
!
! CTR GAUC
! CTCVR GAUC
! PV_CTR
! UV_CTCVR
|-
! MTGR-small
| +0.0036
| +0.0154
| +1.04%
| +0.04%
|-
! MTGR-medium
| +0.0071
| +0.0182
| +2.29%
| +0.62%
|-
! MTGR-large
| +0.0153
| +0.0288
| +1.90%
| +1.02%
|}

<!--T:134-->
<div id="S7" class="section">

<!--T:135-->
<span id="conclusion"></span>

<!--T:136-->
== 7. Conclusion ==

<!--T:137-->
In this paper, we proposed MTGR, a new ranking framework to explore the scaling law in recommendation systems based on HSTU. MTGR combines the advantages of DLRM and GRM, allowing the use of cross-features to ensure model performance while having the same scalability as GRM. MTGR has already been deployed in our scenario and has yielded significant benefits. In the future, we will explore how to extend MTGR to multi-scenario modeling, similar to large language models, to establish a recommendation foundation model with extensive knowledge.

<!--T:138-->
<div id="bib" class="section">

<!--T:139-->
<span id="references"></span>

<!--T:140-->
== References ==

<!--T:141-->
* <span id="bib.bib1">(1)</span>
* <span id="bib.bib2">Chen et al. (2021) Qiwei Chen, Changhua Pei, Shanshan Lv, Chao Li, Junfeng Ge, and Wenwu Ou. 2021. End-to-end user behavior retrieval in click-through rateprediction model. ''arXiv preprint arXiv:2108.04468'' (2021).</span>
* <span id="bib.bib3">Dao et al. (2022) Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. ''Advances in neural information processing systems'' 35 (2022), 16344–16359.</span>
* <span id="bib.bib4">Deng et al. (2025) Jiaxin Deng, Shiyao Wang, Kuo Cai, Lejian Ren, Qigen Hu, Weifeng Ding, Qiang Luo, and Guorui Zhou. 2025. OneRec: Unifying Retrieve and Rank with Generative Recommender and Iterative Preference Alignment. ''arXiv preprint arXiv:2502.18965'' (2025).</span>
* <span id="bib.bib5">Fang et al. (2024) Yan Fang, Jingtao Zhan, Qingyao Ai, Jiaxin Mao, Weihang Su, Jia Chen, and Yiqun Liu. 2024. Scaling laws for dense retrieval. In ''Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval''. 1339–1349.</span>
* <span id="bib.bib6">Ferri et al. (2011) Cesar Ferri, José Hernández-Orallo, and Peter A Flach. 2011. A coherent interpretation of AUC as a measure of aggregated classification performance. In ''Proceedings of the 28th International Conference on Machine Learning (ICML-11)''. 657–664.</span>
* <span id="bib.bib7">Guo et al. (2023) Xingzhuo Guo, Junwei Pan, Ximei Wang, Baixu Chen, Jie Jiang, and Mingsheng Long. 2023. On the embedding collapse when scaling up recommendation models. ''arXiv preprint arXiv:2310.04400'' (2023).</span>
* <span id="bib.bib8">Han et al. (2024) Ruidong Han, Qianzhong Li, He Jiang, Rui Li, Yurou Zhao, Xiang Li, and Wei Lin. 2024. Enhancing CTR Prediction through Sequential Recommendation Pre-training: Introducing the SRP4CTR Framework. In ''Proceedings of the 33rd ACM International Conference on Information and Knowledge Management''. 3777–3781.</span>
* <span id="bib.bib9">Ivchenko et al. (2022) Dmytro Ivchenko, Dennis Van Der Staay, Colin Taylor, Xing Liu, Will Feng, Rahul Kindi, Anirudh Sudarshan, and Shahin Sefati. 2022. Torchrec: a pytorch domain library for recommendation systems. In ''Proceedings of the 16th ACM Conference on Recommender Systems''. 482–483.</span>
* <span id="bib.bib10">Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. ''arXiv preprint arXiv:2001.08361'' (2020).</span>
* <span id="bib.bib11">Krell et al. (2021) Mario Michael Krell, Matej Kosec, Sergio P Perez, and Andrew Fitzgibbon. 2021. Efficient sequence packing without cross-contamination: Accelerating large language models without impacting performance. ''arXiv preprint arXiv:2107.02027'' (2021).</span>
* <span id="bib.bib12">Li et al. (2023) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ''International conference on machine learning''. PMLR, 19730–19742.</span>
* <span id="bib.bib13">Lian et al. (2018) Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun. 2018. xdeepfm: Combining explicit and implicit feature interactions for recommender systems. In ''Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery &amp; data mining''. 1754–1763.</span>
* <span id="bib.bib14">Ma et al. (2018) Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. 2018. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. In ''Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery &amp; data mining''. 1930–1939.</span>
* <span id="bib.bib15">Peebles and Xie (2023) William Peebles and Saining Xie. 2023. Scalable diffusion models with transformers. In ''Proceedings of the IEEE/CVF international conference on computer vision''. 4195–4205.</span>
* <span id="bib.bib16">Pi et al. (2020) Qi Pi, Guorui Zhou, Yujing Zhang, Zhe Wang, Lejian Ren, Ying Fan, Xiaoqiang Zhu, and Kun Gai. 2020. Search-based user interest modeling with lifelong sequential behavior data for click-through rate prediction. In ''Proceedings of the 29th ACM International Conference on Information &amp; Knowledge Management''. 2685–2692.</span>
* <span id="bib.bib17">Shin et al. (2023) Kyuyong Shin, Hanock Kwak, Su Young Kim, Max Nihlén Ramström, Jisu Jeong, Jung-Woo Ha, and Kyung-Min Kim. 2023. Scaling law for recommendation models: Towards general-purpose user representations. In ''Proceedings of the AAAI conference on artificial intelligence'', Vol. 37. 4596–4604.</span>
* <span id="bib.bib18">Si et al. (2024) Zihua Si, Lin Guan, ZhongXiang Sun, Xiaoxue Zang, Jing Lu, Yiqun Hui, Xingchao Cao, Zeyu Yang, Yichen Zheng, Dewei Leng, et al. 2024. Twin v2: Scaling ultra-long user behavior sequence modeling for enhanced ctr prediction at kuaishou. In ''Proceedings of the 33rd ACM International Conference on Information and Knowledge Management''. 4890–4897.</span>
* <span id="bib.bib19">Tang et al. (2020) Hongyan Tang, Junning Liu, Ming Zhao, and Xudong Gong. 2020. Progressive layered extraction (ple): A novel multi-task learning (mtl) model for personalized recommendations. In ''Proceedings of the 14th ACM conference on recommender systems''. 269–278.</span>
* <span id="bib.bib20">Wang et al. (2021) Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. 2021. Dcn v2: Improved deep &amp; cross network and practical lessons for web-scale learning to rank systems. In ''Proceedings of the web conference 2021''. 1785–1797.</span>
* <span id="bib.bib21">Wang et al. (2024) Xu Wang, Jiangxia Cao, Zhiyi Fu, Kun Gai, and Guorui Zhou. 2024. HoME: Hierarchy of Multi-Gate Experts for Multi-Task Learning at Kuaishou. ''arXiv preprint arXiv:2408.05430'' (2024).</span>
* <span id="bib.bib22">Yan et al. (2025) Bencheng Yan, Shilei Liu, Zhiyuan Zeng, Zihao Wang, Yizhen Zhang, Yujin Yuan, Langming Liu, Jiaqi Liu, Di Wang, Wenbo Su, et al. 2025. Unlocking Scaling Law in Industrial Recommendation Systems with a Three-step Paradigm based Large User Model. ''arXiv preprint arXiv:2502.08309'' (2025).</span>
* <span id="bib.bib23">Zhai et al. (2024) Jiaqi Zhai, Lucy Liao, Xing Liu, Yueming Wang, Rui Li, Xuan Cao, Leon Gao, Zhaojie Gong, Fangda Gu, Michael He, et al. 2024. Actions speak louder than words: Trillion-parameter sequential transducers for generative recommendations. ''arXiv preprint arXiv:2402.17152'' (2024).</span>
* <span id="bib.bib24">Zhai et al. (2022) Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. 2022. Scaling vision transformers. In ''Proceedings of the IEEE/CVF conference on computer vision and pattern recognition''. 12104–12113.</span>
* <span id="bib.bib25">Zhang et al. (2024b) Buyun Zhang, Liang Luo, Yuxin Chen, Jade Nie, Xi Liu, Daifeng Guo, Yanli Zhao, Shen Li, Yuchen Hao, Yantao Yao, et al. 2024b. Wukong: Towards a scaling law for large-scale recommendation. ''arXiv preprint arXiv:2403.02545'' (2024).</span>
* <span id="bib.bib26">Zhang et al. (2024a) Wei Zhang, Dai Li, Chen Liang, Fang Zhou, Zhongke Zhang, Xuewei Wang, Ru Li, Yi Zhou, Yaning Huang, Dong Liang, et al. 2024a. Scaling User Modeling: Large-scale Online User Representations for Ads Personalization in Meta. In ''Companion Proceedings of the ACM Web Conference 2024''. 47–55.</span>
* <span id="bib.bib27">Zhou et al. (2018) Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep interest network for click-through rate prediction. In ''Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery &amp; data mining''. 1059–1068.</span>
</translate>

[[Category:NLP]]
[[Category:Research]]
[[Category:Research Papers]]