The machine learning field is developing at a quick pace with the regular release of new models that promise improvements over their predecessors. However, evaluation of a new model for a particular use case is a time -consuming and resource -intensive process. It is a conundrum for online services such as Amazon’s store, which is required to offer their customers advanced technology, but operates in high volume 24 hours a day.
In a paper we presented at this year’s web conference, we offer a solution to this conundrum. Instead of using a single model or a few models, a language model and a graph of neural network-to-process customers, we offer using an ensemble of models whose output is assembled by gradient-boosted decision trees (GBDT (GBDT (GBDT ‘is).
By using Shapley values to determine how much each model contributes to GBDT’s final decision, we can rank the models after use. Depending on the available calculation resources, we only keep as many of the most commonly used models as it is practical to run in parallel.
A new model that has not yet been thoroughly evaluated for a particular use of use can be trained in something, data is available and added to the set where it takes its chances with the existing models. Shapley value analysis can remove it from the ensemble or it may decide that the new model has made an existing model outdated. Either way, the customer gets the benefit of the best current technology.
We tested by approach using our shopping Quries Dataset, a public data set that we released as part of a 2022 challenge at the conference on knowledge discovery and data. The data set consists of millions of query product peirs in three languages, where the ratio of queries and products has been marked agreement to the ESCI scheme (accurate, replacement, complement or irrelevant). We trained three large language models (LLMS) and three graphs of neural networks (GNNs) on the data set and then used three different measurements (accuracy, macro F1 and weightd F1) to compare them with an ensemble of all six using our GBDT -Based approach. Across the line, the ensemble exceeded the individual models, often dramatically.
Esci classification
Historically speaking, information extraction models have been evaluated elevator of the results they return; Amazon developed the Esci scheme as a finer-grained alternative. Given a query a product can be classified as a accurate Match (labeled and/or make specified in the query); As one replace (A product in the same product class, but from another manufacturer); As one Complement (A complementary product, such as a telephone booth when the query is for a phone); Gold like irrelevant (An important classification as it applies to the vast majority of the products for a given inquiry).
There are two most important ways to perform Esci classification: One is to fine-tune a language model, white base its output solely on the text of the product description and the query, and the other is to use a gnn that can be included in observed relationships between products and between products and queries.
For example, special query conditions and so on.
Gnns maps graph information to a representation room in an iterative process that first embedded the data attached to each knot; Then create new embedders that combine the embedders of nodes, their neighbors, and the relationship between them; And so on, usal to distance from one to the oven wheet. GNNS fine-tuned on the Esci task and factor in information in addition to the semantic content of queries and product descriptions.
Model set
At Amazon, we have found that combining output from fine-tuned LLMs and GNNs usually provides the best performance on the Esci task. In our webconf paper, we describe a general method of expanding the number of models we include in our together.
Output from the separate models is assembled by GBDTs. A decision tree is a model that makes a number of binary decisions – USERI, where the value of a particular data function is excected a threshold. The leaves of the tree are correlated with specific data classifications.
To calculate how much each model in our together contributors to the final output, we use Shaplley Additive Explandations, a method based on the game theoretical concept of Shapley values. With Shapley values, we systematically vary input into the GBDT model and spur how each variation is propagated through the decision trees; Formalism Shapley Value provides a way to use this data to estimate overall effects across all possible inputs.
This again allows us to calculate how much each model in the ensemble contributes to the GBDT model’s output. On this basis, we can only choose the most useful models for admission to our ensemble – up to Makever -threshold we consider calculating practical terms.
Running, running an ensemble of models is more computational to exist than running a single model (or a few models, a language model and a gnn). But in our paper, we describe several techniques to make ensemble models more efficient, such as cache labels of previously seen query product peers, for later recycling and pre-determination of the gnn enlightenments to the neighborhoods around often picked up products. Our experience shows that models must be practical for real -time installation.