Robot-ad-click detection is the task of determining whether an ad click on an e-commerce site was initiated by a human or software agent. Its goal is to ensure that advertisers’ campaigns are not invoiced for robotic activity and that human clicks are not invalid. It must act in real time to cause minimal disruption of the advertising experience, and it must be scalable, comprehensive, accurate and able to respond quickly to changing traffic patterns.
At this year’s conference on innovative uses of artificial intelligence (IAAI)-a part of AAAI, the annual meeting of the Association for the Advancement of Artificial Intelligence-Pressed We Slidr, or Detection of Skive Level of Robots, a Real-time Neurale Network Model trained with weak supervision to identify countless clicks on online ads. Slidr has been deployed on Amazon since 2021 and protected advertising campaigns against robotic clicks.
In the paper, we formulate a convex optimization problem that allows SLIDR to obtain optimal performance on individual traffic discs with a budget with overall false positive. We also describe our system design, which enables continuous offline-omskoling and large-scale real-time inference, and we share some of the important lessons we have learned by implementing Slidr, including the use of protective frames to prevent updates of anomal models and disaster recovery mechanisms to mitigate or correct decisions made by a mistaken model.
Challenges
Detection of robotic activity in online advertising faces various challenges: (1) precise soil sandy marks with high coverage are difficult to come up with; (2) bot -behavior patterns are developing continuously; (3) Bot behavior patterns vary significantly across different traffic discs (eg desktop vs, mobile); and (4) false positives reducing ad revenue.
Labels
Since accurate soil truth is not available in scale, we generate data cables by identifying two activities with high HURD RFM scores. RFM scores represent Recency (R), Frequency (F) and Monetary (M) Value of customers’ buying patterns on Amazon. Clicks of both varieties are labeled as human; All remaining clicks are marked as non-human.
Metrics
Due to the lack of reliable soil truth brands, typical measurements such as accuracy cannot be used to evaluate the model performance. So we turn to a trio with more specific measurements.
Invalidity (IVR) Defined as the fraction of total click marked as a robot of the algorithm. IVR is a sign of the recall of our model as a model with a higher IVR is more likely to invalid robotics clicks.
On its own, however, IVR may be misleading as a poorly executing model will invalid human and robotic clicks. That’s why we measure IVR in connection with FALSE-Positive Rate (FPR). We consider the purchase of click as a power of attorney for the distribution of human clicks and define FPR as a fraction of purchase clicks invalid of the algorithm. Here we take two assumptions: (1) All purchase clicks are human, and (2) buying clicks is a representative test of all human clicks.
We also define a more accurate variant of recall by checking the model’s coverage over a heuristic that identifies clicks with a high probability of being a robot. The heuristic labels all clicks in user sessions with more than k Ad -click in one hour as a robot. We call this metric Robot coverage.
A neural model for detection of bots
We are considering different input functions for our model that enable it to disambigue robotic and human behavior:
- User level frequency and Speed ​​counters Calculate quantities and speeds of clicks from users over different time periods. These enable the identification of new robotic attacks involving sudden bursts of clicks.
- User Device counters Keep track of statistics such as number of different sessions or users from an IP. These features help identify IP addresses that can be gateways with many users behind them.
- Click time Track Times Day and Day of the Week, mapped to a Unit Circle. Although human activity follows daily and weekly activity patterns, robot activity often does not.
- Logged status Differentiation between customers and non-logged sessions as we expect much more robotic traffic in the latter.
The neural network is a binary classifier consisting of three fully connected layers with relu activations and L2 control in the intermediate layers.
While training our model, we use trial weights that weigh click equivalent across the day, weekday, logged status and label value. We have found that sample weights are crucial to improving the model’s performance and stability, especially in terms of sparse data rates such as night lessons.
We compare our model against basic lines such as logistical regression and a heuristic rule that calculates speed results of clicks. Both base lines lack the ability to model complex patterns and are therefore unable to perform as well as the neural network.
Calibration
Calibration involves choosing a threshold for the model’s output -probe, over which all clicks are marked as invalid. The model should invalid certain very robotic clicks, but at the same time do not incur a high loss of income by invalid human clicks. Against this is an option to choose the “knee” on the IVR-FPR curve, in addition to which the false positive speed increases sharply compared to the increase in IVR.
But calibration of the model across all traffic discs leads together to different behaviors for different discs. For example, a decision threshold obtained via the total calibration when applied to the desktop disc, under calibrated: A lower probability threshold could invalid several bots. Similarly, when the global decision threshold is used for the mobile disc, it can be overcalibrated: A higher probability threshold may be able to recover some loss of income without compromising the bot coverage.
To ensure justice across all traffic discs, we formulate calibration as a convex optimization problem. We perform joint optimization across all slices by fixing a total FPR budget (an upper limit of FPR for all slices combined) and solve to maximize the combined IVR on all slices together. The optimization must meet two conditions: (1) Each slice has a minimum robotic coverage establishing a lower found for its FPR, and (2) the combined FPR for all slices should not exceed the FPR budget.
Since the IVR-FPR curve for each slice can be approached as a square function of FPR, it finds that solving the joint optimization problem appropriate values ​​for each slice. We have found that calibration at Skive Level is essential for lowering the total FPR and increasing robotic coverage.
Implementation
To quickly adapt to changing bot patterns, we built an offline system that retrains and calibrates the model on a daily basis. For incoming traffic requests, the real-time component calculates the function values ​​using a combination of redis and read-only DB caches and runs the neural network on a horizontal scalable fleet of GPU deposits. To accommodate the real-time restriction, the entire infernic service running on AWS has a P99.9 latency less than five milliseconds.
To address data and model deviations during retraining and calibration, we put certain protective frames on input training data and model performance. For example, when purchase labels are missing for a few hours, the model can learn to invalid a large amount of traffic. Protections at least human density in every hour of one week prevent such behavior.
We have also developed disaster recovery mechanisms, such as fast rollbacks for a previously stable model when observing a sharp metric deviation and a repetition tool that can play traffic through a previously stable model or refresh real -time functions and publish delayed decisions that help prevent events with great influence.
In the future, we plan to add more features to the model, such as learned representations of users, IPS, USERAGENTS and Search queries. We presented our original work in that direction in our Neurips 2022 paper, “Self-monitored Premonition for Table Data on a large scale”. We also plan to experiment with advanced neural architectures such as deep and cross -networks, which can effectively capture functional data interviews.
Recognitions: Muneeb Ahmed