[Query Testing] Winner visibility
complete
Leon Anthony MENGE
As a Search Manager, I aim to streamline the process of identifying the winning version of the Query Test directly from the UI. This enhancement will save time and improve efficiency in decision-making.
Currently, accessing the winning version of the Query Test requires clicking on each test individually, which is time-consuming. To optimize this process, we propose displaying the winning status directly on the UI. For instance, the status could be represented by the color green to indicate that the variant won. (see current version in screenshot)
An additional and helpful feature would be an automatic extension of the testing time if the version wins over control and lacks sufficient data for the algorithm to designate it as the master query. This way, the algorithm can gather more data.
Andreas Wagner
complete
Andreas Wagner
in progress
Andreas Wagner
Hi Leon,
another question that came up during the preparation for DEV.
Right now we do not judge if an Experiment has a winner or not since we always evaluate at least 2 different KPIs (clicks & carts) and in some cases we might end up with a contrary outcome. Therefore we can't flag Experiments by a winner status.
But what we potentially could do is to flag the Experiment if at least one of the KPIs showed a significant positive outcome or not (again green vs. red)
Would this slightly different solution help you in terms of efficiency?
Leon Anthony MENGE
Hi Andreas,
yes, this solution would still help me a lot.
Andreas Wagner
Hi Leon,
the UI is already pretty packed. Would it also be ok if we would color-indicate the status like green check-marks for winner (variant wins) and red check-marks if control wins ?
Regarding longer run-times:
The problem is that in ecommerce we are confronted a lot of different biases (seasonal, weather, marketing,...) and cookie persistence challenges. The longer the experiment/test runs the more likely it will be influenced by these biases. Therefore we abort every experiment that runs longer than 28 days.
Leon Anthony MENGE
Hi Andreas,
yes sure.
Thant's a good point. But should it not be the same for both tested queries?
Andreas Wagner
Leon Anthony MENGE: well we only have a winner when the variant wins significantly otherwise control always wins - therefore 2 colors should be enough -right?
Leon Anthony MENGE
Andreas Wagner yes. Two should be enough. One for variant as a winner and one for control. Tests without a winner can stay neutral (for example). It’s just important that I can easily it
Andreas Wagner
under review