Redirect vs. query testing
planned
Andreas Wagner
planned
Andreas Wagner
under review
Andreas Wagner
Hello everyone, we need your feedback again:
We strongly believe that automatically disabling query testing for clusters with redirects contradicts the data-driven philosophy of searchHub. This is supported by this year's query testing data, which shows that in 25% of the tests (over 10.000 overall), the variant outperformed with an average uplift of over 11%.
To fully unlock this potential for our customers, it's essential to properly enable search-redirect testing. This has been possible since SmartQuery 2.3, thanks to the extended integration. In this setup, 50% of users are directed to the redirected page, while the other 50% see the organic search results page. This approach ensures statistically valid and rapid decision-making.
Disabling query testing for redirects, on the other hand, would prevent customers from understanding whether their redirects outperform or underperform compared to organic search over time—reverting to outdated practices from a decade ago in eCommerce.
Andreas Wagner
Merged in a post:
Deactivate QueryTesting for specific Clusters
Leon Anthony MENGE
As a Search Manager, I want to be able to deactivate QueryTesting for specific Clusters in order to ensure wanted results.
When there is a redirection in a Cluster which is in QueryTesting, the redirection does not work for the testing queries. Because the tested queries have to have a high amount of traffic, the redirected page does lose a lot of traffic. Even though I think it is great that there can still be gathered some data for the search results, this behavior is in some situations not desired. For instance, during an important Campaign, I would have to check the QueryTesting Tab on a daily basis in order to make sure that the redirection for that specific cluster and campaign is working as expected.
It would be helpful to be able to manually deactivate the possibility of QueryTesting for those Clusters.
It might also be helpful to have it deactivated for Clusters which meet certain requirements. One setting could be if a redirection is scheduled (as an indicator for a campaign)
Andreas Wagner
planned
Andreas Wagner
Hi, and thank you for the feedback!
Before we proceed with it, I'd like to ensure we're all aligned:
As you may know, clusters are not stable over time. This means that if a USER manually sets a redirect for a cluster, that action is based on the data available at that specific point in time. If this point in time is in the past, there is a high chance that numerous manual or automatic changes (such as merges or unmerges) have occurred in the meantime.
If we deactivate query testing in these situations, we lose the ability to automatically detect cases where these changes result in poorer performance in terms of KPIs—which is precisely what an Auto-Pilot should be able to do.
Andreas Wagner
Hi Leon,
we are currently reviewing this request and we are having a hard time coming up with a conceptual draft on how a UI or even storing this kind of information could look like since we have to handle so much complexity.
- just an UI with a database would not work as clusters change automatically or due to user interaction
- we have the timed redirects and the redirect-uploads
- different users might want to do competitive actions
therefore we came up with a much simpler approach.
Why not just have a configuration flag which says - exclude queries in clusters with a redirect from testing?
Would this also work for you at least for the near and mid-term future?
Best Andy
Leon Anthony MENGE
Hi Andreas,
do you mean a setting to deactivate it for all clusters with a redirect, or do you mean by flag something like the star for choosing a masterQuery manually on cluster level so that it can be chosen for each cluster?
I would rather spend more time going through queryTesting than loosing those tests on all clusters with redirections. But being able to choose for each would help.
Best regards!
Andreas Wagner
Hi Leon,
A) yes the next version of SmartQuery which we are already working on will exactly support that 50/50 splits.
B) I get your point and I agree it’s valid. I just don’t know yet how to expose such a (deactivation) feature into the UX without confusing the user - have to think about it
Leon Anthony MENGE
Hi Andreas,
A) Can't wait to have it :)
B) Maybe a Tab inside QueryTesting? I'll add a sketch to show you what I mean.
Andreas Wagner
Hi Leon, I think I get what you want but I still have some questions:
- Query Testing's main goal is to find get (statistically significant results) instead of gut-feeling. In order get these we have to explore the result space. Otherwise we would not be able to prove that the "wanted behavior" might not lead to the user satisfaction and business outcome. For example you could never be sure if the campaign / redirect really improved the KPIs beyond a local-optimum. - So how should we balance that?
- We already provide the ability to abort tests manually. If you do it it's guaranteed that the test is not restarted for the next 28 days at least. - Is this not enough?
Leon Anthony MENGE
Hello Andreas,
- I totally get your point, but I think in order to be able to actually test the performance of a redirection / campaign we would need a traffic / user splitting. So that it is actually possible to have for example a 50 / 50 split. What do you think about it?
- This solves 50% of my "issue". Sure, I can stop the test if I want / have to. However, I still have to check if there is one on a regular (if not even daily) basis.
Please don't get me wrong, I think Query Testing is really awesome. There are just those 2% of cases where it's not.