Question 1

Question

How exactly does Sitecore calculate when statistical relevance for the test is reached / when it will be reached? How does Sitecore suggest the number of visitors and number of days for the test to run?

Answer

• Using the Pearson's chi-squared test method, Sitecore calculates the number of visitors needed to validate if any of the alternative experiences has a significantly better result than the original. This number depends on the number of experiences to be tested and the required confidence level.
• When the required number of visitors is reached, Sitecore uses Pearson's method to validate if there is a winner or not. 
• When starting a test, Sitecore shows the required number of visitors needed to get a statistically valid result. If there is historical visitor data for the page, Sitecore forecasts the expected number of days that it will take to finish the test. The forecast is based on historical data for the last month in relation to the needed number of visitors to the content. If traffic allocation is added, this will also has effect on the forecast. 

Question 2

Question

If the number of visits is reached before or after than the minimum or maximum dates and the test result is statistically significant, how long will Sitecore run the test?

Answer

• When the minimum duration is reached, Sitecore will first try to determine a test winner. There should be test data from all the days in one week before declaring a winner.
• When the maximum duration is reached, Sitecore suspends the test and prompts the user to take further action.

Question 3

Question

Why would I set the minimum and maximum durations of the test when Sitecore already predicts how long a test should run?

Answer

• Sitecore provides a forecast for the test, and cannot take into account significant changes in traffic that could occur while the test is running, for example, a new campaign linking to the page.
• For general test governance and SEO, it is best practice to have a maximum duration for the test.