How to reduce the batch size during analytics data aggregation


Description

The following messages can appear in the log files during the analytics data aggregation process. This means that the batch has failed to be stored and will be split and retried. You can safely ignore the WARN messages, however if they appear very often, you can reduce the batch size as described in the Solution section.

Note: If the message is logged at the ERROR level, the solution in this article is not applicable. Please contact Sitecore Support in this case.

WARN SqlReportingStorageProvider: Could not store entire batch. Try to storing in smaller batches.
Exception: System.Data.SqlClient.SqlException
Message: Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
Source: .Net SqlClient Data Provider
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Sitecore.Analytics.Aggregation.SqlReportingStorageProvider.<>c__DisplayClass7_0.<Commit>b__0()
   at Sitecore.Data.DataProviders.NullRetryer.ExecuteNoResult(Action action, Action recover)
   at Sitecore.Analytics.Aggregation.SqlReportingStorageProvider.Commit(SqlDataApi api, IReportingStorageItemBatch batch, Func`2 filter)
   at Sitecore.Analytics.Aggregation.SqlReportingStorageProvider.<>c__DisplayClass2_1.<StoreBatch>b__1()
   at Sitecore.Data.DataProviders.NullRetryer.ExecuteNoResult(Action action, Action recover)
   at Sitecore.Analytics.Aggregation.SqlReportingStorageProvider.StoreBatch(IReportingStorageItemBatch batch)
   at Sitecore.Analytics.Aggregation.SqlReportingStorageProvider.Store(IReportingStorageItemBatch batch)

Solution

To resolve this issue, try decreasing the maximumBatchSize value in the \App_Config\Sitecore\Marketing.xDB\Sitecore.Analytics.Processing.Aggregation.Services.config file:

<aggregator type="Sitecore.Analytics.Aggregation.InteractionAggregationAgent, Sitecore.Analytics.Aggregation">
  ...
  <param desc="maximumBatchSize" type="Sitecore.Analytics.Core.ConfigurationHelper, Sitecore.Analytics.Core" factoryMethod="ToShort" arg0="32" />
  ...
</aggregator>

If you want to rebuild your reporting database, you can also decrease the same value for the <historyWorker> node:

<historyWorker type="Sitecore.Analytics.Aggregation.Data.Processing.InteractionHistoryWorker, Sitecore.Analytics.Aggregation">
  <param desc="maximumBatchSize" type="Sitecore.Analytics.Core.ConfigurationHelper, Sitecore.Analytics.Core" factoryMethod="ToShort" arg0="16" />
</historyWorker>

For more details, see the following topic: Batch aggregation.