%kb_name - %short_descr - Knowledge Portal

Description

This article describes step-by-step scenarios for troubleshooting xConnect certificate issues:

Scenario 1 - Issues with Server Certificate
Scenario 2 - Issues with Client Certificate
Scenario 3 - 403 Forbidden error related to the Client Certificate

Symptoms of the issues might include errors about certificates in log files and the following error messages in the Experience Analytics user interface (UI):

This article describes step-by-step scenarios for troubleshooting the most frequent issues in Experience Analytics application

Scenario 1 - Issues with Server Certificate

Symptoms:

The following errors appear in the log of the CD server:

ERROR Cannot start analytics Tracker
Exception: System.Net.Http.HttpRequestException
Message: An error occurred while sending the request.
Source: Sitecore.Xdb.Common.Web
   at Sitecore.Xdb.Common.Web.Synchronous.SynchronousExtensions.SuspendContextLock[TResult](Func`1 taskFactory)
   at Sitecore.Analytics.DataAccess.Dictionaries.DataStorage.ReferenceDataClientDictionary.EnsureDefinitionType(String definitionTypeName)
   at Sitecore.Analytics.DataAccess.Dictionaries.DataStorage.ReferenceDataClientDictionary.LoadAs[T](Object key)
   at Sitecore.Analytics.DataAccess.Dictionaries.AverageCounterExtensions.MeasureMilliseconds[T](AverageCounter counter, Func`1 func)
   at Sitecore.Analytics.DataAccess.Dictionaries.ReferenceDataDictionary`2.Get(TKey key, LookupStrategy strategy)
   at Sitecore.Analytics.DataAccess.Dictionaries.UserAgentsDictionary.Register(String userAgentName)
   at Sitecore.Analytics.Tracking.CurrentVisitContext.set_UserAgent(String value)
   at Sitecore.Analytics.Pipelines.CreateVisits.InitializeWithRequestData.Process(CreateVisitArgs args)
   at (Object , Object[] )
   at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
   at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain)
   at Sitecore.Analytics.Pipelines.CreateVisits.CreateVisitPipeline.Run(CreateVisitArgs args)
   at Sitecore.Analytics.Tracking.StandardSession.CreateInteraction(HttpContextBase httpContext)
   at Sitecore.Analytics.Pipelines.InitializeTracker.CreateVisit.Process(InitializeTrackerArgs args)
   at (Object , Object[] )
   at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
   at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain)
   at Sitecore.Analytics.Pipelines.InitializeTracker.InitializeTrackerPipeline.Run(InitializeTrackerArgs args)
   at (Object , Object[] )
   at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
   at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain)
   at Sitecore.Analytics.Pipelines.StartTracking.StartTrackingPipeline.Run(StartTrackingArgs args)
   at Sitecore.Analytics.DefaultTracker.StartTracking()

Nested Exception

Exception: System.Net.WebException
Message: The request was aborted: Could not create SSL/TLS secure channel.
Source: System
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)

Solution:

Request your xConnect instance using a browser. If the certificate prompt appears, just close it.

Ensure that the connection is secure:

Connection is secure:
Connection is not secure, connection between Sitecore and xConnect cannot be established:

If the connection is not secure, ensure that the server certificate that is used for the HTTPS site binding has not expired. Also, ensure that the Issued To property of the certificate matches the hostname that you use.

Note: If a certificate thumbprint is copied from the Microsoft Management Console (MMC) application, it might contain invisible symbols in the beginning. Such invisible symbols must be manually removed before applying a certificate thumbprint in a Sitecore XP configuration.

Scenario 2 - Issues with Client Certificate

Symptoms:

The following errors can be found in the logs:

[Experience Analytics]: Failed to synchronize segments. Message: The certificate was not found. Store: My, Location: CurrentUser, FindType: FindByThumbprint, FindValue: 83DCC21BBF54D76F71D7B67EA2319273BCDA8E10, InvalidAllowed: True.. Details:    at Sitecore.Xdb.Common.Web.Synchronous.SynchronousExtensions.SuspendContextLock[TResult](Func`1 taskFactory)
   at Sitecore.ExperienceAnalytics.Core.Repositories.ReferenceData.ReferenceDataSegmentReader.GetAll(NameValueCollection readingPreferences)
   at Sitecore.ExperienceAnalytics.Aggregation.Repositories.AggregationSegmentReader.GetAll(NameValueCollection readingPreferences)
   at Sitecore.ExperienceAnalytics.Client.Deployment.SyncSegmentsManager.GetSegmentsToSynchronize()
   at Sitecore.ExperienceAnalytics.Client.Deployment.SyncSegmentsManager.SynchronizeAllSegments()
   at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.ExperienceAnalytics.Client.Deployment.SyncSegmentsProcessor.<process>d__4.MoveNext()

Exception System.InvalidOperationException: The certificate was not found. Store: My, Location: CurrentUser, FindType: FindByThumbprint, FindValue: 83DCC21BBF54D76F71D7B67EA2319273BCDA8E10, InvalidAllowed: True.
   at Sitecore.Xdb.Common.Web.Synchronous.SynchronousExtensions.SuspendContextLock[TResult](Func`1 taskFactory)
   at Sitecore.ExperienceAnalytics.Core.Repositories.ReferenceData.ReferenceDataSegmentReader.Get(IEnumerable`1 keys, NameValueCollection readingPreferences)
   at Sitecore.ExperienceAnalytics.Aggregation.Repositories.AggregationSegmentReader.Get(IEnumerable`1 keys, NameValueCollection readingPreferences)
   at Sitecore.ExperienceAnalytics.Api.RequestTypeResolver.GetReportSegments()
   at Sitecore.ExperienceAnalytics.Api.RequestTypeResolver.GetRequestType()
   at Sitecore.ExperienceAnalytics.Api.Http.ModelBinding.ReportQueryModelBinder.GetModelFromBindingContext(HttpActionContext actionContext, ModelBindingContext bindingContext)
   at Sitecore.ExperienceAnalytics.Api.Http.ModelBinding.ReportQueryModelBinder.BindModel(HttpActionContext actionContext, ModelBindingContext bindingContext)
   at System.Web.Http.ModelBinding.ModelBinderParameterBinding.ExecuteBindingAsync(ModelMetadataProvider metadataProvider, HttpActionContext actionContext, CancellationToken cancellationToken)
   at System.Web.Http.Controllers.HttpActionBinding.<executebindingasynccore>d__12.MoveNext()
   ...

Exception System.InvalidOperationException: Ensure definition type did not complete successfully. StatusCode: 401, ReasonPhrase: 'Invalid certificate', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Pragma: no-cache
  Cache-Control: no-cache
  Date: Thu, 02 Jan 2020 15:03:36 GMT
  Server: Microsoft-IIS/10.0
  X-AspNet-Version: 4.0.30319
  X-Powered-By: ASP.NET
  Content-Length: 0
  Expires: -1
}
   at Sitecore.Xdb.Common.Web.Synchronous.SynchronousExtensions.SuspendContextLock[TResult](Func`1 taskFactory)
   at Sitecore.ExperienceAnalytics.Core.Repositories.ReferenceData.ReferenceDataSegmentReader.Get(IEnumerable`1 keys, NameValueCollection readingPreferences)
   at Sitecore.ExperienceAnalytics.Aggregation.Repositories.AggregationSegmentReader.Get(IEnumerable`1 keys, NameValueCollection readingPreferences)
   at Sitecore.ExperienceAnalytics.Api.RequestTypeResolver.GetReportSegments()
   at Sitecore.ExperienceAnalytics.Api.RequestTypeResolver.GetRequestType()
   at Sitecore.ExperienceAnalytics.Api.Http.ModelBinding.ReportQueryModelBinder.GetModelFromBindingContext(HttpActionContext actionContext, ModelBindingContext bindingContext)
   at Sitecore.ExperienceAnalytics.Api.Http.ModelBinding.ReportQueryModelBinder.BindModel(HttpActionContext actionContext, ModelBindingContext bindingContext)
   at System.Web.Http.ModelBinding.ModelBinderParameterBinding.ExecuteBindingAsync(ModelMetadataProvider metadataProvider, HttpActionContext actionContext, CancellationToken cancellationToken)
   at System.Web.Http.Controllers.HttpActionBinding.<executebindingasynccore>d__12.MoveNext()
   ...

Solution:

Ensure that the certificate thumbprint in the connection strings of Sitecore XP roles matches the thumbprint of the certificate in use:

<add name="xconnect.collection.certificate" 
connectionString="StoreName=My;StoreLocation=CurrentUser;FindType=FindByThumbprint;FindValue=83DCC21BBF54D76F71D7B67EA2319273BCDA8E19" />

<add name="sitecore.reporting.client.certificate" 
connectionString="StoreName=My;StoreLocation=CurrentUser;FindType=FindByThumbprint;FindValue=83DCC21BBF54D76F71D7B67EA2319273BCDA8E19" />

<add name="xdb.marketingautomation.operations.client.certificate" 
connectionString="StoreName=My;StoreLocation=CurrentUser;FindType=FindByThumbprint;FindValue=83DCC21BBF54D76F71D7B67EA2319273BCDA8E19" />

<add name="xdb.marketingautomation.reporting.client.certificate" 
connectionString="StoreName=My;StoreLocation=CurrentUser;FindType=FindByThumbprint;FindValue=83DCC21BBF54D76F71D7B67EA2319273BCDA8E19" />

<add name="xdb.referencedata.client.certificate" 
connectionString="StoreName=My;StoreLocation=CurrentUser;FindType=FindByThumbprint;FindValue=83DCC21BBF54D76F71D7B67EA2319273BCDA8E19" />

Ensure that the validateCertificateThumbprint value of xConnect and Marketing Automation roles matches the thumbprint of the certificate in use. The setting can be found in the following file: \App_Config\AppSettings.config
Ensure that the xconnect.collection.certificate connection string of the AutomationEngine job has correct certificate thumbprint:
\App_Data\jobs\continuous\AutomationEngine\App_Config\ConnectionStrings.config
[For Sitecore XP 9.1+] Ensure that xconnect.collection.certificate, xconnect.configuration.certificate, and xconnect.search.certificate connection strings of the ProcessingEngine job have the correct thumbprints: \App_Data\jobs\continuous\ProcessingEngine\App_Config\ConnectionStrings.config
Ensure that the AllowInvalidClientCertificates app setting is true on Sitecore XP roles if your certificate is self-signed (the setting can be found in the Web.config file by default):
```
<add key="AllowInvalidClientCertificates" value="True" />
```
Ensure that the AllowInvalidClientCertificates app setting is true on xConnect roles if your certificate is self-signed. The setting can be found in the following files by default:
\App_Config\AppSettings.config
\App_Data\jobs\continuous\JOB_NAME_GOES_HERE\App_Config\AppSettings.config

Note: Thumbprint value needs to be uppercase in the configuration.

Depending on the implementation of the specific solution, do the following:

For on-prem:
1. Check the certificate purpose using PowerShell:
```
$cert = Get-ChildItem -Path Cert:\LocalMachine\My | Where-Object {$_.Thumbprint -match "XXXX...XXXX"} 
foreach($key in $cert.Extensions){if('EnhancedKeyUsages' -in $key.psobject.Properties.Name){ $key.EnhancedKeyUsages.FriendlyName }}
```
  where XXXX...XXXX is the thumbprint from the connection string.
  
  Note that the example uses "LocalMachine" StoreLocation and "My" StoreName. If you have different values in the connection strings, update the script accordingly:
```
<add name="xconnect.collection.certificate" connectionString="StoreName=My;StoreLocation=LocalMachine;FindType=FindByThumbprint;FindValue=XXXX...XXXX" />
```
  The script must output the Client Authentication in the list:
2. Ensure that the process that runs the Sitecore XP and xConnect application has access to the client certificate private keys:
  
  2.1. Open the Windows Run window and enter mmc command.
  
  2.2. In the opened window click File, in the drop-down list select Add/Remove Snap In, among the Available snap-ins select Certificates, click Add. In the pop-up window, select the Computer Account radio button and click Finish.
  
  2.3. In the left-side Console Root area, expand Certificates (Local Computer), expand Personal folder, click the Certificates folder. In the central window area, find your client certificate in the table (to ensure that the certificate is correct, double click it, select Details and select Thumbprint in the list. It must be the same as in the connection strings).
  
  2.4. Right click on the certificate, select All Tasks in the drop-down list, select Manage Private Keys:
  
  2.5. Ensure that the users that run the applications are present in the list:
  
  If the user is missing, click Add, and in the opened window, specify the server root in the From this location field and type the user name in the search window. In the search window, type IIS AppPool\UserName, where UserName is the name of the user that is used for running the application:
  
  Task Manager can be used for finding a user that runs the application. For example, the user name is "sc902.xconnect" here.
3. Ensure that the trusted root does not contain non-self-signed certificates using the following PowerShell script:
```
Get-Childitem cert:\LocalMachine\root -Recurse | Where-Object {$_.Issuer -ne $_.Subject}
```
  If there are any, move them to the Intermediate Certification Authorities:
```
Get-Childitem cert:\LocalMachine\root -Recurse | Where-Object {$_.Issuer -ne $_.Subject} | Move-Item -Destination Cert:\LocalMachine\CA
```
4. Ensure that xConnect instances accept Client Certificates:
  
  4.1. Select xConnect site in IIS.
  
  4.2. Click the SSL Settings icon in the IIS section and ensure that the configuration looks as follows:
For Azure Web Apps:
1. Ensure that Incoming Client certificate is enabled on xConnect services:
2. Ensure that the WEBSITE_LOAD_CERTIFICATES app setting is defined for all Sitecore and xConnect Web Apps and contains the same thumbprint as the certificate:

Scenario 3 - 403 Forbidden error related to the Client Certificate

Symptoms:

The following errors can be found in the IIS logs on a XConnect instance.

Exception: Sitecore.XConnect.XdbCollectionUnavailableException
Message: The HTTP response was not successful: Forbidden
Source: Sitecore.Xdb.Common.Web
at Sitecore.Xdb.Common.Web.Synchronous.SynchronousExtensions.SuspendContextLock[TResult](Func`1 taskFactory)
at Sitecore.XConnect.Client.XConnectSynchronousExtensions.SuspendContextLock(Func`1 taskFactory)
at Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.Initialize(XmlNode configNode)
at Sitecore.Configuration.DefaultFactory.CreateObject(XmlNode configNode, String[] parameters, Boolean assert, IFactoryHelper helper)
at Sitecore.Configuration.DefaultFactory.CreateObject(XmlNode configNode, String[] parameters, Boolean assert)
at Sitecore.Configuration.DefaultFactory.CreateObject(String configPath, String[] parameters, Boolean assert)
at Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.GetClient(String clientConfigPath)
at Sitecore.Analytics.XConnect.DataAccess.BatchEnabledXdbRuntimeContext..ctor(IXdbContextFactory factory)
at Sitecore.Analytics.Events.BatchEventHandler.OnBatchStarting(Object sender, EventArgs args)
at Sitecore.Events.Event.EventSubscribers.RaiseEvent(String eventName, Object[] parameters, EventResult result)

Note: The error with the status code 403.13, the sub-status code 13 is related to a Client Certificate Revocation check the IIS logs.

Exception System.InvalidOperationException: Ensure definition type did not complete successfully. StatusCode: 403, ReasonPhrase: 'Forbidden', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Date: Sun, 1 Mar 2021 05:02:23 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Content-Length: 1377
Content-Type: text/html
}
at Sitecore.Xdb.Common.Web.Synchronous.SynchronousExtensions.SuspendContextLock[TResult](Func`1 taskFactory)
at Sitecore.ExperienceAnalytics.Core.Repositories.ReferenceData.ReferenceDataSegmentReader.Get(IEnumerable`1 keys, NameValueCollection readingPreferences)
at Sitecore.ExperienceAnalytics.Aggregation.Repositories.AggregationSegmentReader.Get(IEnumerable`1 keys, NameValueCollection readingPreferences)
at Sitecore.ExperienceAnalytics.Api.RequestTypeResolver.GetReportSegments()
at Sitecore.ExperienceAnalytics.Api.RequestTypeResolver.GetRequestType()
at Sitecore.ExperienceAnalytics.Api.Http.ModelBinding.ReportQueryModelBinder.GetModelFromBindingContext(HttpActionContext actionContext, ModelBindingContext bindingContext)
at Sitecore.ExperienceAnalytics.Api.Http.ModelBinding.ReportQueryModelBinder.BindModel(HttpActionContext actionContext, ModelBindingContext bindingContext)
at System.Web.Http.ModelBinding.ModelBinderParameterBinding.ExecuteBindingAsync(ModelMetadataProvider metadataProvider, HttpActionContext actionContext, CancellationToken cancellationToken)
at System.Web.Http.Controllers.HttpActionBinding.>ExecuteBindingAsyncCore<d__12.MoveNext()
   ...

Solution:

Use the certificate that is unrevoked and ensure that the root certificates are valid. To perform the validation, follow the steps outlined in the second scenario's Solution.
Alternatively, configure IIS to disable the certificate revocation check as outlined in the following articles:
- Client Certificate Revisited: How to troubleshoot client certificate related issues
- Disable Client Certificate Revocation (CRL) Check on IIS

Troubleshooting xConnect certificate issues

Description

Scenario 1 - Issues with Server Certificate

Scenario 2 - Issues with Client Certificate

Scenario 3 - 403 Forbidden error related to the Client Certificate