How to capture outgoing HTTP(S) requests


Description

Sometimes, for troubleshooting purposes, it is necessary to use web debugging proxies to capture and investigate the HTTP traffic of your website.

A good example of such a proxy is Fiddler.

Fiddler changes proxy server settings for the currently logged in Windows user to allow capturing traffic from web browsers or other applications. Specifically, it changes the system's proxy settings to the 127.0.0.1:8888 address.

However, this does not affect applications such ASP.NET sites, because they use different user accounts and their own proxy settings.

This article describes how to configure an ASP.NET web site to use a proxy, to capture web service calls between the ASP.NET and the web services that the application is using.

Solution

To configure and ASP.NET application to send web service calls through a web debugging proxy follow these steps:

  1. Create or update the global.asax file to treat all remote certificates as valid:
    <%@ Import Namespace="System.Net" %>
    <%@ Import Namespace="System.Net.Security" %>
    <script>
        public void Application_Start()
        {
            ServicePointManager.ServerCertificateValidationCallback = new RemoteCertificateValidationCallback(delegate { return true; });
        }
    </script>
  2. Edit the web.config file to set the proxy settings, for example:
    <system.net>
      <defaultProxy useDefaultCredentials="true" enabled="true">
        <proxy usesystemdefault="false" proxyaddress="http://127.0.0.1:8888" bypassonlocal="false" autoDetect="false"/>
      </defaultProxy>
    </system.net>

    For a detailed explanation of each attribute see Microsoft documentation:
    https://learn.microsoft.com/en-us/dotnet/framework/configure-apps/file-schema/network/proxy-element-network-settings
    https://learn.microsoft.com/en-us/dotnet/framework/configure-apps/file-schema/network/defaultproxy-element-network-settings

  3. Make sure the web debugging proxy can capture and HTTPS traffic. For example, see how to enable the feature for Fiddler Classic:
    https://docs.telerik.com/fiddler/configure-fiddler/tasks/decrypthttps

  4. After the traffic has been captured, it can be saved for further sharing and analysis. For example, as a Session Archive Zip in Fiddler Classic:
    https://docs.telerik.com/fiddler/save-and-load-traffic/tasks/createsaz

  5. Note, to capture requests to the Publishing Service an additional change is required. The configuration of the Standalone or Content Management role must be patched to enable the proxy, for example:
    <?xml version="1.0" encoding="utf-8"?>
    <configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:set="http://www.sitecore.net/xmlconfig/set/" xmlns:role="http://www.sitecore.net/xmlconfig/role/">
        <sitecore role:require="Standalone or ContentManagement">
            <settings>
                <setting name="PublishingService.ProxyAddress" set:value="http://localhost:8888" />
                <setting name="PublishingService.ProxyEnabled" set:value="false" />
            </settings>
        </sitecore>
    </configuration>

Make sure to use the steps above for troubleshooting purposes only and revert the changes when the capturing of requests is no longer needed.