Instrument .NET applications

When instrumenting an existing service or application, there are a few steps along the path towards rich custom instrumentation:

Establish a baseline
Identify business-relevant fields
Establish a pattern for propagating context around
Instrument dependencies

To jump straight to an example .NET applications wired up to send events to Honeycomb, see our examples GitHub repository.

Establish a baseline

What does our service do? And when does it start to do that thing? To learn more about the concepts behind instrumentation, check out our resources in “About instrumentation” first.

In ASP.NET Core, creating a custom middleware is a straightforward way to capture metadata in line with the HTTP handler:

For example, this middleware is all we might need in order to capture one nicely populated event per HTTP request:

// In Startup.cs
app.UseMiddleware<HoneycombMiddleware>();

// In HoneycombMiddleware.cs
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Net.Http;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Http;

namespace dotnet_core_webapi_sample
{
  public class HoneycombMiddleware
  {
    private readonly RequestDelegate _next;
    private readonly HttpClient _client;
    public HoneycombMiddleware(RequestDelegate next)
    {
      _next = next;
      _client = new HttpClient();
      _client.DefaultRequestHeaders.Add("X-Honeycomb-Team","<WRITEKEY>");
    }

    public async Task InvokeAsync(HttpContext context)
    {
      var now = DateTime.UtcNow;
      var stopwatch = new Stopwatch();
      var fields = new Dictionary<string, object> ();
      fields.Add("request.path", context.Request.Path.Value);
      fields.Add("request.method", context.Request.Method);
      fields.Add("request.content_length", context.Request.ContentLength);
      stopwatch.Start();
      await _next.Invoke(context);
      stopwatch.Stop();
      fields.Add("duration_ms", stopwatch.ElapsedMilliseconds);
      fields.Add("response.http_status", context.Response.StatusCode);
      fields.Add("response.content_length", context.Response.ContentLength);
      var dataset = "<DATASET_NAME>";
      // TODO: Think about sending this out of band from the web request
      await _client.PostAsJsonAsync($"https://api.honeycomb.io/1/events/{dataset}", fields);
    }
  }
}

Clone our dotnet-core-webapi example app to see the HoneycombMiddleware in action.

Add custom fields

A baseline of standard fields is nice, but ultimately only we know the best fields by which to slice our production traffic in order to identify and isolate unexpected behavior.

Let’s imagine that our business is an ecommerce platform running in the cloud. These are examples of great types of custom fields to capture in our instrumentation, in order to investigate unexpected behavior (e.g. errors, increases in latencylatency, etc).

Our Infra	Our Deploy	Our Business	Our Execution
hostname	version / build	user/customer	payload characteristics
machine type	feature flags	shopping cart	timers around expensive RPCs

One common pattern for custom fields is to instrument dependencies like database calls by capturing timers (e.g. get_schema_duration_ms) as custom fields. Those timers can then even be paired with custom identifiers to describe—fairly precisely—how our system is behaving. (For example, see this query on our Gatekeeper Tour dataset, in which we compare the amount of time spent waiting for a get_schema call across a couple of different application-specific fields.)

Establish a pattern for propagating context

An important part of capturing events (and eventually building up traces!) is context propagation. Being able to attach metadata to a high-level event from functions enables us to collect the bits of metadata together to fully describe our unit of work.

One such pattern may be using dependency injection to ensure a locally-available HoneycombContext for collecting useful metadata.

First, we define a HoneycombContext class, which contains a simple Dictionary to contain our metadata, then make sure to merge that custom metadata into our basic HTTP fields inside InvokeAsync:

namespace dotnet_core_webapi_sample
{
  public class HoneycombContext
  {
    public Dictionary<string, object> Fields { get; }

    public HoneycombContext()
    {
      Fields = new Dictionary<string, object>();
    }
  }
  public class HoneycombMiddleware
  {
    public async Task InvokeAsync(HttpContext context, HoneycombContext honeycombContext)
    {
    // Collect metadata, call _next
    //...

    foreach (var field in honeycombContext.Fields) {
      fields.Add(field.Key, field.Value);
    }
    var dataset = "dotnet-core-webapi";
    // TODO: Think about sending this out of band from the web request
    await _client.PostAsJsonAsync($"https://api.honeycomb.io/1/events/{dataset}", fields);
    }
  }
}

And we can rely on dependency injection in order to add a scoped HoneycombContext to all of our services:

// in Startup.cs
public void ConfigureServices(IServiceCollection services)
{
  services.AddScoped<HoneycombContext, HoneycombContext>();
  // ...
}

This injection allows any of our controllers to add useful fields to our Honeycomb event, as in the example below:

namespace dotnet_core_webapi_sample.Controllers
{
    [Route("api/[controller]")]
    [ApiController]
    public class AuthController : ControllerBase
    {
        private HoneycombContext Context { get; }

        public AuthController(HoneycombContext context)
        {
            Context = context;
        }

        private bool IsAuthenticated()
        {
            if (context.Cache.Get(context.Request.User) != null)
            {
              Context.Fields.Add("app.auth_hit_cache", true);
              return true;
            }
            return db.Get(context.Request.User) != null;
        }
    }
}

In conclusion

To see a full working .NET application illustrating these principles, please refer to our Examples repository on GitHub.

As much as testing or documentation, instrumentation should be an updated, evolving practice in writing and shipping software.

For more wisdom on the topic of instrumentation, check out the Honeycomb blog.