1. Docs
2. Troubleshoot
3. Recommended Migrations
4. From Beelines to OpenTelemetry
5. Beeline for Ruby

Beeline for Ruby Reference

Table of Contents

The Ruby Beeline for Honeycomb is quick and easy way to instrument your Ruby application. It includes several optional wrappers that automatically instrument HTTP requests and database queries.

It provides automatic tracing out of the box, linking database queries to the HTTP request from which they originated. While this is a great way to get general insights about your app as quickly as possible, as you forge ahead on your observability journey you may find you would like to add new events or traces to add more details specific to your app. The Ruby Beeline provides simple interfaces for adding both.

To see an example of the Ruby Beeline in action, try out the examples.

If you would like to see more options in the Ruby Beeline, please file an issue or vote up an already filed one!

Requirements 

• A Ruby app, preferably one that listens for HTTP requests or makes SQL calls
• Ruby 2.3 or greater
• A Honeycomb API key

You can find your API key in your Environment Settings. If you do not have an API key yet, sign up for a free Honeycomb account.

Quick Installation 

Add Custom Fields To Events 

The Ruby Beeline tracks the current span within the overall trace. This allows you to add as many additional custom fields as you like.

Here is an example of adding a custom field to the current span:

def calculate_big_number
  # ... do work, find a big number
  Honeycomb.add_field('big_num', result)
  # ... do thing with big_num, maybe something fails
  Honeycomb.add_field('error', err)
end

Additional fields are added under the app. namespace. For example, the field above would appear in your event as app.big_num. The namespace groups your fields together to make them easy to find and examine.

If you prefer to avoid the app. namespace, for example in order to overwrite an automatically-populated field, you can have your block take the current span as a parameter and call add_field directly on that instance.

def calculate_another_big_number
  Honeycomb.start_span(name: 'big_number_calculation') do |span|
    # ... do work, find another big_num
    span.add_field('handler.name', 'calc_other_big_num')
  end
end

These additional fields are your opportunity to add important and detailed context to your instrumentation. Put a timer around a section of code, add per-user information, include details about what it took to craft a response, and so on. It is expected that some fields will only be present on some requests. Error handlers are a great example of this; they will obviously only exist when an error has occurred.

It is common practice to add in these fields along the way as they are processed in different levels of middleware. For example, if you have an authentication middleware, it would add a field with the authenticated user’s ID and name as soon as it resolves them. Later on in the call stack, you might add additional fields describing what the user is trying to achieve with this specific HTTP request.

If you are interested in adding custom fields to all spans, use add_field_to_trace instead.

add_field_to_trace adds the field to both the currently active span and all other spans that have yet to be sent and are involved in this trace that occur within this process. Additionally, these fields are packaged up and passed along to downstream processes if they are also using a beeline. This function is good for adding information that is better scoped to the request than this specific unit of work, such as user IDs, globally relevant feature flags, or errors. Fields added here are also prefixed with app.

Adding Spans To a Trace 

We encourage people to think about instrumentation in terms of “units of work”. As your program grows, what constitutes a unit of work is likely to be portions of your overall service rather than an entire run. Spans are a way of breaking up a single external action (say, an HTTP request) into several smaller units in order to gain more insight into your service. Together, many spans make a trace, which you can visualize traces within the Honeycomb query builder.

Adding spans with the Ruby Beeline is easy! Here is an example, where calls to slow_operation get their own span within the trace:

def slow_operation
  Honeycomb.start_span(name: 'slow_operation') do |span|
    # ... go on and do the slow opp, add more data along the way
    span.add_field('interesting_thing', 'banana')
  end
end

Spans always get a few fields:

• a name - in this case slow_operation
• a duration - how much time elapsed between when the span was started and sent
• a service_name - generally configured during Beeline initialization
• several IDs - trace, span, and parent identifiers (UUIDs)

You are always welcome (and encouraged!) to add additional fields to spans using the Honeycomb.add_field method.

Other Integrations 

The Ruby Beeline integrates with other common ruby gems in order to populate spans and traces with useful information.

Active Support 

The Ruby Beeline can listen to any ActiveSupport instrumentation calls in your application to produce spans from subscribed event types. This is the primary means by which the Rails framework is instrumented.

Subscribe And Use The Default Handler 

Add the identifier for the ActiveSupport notification to an array assigned to the Beeline configuration’s notification_events property. The Beeline will generate a span for the notification event with the event’s keys added as fields. Event keys whose values are complex objects will be coerced to strings with #to_s.

Honeycomb.configure do |config|
  config.write_key = 'YOUR_API_KEY'
  config.service_name = 'your-service-name'
  config.notification_events = %w[
    custom_events.are_cool
  ].freeze
end

Customize Notification Handling by Event Type 

To subscribe to a notification event type and customize the fields that will appear on its span, use the Beeline configuration’s #on_notification_event method. Call #on_notification_event with a specific event name, provide a block that receives a name, span, and payload, then customize the span within the block. For example, to add fields to the span for a perform.active_job notification event with values from the Job object on its job key:

Honeycomb.configure do |config|
  config.write_key = "YOUR_API_KEY"
  config.service_name = 'your-service-name'

  # subscribe to some interesting Rails notifications
  config.notification_events = %w[
    sql.active_record
    render_template.action_view
    render_partial.action_view
    render_collection.action_view
    perform.active_job
    process_action.action_controller
    send_file.action_controller
    send_data.action_controller
    deliver.action_mailer
  ].freeze

  # customize the handling of a particular notification
  # it's OK if the name also appears in the list given to #notifications_events!
  config.on_notification_event("perform.active_job") do |name, span, payload|
    span.add_field("#{name}.adapter", payload["adapter"].class)

    job = payload["job"]
    span.add_field("#{name}.job_class", job.class)

    span.add_field("#{name}.provider_id", job.provider_job_id)
    span.add_field("#{name}.scheduled_at", job.scheduled_at)
  end

  # on_notification_event can be called multiple times with different event names
  # subscribe and customize the handling of some custom notification type of your own
  config.on_notification_event("custom_events.are_cool") do |name, span, payload|
    # let's reuse the behavior of the default handler
    config.default_handler.call(name, span, payload)
    # ... and then add to or change the data on the span for this event
    span.add_field("#{name}.reason", "Because you can annotate their spans how you like.")
    span.add_field("#{name}.coolness_factor", payload["level_of_cool"])
  end

AWS 

The Ruby Beeline will automatically instrument any calls made using v2 or v3 of the aws-sdk gem and create a span for each call.

Faraday 

The Ruby Beeline will attempt to propagate outgoing HTTP requests with a trace propagation header. This allows other services that are using a Beeline to produce a distributed trace.

Rack 

The Ruby Beeline provides a piece of rack middleware that can be used in your rack applications. This will create a trace and root span for each request and continue any distributed traces with trace ids included from other Beelines in the header.

app = Rack::Builder.new do |builder|
  builder.use Honeycomb::Rack::Middleware, client: Honeycomb.client
  builder.run RackApp.new
end

Rails 

The Ruby Beeline provides a Rails generator for easy configuration.

bundle exec rails generate honeycomb YOUR_API_KEY --service-name SERVICE_NAME

With the provided API key and service name, the rails generate honeycomb command will generate a Rails initializer with some basic Active Support events selected and a sample presend_hook to demonstrate removing potentially sensitive information from notification events. The Rails Guide for Active Support Instrumentation documents many more types of events to which you can subscribe to send to Honeycomb. The handling of notification events can be customized based on the event identifier; see the Active Support section for details.

Honeycomb.configure do |config|
  config.write_key = 'YOUR_API_KEY'
  config.service_name = 'your-service-name'
  config.presend_hook do |fields|
    if fields["name"] == "redis" && fields.has_key?("redis.command")
      # remove potential PII from the redis command
      if fields["redis.command"].respond_to? :split
        fields["redis.command"] = fields["redis.command"].split.first
      end
    end
    if fields["name"] == "sql.active_record"
      # remove potential PII from the active record events
      fields.delete("sql.active_record.binds")
      fields.delete("sql.active_record.type_casted_binds")
    end
  end
  config.notification_events = %w[
    sql.active_record
    render_template.action_view
    render_partial.action_view
    render_collection.action_view
    process_action.action_controller
    send_file.action_controller
    send_data.action_controller
    deliver.action_mailer
  ].freeze
end

Rake 

The Ruby Beeline will automatically instrument your rake tasks and create a trace and root span for each rake task.

Sequel 

The Ruby Beeline will automatically instrument your sequel database calls and create a span for each call.

Sinatra 

The Ruby Beeline enhances the provided rack middleware to collect extra information for sinatra application.

use Honeycomb::Sinatra::Middleware, client: Honeycomb.client

get '/' do
  'Hello from Honeycomb'
end

Proxy Configuration 

The Beeline will use any https_proxy, http_proxy, and no_proxy environment variables set for the process for all event traffic to Honeycomb.

Note: To use the standard proxy environment variables, ensure that the underlying Ruby Libhoney client is version 1.18.0 or greater. Proxy support was added in Libhoney version 1.14.0 and was configured via a proxy_config Array passed during client initialization. Array-style proxy configuration will be deprecated in Libhoney 2.0.

Honeycomb.configure do |config|
  config.client = Libhoney::Client.new(
    writekey: 'YOUR_API_KEY',
    service_name = 'your-service-name'
    proxy_config: ['myproxy.example.com', 8080])
  )
end

# ... or if you need to authenticate to your proxy

Honeycomb.configure do |config|
  config.client = Libhoney::Client.new(
    writekey: 'YOUR_API_KEY',
    service_name = 'your-service-name'
    proxy_config: ['myproxy.example.com', 8080, 'username', 'password'])
  )
end

Augmenting Or Scrubbing Spans 

If you have some transformations you would like to make on every span before it leaves the process for Honeycomb, the presend_hook is your opportunity to make these changes. Examples are scrubbing sensitive data (for example, you may want to ensure specific fields are dropped or hashed) or augmenting data (like making out-of-band translations between an ID and a more human readable version). Similar to the sample_hook discussed below, you pass the presend_hook a block that will be called on every span with the fields of that span as an argument. The function is free to mutate the hash passed in and those mutations will be what finally gets sent to Honeycomb.

As an example, say we have some HTTP requests that provide sensitive values which have been added to a span. This code will examine all spans just before they are sent to Honeycomb and replace the sensitive values with “[REDACTED]”.

Honeycomb.configure do |config|
  config.presend_hook do |fields|
    fields['credit_card_number'] = '[REDACTED]'
  end
end

Sampling Events 

To sample a portion of events for very high throughput services, include a custom client with a sample_rate in the initialization of the Ruby Beeline. This sends 1/n of all events, so a sample rate of 5 would send 20% of your events.

Honeycomb.configure do |config|
  config.client = Libhoney::Client.new(
    writekey: 'YOUR_API_KEY',
    service_name = 'your-service-name'
    sample_rate: 5 # sends 1/5 (20%) of events
  )
end

The value of sample_rate must be a positive integer.

Sampling is performed by default on a per-trace level in the Ruby Beeline, so adding sampling will not break your traces. Either all spans in a trace will be sent, or no spans in the trace will be sent.

The Honeycomb engine weights query results based on sample rate to ensure that sampled computations return correct results.

Customizing Sampling Logic 

Our Beeline lets you define a sample_hook in order to customize the logic used for deterministic per-trace sampling.

For example, assume you have instrumented an HTTP server. You would like to keep all errored requests and heavily sample healthy traffic (200 response codes). Also, you do not really care about 302 redirects in your app, so want to drop those. You could define a sampler function like so:

class CustomSampler
  extend Honeycomb::DeterministicSampler

  def self.sample(fields)
    # capture everything by default
    sample_rate = 1

    case fields['app.response_code']
    when 302
      return [false, 0]
    when 200
      # only keep 1 out of every 100 traces for successful requests
      sample_rate = 100
    end

    if should_sample(sample_rate, fields['trace.trace_id'])
      return [true, sample_rate]
    end

    return [false, 0]
  end
end

Honeycomb.configure do |config|
  config.sample_hook do |fields|
    CustomSampler.sample(fields)
  end
end

Distributed Trace Propagation 

When a service calls another service, you want to ensure that the relevant trace information is propagated from one service to the other. This allows Honeycomb to connect the two services in a trace.

Distributed tracing enables you to trace and visualize interactions between multiple instrumented services. For example, your users may interact with a front-end API service, which talks to two internal APIs to fulfill their request. In order to have traces connect spans for all these services, it is necessary to propagate trace context between these services, usually by using an HTTP header.

Both the sending and receiving service must use the same propagation format, and both services must be configured to send data to the same Honeycomb environment.

Interoperability With OpenTelemetry 

Trace context propagation with OpenTelemetry is done by sending and parsing headers that conform to the W3C Trace Context specification.

To get Beelines and OpenTelemetry instrumentation to interoperate, you will need to use W3C headers.

The Beeline includes marshal and unmarshal functions that can generate and parse W3C Trace Context headers. Honeycomb Beelines default to using a Honeycomb-specific header format on outgoing requests, but can automatically detect incoming W3C headers and parse them appropriately. In mixed environments where some services are using OpenTelemetry and some are using Beeline, W3C header propagation should be used.

To propagate trace context, a parser hook and propagation hook are needed. The parser hook is responsible for reading the trace propagation context out of incoming HTTP requests from upstream services. The propagation hook is responsible for returning the set of headers to add to outbound HTTP requests to propagate the trace propagation context to downstream services.

Note: Older versions of Honeycomb Beelines required HTTP parsing hooks to properly parse incoming W3C headers. Current versions of Honeycomb Beelines can automatically detect incoming W3C headers and parse them appropriately. Check the release notes for your Beeline version to confirm whether an upgraded version is needed.

To specify that a service should propagate W3C Trace Context Headers with outgoing requests, you must specify a propagation hook in the beeline configuration.

An http_trace_parser_hook is a function that takes a rack env as an argument and returns a Honeycomb::Propagation::Context. The rack env is provided to the function so that the author can make decisions about whether to trust the incoming headers based on information contained in the request (perhaps you do not want to accept headers from the public internet).

An http_trace_propagation_hook is a function that takes a Faraday env and a Honeycomb::Propagation::Context as arguments and returns a hash of name-value pairs representing serialized headers. Similar to the parsing hook described above, the Faraday env and Propagation Context objects are passed to this function so that the author can make decisions about whether to include trace context in the outgoing request (for example, you may not wish to send trace context headers when calling a third-party API).

This is an example of adding these hooks to parse and propagate W3C headers:

require "honeycomb-beeline"
require "honeycomb/propagation/w3c"

Honeycomb.configure do |config|
  ...
  config.http_trace_parser_hook do |env|
    Honeycomb::W3CPropagation::UnmarshalTraceContext.parse_rack_env env
  end
  ...
    config.http_trace_propagation_hook do |env, context|
    Honeycomb::W3CPropagation::MarshalTraceContext.parse_faraday_env env, context
  end
  ...
end

Troubleshooting The Beeline 

No Traces for a Service 

The service name is a required configuration value. If it is unspecified, all trace data will be sent to a default dataset called unknown_service.

Enable Debug Mode 

Add the debug line in the config block.

Honeycomb.configure do |config|
  # any other configuration...
  config.debug = true
end

Use a LogClient 

By using a Libhoney LogClient when configuring the Ruby Beeline you can send the events to STDOUT.

Honeycomb.configure do |config|
  config.client = Libhoney::LogClient.new
end

Using ENV Variables to Control Framework Integrations 

If you are having issues with a specific integration with Rails, Faraday, Sequel or other frameworks, use the following ENV variables to determine which integration may be causing the issue or to disable it entirely.

Use HONEYCOMB_DISABLE_AUTOCONFIGURE=true to stop the Beeline from requiring any of the Beeline integrations. This will still allow you to use the Beeline but without any of the automatic instrumentation.

You can also use HONEYCOMB_INTEGRATIONS=rails,faraday using a comma-separated list to only load specific integrations. This will allow you to be more selective about which automatic instrumentation integration you want to use. The available integration list is: active_support, aws, faraday, rack, rails, railtie, rake, sequel, sinatra.

Customizing Middleware Location in a Rails Application 

The Rails framework integration will automatically insert the Rack middleware before the Rails::Rack::Logger middleware. To insert this in some other location, disable the railtie integration as defined above using the ENV variable HONEYCOMB_INTEGRATIONS. You will then need to insert the Honeycomb::Rails::Middleware into your middleware stack manually.

My Traces Are Showing Missing Root Spans 

There can be a number of reasons for missing root spans. One potential reason could be that there is an upstream service, load balancer, or other proxy propagating W3C trace headers as part of your distributed trace. Since beelines accept both Honeycomb and W3C headers, that service propagating a W3C header will cause “missing span” gaps in your trace if the service is not also configured to send telemetry to Honeycomb. The solution is to either instrument that service and configure it to send telemetry to Honeycomb, or to specify in the downstream service’s beeline configuration that only Honeycomb propagation headers should be parsed.

To override undesired W3C trace header propagation behavior, configure the Beeline to use an http_trace_parser_hook:

require "honeycomb-beeline"
require "honeycomb/propagation/honeycomb"

Honeycomb.configure do |config|
  ...
  config.http_trace_parser_hook do |env|
    Honeycomb::HoneycombPropagation::UnmarshalTraceContext.parse_rack_env env
  end
  ...
end

The above configuration will solely use the Honeycomb format when parsing incoming trace headers. See Distributed Trace Propagation for more details.

Example Event 

Below is a sample event from the Ruby Beeline. This example is an http_server event, generated when your app handles an incoming HTTP request.

{
  "meta.beeline_version": "1.3.0",
  "meta.local_hostname": "myhost",
  "service_name": "my-test-app",
  "meta.package": "rack",
  "meta.package_version": "1.3",
  "name": "http_server",
  "request.route": "GET /dashboard",
  "request.method": "GET",
  "request.path": "/dashboard",
  "request.protocol": "https",
  "request.http_version": "HTTP/1.1",
  "request.host": "my-test-app.example.com",
  "request.remote_addr": "192.168.1.5",
  "request.header.user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36",
  "trace.trace_id": "b694512a-833f-4b35-be5f-6c742ba18e12",
  "trace.span_id": "c35cc326-ed90-4881-a4a8-68526d252f2e",
  "response.status_code": 200,
  "duration_ms": 303.057396
}

Queries to Try 

Here are some examples to get you started querying your app’s behavior:

Which Of My App’s Routes Are The Slowest? 

• GROUP BY: request.path
• VISUALIZE: P99(duration_ms)
• WHERE: name = http_request
• ORDER BY: P99(duration_ms) DESC

Where Is My App Spending The Most Time? 

• GROUP BY: name
• VISUALIZE: SUM(duration_ms)
• ORDER BY: SUM(duration_ms) DESC

Which Users Are Using The Endpoint That I Would Like to Deprecate? (Using a Custom Field user.email) 

• GROUP BY: app.user.email
• VISUALIZE: COUNT
• WHERE: request.path == /my/deprecated/endpoint

Contributions 

Bug fixes and other changes to Beelines are gladly accepted. Please open issues or a pull request with your change via GitHub. Remember to add your name to the CONTRIBUTORS file!

All contributions will be released under the Apache License 2.0.

Did you find what you were looking for?

Yes No

---

If you have questions, join our Pollinators Community Slack!
Pro/Enterprise users can visit Honeycomb Support or email support@honeycomb.io.

Submit