Symbolicate Android Stack Traces with the OpenTelemetry Collector


Use the symbolicator processor in your OpenTelemetry Collector to symbolicate Android stack traces.

Important
This feature is in beta, and we would love your feedback!

The symbolicator processor replaces obfuscated names and addresses in your Android stack traces with symbols from provided Proguard files.

Before You Start 

The symbolicator processor is compatible with Honeycomb OpenTelemetry Android SDK version 0.0.13 and later.

To use the symbolicator processor, you need:

  • OpenTelemetry Collector built with CGO enabled.

  • An environment or container image with glibc support.

    We recommend gcr.io/distroless/cc, a secure and lightweight container image with CGO and glibc support.

If you’re not using the Honeycomb OpenTelemetry Android SDK, make sure your exception data is in the format the processor expects.

Install 

By default, the Honeycomb OpenTelemetry Collector distribution includes the symbolicator processor, so you can skip to the next section if you’re using it. If you use another collector distribution or build your own, it must be built with CGO enabled.

You can install the symbolicator processor by adding it to your OpenTelemetry Collector build configuration file:

processors:
  - gomod: github.com/honeycombio/opentelemetry-collector-symbolicator/symbolicatorprocessor v0.0.0
    name: symbolicatorprocessor
    path: ./symbolicatorprocessor

Proguard Files 

The symbolicator processor requires access to the Proguard file generated by your build process. This file can be stored in your local file system, Amazon S3, or Google Cloud Storage.

To support symbolication, your Proguard file must be versioned with the generated build UUID in the file name. For example: 6A8CB813-45F6-3652-AD33-778FD1EAB196.txt.

Configure a File Store 

Add the proguard_symbolicator as a processor in your OpenTelemetry Collector configuration:

processors:
  proguard_symbolicator:

You can then configure where your Proguard files are stored.

Local File Store 

By default, the symbolicator processor loads Proguard files from a local directory. You can set the file path in your collector configuration:

processors:
  proguard_symbolicator:
    # proguard_store is sets which store to use, in this case local disk
    proguard_store: file_store
    local_store:
        # (optional) path sets the base path of the files, defaults to `.`
        path: /tmp/proguards

Make sure your collector can access the path directory you set, and that file paths in stack traces match the structure of your configured file store.

How the Processor Retrieves Files from Local Disk 

When retrieving files from local disk, the processor:

  1. Gets the base file name from the URL included in the stack trace.
  2. The path, if configured, is joined with the base file name.
  3. Reads the file using the joined path from disk.

Amazon S3 Store 

Optionally, you can load Proguard files from an Amazon S3 bucket. Add to your OpenTelemetry Collector configuration:

processors:
  proguard_symbolicator:
    # proguard_store sets which store to use, in this case S3
    proguard_store: s3_store
    s3_store:
        # name of the bucket the files are stored in
        bucket: proguards-bucket
        # (optional) the bucket's region
        region: us-east-1
        # (optional) prefix is used to nest the files in a sub key of the bucket
        prefix: proguards

Make sure your collector has permission to access the S3 bucket. Also, ensure the file paths in stack traces match the structure used in your file store.

Private Amazon S3 Bucket Authentication 

To use a private Amazon S3 bucket as your file store, set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.

How the Processor Retrieves Files from S3 

When retrieving files from an Amazon S3 bucket, the processor:

  1. Gets the base file name from the URL included in the stack trace.
  2. The prefix, if configured, is joined with the base file name.
  3. Uses this joined path as the key to retrieve the file from the bucket.

Google Cloud Storage Store 

Optionally, you can load Proguard files from a Google Cloud Storage (GCS) bucket. Add to your OpenTelemetry Collector configuration:

processors:
  proguard_symbolicator:
    # proguard_store sets which store to use, in this case GCS
    proguard_store: gcs_store
    gcs_store:
        # the name of the bucket the files are stored in
        bucket: proguards-bucket
        # (optional) prefix is used to nest the files in a sub key of the bucket
        prefix: proguards

Make sure your collector has permission to access the GCS bucket. Also, ensure the file paths in stack traces match the structure used in your file store.

Private Google Cloud Storage Bucket Authentication 

To use a private Google Cloud Storage bucket as your file store, set the GOOGLE_APPLICATION_CREDENTIALS environment variable.

How the Processor Retrieves Files from GCS 

When retrieving files from a Google Cloud Storage bucket, the processor:

  1. Gets the base file name from the URL included in the stack trace.
  2. The prefix, if configured, is joined with the base file name.
  3. Uses this joined path as the key to retrieve the file from the bucket.

Advanced Configuration 

In addition to basic setup, you can customize how the symbolicator processor handles stack traces by configuring attribute mappings and additional processing options.

Tip
After updating the configuration file, restart the OpenTelemetry Collector to apply the changes.

Mapping Attributes 

Use these configuration options to specify which attributes the processor should read from and write to when handling stack traces:

Config Key Description Example Value
symbolicator_failure_attribute_key Signals if the symbolicator fails to fully symbolicate the stack trace exception.symbolicator.failed
classes_attribute_key Which attribute should the classes of the stack trace be sourced from exception.structured_stacktrace.classes
methods_attribute_key Which attribute should the methods of the stack trace be sourced from exception.structured_stacktrace.methods.
lines_attribute_key Which attribute should the lines of the stack trace be sourced from exception.structured_stacktrace.lines
output_stack_trace_key Which attribute should the symbolicated stack trace be populated into exception.stacktrace
preserve_stack_trace After the stack trace has been symbolicated, determines if the original values be preserved as attributes true
original_classes_attribute_key If the stack trace is being preserved, which key should the classes be copied to exception.structured_stacktrace.classes.original
original_methods_attribute_key If the stack trace is being preserved, which key should the methods be copied to exception.structured_stacktrace.methods.original
original_lines_attribute_key If the stack trace is being preserved, which key should the lines be copied to exception.structured_stacktrace.lines.original
original_stack_trace_key If the stack trace is being preserved, which key should it be copied to exception.stacktrace.original
proguard_uuid_attribute_key Which resource attribute should the proguard UUID of a generic stacktrace log be sourced from app.debug.proguard_uuid

Additional Processing Options 

Use these configuration options to control how stack traces are processed and managed:

Config Key Description Example Value
timeout Max duration to wait to symbolicate a stack trace in seconds. 5
proguard_cache_size The maximum number of proguard files to cache. Reduce this if you are running into memory issues with the collector. 128