AWS S3


Note
This feature is available as an add-on for the Honeycomb Enterprise plan. Please contact your Honeycomb account team for details.

Supported Types 

Metrics Logs Traces

​The AWS S3 destination saves telemetry into timestamped JSON files in an S3 bucket.

Configuration 

Parameter Type Default Description
telemetry_types* telemetrySelector ["Logs", "Metrics", "Traces"] Specifies which types of telemetry to export.
region* enum us-east-1 AWS region of the bucket to export telemetry to.
bucket* string "" Name of the S3 Bucket to export telemetry into.
prefix string "" The root directory of the bucket to export telemetry into.
file_prefix string "" Prefix for the name of exported telemetry files.
granularity* enum minute The granularity of the timestamps in the S3 key, either “minute” or “hour”.
compression string "" Whether or not data should be uploaded and stored in S3 as compressed data. Valid values are “gzip” or no value set.
storage_class enum “STANDARD” The storage class to store telemetry data in. Read this AWS documentation for more info.
endpoint string "" The endpoint to use for the S3 destination. Overrides the endpoint constructed from the region and bucket values. Generally not needed.
marshaler_logs enum “otlp_json” Determines the format of data sent to AWS S3. See Marshaler Configuration for more details.
marshaler_metrics enum “otlp_json” Determines the format of data sent to AWS S3. See Marshaler Configuration for more details.
marshaler_traces enum “otlp_json” Determines the format of data sent to AWS S3. See Marshaler Configuration for more details.

*required field

Marshaler Configuration 

The destinations supports configuring the marshaler for each telemetry type. The marshaler determines how the data is formatted before being sent to AWS S3. By default, the otlp_json marshaler is used for all telemetry types.

Marshaler Type Metrics Logs Traces Description
otlp_json Uses the OpenTelemetry Protocol (OTLP) in JSON format.
otlp_proto Uses the OpenTelemetry Protocol (OTLP) in Protocol Buffers format.
sumo_ic Uses the Sumo Logic Ingestion format.
body Export the log body as string.
Note
Note that sumo_ic and body marshaler types are only applicable for logs.

Credentials 

With AWS S3, users are required to provide some form of authentication. There are two ways to configure this: either entering profile credentials manually or using the CLI, or environment variables that specify access keys for user accounts.
​ In the AWS CLI getting started guide it will instruct you to install for your current user or all users.
The observIQ OTel Collector runs as root by default, meaning the AWS CLI and credentials should be installed under the collector system’s root account.

CLI 

The easiest way to configure this is using the aws CLI program provided by AWS. The AWS CLI Getting Started guide describes how to install the CLI and configure it with credentials. The AWS S3 destination uploads telemetry to the specified bucket, so the credentials configured should be associated with an account that has s3:PutObject permissions for that bucket.

Environment Variables 

Alternatively, AWS Environment variables can be specified to override a credentials file. You can modify the collector’s environment variables by configuring a systemd override.
Run sudo systemctl edit observiq-otel-collector and add your access key, secret key, and region:

[Service]
Environment=AWS_ACCESS_KEY_ID=******
Environment=AWS_SECRET_ACCESS_KEY=******
Environment=AWS_DEFAULT_REGION=us-east-1

After making that change, reload Systemd and restart the collector service.

sudo systemctl daemon-reload
sudo systemctl restart observiq-otel-collector