This feature is available as part of the Honeycomb Enterprise plan.
This topic describes the features, options, installation, and configuration of Honeycomb Secure Tenancy, also referred to as Secure Proxy. If you have questions or need assistance with any aspect of installing or configuring this functionality, contact us and we’ll help.
Secure Tenancy provides two masking options to address your security/compliance requirements while delivering access to the fine-grained observability that you need: Event encryption and hashing.
Both options make use of a Honeycomb Secure Proxy running in your infrastructure. No plaintext data ever traverses Honeycomb’s infrastructure and the Honeycomb UI presents complete transparency to authorized members of your team. You have complete control of key rotation and reissuance down to the columnar level from within your own infrastructure.
With Event Encryption, all strings in your datasets are encrypted and the keys are stored in a database on the Secure Proxy running in your infrastructure. When an authorized user accesses Honeycomb, their web browser connects to the Secure Proxy directly and the data is unencrypted for them. Honeycomb infrastructure never has access to the sensitive data in plaintext.
With Event Hashing, all strings in your datasets are hashed and the hash mappings are stored in a database on the Secure Proxy running in your infrastructure. When an authorized user accesses Honeycomb, their browser sends the hashed data to the Secure Proxy running in your environment and receives the un-hashed data back. Again, no plaintext sensitive data reaches the Honeycomb infrastructure.
All strings values sent as part of your events are masked when you use Secure Tenancy. Additionally, most other metadata strings are masked, such as column names, aliases, descriptions, derived column aliases, board names, descriptions, and query captions.
The following strings are sent to third party services for notifications and will not be masked by the secure proxy:
Please only use names and descriptions that you feel comfortable storing in Honeycomb’s database and sending through third parties such as Slack or PagerDuty.
The Honeycomb team will provide you with either a system package (such as a DEB or RPM) or a .tar.gz containing the elements required to run the Secure Proxy.
If we provide a full package, you may not need to create some of the files/directories mentioned here manually, but rather edit their values.
If you use one load balancer, Honeycomb will use it for both encrypting and decrypting data. You may use two to separate encryption and decryption.
Configure the load balancer(s) with a signed certificate from a Certificate Authority, to serve as the entry point for encryption and decryption.
Configure these instances to allow incoming traffic from applications on the internal network(s) where they are running and so that end users can access the endpoint when tunneled in to the company VPN.
The domain name and SSL certificate should be publicly resolvable, but the endpoint can be on a non-publicly-routable IP accessible within VPN.
Your load balancers must support gRPC connections if you intend to use OpenTelemetry SDKs to instrument your applications.
The aforementioned load balancer will forward traffic to these. If possible, place the servers in separate availability zones.
The MySQL database should not be accessible by any programs other than the secure proxy.
Here’s an example configuration using AWS technology:
If you’re planning to deploy using a higher level platform such as OpenShift, Mesos, Kubernetes, etc., please contact us to get additional, more specific guidance for your platform.
/srv/hny./srv/hny/config/.If you choose to install elsewhere you must specify the location of the config file as a command line argument (-hnyconfig flag) to the binary.
$ mkdir -p /srv/hny
$ cd /srv/hny
$ <place file from above step in this directory>
$ tar --strip-components 1 -xjf <provided_tarball>
This is sensitive information and is not meant to be shared outside of the operator(s) of the Secure Proxy.
Run this command to generate a long random hexidecimal token:
$ head -c64 /dev/urandom | openssl dgst -sha256 -hex | sed -e 's/(stdin)= //'
Add the token from the step above to /srv/hny/config/honeycomb.yml. The value must be in quotes.
This is also the file used for other configuration values, like GRPC and HSTS.
# ---- auth_token -----
#
# This token is sent to Honeycomb with each ingest API request. if it doesn't match the value on the
# ingest side (configurable in your team settings on https://ui.honeycomb.io/)
# the request will be rejected.
auth_token: "generated token here"
#
# ---- GRPC (optional) -----
#
# Address for honeycomb to ingest grpc data. If GRPC is used, this value is required,
# including the port.
# api_grpc_base_url: https://api.honeycomb.io:443
#
# Address on which the proxy should listen for grpc requests.
# Defaults to all interfaces
# grpc_listen_addr: :8081
#
# ---- HSTS (optional) -----
#
# Enable HTTP Strict Transport Security (HSTS) (https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security)
# http_strict_transport_security_enabled: false
#
# The maximum time (in seconds) that client browsers should cache the HSTS configuration.
# Must be an integer greater than 0.
# http_strict_transport_security_max_age: 600
If you use a secret management tool such as Vault and wish to keep this secret there, use a # to comment out the auth_token line, and instead set the HONEYCOMB_AUTH_TOKEN environment variable to the correct value when the proxy is started. Environment variables override the values from the config file, so if both are set, the values from the environment will be used.
Once you’ve installed MySQL on the database server, create or edit /srv/hny/config/mysql.yml to include the following information so it is able to connect to the MySQL instance:
user: root
password: ""
host: localhost:3306
database: honeycomb_secure_proxy
maxopenconns: 100
Use this command to create the database:
mysql -u root -e 'create database honeycomb_secure_proxy;'
Run the database migrations to populate the database schemas:
$ cd /srv/hny && \
bin/migrate -url "mysql://<user>:<password>@tcp(<host>:<port>)/honeycomb_secure_proxy" -path ./migrate up
If Secure Proxy is connecting to MySQL over TLS, set the tls key. The following values are supported:
host for ServerNamehost)To specify a non-default location for the MySQL config file, use the -mysqlconfig flag.
Do a pre-flight check to ensure that the Secure Proxy installation and the MySQL database are not accessible from outside the private network(s) they are installed in. The MySQL database contains the keys used to encrypt and decrypt information sent to Honeycomb, secure it from untrusted access.
This step is optional, but recommended.
You can help Honeycomb ensure the quality of our service by sending usage and debugging information from the Secure Proxy to Honeycomb. This data does not contain any sensitive information (such as masked events or encryption keys), and is available for you to access in your own Honeycomb account.
To enable sending proxy metrics, you must create a new team (one that isn’t in high security mode). Log into Honeycomb and go to https://ui.honeycomb.io/teams. On that page you’ll see Create Team with a field to create the new team.
Once you’ve created this team, set the libhoney_writekey parameter in /srv/hny/config/honeycomb.yml with the team’s api key, which is at https://ui.honeycomb.io/account).
Then, let us know the team name. This team for telemetry information is still under your control, and you can delete the dataset at any time you choose.
For systemd, create the following file: /lib/systemd/system/honeycomb_secure_proxy.service
Edit it to include the following:
[Unit]
Description=Honeycomb Secure Proxy
After=network.target
[Service]
ExecStart=/srv/hny/bin/honeycomb_secure_proxy
KillMode=process
Restart=on-failure
[Install]
Alias=honeycomb_secure_proxy honeycomb_secure_proxy.service
Reload the systemd process:
sudo systemctl daemon-reload
Start the system service:
sudo systemctl start honeycomb_secure_proxy
Enable it to start on boot:
sudo systemctl enable honeycomb_secure_proxy
By default the Secure Proxy will run on port 8080.
When you have deployed the Secure Proxy, configure SSL termination on the load balancer that fronts the proxy instances. If HTTP/HTTPS content is mixed, the Honeycomb browser UI will not load.
You must provide your auth token to Honeycomb before sending traffic through the secure proxy. Honeycomb will refuse all traffic when the token is missing or doesn’t match the token provided to the web app.
Go to the Secure Tenancy settings page for the relevant team by clicking the lower left avatar menu, clicking “Team Settings”, and then clicking “Secure Tenancy” in the left navigation area.
Look for the Secure Proxy Auth Token field. Enter the auth token you generated earlier, then click outside the text field. Once you see “OK” appear below the text field, you can send data through your Secure Tenancy installation. These new settings may take a minute or so to propagate through our backend.
In the Secure Proxy Address field, enter the url for your proxy.
Note: No one will be able to connect to the secure proxy directly and decrypt the data. There is a built in security layer that prevents this from happening.
You must send at least one event to Honeycomb using the first API key listed for your team. If you do not do this, Honeycomb won’t recognize your Secure Tenancy proxy and the UI will show an error.
You can use a simple curl command to send the event.
curl -X POST -H "Content-Type: application/json" --header "X-Honeycomb-Team: <api key>" \
--data '{"test":0}' "<Secure Proxy URL:port>/1/events/test_dataset"
All integrations should point to the proxy instead of directly to the Honeycomb API. For instance, if you’re using honeytail, add the –api_host flag for initial testing:
--api_host=https://our-staging-secure-proxy.yourcompany.com:8080
For language SDKs and Beelines, all config objects have an APIHost property which can be set to the same effect.
Once data is sent, you should see a new dataset appear and be able to query it in the Honeycomb UI.
Once you’ve sent some data to a dataset provisioned under Secure Tenancy, you can specify the fields you want Honeycomb to use to display trace waterfall diagrams. For more information about tracing in Honeycomb, review the Tracing documentation.
If you are a team owner, you can do this from the Definitions tab for that dataset’s settings:
The secure tenancy model means that Honeycomb doesn’t know the columns to use for different features, so you’ll have to tell us. Once you’ve sent some data to a dataset provisioned under Secure Tenancy, navigate to the Definitions tab in that dataset’s details. For more information about configuring dataset definitions, review the Tracing and Home documentation.
As a whole, Honeycomb will continue to work the same way it does for non-secure tenancy. However there are some limitations imposed by the Secure Proxy:
In the Secure Tenancy model, Honeycomb does not have access to the meaning of string values for columns. As a result, some derived column functions are not available or operate slightly differently under secure tenancy.
None of the string functions are available; in addition, functions that coerce strings to values will not do so.
The following functions can be used in derived column expressions.
We’ll reach out when there’s a new release of the proxy. To download the new version, go to your Secure Tenancy settings.
There is a health check endpoint at /x/healthcheck that checks the secure proxy’s dependencies: the Honeycomb API and the secure proxy’s database.
The health check will return a 200 if both dependencies are OK, or a 500 if any of the dependencies are down. Both 200 and 500 responses return JSON with more details.
To rotate your team’s API Key, go to your Team Settings page and create a new API Key. You can start sending data right away with this new key. In order to successfully unmask your data, you should disable API keys that are not being used.
Honeycomb uses the first enabled API key available to unmask your data. This API key must be configured with permission to send events.
Packaged with your Secure Tenancy installation is the honeycomb_keys binary. This small program is used to rotate the Secure Tenancy keys used for masking (and stored in MySQL) if needed.
To list the keys:
$ honeycomb_keys list
From that list, you can rotate a key with rotate:
$ honeycomb_keys rotate <key>
Contact your Honeycomb account manager with any questions or you need assistance with any part of this installation process.