We use cookies or similar technologies to personalize your online experience and tailor marketing to you. Many of our product features require cookies to function properly. Your use of this site and online product constitutes your consent to these personalization technologies. Read our Privacy Policy to find out more.

X

Secure Tenancy

This topic describes the features, options, installation, and configuration of Honeycomb Secure Tenancy, also referred to as Secure Proxy. If you have questions or need assistance with any aspect of installing or configuring this functionality, contact us and we’ll help.

About Secure Tenancy

Secure Tenancy provides two options to address your security/compliance requirements while delivering access to the fine-grained observability that you need.

Both options make use of a Honeycomb Secure Proxy running in your infrastructure. No plaintext data ever traverses Honeycomb’s infrastructure and the Honeycomb UI presents complete transparency to authorized members of your team. You have complete control of key rotation and reissuance down to the columnar level from within your own infrastructure.

Option 1: Event encryption

With Event Encryption, your datasets are encrypted and the keys are stored in a database on the Secure Proxy running in your infrastructure. When an authorized user accesses Honeycomb, their web browser connects to the Secure Proxy directly and the data is unencrypted for them. Honeycomb infrastructure never has access to the sensitive data in plaintext.

Option 2: Event hashing

With Event Hashing, your datasets are hashed and the hash mappings are stored in a database on the Secure Proxy running in your infrastructure. When an authorized user accesses Honeycomb, their browser sends the hashed data to the Secure Proxy running in your environment and receives the unhashed data back. Again, no plaintext data reaches the Honeycomb infrastructure.

How the data flows

data flow in Secure tenancy
  1. Your data is masked by the Secure Proxy before it leaves your network.
  2. The masked data is sent to Honeycomb while the keys remain on-premises.
  3. An authorized user queries the data in Honeycomb via their browser.
  4. The user’s browser sends the masked data to the Secure Proxy and receives unmasked data.

A load balancer configured to serve traffic over HTTPS

(Or two, if you wish to split encrypting and decrypting)

Configure the load balancer(s) with a signed certificate from a Certificate Authority. It (they) will serve as the entry point for both encryption and decryption.

Configure these instances to allow incoming traffic from applications on the internal network(s) where they are running and so that end users can access the endpoint when tunneled in to the company VPN.

The domain name and SSL certificate should be publicly resolvable, but the endpoint can be on a non-publicly-routable IP accessible within VPN.

Two servers provisioned for the Secure Proxy itself to run on

The aforementioned LB will forward traffic to these. If possible, place the servers in separate availability zones. * Recommended minimum size: 1 Core, 4 GB RAM * Recommended OS: Debian or Ubuntu

A MySQL server

The MySQL database should not be accessible by any programs other than the secure proxy.

Here’s an example configuration using AWS technology:

example Secure Tenancy deployment architecture in AWS

Note: If you’re planning to deploy to a higher level platform such as OpenShift, Mesos, Kubernetes, etc., contact us to get additional, more specific guidance for your platform.

Install and configure Secure Tenancy

The Honeycomb team will provide you with either a system package (such as a DEB or RPM) or a .tar.gz containing the elements required to run the Secure Proxy.

Note: If we provide a full package, you may not need to create some of the files/directories mentioned here manually, but rather edit their values.

Create and populate the installation directory:

  • The default location for the install is /srv/hny.
  • The default location for configs is /srv/hny/config/.

    If you choose to install elsewhere you must specify the location of the config file as a command line argument (-hnyconfig flag) to the binary.

    $ mkdir -p /srv/hny
    $ cd /srv/hny
    $ <place file from above step in this directory>
    $ tar --strip-components 1 -xjf <provided_tarball>

    Generate the auth token:

    Note: This is sensitive information and is not meant to be shared outside of the operator(s) of the Secure Proxy.

    Run this command and make a note of its output:

    $ head -c64 /dev/urandom | openssl dgst -sha256 -hex | sed -e 's/(stdin)= //' Add the key/token to the config file

    Add the hash from the step above to /srv/hny/config/honeycomb.yml. The value must be in quotes: auth_token: "<token>"

    Note: If you use a secret management tool such as Vault and wish to keep this secret there, do not add this line and instead set the HONEYCOMB_AUTH_TOKEN environment variable to the correct value when the proxy is started. Environment variables override the values from the config file, so if both are set, the values from the environment will be used.

    Configure and setup mysql PARAMETERS

    Once you’ve installed MySQL on the database sever, create or edit /srv/hny/config/mysql.yml to include the following information so it is able to connect to the MySQL instance:

    user: root
    password: ""
    host: localhost:3306
    database: honeycomb_secure_proxy
    maxopenconns: 100

    Create and configure a database called honeycomb_secure_proxy:

    Use this command to create the database:

    mysql -u root -e 'create database honeycomb_secure_proxy;'

    Run the database migrations to populate the database schemas:

    cd /srv/hny && \
    bin/migrate -url "mysql://<user>:<password>@tcp(<host>:<port>)/honeycomb_secure_proxy" -path ./migrate up

    If Secure Proxy is connecting to MySQL over TLS, set the tls key. The following values are supported:

  • false: disables TLS (default)
  • true: enables TLS, uses the hostname portion of host for ServerName
  • skip-verify: enables TLS, doesn’t verify ServerName matches the cert (anything-else): uses the value as ServerName (even if it doesn’t match the hostname portion of host)

    To specify a non-default location for the MySQL config file, use the -mysqlconfig flag.

    Confirm Secure Proxy and the MySQL instance are not accessible from outside your network(s):

    Do a pre-flight check to ensure that the Secure Proxy installation and the MySQL database it depends on are not accessible from outside the private network(s) they are installed in. The MySQL database contains the keys used to encrypt and decrypt information sent to Honeycomb, secure it from untrusted access.

    Enable sending of proxy metrics to Honeycomb

    This step is optional, but recommended.

    You can help Honeycomb ensure the quality of our service by sending usage and debugging information from the Secure Proxy to Honeycomb. This data does not contain any sensitive information (such as masked events or encryption keys), and is available for you to access in your own Honeycomb account.

    To enable sending proxy metrics, create a separate team (one that isn’t in high security mode). From https://ui.honeycomb.io/, scroll to the bottom of the page and click +New Team.

    Once you’ve created this team, set the libhoney_writekey parameter in /srv/hny/config/honeycomb.yml, which is at https://ui.honeycomb.io/account).

    libhoney_writekey: "<writekey>"`

    Then, let us know the team name. This team for telemetry information is still under your control, and you can delete the dataset at any time you choose.

    Create a system service for the proxy (systemd)

    Create the following file: /lib/systemd/system/honeycomb_secure_proxy.service

    Edit it to include the following:

    [Unit]
    Description=Honeycomb Secure Proxy
    After=network.target
    
    [Service]
    ExecStart=/srv/hny/bin/honeycomb_secure_proxy
    KillMode=process
    Restart=on-failure
    
    [Install]
    Alias=honeycomb_secure_proxy honeycomb_secure_proxy.service

    Start the system service:

    sudo systemctl start honeycomb_secure_proxy
    

    Enable it to start on boot:

    sudo systemctl enable honeycomb_secure_proxy
    

    By default the Secure Proxy will run on port 8080.

    Configure SSL termination either through NGINX or ELB/ALB/etc

    When you have deployed the Secure Proxy, configure SSL termination on the load balancer that fronts the proxy instances. If HTTP/HTTPS content is mixed, the Honeycomb browser UI will not load.

    Configure your auth token in the UI

    You must provide your autho token to Honeycomb before sending traffic through the secure proxy. Honeycomb will refuse all traffic when the token is missing or doesn’t match the token provided to the web app.

    Go to the settings page for the relevant team by clicking the upper right hand avatar menu, clicking “All Teams”, and then clicking the “Settings” link for that team.

    Look for the High Security Token field. Enter the auth token you generated earlier, then click outside the text field. Once you see “OK” appear below the text field, you can send data through your Secure Tenancy installation. These new settings may take a minute or so to propagate through our backend.

    In the High Security Proxy URL field, enter the endpoint for your proxy (which should be available on company VPN).

    Configure the integrations you will use to send data to the proxy instead of to the normal Honeycomb endpoint

    For instance, if you’re using honeytail, add the –api_host flag for initial testing:

    --api_host=https://our-staging-secure-proxy.yourcompany.com:8080
    

    For language SDKs and Beelines, all config objects have an APIHost property which can be set to the same effect.

    Once data is sent, you should see a new dataset appear and be able to query it in the Honeycomb UI.

    Contact your Honeycomb account manager with any questions or you need assistance with any part of this installation process.