sudo initctl start honeytail
sudo systemctl start honeytail
Our connector pulls your MySQL logs into Honeycomb for analysis, so you can finally get a quick handle on the database queries triggered by your application logic. It surfaces attributes like:
Honeycomb is unique in its ability to calculate metrics and statistics on the fly, while retaining the full-resolution log lines (and the original MySQL query that started it all!).
Once you have got data flowing, be sure to take a look at our starter queries! Our entry points will help you see how we recommend comparing lock retention by normalized query, scan efficiency by collection, or read vs. write distribution by host.
The agent used to translate logs to events and send them to Honeycomb is called honeytail
.
Before running honeytail
, you will want to turn slow query logging on for all queries if possible.
To turn on slow query logging for your MySQL host, run the following in your MySQL shell:
mysql> SET GLOBAL slow_query_log = 'ON';
Set the threshold for a query to be considered a “slow” query to 0
(the default is 10
):
mysql> SET GLOBAL long_query_time = 0;
And verify the slow query log’s location via:
mysql> SELECT @@GLOBAL.slow_query_log_file;
On your MySQL host, download and install the latest honeytail
by running:
Download the honeytail_1.8.2_amd64.deb
package.
wget -q https://honeycomb.io/download/honeytail/v1.8.2/honeytail_1.8.2_amd64.deb
Verify the package.
echo 'c9101060b782c34045f359bfe7b7a0a95c6a54eb97ed51777b053a769afa823b honeytail_1.8.2_amd64.deb' | sha256sum -c
Install the package.
sudo dpkg -i honeytail_1.8.2_amd64.deb
The packages install honeytail
, its config file /etc/honeytail/honeytail.conf
,
and some start scripts.
Build honeytail
from source if you need it in an unpackaged form or for ad-hoc use.
Download the honeytail_1.8.2_arm64.deb
package.
wget -q https://honeycomb.io/download/honeytail/v1.8.2/honeytail_1.8.2_arm64.deb
Verify the package.
echo 'c5a03a2ef4b850be35fb3809173efd3d6e8b16c52b770f61691355b7d39505b0 honeytail_1.8.2_arm64.deb' | sha256sum -c
Install the package.
sudo dpkg -i honeytail_1.8.2_arm64.deb
The packages install honeytail
, its config file /etc/honeytail/honeytail.conf
,
and some start scripts.
Build honeytail
from source if you need it in an unpackaged form or for ad-hoc use.
Download the honeytail-1.8.2-1.x86_64.rpm
package.
wget -q https://honeycomb.io/download/honeytail/v1.8.2/honeytail-1.8.2-1.x86_64.rpm
Verify the package.
echo 'bccdedc9aa3776b827899cc0003b9f23f3a9c3b366ce2c8de9d2d943a474d306 honeytail-1.8.2-1.x86_64.rpm' | sha256sum -c
Install the package.
sudo rpm -i honeytail-1.8.2-1.x86_64.rpm
The packages install honeytail
, its config file /etc/honeytail/honeytail.conf
,
and some start scripts.
Build honeytail
from source if you need it in an unpackaged form or for ad-hoc use.
Download the 1.8.2 binary.
wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.8.2/honeytail-linux-amd64
Verify the binary.
echo 'd774112265ee8e98c6221232461cf36c35faf844005cc98b43b55bb375761766 honeytail' | shasum -a 256 -c
Set the permissions to allow execution.
chmod 755 ./honeytail
Download the 1.8.2 binary.
wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.8.2/honeytail-linux-arm64
Verify the binary.
echo 'c5a57a729b0ccf4ca0f2287c862538812604f5fd67d102372e91215701afdbe1 honeytail' | shasum -a 256 -c
Set the permissions to allow execution.
chmod 755 ./honeytail
Download the 1.8.2 binary.
wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.8.2/honeytail-darwin-amd64
Verify the binary.
echo '49c976e9b98fa238c265ec6afce35d41767a3f8d67defa547bff933485f40a83 honeytail' | shasum -a 256 -c
Set the permissions to allow execution.
chmod 755 ./honeytail
Clone the Honeytail repository.
git clone https://github.com/honeycombio/honeytail
Install from source.
cd honeytail; go install
Make sure you have enabled MySQL query logging before running honeytail
.
To consume the current MySQL slow query log from the beginning, run:
honeytail --writekey=YOUR_API_KEY --dataset=MySQL --parser=mysql \
--file=/usr/local/var/mysql/myhost-slow.log \
--tail.read_from=beginning
First, check out honeytail
Troubleshooting for general debugging tips.
--debug
does not seem to show anything useful Take a look at the --file
being handed to honeytail
and make sure they look like MySQL slow query logs, with blocks of comments containing metadata alternating with the MySQL commands issued.
An example excerpt from a MySQL slow query log might look like:
# Time: 151008 0:31:03
# User@Host: rails[rails] @ [10.252.10.158]
# Query_time: 0.000547 Lock_time: 0.000019 Rows_sent: 1 Rows_examined: 938
use rails;
SET timestamp=1444264263;
SELECT `app_data`.* FROM `app_data` WHERE (`app_data`.user_id = 69213) LIMIT 1;
If your log file looks like a normal MySQL output log but honeytail
is still failing to send events to Honeycomb, let us know!
Join our Pollinators Community Slack to ask questions and learn more.
Did you remember to SET
the GLOBAL long_query_time
?
Our parser relies on reading your server’s slow query logs, which contain much more valuable metadata than the general log—and the default slow query threshold is 10 seconds.
Try checking the output of:
mysql> SELECT @@GLOBAL.long_query_time;
If it is not 0
, take another look at the steps to Configure MySQL Query Logging.
Join our Pollinators Community Slack to ask questions and learn more.
To run honeytail
continuously as a daemon process, first modify the config file /etc/honeytail/honeytail.conf
and uncomment and set:
ParserName
to mysql
WriteKey
to your API key, available from the account pageLogFiles
to the path for your MySQL slow query log file, often located at /usr/local/var/mysql/myhost-slow.log
Dataset
to the name of the dataset you wish to create with this log file.Then start honeytail
using upstart
or systemd
:
sudo initctl start honeytail
sudo systemctl start honeytail
You may have archived logs that you would like to import into Honeycomb.
If you have a MySQL logfile located at /usr/local/var/mysql/myhost-slow.16.log
, you can backfill using this command:
honeytail --writekey=YOUR_API_KEY --dataset=MySQL --parser=mysql \
--file=/usr/local/var/mysql/myhost-slow.16.log \
--backfill
This command can be used at any point to backfill from archived log files.
You can read more about honeytail
’s backfill behavior here.
honeytail
does not unzip log files, so you will need to do this before backfilling.Once you have finished backfilling your old logs, we recommend transitioning to the default streaming behavior to stream live logs to Honeycomb.
While we believe strongly in the value of being able to track down the precise query causing a problem, we understand the concerns of exporting log data, which may contain sensitive user information.
With that in mind, we recommend using honeytail
’s MySQL parser, but adding a --scrub_field=query
flag to hash the concrete query
value.
The normalized_query
attribute will still be representative of the shape of the query, and identifying patterns including specific queries will still be possible—but the sensitive information will be completely obscured before leaving your servers.
More information about dropping or scrubbing sensitive fields can be found here.
Ingesting a MySQL log line (resulting from a SELECT
with a JOIN
):
# Time: 161019 18:30:00
# User@Host: rdsadmin[rdsadmin] @ localhost [127.0.0.1] Id: 1
# Query_time: 1.294391 Lock_time: 0.000119 Rows_sent: 4049 Rows_examined: 4049
SET timestamp=1476901800;
SELECT teams.* FROM teams INNER JOIN users_teams ON team_id=teams.id WHERE user_id=21782 AND slug='foobar' LIMIT 1
will produce an event for Honeycomb that looks like:
field name | value | type |
---|---|---|
client | string | localhost |
client_ip | string | 127.0.0.1 |
lock_time | float | 0.000119 |
normalized_query | string | select teams._ from teams inner join users_teams on team_id = teams.id where user_id = ? and slug = ? limit ? |
query | string | SELECT teams.* FROM teams INNER JOIN users_teams ON team_id=teams.id WHERE user_id=21782 AND slug='foobar' LIMIT 1 |
query_time | float | 1.294391 |
rows_examined | float | 4049 |
rows_sent | float | 4049 |
statement | string | select |
tables | string | teams users_teams |
user | string | rdsadmin |
Numbers are ingested as floats by default in Honeycomb, though you can coerce a field to integers in the Schema section of your dataset’s Overview.
You can find more on our MySQL query normalization in our mysqltools
repository.
Honeytail is open source and Apache 2.0 licensed.