1. Docs
2. Send Data to Honeycomb
3. Logs
4. Structured
5. PostgreSQL

Send PostgreSQL Logs

Table of Contents

Our connector pulls your PostgreSQL logs into Honeycomb for analysis, so you can analyze PostgreSQL traffic on your machines and finally get a quick handle on the database queries triggered by your application logic. It surfaces attributes like:

• The normalized query shape
• Time spent executing the query
• Transaction ID
• Client information
• … and more!

Honeycomb is unique in its ability to calculate metrics and statistics on the fly, while retaining the full-resolution log lines (and the original query that started it all!).

The agent used to translate logs to events and send them to Honeycomb is called honeytail.

Configure PostgreSQL Query Logging 

Before running honeytail, turn slow query logging on for all queries if possible. To turn on slow query logging, edit your postgresql.conf and set:

log_min_duration_statement = 0
log_statement='none'

Alternatively, you can set this from the psql shell by running

ALTER SYSTEM SET log_min_duration_statement=0;
ALTER SYSTEM SET log_statement='none';
SELECT pg_reload_conf();

Finally, take note of the value of the log_line_prefix configuration line. It will look something like this:

log_line_prefix = '%t [%p-%l] %q%u@%d '

Install and Run Honeytail 

On your PostgreSQL host, download and install the latest honeytail by running:

wget -q https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0_amd64.deb

echo '3db441215f97eaed068aa0531c986cf5405957e3e8e26b22c16b571091caf917  honeytail_1.10.0_amd64.deb' | sha256sum -c

sudo dpkg -i honeytail_1.10.0_amd64.deb

wget -q https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0_arm64.deb

echo '4220756e5a941cde6a484cb4cfde184eb189aaf29170df301a874eb143e960ed  honeytail_1.10.0_arm64.deb' | sha256sum -c

sudo dpkg -i honeytail_1.10.0_arm64.deb

wget -q https://honeycomb.io/download/honeytail/v1.10.0/honeytail-1.10.0-1.x86_64.rpm

echo 'b23215a9301b20b2e2262a0823c9e761e8b57e1a62fd5cec35f697fce41fa863  honeytail-1.10.0-1.x86_64.rpm' | sha256sum -c

sudo rpm -i honeytail-1.10.0-1.x86_64.rpm

wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.10.0/honeytail-linux-amd64

echo 'c9cc7dd1aa2b12afeb30b089061870f3407d2df0119e7c2807fec648b603e2d5  honeytail' | shasum -a 256 -c

chmod 755 ./honeytail

wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.10.0/honeytail-linux-arm64

echo '1dd37227788548c4ed44592554e3c90e374c4d796c444dde9f372db8618bc7fa  honeytail' | shasum -a 256 -c

chmod 755 ./honeytail

wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.10.0/honeytail-darwin-amd64

echo '9a3da0f48fe21b1e610ac6b63130dfb8118a9a0ec16abae13350edba02d85e4d  honeytail' | shasum -a 256 -c

chmod 755 ./honeytail

git clone https://github.com/honeycombio/honeytail

cd honeytail; go install

Make sure you have enabled query logging before running honeytail.

To consume the current slow query log from the beginning, run:

honeytail \
    --writekey=YOUR_API_KEY \
    --dataset=postgres-queries --parser=postgresql \
    --postgresql.log_line_prefix=YOUR_LOG_LINE_PREFIX \
    --file=/var/log/postgresql/postgresql-9.5-main.log \
    --tail.read_from=beginning

Troubleshooting 

Check out honeytail Troubleshooting for debugging tips.

Run Honeytail Continuously 

To run honeytail continuously as a daemon process, first modify the configuration file /etc/honeytail/honeytail.conf and uncomment and set:

• ParserName to postgresql
• WriteKey to your API key, available from the account page
• LogFiles to the path for your PostgreSQL log file.
• Dataset to the name of the dataset you wish to create with this log file.

Then start honeytail using upstart or systemd:

sudo initctl start honeytail

sudo systemctl start honeytail

Backfill Archived Logs 

You may have archived logs that you would like to import into Honeycomb. If you have a log file located at /var/log/postgresql/postgresql-main.log, you can backfill using this command:

honeytail \
    --writekey=YOUR_API_KEY \
    --dataset=PostgreSQL \
    --parser=postgresql \
    --file=/var/log/postgresql/postgresql-main.log \
    --postgresql.log_line_prefix=YOUR_CONFIGURED_LOG_LINE_PREFIX \
    --backfill

This command can be used at any point to backfill from archived log files. You can read more about honeytail’s backfill behavior here.

Once you have finished backfilling your old logs, we recommend transitioning to the default streaming behavior to stream live logs to Honeycomb.

Scrub Personally Identifiable Information 

While we believe strongly in the value of being able to track down the precise query causing a problem, we understand the concerns of exporting log data, which may contain sensitive user information.

With that in mind, we recommend using honeytail’s PostgreSQL parser, but adding a --scrub_field=query flag to hash the concrete query value. The normalized_query attribute will still be representative of the shape of the query, and identifying patterns including specific queries will still be possible—but the sensitive information will be completely obscured before leaving your servers.

More information about dropping or scrubbing sensitive fields can be found here.

Open Source 

Honeytail is open source and Apache 2.0 licensed.

Did you find what you were looking for?

Yes No

---

If you have questions, join our Pollinators Community Slack!
Pro/Enterprise users can visit Honeycomb Support or email support@honeycomb.io.

Submit