OpenStack Lumberjack – Part 2 OpenStack Services

Configuring OpenStack for Remote Logging

Logging for OpenStack has come quite a ways. What I’m going to attempt to do over a few posts, is recreate and expand a bit on what was discussed at this last OpenStack Summit with regard to Log Management and Mining in OpenStack. For now, that means installing rsyslogd and setting it up to accept remote connections.

Getting Started

For this work, you will need to have followed all the steps from here.

Once your environment is online and rsyslog is configured, we will need to change a few configuration lines for each OpenStack service. In the example we’re working with, the majority of the services are installed on the controller node. If doing this in your environment, you will need to adjust accordingly.

First, we will configure the services logging for all OpenStack services. Next, we will log into our remote hosts and configure rsyslog to ship the logs to our controller node.

Note: No additional config will be needed on the controller to send things off to rsyslog, as it’s all local.

Configuring Logging for OpenStack Services

As the configuration for each service is actually very similar, we will keep things simple and show you how to do it for the first service and leave it an exercise of the reader to enable / configure logging on the remaining services.

1. Log into the controller
2. Open /etc/[servicename]/[servicename].conf
3. Add or edit the following lines:

use_syslog = True
syslog_log_facility = LOG_LOCAL0

1. Restart the relevant services. On Ubuntu for nova this generally works like:
   sudo restart nova-api

What this does, is tell the service to send it’s logs to the syslog facility on the host. You will need to do this for each OpenStack service on each host you have set up.

This is where having a config management tool like Chef or Puppet becomes super handy.

Configuring rsyslog on the Remote Hosts

Now that you have configured all of the OpenStack services to send logs to syslog, the next step is to configure rsyslog on each host to ship logs back to the controller. We will show you how to do that on the compute host of our example setup, and leave it as an exercise of the reader to configure the remainder. To set up remote log shipping in rsyslog on the compute node, you can use the following steps:

1. Log into the compute node
   vagrant ssh compute
2. Open /etc/rsyslog.conf
3. Add a line that looks similar to:
   *.* @@controller:514
4. Finally, restart syslog on all the things:
   sudo restart rsyslog

Note: This ships all the logs from the host to the controller. Narrow as you see fit.
Note 2: The format is @@hostnameoripaddress:port for the syslog server. What is above is what works for our environment.

Verify Remote Logs are Working

Now that you have configured the services and restarted syslog on all the relevant hosts, things should be flowing in. To check this, you can do the following:

vagrant ssh controller
sudo su -
tail -f /var/log/syslog

You should start to see messages similar to:

Jun 13 23:10:15 controller 2014-06-13 23:10:15.799 10821 INFO nova.osapi_compute.wsgi.server [req-4cd20914-e9b9-42a3-bf04-c33c5a515421 624b7198901144f4a4aef8fd90821d56 8892c8b43e55425ba0860dd1003e69bd] 172.16.0.200 "GET /v2/8892c8b43e55425ba0860dd1003e69bd/limits?reserved=1 HTTP/1.1" status: 200 len: 604 time: 0.0333149
Jun 13 23:10:15 controller 2014-06-13 23:10:15.843 10821 INFO nova.osapi_compute.wsgi.server [req-73ed81b6-4dbf-4528-b3bc-e0d4a937070a 624b7198901144f4a4aef8fd90821d56 8892c8b43e55425ba0860dd1003e69bd] 172.16.0.200 "GET /v2/8892c8b43e55425ba0860dd1003e69bd/limits?reserved=1 HTTP/1.1" status: 200 len: 604 time: 0.0366650
Jun 13 23:10:35 compute vmsvc[8575]: [ warning] [guestinfo] RecordRoutingInfo: Unable to collect IPv4 routing table.
Jun 13 23:10:35 compute vmsvc[8575]: [ warning] [guestinfo] RecordRoutingInfo: Unable to collect IPv6 routing table.
Jun 13 23:10:35 compute vmsvc[8575]: [ warning] [guestinfo] Failed to get nic info.

Summary

What we did in this post, was configure the various OpenStack services to send their logs to syslog. Additionally we set up rsyslog on the remote hosts to forward all of their logs to our controller. So now we have all of our logs collecting in a central location.

In the next post, we’ll install a few tools that will help us actually do something with these logs.

Resources

This post here was super helpful for some of the bits that just weren’t fitting into place.