When OpenStack is deployed, it can grow to be quite complex and have any number of service interdependencies which makes monitoring it difficult. The Mirantis blog does a much better job of describing this than I could. This series will cover installing and configuring Sensu & it’s respective checks to monitor your OpenStack installation. This post specifically will cover installing both the Sensu Server & Sensu client..
Installing Sensu Monitoring – Server
The Sensu monitoring app is broken into a number of components:
- Server
- Client
- API
- Dashboard
In this section we will cover installing the Sensu-Server which will also include the API and Dashboard components.
Getting Started
Go ahead and build an Ubuntu 12.04 VM and sudo to root. Have one already? Good, let’s roll.
How to do it
To install sensu server, execute the following commands:
export DEBIAN_FRONTEND=noninteractive
apt-get update
apt-get install -y wget vim curl git
######################
# Setup package keys #
######################
# Rabbit
echo “deb http://www.rabbitmq.com/debian/ testing main” > /etc/apt/sources.list.d/rabbitmq.list
curl -L -o ~/rabbitmq-signing-key-public.asc http://www.rabbitmq.com/rabbitmq-signing-key-public.asc
apt-key add ~/rabbitmq-signing-key-public.asc
# Sensu
wget -q http://repos.sensuapp.org/apt/pubkey.gpg -O- | sudo apt-key add –
echo ” deb http://repos.sensuapp.org/apt sensu main” >> /etc/apt/sources.list
########################
# Install Dependancies #
########################
# erlang & rabbit
apt-get update
apt-get -y install erlang-nox
apt-get -y –allow-unauthenticated –force-yes install rabbitmq-server
####################
# Configure Rabbit #
####################
# Generate SSL Certs
git clone git://github.com/joemiller/joemiller.me-intro-to-sensu.git
cd joemiller.me-intro-to-sensu/
./ssl_certs.sh clean
./ssl_certs.sh generate
mkdir -p /etc/rabbitmq/ssl
cp ./server_key.pem /etc/rabbitmq/ssl/
cp ./server_cert.pem /etc/rabbitmq/ssl/
cp ./testca/cacert.pem /etc/rabbitmq/ssl/
sudo cat > /etc/rabbitmq/rabbitmq.config <<EOF
[
{rabbit, [
{ssl_listeners, [5671]},
{ssl_options, [{cacertfile,”/etc/rabbitmq/ssl/cacert.pem”},
{certfile,”/etc/rabbitmq/ssl/server_cert.pem”},
{keyfile,”/etc/rabbitmq/ssl/server_key.pem”},
{verify,verify_peer},
{fail_if_no_peer_cert,true}]}
]}
].
EOF
# Install the mgmt console
rabbitmq-plugins enable rabbitmq_management
# Restart rabbit
update-rc.d rabbitmq-server defaults
/etc/init.d/rabbitmq-server start
# Configure rabbit for sensu
rabbitmqctl add_vhost /sensu
rabbitmqctl add_user sensu mypass
rabbitmqctl set_permissions -p /sensu sensu “.*” “.*” “.*”
#################
# Install Sensu #
#################
apt-get install -y redis-server sensu
# Enable Sensu services
update-rc.d sensu-server defaults
update-rc.d sensu-api defaults
update-rc.d sensu-client defaults
update-rc.d sensu-dashboard defaults
# Configure Sensu
# SSL
mkdir -p /etc/sensu/ssl/
cp ~/joemiller.me-intro-to-sensu/client_key.pem /etc/sensu/ssl/
cp ~/joemiller.me-intro-to-sensu/client_cert.pem /etc/sensu/ssl/
# Create just enough sensu to start /etc/sensu/config.json
sudo cat > /etc/sensu/config.json <<EOF
{
“rabbitmq”: {
“ssl”: {
“private_key_file”: “/etc/sensu/ssl/client_key.pem”,
“cert_chain_file”: “/etc/sensu/ssl/client_cert.pem”
},
“port”: 5671,
“host”: “localhost”,
“user”: “sensu”,
“password”: “mypass”,
“vhost”: “/sensu”
},
“redis”: {
“host”: “localhost”,
“port”: 6379
},
“api”: {
“host”: “localhost”,
“port”: 4567
},
“dashboard”: {
“host”: “localhost”,
“port”: 8080,
“user”: “admin”,
“password”: “secret”
},
“handlers”: {
“default”: {
“type”: “pipe”,
“command”: “true”
}
}
}
EOF
# /etc/sensu/conf.d/client.json
export MY_IP=$(ifconfig eth1 | awk ‘/inet addr/ {split ($2,A,”:”); print A[2]}’)
sudo cat > /etc/sensu/conf.d/client.json <<EOF
{
“client”: {
“name”: “sensu-server.dom.tld”,
“address”: “${MY_IP}”,
“subscriptions”: [ “test” ]
}
}
EOF
# start / restart services
sudo /etc/init.d/rabbitmq-server restart
sudo /etc/init.d/sensu-server start
sudo /etc/init.d/sensu-api start
sudo /etc/init.d/sensu-client start
sudo /etc/init.d/sensu-dashboard start
How it worked
The script above had a number of components to it. First we added the required keys so we can install our various packages. From there we installed and configured RabbitMQ, the service Sensu uses as a message passing middleware.
Next we install the Sensu package:
apt-get install -y redis-server sensu
# Enable Sensu services
update-rc.d sensu-server defaults
update-rc.d sensu-api defaults
update-rc.d sensu-client defaults
update-rc.d sensu-dashboard defaults
From there we setup our configuration files and finally start the services. A few things of note: get the location of your SSL certs as well as the IP/port of your rabbit server correct, otherwise there will be pain later. Also, you will want to restart rabbit before restarting the sensu services, otherwise they may not start properly.
Installing Sensu Monitoring – Client
The installation for the Sensu monitoring client is a bit less involved, but still has some caveats we need to pay attention to.
Getting started
In our “example” setup each OpenStack node will need the client installed. Basically, anywhere you have an OpenStack service running, being Nova-Compute or the various Nova, Keystone, Cinder, etc APIs. Log into one of these nodes, sudo to root and you should be good to go. Note: I’m assuming Ubuntu 12.04 here.
How to do it
To install the Sensu client, do the following:
# Sensu
wget -q http://repos.sensuapp.org/apt/pubkey.gpg -O- | sudo apt-key add –
echo ” deb http://repos.sensuapp.org/apt sensu main” >> /etc/apt/sources.list
apt-get update
apt-get install -y redis-server sensu
# Enable Sensu services
update-rc.d sensu-server defaults
update-rc.d sensu-api defaults
update-rc.d sensu-client defaults
update-rc.d sensu-dashboard defaults
# Configure Sensu
mkdir -p /etc/sensu/ssl/
scp user@sensu_server:/ssl/client_key.pem /etc/sensu/ssl/
scp user@sensu_server:/ssl/client_cert.pem /etc/sensu/ssl/
# Create just enough sensu to start /etc/sensu/config.json
sudo cat > /etc/sensu/config.json <<EOF
{
“rabbitmq”: {
“ssl”: {
“private_key_file”: “/etc/sensu/ssl/client_key.pem”,
“cert_chain_file”: “/etc/sensu/ssl/client_cert.pem”
},
“port”: 5671,
“host”: “Sensu_Server_IP“,
“user”: “sensu”,
“password”: “mypass”,
“vhost”: “/sensu”
}
}
EOF
# /etc/sensu/conf.d/client.json
export MY_IP=$(ifconfig eth1 | awk ‘/inet addr/ {split ($2,A,”:”); print A[2]}’)
export HOST_NAME=`hostname`
sudo cat > /etc/sensu/conf.d/client.json <<EOF
{
“client”: {
“name”: “${HOST_NAME}”,
“address”: “${MY_IP}”,
“subscriptions”: [ “test” ]
}
}
EOF
# start / restart services
sudo /etc/init.d/sensu-client start
How it worked
On each of the OpenStack nodes we’ve built, we again added the apt-key for the Sensu omnibus installer. We then pulled in the packages needed to install Sensu. Next we copied our SSL certificates to the node and configured Sensu. In the configuration files it’s important again to get the location of your SSL files correct. Additionally, to change the IP address of the Rabbit server to where we installed the Sensu service. Lastly we restarted the services.
Summary
Ok, so there isn’t a whole heck of a lot of OpenStack yet. What we have built however, is a Sensu monitoring server as well as provided the blueprint on how to install a bunch of clients. In turn this provides the framework needed to add our assorted monitoring checks down the line.
As always, if you have questions, drop a line in the comments. Also you can follow me on twitter here.
I tried installing the Sensu build from your github. Some minor problem on conflicting subnet between vmnet1 vmnet2
I don’t know if that is a default ip range 172.16.52.x or I add it while doing some testing. Probably a heads up to look for conflict subnet and how to solve it. I got it to work after changing the ip range to 172.17.52.x and restart. Other than that it seems to work fine.