I'm sure that all of you faced with performance issue in case of
Zenoss monitoring implementation. This post will show you how to avoid
such as issues.
Each monitoring system has limits and I assure you that sooner or later you run into such problems. There is one sensible solution: "Scalable Monitoring Infrastructure".
Advantages of the solution
- reduce network traffic between datacenters;
- central place of the management;
- devide network, CPU, Memory load between separated pollers;
- reliability in case of failure (not automatticly but possible to do);
| HOST | IPADDRESS | NETMASK | GATEWAY | TYPE | WEB | DB | LOCATION |
| zenPL | 192.168.20.254 | 255.255.255.0 | 192.168.20.1 | MAIN | YES | YES | Poland |
| zenHU | 192.168.20.252 | 255.255.255.0 | 192.168.20.1 | POLLER | NO | NO | Hungary |
| zenDE | 192.168.20.253 | 255.255.255.0 | 192.168.20.1 | POLLER | NO | NO | Germany |
- zenPL - this is main Zenoss server with all components - zenoss daemons, MySQL (Location - Poland)
- zenHU - this is remote collector running without zenoss frontend and MySQL (Location - Hungary)
- zenDE - this is remote collector identical to zenHU
This is not installation guide! I will not show/explain how to properly install Zenoss Core. More details about installation process you can find here:
Zenoss Installation Guide
On zenPL server you should proceed according to Zenoss Installation Guide bu instead of remote collectors you should remember about disable zenoss daemon after successfull instalation. Remember also about disable MySQL on zenHU and zenDE. Please use following commands:
service zenoss stop
service mysqld stop
Attention!
Before final poller implementation good practise is a reboot all zenoss infrastructure server.
Zenoss team has improved core product! Belive me you do not have to manually edit configuration files which placed in $ZENHOME/etc (zencommand.conf, zenhub.conf, etc.). In previous versions it was a nightmare. Edition of 15~20 config files across 3 or more servers may cause a problems.
Do not forget about zenrender activation for remote pollers (zenHU, zenDE)!
Without it zenoss main server will be not able to present graphs. It means that central server will have no access to remote RRD's files located on remote pollers.
Perform following procedure:
- switch to zenoss user:
su - zenoss
- modify $ZENHOME/bin/zenoss
uncomment line 43:
42 C="$C zenmodeler"
43 #C="$C zenrender" <--- uncomment!
44 C="$C zenperfsnmp
42 C="$C zenmodeler"
43 C="$C zenrender" <--- line has been uncommented!
44 C="$C zenperfsnmp
You don't need to restart daemons on zenHU and zenDE because all of them should be down.
Last mandatory requirements is distribute ssh rsa key based authentication for zenoss user between main and poller servers. You should also remember about additional sshd additional configuration.
- generate private key on zenPL and distribute for all remote poolers:
su - zenoss
ssh-keygen -t rsa -b 4096
ssh-copy-id -i zenHU
ssh-copy-id -i zenDE
ssh-copy-id -i zenPL
- testing & accept fingerprints:
ssh zenHU
ssh zenDE
ssh zenPL
- modify /etc/ssh/sshd_config and sshd restart
On all of zenoss infrastructure servers perform below procedure:
http://zenPL:8080
On the top bar menu go to Advanced -> from sub-menu choose Collectors:
Next please type FQDN hostname or IP address. This must be remote IP not localhost! Please remember about it otherwise it will never work!
After "click" OK some outcome should appear in separate popup window. Please trace the result very carefully. At the end of output should be "command finished successfully"
You should repeat above actions for second remote poller which is located in Germany.
At this moment you should see 3 collectors on the list:
- localhost which is exacly zenPL collector;
- zenHU remote collector Hungary;
- zenDE remote collector Germany;
From now if you will add new device you will have possibility to assign new monitored device into designated collector.
I hope that this article will be usefull for newbie Zenoss administrators.
There is also second way to configure remote collector. In this case you should only install zenpack for distributed remote collector. Particular zenpach can be found on following webpage: http://wiki.zenoss.org/download/zenpacks/ZenPacks.community.DistributedCollectors/2.0.2/ZenPacks.community.DistributedCollectors-2.0.2.egg
Attention!
Install latest version previous has bugs.
Please download Distributed Collector ZenPack and install it on each on monitoring server. You can di it in two ways 1st: command line, 2nd: webgui. I will present both methods:
1. Zenpack commandline installation:
Log to zenPL, zenDE and zenHU and run as zenoss user following command:
sudo su - zenoss
cd /home/zenoss/download/Zenpacks
zenpack --install ZenPacks.community.DistributedCollectors-2.0.2.egg
zenoss restart
Please remember to restart zenoss afterwards. Anyway distributed collector will be not active in the system.
You can stop configure process right now. If your station is in the same subnet you don't need to do additional steps otherwise you should proceed as follow:
2. Modify Zenrender URL - customize zenrender.
Open main zenoss webpage from zenPL:
http://zenPL:8080
ADVANCED->Collectors->zenHU->Edit->Render URL
change previous value to:
/remote-collector/zenHU for collector zenHU
/remote-collector/zenDE for collector zenDE
3. Configure https proxy server as central webgui.
We want to hide zenoss zenrender and collectors behind 1 https proxy server. I choose to this role apache 2.2 but you can use nginx as well.
If you want to use default ssl configuration please consider to add following entries into /etc/httpd/conf.d/ssl.conf:
ProxyPass /remote-collector/zenHU/ http://zenHU:8091/
ProxyPass /remote-collector/zenHU/ http://zenDE:8091/
ProxyPass / http://localhost:8080/VirtualHostBase/https/zenPL:443/
First two entries are proxy pass rules to external zenoss collectors (zenrender). We would like to see all graphs on zenPL. That's why I choose zenPL as point for presentation layer. It will be central server with http webgui. 3rd entry is going to realize central web server function.
Do not forget about httpd apache restart. You can use following command to refresh apache configuration:
service httpd restart
On zenPL server you should proceed according to Zenoss Installation Guide bu instead of remote collectors you should remember about disable zenoss daemon after successfull instalation. Remember also about disable MySQL on zenHU and zenDE. Please use following commands:
service zenoss stop
service mysqld stop
Attention!
Before final poller implementation good practise is a reboot all zenoss infrastructure server.
Zenoss team has improved core product! Belive me you do not have to manually edit configuration files which placed in $ZENHOME/etc (zencommand.conf, zenhub.conf, etc.). In previous versions it was a nightmare. Edition of 15~20 config files across 3 or more servers may cause a problems.
Do not forget about zenrender activation for remote pollers (zenHU, zenDE)!
Without it zenoss main server will be not able to present graphs. It means that central server will have no access to remote RRD's files located on remote pollers.
Perform following procedure:
- switch to zenoss user:
su - zenoss
- modify $ZENHOME/bin/zenoss
uncomment line 43:
42 C="$C zenmodeler"
43 #C="$C zenrender" <--- uncomment!
44 C="$C zenperfsnmp
42 C="$C zenmodeler"
43 C="$C zenrender" <--- line has been uncommented!
44 C="$C zenperfsnmp
You don't need to restart daemons on zenHU and zenDE because all of them should be down.
Last mandatory requirements is distribute ssh rsa key based authentication for zenoss user between main and poller servers. You should also remember about additional sshd additional configuration.
- generate private key on zenPL and distribute for all remote poolers:
su - zenoss
ssh-keygen -t rsa -b 4096
ssh-copy-id -i zenHU
ssh-copy-id -i zenDE
ssh-copy-id -i zenPL
- testing & accept fingerprints:
ssh zenHU
ssh zenDE
ssh zenPL
- modify /etc/ssh/sshd_config and sshd restart
On all of zenoss infrastructure servers perform below procedure:
- activate PermitUserEnvironment:uncomment and change value PermitUserEnvironment to yes:PermitUserEnvironment yes
- restart sshd:service sshd restart
http://zenPL:8080
On the top bar menu go to Advanced -> from sub-menu choose Collectors:
Next and final step is add remote collector. Please click cog icon and from popup menu Add Remote Monitor
After "click" OK some outcome should appear in separate popup window. Please trace the result very carefully. At the end of output should be "command finished successfully"
You should repeat above actions for second remote poller which is located in Germany.
At this moment you should see 3 collectors on the list:
- localhost which is exacly zenPL collector;
- zenHU remote collector Hungary;
- zenDE remote collector Germany;
From now if you will add new device you will have possibility to assign new monitored device into designated collector.
I hope that this article will be usefull for newbie Zenoss administrators.
There is also second way to configure remote collector. In this case you should only install zenpack for distributed remote collector. Particular zenpach can be found on following webpage: http://wiki.zenoss.org/download/zenpacks/ZenPacks.community.DistributedCollectors/2.0.2/ZenPacks.community.DistributedCollectors-2.0.2.egg
Attention!
Install latest version previous has bugs.
Please download Distributed Collector ZenPack and install it on each on monitoring server. You can di it in two ways 1st: command line, 2nd: webgui. I will present both methods:
1. Zenpack commandline installation:
Log to zenPL, zenDE and zenHU and run as zenoss user following command:
sudo su - zenoss
cd /home/zenoss/download/Zenpacks
zenpack --install ZenPacks.community.DistributedCollectors-2.0.2.egg
zenoss restart
Please remember to restart zenoss afterwards. Anyway distributed collector will be not active in the system.
You can stop configure process right now. If your station is in the same subnet you don't need to do additional steps otherwise you should proceed as follow:
2. Modify Zenrender URL - customize zenrender.
Open main zenoss webpage from zenPL:
http://zenPL:8080
ADVANCED->Collectors->zenHU->Edit->Render URL
change previous value to:
/remote-collector/zenHU for collector zenHU
/remote-collector/zenDE for collector zenDE
3. Configure https proxy server as central webgui.
We want to hide zenoss zenrender and collectors behind 1 https proxy server. I choose to this role apache 2.2 but you can use nginx as well.
If you want to use default ssl configuration please consider to add following entries into /etc/httpd/conf.d/ssl.conf:
ProxyPass /remote-collector/zenHU/ http://zenHU:8091/
ProxyPass /remote-collector/zenHU/ http://zenDE:8091/
ProxyPass / http://localhost:8080/VirtualHostBase/https/zenPL:443/
First two entries are proxy pass rules to external zenoss collectors (zenrender). We would like to see all graphs on zenPL. That's why I choose zenPL as point for presentation layer. It will be central server with http webgui. 3rd entry is going to realize central web server function.
Do not forget about httpd apache restart. You can use following command to refresh apache configuration:
service httpd restart
