LJ: Why Your Server Monitoring (Still) Sucks

Five observations about why your your server monitoring still stinks by a monitoring specialist-turned-consultant.

REF: https://www.linuxjournal.com/content/why-your-server-monitoring-still-sucks

New Check_MK stable release 1.4

Changes in all Check_MK Editions:

WATO:
* 4747 FIX: Fixed resetting global settings to factory defaults
* 4750 FIX: Ensure uploaded icons are saved with their filename instead of paths

User interface:
* 4702 FIX: Quicksearch: fixed minor parsing issue when using specific filters
* 4746 FIX: Fixed using HW/SW inventory filters in views without inventory painters

Livestatus:
* 4430 FIX: Fixed logwatch-related Livestatus queries in NEB module

HW/SW inventory:
* 4568 FIX: mk_inventory.vbs: fixed getNetworkAdpater function to prevent errors caused by null arrays

Checks & agents:
* 4742 FIX: wmi_cpuload: Added required tables to prevent crash
* 4748 FIX: solaris_fmadm: Check was only discovered when an error is detected
* 4664 FIX: netapp_api_qtree_quota: Fixed NameError: name 'is_digit' is not defined
* 4740 FIX: mysql: Fixed crash if instance is not available anymore
* 4633 FIX: jolokia_metrics.uptime: No longer crash when agent info is incomplete
* 4739 FIX: hp_proliant_da_phydrv: Implemented new possible values
* 4632 FIX: fsc_subsystems: No longer crash when statuscode is missing
* 4743 FIX: f5_bigip_conns: Fixed crash if empty values
* 4741 FIX: dell_powerconnect_temp: Fixed missing reference for computing the temperature
* 4631 FIX: cisco_ace_rserver: Fix broken IP address parsing
NOTE: Please refer to the migration notes!
* 4663 FIX: check_mailboxes: Removed useless report_age which causes TypeError: float() argument must be a string or a number
* 4735 FIX: check_mailboxes: Fixed broken check when setting "connect timeout" option
* 4734 FIX: check_mail check_mail_loop check_mailboxes check_sql: Passwords from password store were not used correctly

REF: http://lists.mathias-kettner.de/pipermail/checkmk-announce/2017-May/000205.html

Custom Snort Rules

For example, this specifies a logical message whenever Snort notices any traffic that is sent from 192.168.1.35:

alert tcp 192.168.1.35 any -> any any (msg:"Traffic from 192.168.1.35";)

REF: http://archive.oreilly.com/pub/h/1393

Check_MK: Windows mapped drives

philipp.bieber at gildemeister.com [21.08.2013 08:53]:
> Hi James,
> the problem is (probably) that the check_mk agent is run under the LOCAL
> SYSTEM Account, which has no access to those mapped drives.
> You can either go with Bill's proposal and use legacy checks, or you may
> look into creating a custom SNMP check to get the quotas of your EMC...
> ( -> http://mathias-kettner.de/checkmk_devel_snmpbased.html )
> 
> We did something similar with our NetApps....

Maybe you can create a batch file that mounts all the shares. Then you
tell your system to mount them as SYSTEM during startup.

We had to do this for one server, and it works fine. A description can
be found on
<http://social.technet.microsoft.com/Forums/windowsserver/en-US/44445dc3-0185-407f-b8ad-711f22cd4ec6/windows-nfs-client-reconnect-to-nfs-target-on-reboot>
(it works with non-nfs shares as well).

HTH, Werner

REF: http://lists.mathias-kettner.de/pipermail/checkmk-en/2013-August/010151.html

Check_MK: Agents

Check_MK agent consumes very few resource since they are only small scripts parsing system info. But remember that if the parsed resource such as log files are too large, agent performance will be affected. This can be monitored by agent execution time graph.
 
The check_mk agent for Linux consists of only two files: a shell skript called check_mk_agent.linux and a configuration file for xinetd.conf, both of which can be found in the subdirectory agents. xinetd is an improved version of the classical inetd and a is available or even standard on most current linux distributions.

REF: https://mathias-kettner.de/checkmk_linuxagent.html

Check_MK: BI & Acknowledgement

6. Acknowledgement

When a host or service is in a non-OK state then the user can acknowledge that problem. Just like the scheduled downtimes, the acknowledgement is saved as an additional attribute. As of Version 1.2.5i1Check_MK BI now also aggregates acknowledgement information up to the top node. The following algorithm is being used for computing this from the states and acknowledgements of the nodes:

A BI aggregation is acknowledged if it would have an OK state under the assumption that all acknowledged hosts and services would be UP or OK resp.

Note: You cannot directly acknowledge a BI aggregate that is in a problem state. You need to acknowledge its underlying host and service problems.

REF: https://mathias-kettner.de/checkmk_bi.html

Check_MK: customized Windows event log

REF: http://lists.mathias-kettner.de/pipermail/checkmk-en/2013-July/009851.html

>>> If you create a file called check_mk.ini in the agent directory then
>>> you can configure which eventlogs and which levels to process. Here is an
>>> example:
>>> check_mk.ini
>>>
>>>
>>>
>>> [logwatch]
>>>     # From the Application log send only critical messages
>>>     *logfile application = crit*
>>>
>>>     # From the Security log send all messages
>>>     *logfile security = all*
>>>
>>>     # Do not process other event logs at all
>>>     *logfile * = off*
>>>

Check_MK: zpool status

ZFS status can be monitored by zpool from Check_MK. Also available for LInux now.

ZFS Storage Pool status

Distribution:	official part of Check_MK
License:	GPL
Supported Agents:	Solaris

Checks the current state of a ZFS storage pool. The information is read by the agent from /usr/sbin/zpool status -x. If the zpool is healthy you're OK. If an inventoried zpool has been destroyed or has an error the check goes CRITICAL. If the extend state information from zpool status indicates any CRC or other errors, the check will go to WARNING. 

Discovery

The inventory will create one service per host if there are any pools.

REF: https://mathias-kettner.de/checkmk_check_zpool_status.html

Check_MK: network topology

Network topology may be achieved by scanning host parents and NagVis Addon.

REF: https://mathias-kettner.de/checkmk_scan_parents.html

Check_MK: virtual hosts

There're plugins for Nagios / Check_MK to monitor virtual hosts such as ESX or PVE. for example, running VMs as well as their CPU and RAM usage could be checked via the official scripts from proxmox:

Client-Check

/usr/lib/check_mk_agent/plugins/mh_qemu

Plugin

/omd/versions/0.44/share/check_mk/checks/qemu

REF: https://pve.proxmox.com/wiki/Nagios_check_mk

graphite: easy metrics

Graphite, or Grafana, is a good tool for developers to generate beautiful graphs for metrics easily. Compared with other MIS-oriented tools, it may be more flexible for customized inputs for specified application development. Nagios or Zabbix may still be better choice for standard checks such as SNMP or agent inputs.   

REF: 

http://graphite.readthedocs.io/en/latest/tools.html

Syslog: graylog for dashboards

Graylog is a pretty easy-to-use search & dashboard system. if we need to collect lots of data with rather simple dashboards and user management, it's enough for this purpose. setup of appliance is also easy. remote logging from syslog is enabled by default. Advanced analysis still requires Splunk.

Syslog: logger xferlog

xferlog can be filtered then written into syslog with the utility 'logger'. Here's a script with check_log style. Column 8 and 15 is the ip / id pair:

xferlog=/var/log/proftpd/xferlog

if [ ! -f ${xferlog}.old ]; then
 cp $xferlog ${xferlog}.old
fi

diff $xferlog ${xferlog}.old | grep "<" | awk '{print $8 " " $15}' | while read LINE
do
logger $LINE
done

rm -rf ${xferlog}.old
cp $xferlog ${xferlog}.old

Syslog: push data / rsync

Hosts behind NAT or without fixed ip may be the reason we want them to 'push data' for our monitoring. However, letting clients 'phone back' isn't a very good idea for security or stability. Here are some workarounds.

• syslog. using remote logging but no authentication or encryption is offered. use 
• rsync. push data with script running cron job.

REF: http://lists.mathias-kettner.de/pipermail/checkmk-en/2012-June/006232.html

Syslog: with logwatch

There isn't many options in conf for further syslog processing. So remote logging to another server, then using Cacti syslog plugin or Check_MK/Nagios logwatch for filtering or alert is the practical way.

# Remote logging
#*.* @log.server.ip:514

REF: http://www.netadmin.com.tw/article_content.aspx?sn=1609300018