I was working on a simple and unrisky admin task: connecting a server to our monitoring system (zabbix). A perfect no-brainer to finish the week. The server is a CentOS 7.3 and has been installed in old-style with no configuration management :-(. I grab the last version of the Zabbix agent on the Internet and install it.
wget https://repo.zabbix.com/zabbix/4.0/rhel/7/x86_64/zabbix-agent-4.0.3-1.el7.x86_64.rpm
rpm -i zabbix-agent-4.0.3-1.el7.x86_64.rpm
systemctl daemon-reload
systemctl start zabbix-agent
Things didn't go as expected. Zabbix agent was not running. I discover a mismatch between pid file declared in Zabbix configuration and the one .
egrep pid /etc/zabbix/zabbix_agentd.conf
# PidFile=/tmp/zabbix_agentd.pid
PidFile=/var/run/zabbix/zabbix_agentd.pid
egrep pid /usr/lib/systemd/system/zabbix-agent.service
PIDFile=/run/zabbix/zabbix_agentd.pid
I fix it by customizing the systemd unit and set the correct path for the PID file.
cp /usr/lib/systemd/system/zabbix-agent.service /etc/systemd/system
sed -i s|/run|/var/run| /etc/systemd/system/zabbix-agent.service
systemctl restart zabbix-agent
Arg! It still do not work.
journalctl -u zabbix-agent
[...]
24518:20190308:160332.924 using configuration file: /etc/zabbix/zabbix_agentd.conf
24518:20190308:160332.924 cannot set resource limit: [13] Permission denied
24518:20190308:160332.924 cannot disable core dump, exiting...
What? Why Zabbix want disable the core dump. Are they so confident that the software never crashed? :-) Anyway. I checked ulimit: fine. I finally found a relevant entry in the SELinux log:
type=AVC msg=audit(1552057474.427:23308): avc: denied { setrlimit } for pid=24560 comm="zabbix_agentd" scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:system_r:zabbix_agent_t:s0 tclass=process
type=SYSCALL msg=audit(1552057474.427:23308): arch=c000003e syscall=160 success=no exit=-13 a0=4 a1=7ffc3c5ff9b0 a2=0 a3=8 items=0 ppid=24559 pid=24560 auid=4294967295 uid=996 gid=992 euid=996 suid=996 fsuid=996 egid=992 sgid=992 fsgid=992 tty=(none) ses=4294967295 comm="zabbix_agentd" exe="/usr/sbin/zabbix_agentd" subj=system_u:system_r:zabbix_agent_t:s0 key=(null)
SELinux rules are not known as readable and user friendly. So I choose to
disabled SELinux. It is acceptable for an internal server. I edit the file in
/etc and reboot. It took a very long time that alert me that something goes
wrong. I went to the WMWare console and discovered the failure:
It was supposed to be a no-brainer, not a complete failure! Solution at the next post.
