Centralised logging for nextcloud server - 5 Log management

 With most of the functional setup complete for nextcloud (as documented on my previous blog entries) It's time to move on to centralised log management and application monitoring. Im going to choose EFK as my logging stack, which I believe is one of the most popular log management stacks in the world.

However instead of hosting and maintaining my own EFK stack, I'm going to rely on a 3rd party such as logz.io or sematext for hosting the stack for me. both these providers seems to have good coverage in functionality and comes with a free tier for trying out the full feature set. This leaves me with having to install and configure fluent.d on my server.

Heres the steps I followed:

1. Download and install fluent.d
1. check pre installation checkpoints https://docs.fluentd.org/installation/before-install
1. I didn't end up changing anything on my server for now.
2. Follow the installation guide
1. curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-bionic-td-agent4.sh | sh
2. remove all config at /etc/td-agent/td-agent.conf
3. systemctl restart td-agent
2. Setup input source for syslog
1. https://docs.fluentd.org/input/syslog
2. update /etc/td-agent/td-agent.conf
3. update /etc/rsyslogd.conf
4. restart both services
3. Setup input source apache
1. https://docs.fluentd.org/v/0.12/articles/recipe-apache-logs-to-elasticsearch
2. Setup sources for access and error logs
4. Setup input source MySQL
1. MYSQL error log is now managed via syslog, nothing to do on td-agent.conf
5. Setup input source Nextcloud
1. setup tail source for nextcloud log at /var/log/nextcloud.log
6. Setup output source for elastic (logz.io)
1. sudo /usr/sbin/td-agent-gem install fluent-plugin-elasticsearch
2. sudo /usr/sbin/td-agent-gem install fluentd fluent-plugin-logzio
3. Add relevant match clause to /etc/td-config/td-agent.conf
7. Give read access to <user> on var/log
1. sudo chmod -R o+r /var/log/
2. sudo chmod og+rx /var/log/apache2
3. sudo chmod og+rx /var/log/mysql
8. Give td-agent user access to log files
1. sudo usermod -a -G adm td-agent
9. Give td-agent user access to apache2 log files
1. sudo setfacl -m u:td-agent:rx /var/log/apache2
2. sudo vi /etc/logrotate.d/apache2
3. add the config to update ACL to error and other-vhosts-access files
10. Setup logrotate for nextcloud log and give td-agent user access
1. sudo setfacl -m u:td-agent:rx /var/log/nextcloud.log
2. sudo vi /etc/logrotate.d/nextcloud
3. add logrotate config (https://help.nextcloud.com/t/logfile-does-not-rotate/29342/7)
4. add the config to update ACL to nextcloud.log file
11. TODO: Add backup/archive logic for logs  
1. Finalise how to archive the logs to a preferred cloud storage 

Additional reading:

• https://docs.datadoghq.com/logs/guide/setting-file-permissions-for-rotating-logs/