How To Build and Testing Fluentd With Docker

by

Fluentd is used by many companies around the log to process the logs, and then ship them to log collectors, such as Elasticsearch, Datadog, Dynatrace etc. The beauty of using docker for build and testing fluentd, we don’t need to install anything, we can set up different versions and install a lot of plugins without affecting your computer.

In case you didn’t have docker installed on your linux machine, follow the how to install docker on Linux

1. Build Docker Image

Depending on your requirement, you may need to install some plugin, create a Dockerfile file with below code

FROM fluent/fluentd
RUN gem install fluent-plugin-elasticsearch
RUN gem install fluent-plugin-datadog

then build the docker image, using

docker build . -t atetux-fluentd

change atetux-fluentd tag with anything you want.
docker build fluentd
If you don’t need to send any log to Log Collector, then we can just use the default fluent/fluentd images

2. Run Fluentd

The default images need fluentd config to run, so we’ll create a simple one, then we’ll customize that later. Create a fluentd.conf file with following code

<source>
  @type forward
</source>
 
<match **>
  @type stdout
</match>

the run the docker image, which we build earlier.

docker run --name=atetux-fluentd  -it -p 24224:24224 -v $PWD/fluentd.conf:/fluentd/etc/fluentd.conf -e FLUENTD_CONF=fluentd.conf atetux-fluentd

docker run fluentd
port 24224 is the default port for docker logging, so we may send any docker log to this fluentd, without much work, which is make our life easier to testing a lot of different application from docker.

If you change the fluentd config, you may need to restart the pod, but we can do the trick using kill, which is a lot faster than restarting pod. Login to the fluentd pod, the run kill -HUP 1

docker exec -it atetux-fluentd sh

From another terminal, we’ll feed the logs into fluentd pod

docker run --log-driver=fluentd alpine:latest echo "testing fluentd on docker"

these pod will return testing fluentd on docker, but if you check on another terminal which used to run fluentd, it’ll become a json output

2021-12-11 14:14:07.000000000 +0000 4936f2ebbb4f: {"container_id":"4936f2ebbb4f4196d5215daaee9f0d0eaf3c401c606f8927d02ab1a427932c08","container_name":"/zealous_swirles","source":"stdout","log":"testing fluentd on docker"}

log formatted by fluentd

3. Feed the Logs From File

To test different variation of logs, we can copy the logs info files then feed the logs to fluentd to see how it formatted, create a file name raw.log

2021/12/10 14:20:27 [error] 22101#22101: *1890125 directory index of "/var/www/domain/atetux.com/wp-admin/css/" is forbidden, client: 127.0.0.1, server: atetux.com, request: "GET /wp-admin/css/ HTTP/2.0", host: "atetux.com", referrer: "google.com"
server error
server response 500, microservices not found.

the run the docker

docker run -v $PWD:/tmp --log-driver=fluentd -it alpine:latest cat /tmp/raw.log

Leave a Comment