Objective: Configure a Graylog server to accept GELF-formatted logs from an Apache web server. Both servers will be running on Docker, although we will adapt the Apache Docker setup instructions to set up an existing (non-Docker) Apache server running on Ubuntu 14.04. The logs are going to be sent to the Graylog server using UDP port 12201.

Docker Setup

version: '2'
services:
  apache:
    depends_on:
      - graylog
    image: "wakproductions/apache2-graylog-config"
    volumes:
      - ~/Development/graylog-local/install-demos/apache2/etc/apache2/mods-enabled/log_gelf.conf:/etc/apache2/mods-enabled/log_gelf.conf
    ports:
      - "3000:80"
  mongo:
    image: "mongo:3"
    volumes:
      - ~/Development/graylog-local/data/mongo:/data/db
  elasticsearch:
    image: "elasticsearch:2"
    command: "elasticsearch -Des.cluster.name='graylog'"
    volumes:
      - ~/Development/graylog-local/data/elasticsearch:/usr/share/elasticsearch/data
  graylog:
    image: "graylog2/server"
    volumes:
      - ~/Development/graylog-local/data/journal:/usr/share/graylog/data/journal
      - ~/Development/graylog-local/config:/usr/share/graylog/data/config
    environment:
      # GRAYLOG_WEB_ENDPOINT_URI: "http://127.0.0.1:9000/api/" # didn't like having this config here (tried looking for REST server at this address) - could be a bug
      GRAYLOG_REST_TRANSPORT_URI: "http://127.0.0.1:12900"
    depends_on:
      - mongo
      - elasticsearch
    ports:
      - "9000:9000"
      - "12900:12900"
      - "5555:5555"
      - "12201:12201/udp"
      - "12201:12201"

docker-compose.yml

If using a preexisting Apache server, you could remove the apache block and dependency from the file. You'll just have to mirror the image setup instructions below to get the GELF module installed on your Apache server.

Apache Docker Image Setup (wakproductions/apache2-graylog-config)

For Apache to submit GELF logs to Graylog, there is a community-supported module named apache-mod_log_gelf.

FROM webdevops/apache:ubuntu-14.04

ENV WEB_DOCUMENT_INDEX index.html
ADD app /app

RUN apt-get update && apt-get upgrade

RUN a2dismod mpm_event
RUN a2enmod mpm_prefork && apt-get install libjson-c2 zlib1g

RUN wget https://github.com/Graylog2/apache-mod_log_gelf/releases/download/0.2.0/libapache2-mod-gelf_0.2.0-1_amd64.ubuntu.deb
RUN dpkg -i libapache2-mod-gelf_0.2.0-1_amd64.ubuntu.deb
RUN a2enmod log_gelf && restart apache

Dockerfile

GelfEnabled On
GelfUrl "udp://graylog:12201"

/etc/apache2/mods-enabled/log_gelf.conf Reference

Troubleshooting docker build

ERROR: Module mpm_event is enabled - cannot proceed due to conflicts. It needs to be disabled first!

I found a clue to fixing this problem here. It was resolved by disabling mpm_event via the command a2dismod mpm_event.

dpkg: error processing archive libapache2-mod-gelf0.1.0-1amd64.deb (--install): cannot access archive: No such file or directory

The instructions on the apache-mod_log_gelf page were misleading. I overlooked the part that read: "Download a package for your operating system from here Update Apache2 to the latests version and use mpm_prefork."

To fix the error you have to follow that link and download the latest version of module into the working directory on the Apache server. So I added the following line:

RUN wget https://github.com/Graylog2/apache-mod_log_gelf/releases/download/0.2.0/libapache2-mod-gelf_0.2.0-1_amd64.ubuntu.deb

Docker Networking Gotcha

Getting the Apache server to see the Graylog server

It took me a while to figure out how to get the Apache server to talk to the Graylog server. Initially, I set the GelfUrl "udp:127.0.0.1:12201" but the Graylog server was not receiving anything. The problem was that this was the equivalent to the Apache server trying to connect to itself on UDP 12201. You have to use the bridge network IP address of the Graylog server, or refer to it by the network hostname assigned by the Docker network. Each container has a hostname respective to the name you assign it in the docker-compose.yml file.

Useful Terminal Commands

Build Apache server image: docker build -t wakproductions/apache2-graylog-config .

Send test message on port 5555 to Graylog (non-GELF) from Apache docker server echo "hello from apache" | nc -vz graylog 5555

Send GELF test message on port 12201 to Graylog from Apache docker server echo -e '{"version": "1.1","host":"example.org","shortmessage":"Short message","fullmessage":"Backtrace here\n\nmore stuff","level":1,"userid":9001,"someinfo":"foo","someenv_var":"bar"}\0' | nc -u graylog 12201

Sample Graylog Configuration