Proper end-to-end syslog logging with Apache httpd

Table of content

  • Context
  • Objectives
  • Folder structures and relevant files
  • Configurations
    • /etc/httpd/conf.d/httpd_base_module.conf
    • /etc/httpd/conf.d/
    • /opt/fcrouzat/syslog/
    • /etc/rsyslog.d/zz-10-httpd.conf
    • Remote logging: /etc/rsyslog.d/zz-09-remote-httpd.conf
  • Caveats
  • Conclusions


For an unknown reason, Apache httpd only supports syslog since 2.4 and only for error logs.
On top of that, default log format is very poor and (to me) doesn’t mean anything.
Add these two informations and you’ll give any sysadmin a headache.

Fortunately there is a way around these issues by clever usage of the named pipes. This W/A is not so complicated but requires to be careful and a properly configured stack.


Using this workaround I will demonstrate how to perform proper end-to-end syslog logging with both httpd 2.2 and 2.4.
This is very helpful in PCI-DSS compliant environment because it allow you to easily centralize Apache httpd logs. As a reminder: unlike many other regulations and standards, PCI DSS explicitly requires organization to implement log management. Requirement 10 explicitly requires organizations stating: logging mechanisms and the ability to track user activities are critical in preventing, detecting, or minimizing the impact of a data compromise. The presence of logs in all environments allows thorough tracking, alerting, and analysis when something does go wrong. Determining the cause of a compromise is very difficult, if not impossible, without system activity logs.[1]

In addition to proper logging through syslog (both local and to remote hosts), and since we are about to modify our Apache httpd configuration for the better, we will use this opportunity to improve various things that always bugged me with httpd: having a maintained and useful LogFormat and having proper filenames for the local log files.

Testing and validation have been done under the following configurations:

  • RHEL 6.x + httpd-2.2
  • RHEL 7.x + httpd-2.4

Depending on how you might have already tweaked your LogFormat, your logs should looks like this: - - [21/Jan/2016:16:42:25 +0100] "GET /wp-content/ HTTP/1.1" 200 13776 "" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0"

It is not syslog-compliant, date format is quite unexpected, there is no app-name, and it is probably missing interesting informations. In addition, this is for example the combined LogFormat, but there are others and they are all different, you cannot find a discriminating way to filter them or manipulate them: remember, no app-name …
Assuming you’d like to parse or analyze these logs, you’d have to write very specific decoder because you cannot just use a filter on the app-name. The same goes for sending these logs to remote hosts over syslog, you cannot easily filter to only send Apache httpd logs without writing complex decoders…

We want at least to do the following: remove the poorly written datetime and add a valid syslog header with a deterministic app-name in front of the untouched message. A log line would then look like this:

Jan 15 03:28:07 httpd: - - "GET /wp-content/ HTTP/1.1" 200 13776 "" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0"

But while doing so, why not also improve everything by replacing the default LogFormat (combined) with one of our own that is a lot more powerful. Of course, feel free to adapt my “powerful” LogFormat to your needs. A log line would then look like this:

Jan 15 03:28:07 httpd: TLSv1.2(ECDHE-RSA-AES256-GCM-SHA384) "GET /index.html HTTP/1.1" 200 11301B "- - -"

Folder structures and relevant files

You can obviously organize your folder structure as you wish and name your files as you intend to. But, to be consistent I’ll provide the filenames I have been using.
All my configuration is done in /etc/httpd/conf.d/ and all *.conf files are included from httpd’ default configuration /etc/httpd/conf/httpd.conf. I never edit this package-shipped file. Same applies for rsyslog with /etc/rsyslog.d/ included from package-shipped /etc/rsyslog.conf

Here are the relevant files we are going to use:

  • /etc/httpd/conf.d/httpd_base_module.conf : define LogFormat and stuff module dependent
  • /etc/httpd/conf.d/ : virtualhost file for
  • /etc/rsyslog.d/zz-10-httpd.conf : define rsyslog httpd-related configurations
  • /opt/fcrouzat/syslog/ : syslog wrapper for httpd



First, create a new LogFormat that match your needs, feel free to define your own but do not includes date, time or app-name. This has to be the raw message. The syslog header will be added later.

    LogFormat "%h:%{remote}p %{Host}i:%{canonical}p %{SSL_PROTOCOL}x(%{SSL_CIPHER}x) \"%m %U %H\" %>s %BB \"%!200,302,304{Referer}i %!200,302,304{User-Agent}i %!200,302,304{cookie}i\"" default


We have to modify our virtualhost(s) to use the new LogFormat and also the named pipe.
Apache httpd must never write directly to disk anymore, everything has to be piped.
This named pipe execute a custom-made script that takes 3 arguments:

  • Virtualhost FQDN
  • Log type: value can be access or error
  • Encryption: value can be http or https
    CustomLog "|/opt/fcrouzat/syslog/ access https" default
    ErrorLog "|/opt/fcrouzat/syslog/ error https"


This is the wrapper used as a named pipe by httpd. Every single line of log is piped to this script via stdout.
This script uses logger to address rsyslog.
Argument 2 is used to determine if we are handling access or error logs.
Argument 3 is used to determine if we are handling http or https logs.
Argument 1 is passed after the fixed part to have the knowledge of the virtualhost name in rsyslog as you’ll see after
This is useful to use because we will use that to define filename pattern for our logs such as:

  • /var/log/httpd/<fqdn>.access.log
  • /var/log/httpd/<fqdn>.error.log
  • /var/log/httpd/<fqdn>.ssl.access.log
  • /var/log/httpd/<fqdn>.ssl.error.log

This is purely cosmetic and is not really syslog-related but having properly named logfile has shown to be useful.


# Log through syslog using logger and set the "app-name" protocol header with
# "httpd-(access|error)-http[s]-".

while read line ; do
    [[ $2 == access ]] && [[ $3 == http ]]  && /bin/logger -p  -t "httpd-access-http-${1}"
    [[ $2 == access ]] && [[ $3 == https ]] && /bin/logger -p  -t "httpd-access-https-${1}"
    [[ $2 == error ]]  && [[ $3 == http ]]  && /bin/logger -p daemon.error -t "httpd-error-http-${1}"
    [[ $2 == error ]]  && [[ $3 == https ]] && /bin/logger -p daemon.error -t "httpd-error-https-${1}"


Here’s the deal: we now have httpd logging through a named piped that uses logger to perform syslog. We also have relevant informations from the app-name that has been set with the -t flag from logger. We can use all these properties to add the relevant syslog header on httpd raw message and log to a properly named logfile.

$PreserveFQDN on

# Log Format template: add a proper syslog header and append %msg%

$template HttpdFormat,"%timegenerated% %hostname% httpd: %msg%\n"
$template HttpdRemoteFormat,"<%PRI%>%timegenerated% %hostname% httpd: %msg%\n"

# Dynamic app-name logfile templates

# Remove "httpd-access-http-" from %app-name% to extract the virtualhost name
$template HttpAccessLog,"/var/log/httpd/%app-name:19:$:%.access.log"
# Remove "httpd-access-https-" from %app-name% to extract the virtualhost name
$template HttpsAccessLog,"/var/log/httpd/%app-name:20:$:%.ssl.access.log"
# Remove "httpd-error-http-" from %app-name% to extract the virtualhost name
$template HttpErrorLog,"/var/log/httpd/%app-name:18:$:%.error.log"
# Remove "httpd-error-https-" from %app-name% to extract the virtualhost name
$template HttpsErrorLog,"/var/log/httpd/%app-name:19:$:%.ssl.error.log"

# Syslog appname-based routing with proper template depending on access/error and http/https

if $app-name startswith 'httpd-access-http-' then ?HttpAccessLog;HttpdFormat
& stop

if $app-name startswith 'httpd-access-https-' then ?HttpsAccessLog;HttpdFormat
& stop

if $app-name startswith 'httpd-error-http-' then ?HttpErrorLog;HttpdFormat
& stop

if $app-name startswith 'httpd-error-https-' then ?HttpsErrorLog;HttpdFormat
& stop

Remote logging: /etc/rsyslog.d/zz-09-remote-httpd.conf

If you want to forward httpd logs to a remote host now that they are properly formatted, just add the following configuration prior to the & stop instructions, hence the numbering on the filename. We are also using a dedicated syslog template for remote messages that preserve the syslog priority by using <%PRI%>.
This setup is using TCP syslog forward with failover.

if $app-name startswith 'httpd-' then {
    $ActionExecOnlyWhenPreviousIsSuspended on
    $ActionExecOnlyWhenPreviousIsSuspended off
# DEFAULT (all)
else {
    $ActionExecOnlyWhenPreviousIsSuspended on
    $ActionExecOnlyWhenPreviousIsSuspended off


There is at least one caveat with this setup: for an unknown reason, with RHEL6+httpd-2.2 there is an extra leading space in the %msg% part of the log when piped. Because of that, you have to replace httpd: %msg%\n" with httpd:%msg%\n" in the LogFormat section of the rsyslog configuration to have a properly formatted message on the wire and in the logs.


Besides TL;DR and for those who survived, as we have seen in the introduction, having a proper end-to-end configuration for httpd and syslog is not available out-of-the-box and requires many adjustments and maybe skills. Thanks to this setup we have been able to make httpd log through syslog and also do a little cleanup of the configuration: we have properly named logfiles and relevant content, we can filter locally or remotely on the app-name and write decoders from there.

As for stressing this setup: I am currently running this on different servers with approx. 2,000,000 lines of log every day each and without any latency on RHEL7.x + httpd-2.4 and SELinux=enforcing.

[1] –

Cisco AAA using TACACS+ with syslog and PAM support for CentOS/RedHat

Table of content

  • Context
  • TACACS+ Server installation
  • TACACS+ Server Configuration
  • Running TACACS+


In order to provide centralized authentication to network devices, radius is commonly used and works very well. It is generic and be used as a “proxy” to any kind of authentication backend which is great. If you have a very strict security policy or are running network devices in a very strict environment (PCI-DSS is a good example) it is mandatory to provides audit logs of all actions executed by privileged users and if possible, by anyone.

If ever your company is working with Cisco equipments, then it appears it is not possible to use radius for accounting and you have to use … TACACS… While you are stuck with a proprietary solution, the “good” news is that you can plug TACACS with PAM and use it for every A in AAA: authentication, authorization and accouting. Here’s how:

TACACS+ Server installation

It is important to have pam-devel installed during the compilation so that tac_plus is compiled with PAM support.
Verify in your ./configure output that PAM support has been enabled.

cd /usr/local/src/
tar xvzf tacacs+-F4.0.4.27a.tar.gz
cd tacacs+-F4.0.4.27a

yum install flex bison tcp_wrappers-devel tcp_wrappers-lib gcc pam-devel

./configure --prefix=/opt/tacacs+
make install
chmod 750 /opt/tacacs+
mkdir /opt/tacacs+/etc/

yum remove flex bison tcp_wrappers-devel tcp_wrappers-lib gcc cpp mpfr ppl cloog-ppl pam-devel

TACACS+ Server Configuration

  • /opt/tacacs+/etc/tac_plus.conf:

accounting syslog
logging = local2

# read only group
# not sure it works, haven't used it yet
group = readonly {
    default service = deny
    service = exec {
            priv-lvl = 0
    cmd=show {
            permit .*
    cmd=enable {
                permit .*
    cmd=exit {
                permit .*

# admin group
group = admins {
        default service = permit
        login = PAM
        service = exec {
             priv-lvl = 15

# Create a block for every admin user you have
user = fcrouzat {
        member = admins
  • /etc/pam.d/tac_plus:
    auth       include      system-auth
    account    required
    account    include      system-auth
    password   include      system-auth
    session    optional force revoke
    session    include      system-auth
    session    required

Next step is to open firewalling between your Cisco(s) and your TACACS+ server(s) for the TCP/49 port.
Now back to the system, we must configuration rsyslogd /etc/rsyslog.d/tacacs.conf

if $app-name == 'tac_plus' then /var/log/tacacs.log
& stop

… and logrotate /etc/logrotate.d/tacacs

/var/log/tacacs.log {
    rotate 4
    create 0640 root root
        /bin/kill -HUP `cat /var/run/ 2>/dev/null` 2>/dev/null || true

Client configuration (Cisco)

! use tacacs+ for authentication
aaa authentication login default local group tacacs+
! authorization if authenticated hence with tacacs+
aaa authorization exec default if-authenticated

! accounting with tacacs+
aaa accounting commands 0 default
 action-type start-stop
 group tacacs+
! accounting with tacacs+
aaa accounting commands 1 default
 action-type start-stop
 group tacacs+
! accounting with tacacs+
aaa accounting commands 15 default
 action-type start-stop
 group tacacs+
! tacacs+ server configuration
tacacs server logrelay01xxx
 address ipv4 192.168.XX.YY

Running TACACS+

Finally you can run the server in debug mode:

/opt/tacacs+/bin/tac_plus -C /opt/tacacs+/etc/tac_plus.conf -L -p 49 -d128 -g

… or use the very basic following initscript[1] which works for EL6:

wget -O /etc/init.d/tac_plus
restorecon -RvF /etc/init.d/tac_plus
chmod 755 /etc/init.d/tac_plus
chkconfig tac_plus on
chkconfig --list tac_plus
service tac_plus start

Finally, you can tail the TACACS+ server logs and complete the setup by logging on a Cisco and verify that authentication happen, then authorization and finally accounting are working:

Feb 22 15:38:13 tac_plus[25833]:    fcrouzat    tty1    stop    task_id=124    timezone=UTC    service=shell    start_time=1456152071    priv-lvl=1    cmd=show vlan brief 
Feb 22 15:38:17 tac_plus[25834]: connect from []
Feb 22 15:38:17 tac_plus[25834]:    fcrouzat    tty1    stop    task_id=125    timezone=UTC    service=shell    start_time=1456152076    priv-lvl=1    cmd=show interfaces trunk 

[1] –

[Updated 2015-03-10] How to work with root shells in a PCI-DSS 10.2.2 compliant environment

Table of content

  • Context and objectives
  • PCI-DSS 10.2.2
  • How it limits your productivity
  • Solutions
    • PAM
      • Concepts
      • How it works
      • Examples of output

Context and objectives

If you ever worked as a system administrator in a Linux PCI-DSS environment, you know it’s sometimes (often) difficult to administrate your servers from command-line because of PCI-DSS Requirement 10.2.2 which forbids you to open a real root shell, whatever the way, because actions taken in a root shells are not logged anywhere and you must log every privilege usage. That’s why most of us end-up using sudo, but it is limited and not always easy to work with, as I will explain later (PATH completion, wildcards, etc…)

The purpose of this post is to provide a way for administrator to open a root shell (whatever the way) and work in that environment while being compliant with 10.2.2, ie: having all actions taken in this shell logged and centralized.

PCI-DSS 10.2.2

PCI-DSS Requirement 10 : Track and monitor all access to network resources and cardholder data

PCI-DSS Requirements Testing Procedures Guidance
10.2.2 All actions taken by any individual with root or administrative privileges 10.2.2 Verify all actions taken by any individual with root or administrative privileges are logged. Accounts with increased privileges, such as the “administrator” or “root” account, have the potential to greatly impact the security or operational functionality of a system. Without a log of the activities performed, an organization is unable to trace any issues resulting from an administrative mistake or misuse of privilege back to the specific action and individual.

How it limits your productivity

Because of that, it is impossible to use wildcard facilities or auto-completion from the CLI. For example, if the /etc/bar/ folder is not publicly readable, fgrep foo /etc/bar/*.xml will not work as the wildcard will not expand. You’d have to use:

  • ls /etc/bar, then one grep foo per XML file,
  • or grep -R foo /etc/bar/ (which includes non-xml files and subdirectories),
  • or find /etc/bar -name '*.xml' -maxdepth 1 -exec grep foo {} \+ (which is far from trivial)
  • or any other half-smart workaround which sucks anyway…

It would be a lot easier if you could just gain root privilege using sudo -s, sudo -i or maybe for old-schooler that haven’t read the man page of sudo: sudo su - # Eww!. Direct logins as root (ssh are still forbidden and using su is strongly discouraged. Remember that we are trying to find a way to gain root privileges, not to tweak PCI-DSS and PCI-DSS states that you must not log-in as root and you shall not have the knowledge of the entire root password thus making su not usable without having someone entering his other half of the password. Ok so sudo -s would be allowed if only we could provide logs for every actions taken in the privileged shell… How to do that ? Well, let’s check possible solutions. We will also check how to generalize this concept to any shells so we can track any commands anywhere, anytime … !



At first, I looked at PAM and how to log every keystrokes entered within a root shell.
To track such actions we need to configure the following:

  • the PAM module:
  • the auditd service: /sbin/chkconfig auditd on && service auditd start
  • append to /etc/pam.d/system-auth: session required disable=* enable=root
  • append to /etc/pam.d/su and /etc/pam.d/su-l: session required enable=root
  • append to /etc/pam.d/sudo and /etc/pam.d/sudo-i: session required open_only enable=root

It works, I tried it. I could see all keystrokes in /var/log/audit/audit.log or using aureport --tty -if /var/log/audit/audit.log (because you cannot read them in plain-text in the logs …)

The downsides are:

  • It is not trivial to deploy
  • You cannot exploit logs right-away, you need a tool: aureport
  • There is a lot of noise because you log keystrokes, not commands, so you can see lots of “<^C><^C><Up><Del>“….
  • It logs every keystrokes ! Including kerberos passwords, MySQL passwords, anything you enter in a password prompt even if echoing is off is logged because it is a keystroke

Update 2015-03-10: pam_tty_audit has been patched to support password mode and not log the keystrokes while in “password-mode”. You can find the patch here and the Red-Hat documentation updated accordingly here

    It is great for security. To be honest I believe that everyone should enable this feature not only for root but for everyone so that if ever one of you application gets hacked (say apache through a poor website) then you can retrieve every keystrokes and commands entered by the hacker while he was exploiting an unprivileged-shell (eg: id=apache)

    So okay, it’s great news for security and one can generalize this concept without risking to store plain-text passwords but it’s not enough… As someone said “we have to go deeper”. With pam_tty_audit only me and my fellow coworkers could open root shells and start working but it will be a pain in the ass to review who did what because the produced logs are not easy to read (you need aureport –tty …). We are used to read /var/log/secure whenever we want to know who restarted a daemon and such… In addition of the security layer added by pam_tty_audit we need a solution that produces sudo-like logs for legitimate root shells, this is where bash PROMPT_COMMAND enters the game.


    The idea is to make a clever usage of the PROMPT_COMMAND bash variable. The man page says: If set, the value is interpreted as a command to execute before the printing of each primary prompt ($PS1).


    It means that you can execute a custom command every time a command is entered in a bash shell (each time Enter is pressed). Using this feature and a script which use logger we can easily recreate sudo-like logs for any shells, including root’s. This is the first step.

    If you think a little more, since there are ways to keep track of the real source-user (initial user who opened a shell in a shell in a shell in a shell…. etc), you can even track down anyone changing identity (sudo -u foo -s). It also means that in case someone uses N-level of shells before accessing the root shell, like with sudo -u foo -s ; sudo sudo su -, you can still know the real source-user and keep track of his privileged actions accordingly.

    For logging purpose, I had to chose an app-name for both cases: root shells and switched-identity shells. For this example, I chose su-company and chusr-company. Replace “company” by whatever you want. I also defined and hard-coded a log format, but you can easily change that by editing the logger lines, obviously. I did that so I could write Ossec decoders and rules to match these logs.

    How it works

    You just have to set the variable system-wide and assign a bash function to it which will do the testing and logging parts.

    • /etc/profile.d/
    # Written by Florian Crouzat &amp; Anwar El Fatayri
    # Contact: 
    # Feel free to do whatever you want with this file.
    # Just make sure to credit what deserve credits.
    # Warning: do not use a shebang if you are to place this script in /etc/profile.d/
    # Get informations about the parent of the current process via PPID.
    # The idea is to go up in the process-tree until you found the first login-shell
    function get_ppid_information() {
            # Get informations about the PPID
            command=$(ps -o cmd= -p $ppid)
            user=$(ps -o user= -p $ppid)
            # Get the PPID of the PPID for the next iteration
            ppid=$(ps -o ppid= -p $ppid)
    # Init: get the parent PID of the current shell using $PPID
    # First-pass: Get informations about our PPID (command and user) and initialize future PPID
    # Then, travel the process tree until we get the first user that logged in
    while true ; do
             [[ $command != *bash* ]] &amp;&amp; [[ $command != *su* ]] &amp;&amp; break || get_ppid_information
    function log_root_shell_cmd() {
            # Get the last command from history properly (delete white spaces) using bash's builtin fc
            shell_cmd=$(fc -ln | tail -n1 | sed 's/^\t //')
            # If the last command has been repeated, then skip logging
            if [[ $previous_shell_cmd != $shell_cmd ]] ; then
                    # If this is a root shell, we log.
                    if [ $UID -eq  0 ] ; then
                            logger -p authpriv.crit -t su-company "TTY=$SSH_TTY ; PWD=$PWD ; USER=$user ; COMMAND=$shell_cmd"
                    # If this is a user shell, but the source-user and current user differ, we log.
                    elif [[ $UID != 0 ]] &amp;&amp; [[ $user != $USER ]] ; then
                            logger -p authpriv.crit -t chusr-company "TTY=$SSH_TTY ; PWD=$PWD ; SRC_USER=$user ; USER=$LOGNAME ; COMMAND=$shell_cmd"
                    # Save the last command
                    export previous_shell_cmd
    # Append log_root_shell_cmd function to PROMPT_COMMAND
    PROMPT_COMMAND="${PROMPT_COMMAND:-:} ; log_root_shell_cmd"
    • Raw script available for download here.

    Maybe you are thinking that it is very easy to hide your tracks and cancel this behavior, don’t over-think it, it is !
    You can use any other shell, for example sh or ksh which don’t source the same files, you can unset the variable, edit the script, etc. But, as I said earlier in this post, the intent is not to protect yourself from a malicious root user because you just can’t, as soon as someone is root he can always stop any tracking process you created, whatever the complexity. You’ll just log the “stop” command, and after that, you are in the dark. So, don’t loose time trying to be smart against malicious root users they can cancel whatever you do, and focus on the main idea: simplify the life of your goodwill sysadmins which will not try to hide things from you and just want to work in a better environment.

    Examples of usage
    • For root shells:
    # The actions
    (11:48) (florian@bar.wnd) (~) $ sudo -s # let's open a root shell
    (11:49) (root@bar.wnd) (/home/florian) # ls -al /etc/pki/tls/private/*.key # yay, completion !
    (11:49) (root@bar.wnd) (/home/florian) # whoami
    # And the logs ...
    (11:49) (florian@bar.wnd) (~) $ sudo tail -n20 /var/log/secure
    May  7 05:48:53 bar sudo:  florian : TTY=pts/1 ; PWD=/home/florian ; USER=root ; COMMAND=/bin/bash
    May  7 11:49:02 bar su-company: TTY= ; PWD=/home/florian ; USER=florian ; COMMAND=ls -al /etc/pki/tls/private/*.key
    May  7 11:49:08 bar su-company: TTY= ; PWD=/home/florian ; USER=florian ; COMMAND=whoami
    • For switched-identity shells:
    # The actions
    (11:53) (florian@bar.wnd) (~) $ sudo -u jpc -i # let's take-over someone else identity
    [jpc@bar ~]$ id # and do harmless stuff in this shell. Yet, it's good to know.
    uid=501(jpc) gid=501(jpc) groups=501(jpc),504(sftp)
    # And the logs ...
    (11:53) (florian@bar.wnd) (~) $ sudo tail -n20 /var/log/secure
    May  7 05:53:28 bar sudo:  florian : TTY=pts/1 ; PWD=/home/florian ; USER=jpc ; COMMAND=/bin/bash
    May  7 05:53:31 bar chusr-company: TTY= ; PWD=/home/jpc ; SRC_USER=florian ; USER=jpc ; COMMAND=id

    Feel free to ask any questions in the comments, and to provides patches by email, I’ll surely integrate them and mention your name.