[Updated 2015-03-10] How to work with root shells in a PCI-DSS 10.2.2 compliant environment

Table of content

  • Context and objectives
  • PCI-DSS 10.2.2
  • How it limits your productivity
  • Solutions
    • PAM
    • Bash PROMPT_COMMAND
      • Concepts
      • How it works
      • Examples of output

Context and objectives

If you ever worked as a system administrator in a Linux PCI-DSS environment, you know it’s sometimes (often) difficult to administrate your servers from command-line because of PCI-DSS Requirement 10.2.2 which forbids you to open a real root shell, whatever the way, because actions taken in a root shells are not logged anywhere and you must log every privilege usage. That’s why most of us end-up using sudo, but it is limited and not always easy to work with, as I will explain later (PATH completion, wildcards, etc…)

The purpose of this post is to provide a way for administrator to open a root shell (whatever the way) and work in that environment while being compliant with 10.2.2, ie: having all actions taken in this shell logged and centralized.

PCI-DSS 10.2.2

PCI-DSS Requirement 10 : Track and monitor all access to network resources and cardholder data

PCI-DSS Requirements Testing Procedures Guidance
10.2.2 All actions taken by any individual with root or administrative privileges 10.2.2 Verify all actions taken by any individual with root or administrative privileges are logged. Accounts with increased privileges, such as the “administrator” or “root” account, have the potential to greatly impact the security or operational functionality of a system. Without a log of the activities performed, an organization is unable to trace any issues resulting from an administrative mistake or misuse of privilege back to the specific action and individual.

How it limits your productivity

Because of that, it is impossible to use wildcard facilities or auto-completion from the CLI. For example, if the /etc/bar/ folder is not publicly readable, fgrep foo /etc/bar/*.xml will not work as the wildcard will not expand. You’d have to use:

  • ls /etc/bar, then one grep foo per XML file,
  • or grep -R foo /etc/bar/ (which includes non-xml files and subdirectories),
  • or find /etc/bar -name '*.xml' -maxdepth 1 -exec grep foo {} \+ (which is far from trivial)
  • or any other half-smart workaround which sucks anyway…

It would be a lot easier if you could just gain root privilege using sudo -s, sudo -i or maybe for old-schooler that haven’t read the man page of sudo: sudo su - # Eww!. Direct logins as root (ssh root@foo.bar are still forbidden and using su is strongly discouraged. Remember that we are trying to find a way to gain root privileges, not to tweak PCI-DSS and PCI-DSS states that you must not log-in as root and you shall not have the knowledge of the entire root password thus making su not usable without having someone entering his other half of the password. Ok so sudo -s would be allowed if only we could provide logs for every actions taken in the privileged shell… How to do that ? Well, let’s check possible solutions. We will also check how to generalize this concept to any shells so we can track any commands anywhere, anytime … !

Solutions

PAM

At first, I looked at PAM and how to log every keystrokes entered within a root shell.
To track such actions we need to configure the following:

  • the PAM module: pam_tty_audit.so
  • the auditd service: /sbin/chkconfig auditd on && service auditd start
  • append to /etc/pam.d/system-auth: session required pam_tty_audit.so disable=* enable=root
  • append to /etc/pam.d/su and /etc/pam.d/su-l: session required pam_tty_audit.so enable=root
  • append to /etc/pam.d/sudo and /etc/pam.d/sudo-i: session required pam_tty_audit.so open_only enable=root

It works, I tried it. I could see all keystrokes in /var/log/audit/audit.log or using aureport --tty -if /var/log/audit/audit.log (because you cannot read them in plain-text in the logs …)

The downsides are:

  • It is not trivial to deploy
  • You cannot exploit logs right-away, you need a tool: aureport
  • There is a lot of noise because you log keystrokes, not commands, so you can see lots of “<^C><^C><Up><Del>“….
  • It logs every keystrokes ! Including kerberos passwords, MySQL passwords, anything you enter in a password prompt even if echoing is off is logged because it is a keystroke

Update 2015-03-10: pam_tty_audit has been patched to support password mode and not log the keystrokes while in “password-mode”. You can find the patch here https://www.redhat.com/archives/linux-audit/2013-May/msg00007.html and the Red-Hat documentation updated accordingly here https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Security_Guide/sec-Configuring_PAM_for_Auditing.html

    It is great for security. To be honest I believe that everyone should enable this feature not only for root but for everyone so that if ever one of you application gets hacked (say apache through a poor website) then you can retrieve every keystrokes and commands entered by the hacker while he was exploiting an unprivileged-shell (eg: id=apache)

    So okay, it’s great news for security and one can generalize this concept without risking to store plain-text passwords but it’s not enough… As someone said “we have to go deeper”. With pam_tty_audit only me and my fellow coworkers could open root shells and start working but it will be a pain in the ass to review who did what because the produced logs are not easy to read (you need aureport –tty …). We are used to read /var/log/secure whenever we want to know who restarted a daemon and such… In addition of the security layer added by pam_tty_audit we need a solution that produces sudo-like logs for legitimate root shells, this is where bash PROMPT_COMMAND enters the game.

    Bash PROMPT_COMMAND

    The idea is to make a clever usage of the PROMPT_COMMAND bash variable. The man page says: If set, the value is interpreted as a command to execute before the printing of each primary prompt ($PS1).

    Concepts

    It means that you can execute a custom command every time a command is entered in a bash shell (each time Enter is pressed). Using this feature and a script which use logger we can easily recreate sudo-like logs for any shells, including root’s. This is the first step.

    If you think a little more, since there are ways to keep track of the real source-user (initial user who opened a shell in a shell in a shell in a shell…. etc), you can even track down anyone changing identity (sudo -u foo -s). It also means that in case someone uses N-level of shells before accessing the root shell, like with sudo -u foo -s ; sudo sudo su -, you can still know the real source-user and keep track of his privileged actions accordingly.

    For logging purpose, I had to chose an app-name for both cases: root shells and switched-identity shells. For this example, I chose su-company and chusr-company. Replace “company” by whatever you want. I also defined and hard-coded a log format, but you can easily change that by editing the logger lines, obviously. I did that so I could write Ossec decoders and rules to match these logs.

    How it works

    You just have to set the variable system-wide and assign a bash function to it which will do the testing and logging parts.

    • /etc/profile.d/zz-log-root-cmd.sh
    ##################
    # Written by Florian Crouzat &amp; Anwar El Fatayri
    # Contact: 
    # Feel free to do whatever you want with this file.
    # Just make sure to credit what deserve credits.
     
    # Warning: do not use a shebang if you are to place this script in /etc/profile.d/
    ##################
     
    # Get informations about the parent of the current process via PPID.
    # The idea is to go up in the process-tree until you found the first login-shell
    function get_ppid_information() {
            # Get informations about the PPID
            command=$(ps -o cmd= -p $ppid)
            user=$(ps -o user= -p $ppid)
            # Get the PPID of the PPID for the next iteration
            ppid=$(ps -o ppid= -p $ppid)
    }
     
    # Init: get the parent PID of the current shell using $PPID
    ppid=$PPID
     
    # First-pass: Get informations about our PPID (command and user) and initialize future PPID
    get_ppid_information
     
    # Then, travel the process tree until we get the first user that logged in
    while true ; do
             [[ $command != *bash* ]] &amp;&amp; [[ $command != *su* ]] &amp;&amp; break || get_ppid_information
    done
     
    function log_root_shell_cmd() {
     
            # Get the last command from history properly (delete white spaces) using bash's builtin fc
            shell_cmd=$(fc -ln | tail -n1 | sed 's/^\t //')
     
            # If the last command has been repeated, then skip logging
            if [[ $previous_shell_cmd != $shell_cmd ]] ; then
     
                    # If this is a root shell, we log.
                    if [ $UID -eq  0 ] ; then
                            logger -p authpriv.crit -t su-company "TTY=$SSH_TTY ; PWD=$PWD ; USER=$user ; COMMAND=$shell_cmd"
                    # If this is a user shell, but the source-user and current user differ, we log.
                    elif [[ $UID != 0 ]] &amp;&amp; [[ $user != $USER ]] ; then
                            logger -p authpriv.crit -t chusr-company "TTY=$SSH_TTY ; PWD=$PWD ; SRC_USER=$user ; USER=$LOGNAME ; COMMAND=$shell_cmd"
                    fi
     
                    # Save the last command
                    previous_shell_cmd=$shell_cmd
                    export previous_shell_cmd
            fi
    }
     
    # Append log_root_shell_cmd function to PROMPT_COMMAND
    PROMPT_COMMAND="${PROMPT_COMMAND:-:} ; log_root_shell_cmd"
    • Raw script available for download here.

    Maybe you are thinking that it is very easy to hide your tracks and cancel this behavior, don’t over-think it, it is !
    You can use any other shell, for example sh or ksh which don’t source the same files, you can unset the variable, edit the script, etc. But, as I said earlier in this post, the intent is not to protect yourself from a malicious root user because you just can’t, as soon as someone is root he can always stop any tracking process you created, whatever the complexity. You’ll just log the “stop” command, and after that, you are in the dark. So, don’t loose time trying to be smart against malicious root users they can cancel whatever you do, and focus on the main idea: simplify the life of your goodwill sysadmins which will not try to hide things from you and just want to work in a better environment.

    Examples of usage
    • For root shells:
    # The actions
    (11:48) (florian@bar.wnd) (~) $ sudo -s # let's open a root shell
    (11:49) (root@bar.wnd) (/home/florian) # ls -al /etc/pki/tls/private/*.key # yay, completion !
    [...]
    (11:49) (root@bar.wnd) (/home/florian) # whoami
    root
    
    # And the logs ...
    (11:49) (florian@bar.wnd) (~) $ sudo tail -n20 /var/log/secure
    [...]
    May  7 05:48:53 bar sudo:  florian : TTY=pts/1 ; PWD=/home/florian ; USER=root ; COMMAND=/bin/bash
    May  7 11:49:02 bar su-company: TTY= ; PWD=/home/florian ; USER=florian ; COMMAND=ls -al /etc/pki/tls/private/*.key
    May  7 11:49:08 bar su-company: TTY= ; PWD=/home/florian ; USER=florian ; COMMAND=whoami
    • For switched-identity shells:
    # The actions
    (11:53) (florian@bar.wnd) (~) $ sudo -u jpc -i # let's take-over someone else identity
    [jpc@bar ~]$ id # and do harmless stuff in this shell. Yet, it's good to know.
    uid=501(jpc) gid=501(jpc) groups=501(jpc),504(sftp)
    
    # And the logs ...
    (11:53) (florian@bar.wnd) (~) $ sudo tail -n20 /var/log/secure
    [...]
    May  7 05:53:28 bar sudo:  florian : TTY=pts/1 ; PWD=/home/florian ; USER=jpc ; COMMAND=/bin/bash
    May  7 05:53:31 bar chusr-company: TTY= ; PWD=/home/jpc ; SRC_USER=florian ; USER=jpc ; COMMAND=id

    Feel free to ask any questions in the comments, and to provides patches by email, I’ll surely integrate them and mention your name.

    7 comments so far.

    1. Any chance you could upload the raw script file for download? I think I’m experiencing a weird character issue from copy pasting… I fixed the big ones, but abrtd seems to be filing crash reports for bash when this fella is running inside /etc/profile.d/ on CentOS 6.5.

      Thanks,
      Brian

      • Sure thing Brian, I updated the article with more content and some clarifications and also provided a direct link to the script for download. I hope it will work on your side, it certainly works on mine on a couple thousands of CentOS boxes. Keep me posted! Florian

    2. Hi Florian,

      Thanks for putting that up! Let me give you a little more information about what I’m seeing.

      If you were to install and enable the automatic bug reporting tools (abrtd & abrt-addon-ccpp), the ccpp crash collector often reports crashes for bash, often directly after running any command. I seem to have isolated it to this script, as once I remove it from profile.d, log out and back in, ABRT quiets down.

      • Hey again Brian. Sadly I do not use these tools (yet?) and thus have not been able to reproduce.
        I’ll try to see what I can do to try and reproduce on a random box out there.

        Since this bash code is recursive and executed after each command I only see two options as why it could “crash”:
        * abrtd is not happy with recursion, even though it’s mostly one iteration (except if you use sudo sudo sudo sudo sudo -s…)
        * you found a corner case where the exit condition is not matched and the recursion never stops: I’m interested in understanding that and how you end-up in such a use-case

        I’ll let you know if I manage to reproduce, in the meantime you could check your @/var/log/secure@ logs when it crashes to understand how many layers of subshells you are running and check your hosts metrics (mostly RAM/CPU I’d say as an infinite loop would probably consume them…). Keep me posted.

        • I’ll give it a go ahead and let you know, Florian.

          TBH, I didn’t even know my production instances were running ABRT, this sort of let me know. For the time being, I just committed a new build script that removes ABRT, as it may cause more problems than its worth, and I cannot interpret the crash dumps anyway.

          To test locally, all you’d have to do is yum install -y abrt-addon-ccpp, and service abrt-ccpp start. I’m also running OSSEC.

          • As another follow up, I’ve found that when you change users, it always logs the previous command run by said user, which if you left the session properly last time, is likely exit. essentially, if you look at your command history, whatever the last command executed was.

            • Heh! I knew about this issue but I assumed this false positive was not much to pay to have the ability to work with real root shells like “sudo -s” ;)
              I’m interested if you find a way not to log the last-shell-last-command though.

    Share your thoughts

    *