Ossec, jboss, log4j and syslog are on a plane…

As long as java is concerned you will always face the trouble of processing stacktraces. If you use Log4j and PatternLayouts, you’ll soon find that your pattern is only applied to the first line of the stacktrace.

It gets worse if you use ossec[1], a powerful log analyzer with real-time alerting. Ossec ships with default rules, especially rule 1002 “Unknown problem(s) in the system” that matches generic error-related words such as “critical|failure|segfault|error” against your logs.

Say you’ve written a Java decoder for Ossec matching your PatternLayout. Then, you’d probably like to override the 1002 rule (to change subject, ignore stacktraces….) when logs are decoded as Java but you can’t create such a rule because stacktraces aren’t decoded as Java logs, remember: the PatternLayout hasn’t been applied ! Finally, you’ll be sent tons of “Unknown problem(s) in the system” alerts when one of your Java application prints a stacktrace.

As far as I know, in order to have the same layout for all Java log lines and stacktraces, you are left with only one solution: use a Syslog log4j appender, and use rsyslog templates capabilities to prefix each log line with something that resemble Syslog default formating and can be handled by Ossec. In short: extract the “header” from your log4j Layout and move it in the rsyslog template.

  • Log4 configuration

Only “raw” message.

<appender name="SYSLOG">
    <param name="Facility" value="LOCAL2" />
    <param name="FacilityPrinting" value="true" />
    <param name="SyslogHost" value="127.0.0.1" />
    <param name="MaxLineSize" value="4096" />
    <param name="Threshold" value="INFO" />
    <layout class="org.apache.log4j.PatternLayout">
        <param name="ConversionPattern"
             value="%-5p [%X{session}] [%c{1} %X{uri}] - %m%n" />
    </layout>
</appender>
  • Rsyslog configuration

Prepend a Syslog-like pattern to each line.

// Generate: 12 Apr 13:37:42 host.example.com jboss: applicative crap
$template jbossFormat,"%timegenerated% host.example.com jboss: %msg%\n"
local2.*    /var/log/app/server.log;jbossFormat
  • Ossec decoder

Now you can accurately decode 100% of the lines, even stacktrace, they all are prefixed.

<decoder name="jboss">
    <program_name>jboss</program_name>
</decoder>
  • Ossec rule

And you can choose to ignore stacktrace and false positive.

<rule id="100002" level="0">
   <if_sid>1002</if_sid>
   <decoded_as>jboss</decoded_as>
   <description>Ignoring unknown problems for Jboss logs</description>
</rule>

Of course, any better solution or suggestion is much appreciated.

[1] — http://www.ossec.net/

No comments yet, d'oh!

Share your thoughts

*