Posts

Firegen Custom Log Patterns Explained

When you select one of your logs, if we did not ship a pre-configured log pattern for it, you get a “None of the known patterns matched any of the log entries.” message:

Test started on 8:35:50 AM
Lines analyzed: 1001
None of the known patterns matched any of the log entries.
Last analyzed line: <166>Jul 07 2015 00:00:33 : %ASA-6-302016: Teardown UDP connection 7980317 for dmz:10.2.174.218/1041 to inside:10.1.174.152/53 duration 0:00:00 bytes 183

Please send a small log sample (or just the line listed above) to support@firegen.com and the proper log pattern will be sent to you. If the log that you are trying to analyze is not the list of firewall logs supported by Firegen, again send a full log to mailto:support@firegen.com and we will add support for it.

Check http://www.firegen.com for updates!

The analyzer will display the last log entry that it tried to identify, in this case:

<166>Jul 07 2015 00:00:33 : %ASA-6-302016: Teardown UDP connection 7980317 for dmz:10.2.174.218/1041 to inside:10.1.174.152/53 duration 0:00:00 bytes 183

The log entry pattern syntax represents a regular expression that has to extract 7 details from the log entry:
– Firewall identification
– Year
– Month
– Day
– Time
– Log entry code
– Log entry detailed message

Some syslog servers are not recording all of these (most of the are able to but simply are not configured to do it). In the example above, the syslog server is not recording the IP address or the host name of the device (the firewall) sending the syslog messages. In this case, we will choose the <166> as the identifier (since there is nothing else).

So, now we have to create the regular expression that would extract the following:

1. Firewall: 166
2. Year: 2005
3. Month: Jul
4. Day: 07
5. Time: 00:00:33
6. Code: 6-302016 (for Cisco Pix/ASA/FWSM we discard the %ASA/%Pix/%FWSM code prefix)
7. Message: Teardown UDP connection 7980317 for dmz:10.2.174.218/1041 to inside:10.1.174.152/53 duration 0:00:00 bytes 183

A full tutorial in using regular expressions is beyond the scope of this article but there are many resources available on the Internet (as well as standalone tools). However, here is, shortly, how we wrote the regular expression for this log entry:

<(.*?)>(\D{3}) (\d\d) (\d{4}) (\d\d:\d\d:\d\d).*?(\d-\d{6}): (.+)

1. <(.*?)> – will match any set of characters between the < > tags – this will get us the “166” firewall id 2. (\D{3}) – will match any set of 3 non-digit characters (\D) – this will extract the month: Jul
3. (\d\d) – will match the next set of 2 digits (\d) – the day: 07
4. (\d{4}) – will extract a set of 4 consecutive digits – the year: 2005
5. (\d\d:\d\d:\d\d) – will match 3 pairs of digits separated by a colon – in our case the time: 00:00:33
6. .*? – will match some information that we do not need and that is everything between the time and the next piece of useful information (the code). Since is not in parenthesis, it will not be “extracted” in a match group
7. (\d-\d{6}): – will extract any set of characters that match a digit followed by a dash and then a set of 6 digits, in our case this would be the Cisco ASA entry code: 6-302016. Note that the : is outside the parenthesis so it will not be included in the “capture”
8. (.+) – will get us all the characters – a dot matches any character and the + indicates that everything like the one before should be included in the match. In our case is the rest of the message: Teardown UDP connection 7980317 for dmz:10.2.174.218/1041 to inside:10.1.174.152/53 duration 0:00:00 bytes 183

Once we enter this log pattern, save and click “Test pattern” we will get the following:

Test started on 8:56:04 AM
Lines analyzed: 1
Matching groups:
Matching group 1: 166
Matching group 2: Jul
Matching group 3: 06
Matching group 4: 2015
Matching group 5: 23:58:46
Matching group 6: 6-302016
Matching group 7: Teardown UDP connection 7979511 for outside:10.3.174.12/138 to inside:10.1.174.173/138 duration 0:02:03 bytes 176 (uvcs\profs)

Please verify if the following detected information is correct:
Firewall: 166
Year: Jul
Month: 06
Day: 2015
Hour: 23:58:46
Code: 6-302016
Message: Teardown UDP connection 7979511 for outside:10.3.174.12/138 to inside:10.1.174.173/138 duration 0:02:03 bytes 176 (uvcs\profs)

Last analyzed line: <166>Jul 06 2015 23:58:46 : %ASA-6-302016: Teardown UDP connection 7979511 for outside:10.3.174.12/138 to inside:10.1.174.173/138 duration 0:02:03 bytes 176 (uvcs\profs)

The first part of this message show the “matching groups”, that is the set of data that we extracted (by using parenthesis) from the log entry. As you can see it does extract the information that we need. However, Firegen needs to know which one is which, which group is the firewall id, which one the day, the year, etc… This is where the Field mapping setting gets to be used. The field mapping consists in the number of 1 to 7, comma separated (no spaces) that has to indicate which of the matching groups corresponds to the required fields, in this order: firewall id, year, month, day, time, code and message.

The first digit in the Field mapping should indicate which mapping group is the firewall id, the next one the year, then the month, day, time, code and message. Since the default Field mapping is 1,2,3,4,5,6,7, Firegen will consider that the first set of information captured is the firewall, the second one the year, third one the month, forth one the day and so on, and this will be displayed under the “Please verify if the following…” Looking back at the first section of the message, we need to create the Field mapping syntax to match the order required by Firegen: firewall,year,month,day,time,code,message:

Firewall: Matching group 1: 166
Year: Matching group 4: 2005
Month: Matching group 2: Jul
Day: Matching group 3: 06
Time: Matching group 5: 23:58:46
Code: Matching group 6: 6-302016
Message: Matching group 7:
Teardown UDP connection 7979511 for outside:10.3.174.12/138 to inside:10.1.174.173/138 duration 0:02:03 bytes 176 (uvcs\profs)

Considering the list above, the Field mapping syntax should be: 1,4,2,3,5,6,7. If we enter this in the “Field mapping” setting, click Save and then Test pattern we should now get:

Test started on 9:04:38 AM
Lines analyzed: 1

Matching groups:
Matching group 1: 166
Matching group 2: Jul
Matching group 3: 06
Matching group 4: 2015
Matching group 5: 23:58:46
Matching group 6: 6-302016
Matching group 7: Teardown UDP connection 7979511 for outside:10.3.174.12/138 to inside:10.1.174.173/138 duration 0:02:03 bytes 176 (uvcs\profs)

Please verify if the following detected information is correct:

Firewall: 166
Year: 2015
Month: Jul
Day: 06
Hour: 23:58:46
Code: 6-302016
Message: Teardown UDP connection 7979511 for outside:10.3.174.12/138 to inside:10.1.174.173/138 duration 0:02:03 bytes 176 (uvcs\profs) Last analyzed line: <166>Jul 06 2005 23:58:46 : %PIX-6-302016: Teardown UDP connection 7979511 for outside:10.3.174.12/138 to inside:10.1.174.173/138 duration 0:02:03 bytes 176 (uvcs\profs)

Now the information seems to match the fields required by Firegen so we are ready to analyze the log.

Note: Log entries that do not contain the year should be treated in a similar fashion, however the year will always be set to matching group 7 (so the second entry in the Field mapping should be 7 and the regular expression should only extract the remaining 6 groups).