Registered: Jul 2001
Location: Norfolk, VA
Posts: 4678

So, I've got a tasker at work where I'm looking over Raptor (Symantec) firewall logs. These are content-filtering firewalls with several (20 or so) daemons running, like httpd which (oddly enough) examines http/https traffic.

Typically I work with router logs, like Cisco netflow logs, which are delimited. That is, the third column is always going to be (e.g.) the source IP, in the entire file, so it's really easy to mine with fp trees or apriori.

These raptor logs, on the other hand...
So far as I can tell, they were never meant to be analyzed. Instead of delimited text you have an entry like this:
httpd src=192.168.1.1/4675 dst=192.168.2.1/80 sent=192 rcvd=3000

This is sort of typical of what you might see for a web session for an intranet web server.

But if the traffic transit the firewall you also see information about the source & destination interfaces:

httpd src=192.168.1.1/4675 dst=195.224.113.251/80 srcif=192.168.100.1 dstif=34.80.92.1 sent=176 rcvd=4650

I'm trying to parse these out using grep, sed, and awk to get a plain delimited file.

There is a utility called 'flatten' which can be used to do this, but Symantec won't send me one because WE don't have a license. I tried using sawmill and the result was just utter trash--it only pays attention to a few of the daemons, like httpd, ignoring or mangling dns and icmp traffic in the process.

So, at first I was just figuring, well, all the httpd traffic must be similar. Then I noticed that there were extra fields depending on whether or not the traffic transited the firewall, so I grepped those out separately and used awk to add the missing fields to the others.

Then I noticed that the order of the fields is not constant. Sometimes "srcif" will precede "dstif" and sometimes it won't. Sometimes the firewall records sent and received bytes and sometimes it doesn't. Sometimes the firewall rule that was triggered is noted--and sometimes not. When it is, sometimes it's listed before, say, the protocol, and sometimes after.

So now what I need to do is figure out some way to find, on a line, the field beginning with "src=" and place that as field $2 in awk, then find the one matching "dst=" and place that as $3, and so on and so forth. Is this possible using grep/sed/awk or am I barking up the wrong tree?

Report this post to a moderator | IP: Logged

Registered: Nov 2000
Location: UK
Posts: 4736

As I said in chat(but I'm not sure if you saw it), this isn't actually all that easy to implement as a oneliner. This is probably best off as a perl script.

You did miss one example though, and that is a line containing for a protocol and a firewall ruleset column.

Report this post to a moderator | IP: Logged

Registered: Jul 2001
Location: Norfolk, VA
Posts: 4678

Well, I didn't get what I initially wanted, but

grep -i 'httpd' input.txt | awk 'BEGIN {FS=" arg="};{print $2}' | awk '{print $1}' > output.txt

gave me a list of all urls requested, which is what I needed today.

Report this post to a moderator | IP: Logged

Registered: Nov 2000
Location: UK
Posts: 4736

code:

#!/usr/bin/perl
#
# Simple script to attempt to remunge raptor firewall logs
#
# The only sample I have of the log format:
# httpd src=192.168.1.1/4675 dst=192.168.2.1/80 sent=192 rcvd=3000
# httpd src=192.168.1.1/4675 dst=195.224.113.251/80 srcif=192.168.100.1 dstif=34.80.92.1 sent=176 rcvd=4650
#
# There are apparently other variations, but hopefully the script will be simple
# enough to alter to compensate for them

use strict; # I like pedantry

while(<STDIN> ) {
        my($protocol,$src,$dst,$srcif,$dstif,$sent,$rcvd);

        chomp;
        my @linesplit = split(/ /, $_);
        $protocol = $linesplit[0]; # assuming the first value is always the protocol
        foreach (@linesplit) {
                $src = $_ if s/src=//;
                $dst = $_ if s/dst=//;
                $srcif = $_ if s/srcif=//;
                $dstif = $_ if s/dstif=//;
                $sent = $_ if s/sent=//;
                $rcvd = $_ if s/rcvd=//;
        }
        # Print it
        print "$protocol,$src,$dst,$srcif,$dstif,$sent,$rcvd\n";
}

Last edited by macker on 12-17-2005 at 04:03 PM

Report this post to a moderator | IP: Logged

Registered: Jul 2001
Location: Norfolk, VA
Posts: 4678

Thanks.

You'll have to PM me your name if you want credit when this is used. Of course, that means it'll be circulated among certain places you might not want. Let me know.

Report this post to a moderator | IP: Logged

Registered: Feb 2004
Location: in colorado somewhere!
Posts: 25592

You lost me with "So."

__________________

[Ignore Sig]

EEEEEEEEEEEEE!!!!!

Report this post to a moderator | IP: Logged

Registered: Nov 2000
Location: UK
Posts: 4736

Don't worry about credit and circulate as you wish.

Report this post to a moderator | IP: Logged

Registered: Dec 2002
Location:
Posts: 3063

another possibility:

quote:

---

#!/usr/bin/perl -w
use strict;

my $inputString = shift;
my @fields = split /,/, $inputString;
print "Printing fields: " . join(', ', @fields). "\n";

while(<STDIN> )
{
chomp;
my %data = split /[ =]/, "protocol=$_";
my @output;
foreach my $field (@fields)
{
push(@output, $data{$field}) if(defined $data{$field});
}
next unless(@output);
print join(',', @output) . "\n";
}

---

Report this post to a moderator | IP: Logged

Registered: Nov 2000
Location: UK
Posts: 4736

quote:

---

Originally posted by zim
my $inputString = shift;

---

Java slut

Report this post to a moderator | IP: Logged

Registered: Aug 2001
Location: Hilbert Space
Posts: 35561

perlvert.

Report this post to a moderator | IP: Logged

Registered: Dec 2002
Location:
Posts: 3063

actually, i don't know a bit of java, was just trying to make it easier to digest.

Report this post to a moderator | IP: Logged

Registered: Dec 2002
Location:
Posts: 3063

code:

#!/usr/bin/perl
# New RFC 3092 Compliant Code!
my $foo = shift;
my @bar = split /,/, $foo;
print join(',', @bar). "\n";

while(chomp ( my $foobar = <> ))
  {
    my (%baz, @quux);
    %baz = split /[ =]/, 'protocol=' . $foobar;
    foreach my $qux (@bar)
      {
        push @quux, $baz{$qux} if defined $baz{$qux};
        push @quux, q{} unless defined $baz{$qux};
      }
    print join(',', @quux) . "\n";
  }

Last edited by zim on 12-21-2005 at 05:38 AM

Report this post to a moderator | IP: Logged

Registered: Nov 2000
Location: UK
Posts: 4736

It's better, but your bracket spacing scheme is inconsistant now . You do get points for using a hash though(as hashes are great).

Report this post to a moderator | IP: Logged

Registered: Aug 2001
Location: Hilbert Space
Posts: 35561

And it's on more than one line.

Report this post to a moderator | IP: Logged

Registered: Feb 2004
Location: in colorado somewhere!
Posts: 25592

I feel this is fitting here.

Attachment: nerds.jpg
This has been downloaded 43 time(s).

__________________

[Ignore Sig]

EEEEEEEEEEEEE!!!!!

Report this post to a moderator | IP: Logged

Registered: Dec 2002
Location:
Posts: 3063

quote:

---

Originally posted by macker
It's better, but your bracket spacing scheme is inconsistant now . You do get points for using a hash though(as hashes are great).

---

Report this post to a moderator | IP: Logged

Registered: Jul 2001
Location: Norfolk, VA
Posts: 4678

I've been playing around with macker's script. It is pretty easy to modify. I think I will have it processing the bulk logs by this evening.

One thing however--

As it turns out, the first few fields in every entry are the same:

quote:

---

Month Day Time Firewall_Name Daemon Entry_Type

---

Report this post to a moderator | IP: Logged

Registered: Nov 2000
Location: UK
Posts: 4736

$cruft = join(' ', @linesplit[0..5]);

Report this post to a moderator | IP: Logged

Registered: Nov 2000
Location: UK
Posts: 4736

quote:

---

Originally posted by zim
how is it inconsistant?

---

Report this post to a moderator | IP: Logged

Registered: Jul 2001
Location: Norfolk, VA
Posts: 4678

Danke!

Report this post to a moderator | IP: Logged