Tuesday, July 20, 2010

Innovation -- You Keep Using That Word...

So, this week, the OISF has been on a media blitz about Suricata, their open-source Intrusion Detection System.  As always, my preference is for you to review the information yourself, so before I give you my thoughts about the state of Suricata, here are some links:

http://www.computerworld.com/s/article/9179436/DHS_vendors_unveil_open_source_intrusion_detection_engine?taxonomyId=82
http://www.darkreading.com/smb-security/security/intrusion-prevention/showArticle.jhtml?articleID=225900192&cid=nl_DR_DAILY_2010-07-20_h
http://www.openinfosecfoundation.org/

When I talked to Matt Jonkman about Suricata this past December, I was excited.  He talks a good game and his pitch of Suricata as a "Next Generation Intrusion Detection and Prevention Engine" was catching.  It is always good to step outside the box and take a good hard look at how any industry is approaching things and innovation is always welcome.  The fact that Suricata is Open Source and driven by government money meant that the innovation would be available and easily ported to Snort, helping not just Suricata users, but Snort users as well.  To me, it looked win-win all the way.

I was so impressed by the tack that Suricata appeared to be taking that I gave my best wishes to them on their first release, back in December:
"Congrats to Matt Jonkman and the team at OISF.  It's a big step, and I look forward to seeing your work (after then new year :))

Matt". -- [Snort-users] Suricata IDS Available for Download!, 12/31/09
http://marc.info/?l=snort-users&m=126229251430261&w=2

But at this point, having worked with Suricata and looked at what the OISF has actually delivered, I'm just disappointed with where they've ended up and what they've delivered.

Suricata's developers harp on a lot of different issues, some of which are valid, and some are simply wrong.  More than anything else, they beat the multi-threaded drum:

".For example, Suricata's multi-threaded architecture can support high performance multi-core and multiprocesser systems, Jonkman said." -- (Computerworld, above)

 "This is 2010 and not a single IDS supports multi-threading"
"The flaw in every IDS is that it is single threaded"
"Multi threading alone is worth moving over to Suricata for".
-- (Suricata team, various)

We've talked about this before, with an extensive, technical discussion on multi-threading and IDS from Marty (http://vrt-sourcefire.blogspot.com/2010/06/single-threaded-data-processing.html).  Go there and check out what Marty has to say.  The essence of his post is that there are sound architecture reasons why traditional multi-threading is not the appropriate approach for IDS implementations.  Certainly, as commodity hardware evolves (we do work very closely with Intel), we'll continue to evaluate the technology.  In addition to the long technical brief, I'd point out this simple fact of life:  IDS vendors are judged on both detection accuracy and speed.  No report about an IDS engine comes out that doesn't identify how many GB/s throughput the engine was capable of sustaining.  Trust me, if multi-threading were the answer, the industry would have moved there in short order.

Of course, there is no third-party review of Suricata's performance.  I'm going to give you some numbers, but I don't expect you to put any special weight on them, they are more to show that we have looked at the engine and haven't found anything that we would take from it.  I asked Chris McBee, one of the researchers here on the VRT to install Snort and Suricata on the same box.  I told him to make it run, take all the steps necessary to maximize performance.  He even changed out the network card to support the pf_ring buffer.  We did not, however install a CUDA-capable network video card (edited after publish, my bad -- molney), since, in the words of one Suricata developer (it should be noted that the 1.0.0 release notes indicated an increase in performance):

"Currently on my desktop CUDA actually slows things down"
--http://www.inliniac.net/blog/2010/02/20/suricata-has-experimental-cuda-support.html

So the key to the test is that they were on the same, commodity hardware.  We ran Snort first, then Suricata on the same box.  In each test, Snort and Suricata were loaded with the latest default open source ruleset from the VRT.  In the case of Suricata, some rules that used unsupported options failed to load, and there is no .SO rule support, so none of those rules ran either.  This means that Snort was running a larger ruleset than Suricata.

Test Set 1:  The same packet set sent at between 200 and 5200 MB/s (30 runs total)
       Total Packets sent:  30,000,000 (Across all 30 tests)
       Total Packets dropped, Snort:  61264  (0.0204%)
       Total Packets dropped, Suricata:  17,438,542 (58%)

Test Set 2: The same packet set sent at between 200 and 5200 MB/s (30 runs total) (Hyperthreading Disabled)
       Total Packets sent:  30,000,000 (Across all 30 tests)
       Total Packets dropped, Snort: 511 (0.0017%)
       Total Packets dropped, Suricata: 15,714,211 (52%)

Now, we did rerun the tests when Suricata 1.0 was released, briefly:

"Suricata peaked at about 300 Mb/s without dropping packets, provided no rules are loaded.
With rules loaded, Suricata runs up to about 200Mb/s.
Snort, with rules, hits 894Mb/s with no drops" -- Internal VRT Report on Suricata Performance

Suricata's performance isn't just bad; it's hideously, unforgivably bad.  This is especially true for a project that is hawking the performance issue.  There are other issues I have with Suricata, but I think the one that is most concerning is the following statement:

"We made a conscious decision last year that we're not going to go with an obfuscated rule set, we're not going to pick up an SO rule format" -- Matt Jonkman

This decision is going to limit any detection capability to the rule language provided by Suricata.  In today's incredibly complex environment, this is an unacceptable limitation.  Sourcefire developed the C capability to give the VRT and high-end response organizations the ability to build detection with all of the power of C available to them and not lock them into our the Snort rules language.  The loss of this capability dramatically reduces the usefulness of Suricata.  As an administrative issue, for organizations working in a TS/SCI environment, rules often need to be provided in an obfuscated format to protect against unnecessary disclosure of data.  This isn't available without .so rules.

All of this came up in a discussion of how to engage vendors to get data to write rules.  It should be noted that Sourcefire is a partner with several large vendors, including the most targeted:  Microsoft.  In agreement with these vendors, we protect our rule set for vulnerabilities that are not being exploited in the wild to help protect against exploit development.  We do this by obfuscating the rules associated with data we receive from them into .so rules.  There is no impact to performance by doing this, but it does pose a hurdle to potential attackers looking to glean information about potential vulnerabilities from how we go about detecting them.

Matt Jonkman specifically said: "We're hoping to get enough market share to say to vendors if they want to use our product if they want to feel safe, we're going to have to be transparent enough that rules can be written".  This is simply not going to happen, and trying to adjust reality to justify your philosophical approach will simply result in a product that is not usable.

Look, I generally work very hard to avoid looking like a corporate shill, I try to keep to purely technical information.  But in this case, I have to call a spade a spade, even if it leaves you thinking I'm just a mouth piece.  But know that I honestly believe this:

The OISF has spent nearly a million dollars to fulfill their obligation to the DHS to deliver the next generation in IDS engines.  They have since engaged in all manner of wishful thinking, self-aggrandizement and Snort bashing.  They've failed, utterly, to deliver on their promises.  This is forgivable on the performance front, that problem is non-trivial.  But in the end, what they've built is a poorly functioning Snort-clone, missing the most powerful detection capability that Snort has.  There isn't anything in the way of innovation; they are taking the same approach as everyone else from a detection standpoint.  Simply put, rehashing isn't innovation.

If you want to see what innovation looks like, come to Vegas and let the VRT show you the Razorback system.  It isn't Snort, it isn't ClamAV, and it isn't Suricata.  It's a new approach to the detection problem, and was built from the ground up in close collaboration with groups that are facing APT-level threats.  It may not be perfect, it may not even be the right answer (but we think it is), but it is truly innovative.  

And we didn't even cost you a million dollars.
Add to Technorati Favorites Digg! This

8 comments:

Annie said...

Can you share more information about the test set?

ruleset version and test packet (tcp/80? 64bytes?)

Kint said...

Ohhhh, Razorback? What is that? Are you allowed to talk about it before Vegas? Some of us can't attend :/

Nigel Houghton said...

Kint, you can find some information on Razorback here: http://labs.snort.org/razorback/

You should note however, that the material there is old now, the DefCon release will be much larger in scale and has many more features and a lot more code.

We will update that page at the time of the DefCon talk and give links to documentation, downloads, mailing lists, bug tracking etc...

oracle_12 said...

Suricata does run the same rules as Snort, I'm not sure what you're talking about. As for speed, unless you release the exact details of your tests no one is going to believe your results. I've done testing that shows a 6 times speed improvement when using Suricata over Snort.

Matt Olney said...

Annie:
I've talked to Chris about putting together a little entry about how we did our testing, I think its a valid request. I already know we'll have trouble providing the pcaps, since we used in-house traffic, but we will be able to provide statistics on the traffic (protocol distribution, etc...)

oracle_12:
what I said was " In the case of Suricata, some rules that used unsupported options failed to load"

Here is a list of issues that came up in our testing (thanks McBee):

unknown rule keyword: http_state_code
unknown rule keyword file_data
unknown regex modifier 'H'


oracle_12, what I said was "In the case of Suricata, some rules that used unsupported options failed to load". What that means is that there are rule options that Suriata does not support, and this will continue to be the case as we go forward.

Here is a list of issues that came up in our testing (thanks McBee):

unknown rule keyword: http_state_code
unknown rule keyword file_data
unknown regex modifier 'H'

fast_pattern shouldn't be supplied with a value (fast_pattern:only) http_uri cannot be used with "fast_pattern" currently.Unsetting fast_pattern on this modifier.

http_header cannot be used with "fast_pattern" currently.Unsetting fast_pattern on this modifier.

http_method cannot be used with "fast_pattern" currently.Unsetting fast_pattern on this modifier.

As to the performance numbers, like I said: "I don't expect you to put any special weight on them". They will almost certainly get better, but the core issues such as the overall lack of "new" and their decision to not support .so rules are much more concerning.

lt said...

The methodology and some basic hardware info (even just the number of cores) would indeed be interesting. We do often tend to hear sourcefire people getting X gigabits out of snort by running multiple instances on multiple traffic queues pinned to certain cores. The trouble is though the average user has no idea how to do this. The performance a user gets when they run a single instance of snort with default config and compare a single instance of suricata in a default config is rather different to the figures you state here.

.so rules are an interesting one - obviously there are cases where you're trying to do something you can't do with a snort rule. In those cases .so rules may be useful - though suricatas philosophy seems to be to extent the rule language to cover such cases.

The majority of .so rules however appear to be .so simply because they contains some super-secret data that someone doesn't want us to see. These binary-only rules are really quite annoying - the whole reason many of us started using snort in the first place was because we could see the rule. If there was a false positive we could look and understand why - it was one of the major selling points of snort.

Then there's the issue of supported platforms - if you want to use the binary only .so rules you're suddenly limited to a small subset of platforms. Many sensors simply cannot use the binary .so rules at all - snort used to be embedded in several linux firewall distros for example - what use is a binary rule to such users?

Martin said...

Marty, (and therefore Sourcefire), came to fame by being the street-fighting, open-sourcing underdog in a market of stodgy Goliaths. Now Sourcefire puts out closed-source code (yes, .SO rules make you closed-source) and has had a very un-underdog IPO. So, don't be surprised when the crowd roots for the new underdog. Take it like a man, because after all, now you are the man.

cw said...

Seems the press blitz was overhyped, but I don't see how there can be much realistic comparison between the much more mature snort and the infant Suricata. But you've got to hand it to Matt Jonkman to helping the ET project for so many years push out sigs that didn't match the VRT's focus areas. Yes, some of those sigs sucked but some were and continue to be very useful. Regarding the .SO rules, I've wondered (in theory) how one might deobfuscate some parts of their functionality with dynamic debugging. You'd have to have enough of a trigger, which could already be enough to extrapolate the issue, or rely on incoming targeted or in-the-wild attacks to do it for you. And with the various leetness on the VRT you probably already countered this idea. BTW Kudos on the Razorback work, looks like the future. @curtw