Wednesday, December 30, 2009

Matt's Guide to Vendor Response

Well...it's that weird period between Christmas and New Years, and I've realized that I hadn't gotten anything for those wonderful people that keep the VRT employed. So as a gift to you, software vendor, I present Matt's Guide to Vendor Response. Now...this is a complicated subject. It has to be, because every company I've dealt with has had to work through it painfully. Hopefully, this guide should help whoever follows in the footsteps of vendors past to more easily move from "security through no one is looking at me" to a company who operates in balance with business drivers and obligations to customers.

First, let’s cover some concepts:

VALUE OF VULNERABILITY -- The most important thing to understand about vulnerabilities is that they have value. Depending on several factors (the same ones your end users use to gauge risk: ease of exploit, how widely deployed the software is, remote/local etc...) it can be worth a great deal. When someone comes to you with vulnerability data, you should be grateful. There are a number of commercial entities that will purchase vulnerability data and most of them will then pass it on to you. But there are a number of other avenues that a security researcher could take. Various government agencies in the U.S. Government will purchase vulnerability data, especially if it’s been researched fully to the point of exploit. Other organizations around the world, both governmental and non-governmental, will also pay money for vulnerabilities.

Know that when I say pay money, I mean good money. (Matt Watchinski in a presentation to DojoSec about the Adobe JBIG2 issue estimated the value of the bug from between 75 to 100 thousand dollars). Here's a pro-tip: Head to Vegas and hit some of the late night DEFCON parties. Search through the crowd and find the group wearing the custom suits, nice watches and Italian shoes. Congrats, you've found some of the real deal. (By the way, don't discount the dude in the green shirt with green spiky hair...he's just keeping it on the down-low). Here is my point, when you get vulnerability data, someone has given up what could be a substantial sum of money to provide you with that data. Treat them well. Be respectful, when you announce the patch, put their name in it. They've done you and your customers a great service, and it costs you nothing more than having someone on your staff act like a half-way decent human being.

IMPACT OF 0-DAY -- I really believe that a lot of the software development community simply haven't internalized what an 0-day means in terms of impact to customers. When there is an in-the-wild exploit of your software, you should turn all of your resources to alleviating that problem. Most operations folks flail when 0-day comes out, no one will give them enough information to mitigate the issue, they are at the mercy of the vendor's patch cycle and at the core of it all is a gaping hole into their data. Depending on your application, this could be state secrets, medical records, credit card numbers etc... So when you sit in your meeting to figure out how to handle the mess that the failure in your secure software development lifecycle (you do have one of those, oh...please don't make Michael Howard cry...) has caused, keep in mind the real world impact (big or small...) that the problem is causing.

Now, I'm going to keep this simple. There are a ton of policies and spread sheets and threat matrices and God knows what else that some CISSP somewhere will help you build. But here is how, at the absolute root of the issue, to handle security issues in your products:

CASE 1: You've been confidentially notified by a security researcher that there is vulnerability in your product.

RESPONSE:

First, thank the researcher and let him have a point of contact on your security team to discuss the issue. As soon as your triage of the bug is done, notify the researcher of the timetable to patch. Feel free to put in whatever caveats you need to, but keep that communications channel open. You never know when it will be useful to you.

Second. Don't tell anyone who doesn't need to know anything about the vulnerability. Configure your bug handling software to have a security flag that allows viewing only by a dedicated security response development team. If you have an external bug tracking system, remove immediately any information about the bug. Many of you might wonder why this is...especially after I've repeatedly demanded that as much information as possible be given to operations folks. But here is the deal...as soon as any information about an unpatched vulnerability is out, attackers will start the process of converting that information into a working exploit.

For example, when the Adobe JBIG vulnerability came out, Sourcefire was only aware that there was an issue in PDF files that had JBIG tags. So Lurene and Matt set the fuzzer on it, starting at the JBIG tag. It took less than 10 minutes to find the vulnerability, and a short time later we understood the execution path. We're good...but so are the attackers. So until such time that there is information in public, lock down any info you have.

Third. Triage appropriately and patch as soon as is reasonable. I'm not suggesting that you bend over backwards to get a patch out tomorrow for undisclosed bugs. Keep an eye on your security feeds, notify your security vendor partners to be on the lookout for particular strings or conditions and get to work on patching within your normal development framework. Just ensure that you have a process in place to expedite the patch process if the vulnerability becomes actively exploited or the vulnerability details are released.

A quick note on "security vendor partners". If you, as a company, are truly concerned about the operational impact of vulnerabilities in your software on your customers, you should create relationship with security vendors NOW so that they can help protect your customers when vulnerabilities arise. Contact IPS/HIDS/SIM/AV vendors so that when you are under the gun, you have both resources you can reach out to and the ability to tell your customers that while you are working on the patch, their security vendors can provide coverage to protect them. This goes a long way towards making security operations people feel better about your product. Microsoft's MAPP program (http://www.microsoft.com/security/msrc/collaboration/mapp.aspx) that Sourcefire participates in is an excellent example of this kind of program.

Finally, deliver the patch. When delivering the patch, remember to credit the researcher and, if appropriate, his or her company. Also, understand that once you deliver the patch, sufficient data is now in the public arena to create an exploit. So provide as much data as you can so operations folks can make an intelligent decision on how quickly to roll out the patch, and what mitigations are available prior to that patch roll out. Remember that, for better or worse, if there is no active exploit, many organizations require extensive testing prior to a patch roll-out, so give the ops team options to handle things prior to patching.

CASE 2: Information has been publically disclosed regarding a vulnerability, but you have seen no active attacks.

RESPONSE:

This is the tricky one, and is one of the reasons you need to have a dedicated team with the appropriate skill set to handle security issues. First, you need to interpret the level of detail that has been provided to the public. If there is a Metasploit module, just skip straight down to active attacks, because it's on like Donkey Kong (TM, or something). If it is an announcement to bugtraq or full-disclosure, have someone who understands exploitation look at the data and help you determine the risk. If you are completely clueless on exploit development (and really, you have to be willing to admit this) reach out to your security vendor partners for assistance.

Consider reaching out to the original poster of the data. Yes, this can hurt your pride, but remember that your company has caused (minor|moderate|severe) issues and you should feel some obligation to suck it up and fix it. Remember at this point few people are really going to beat you up over the fact that your code has bugs (whose doesn't?) but it’s how you respond that will define how you are perceived.

Now you've got enough data to make some decisions. Based on that data, and your policies, pick from the following:

1) If you feel that the level of disclosure is such that it creates a near-immediate threat to your customers, you need to skip immediately to the active exploit case.

2) Announce the fact that you are aware of the vulnerability and that a patch is in the works. Give a set of mitigation options for people to use until a patch comes out. Provide no unnecessary data. Engage your security vendor partners.

3) Announce the fact that you are aware of the vulnerability and that a patch is in the works. Provide all the technical details you can including locations of exploit code, how error logs look when the exploit is attempted, mitigation strategies and suggested detection techniques. Engage your security vendor partners.

Number three will probably look pretty extreme to some folks, particularly of the severe white-hat persuasion. But let me tell you a secret: There are no secrets. If the vulnerability is out there, you and every ops group on the planet are in a race against the hackers. In all likelihood, in some areas, you've lost the race already. But either way the only fair way to treat customers when the threat is imminent is to provide them every tool they can to get in place to weather the storm while you get your patching in order.

I'm going to bang on Adobe pretty hard here in a minute, so let me point out something they did that was actually close to right. There is exploit code available for Adobe Illustrator CS3 and CS4 for an .eps file parsing overflow. It has not, to the best of my knowledge, been seen in the wild, but it would be an excellent targeted attack vector. Adobe announced that it was aware of the issue, gave a date for the patch and advised that customers not open .eps files from untrusted or unknown providers.

Now, while I would love a better set of mitigations, I'm not sure what you could do in Illustrator. But I have enough information at this point to write mail filters, IDS rules and log parsers looking for .eps transfers. (You can find the notification here: http://www.adobe.com/support/security/advisories/apsa09-06.html)

All that being said, here is why I said "close to right". There is publicly available exploit code that includes the Metasploit windows/adduser and windows/shell_bind_tcp shellcodes. You can find that code here: http://www.exploit-db.com/exploits/10281. This is important because it allows you to write much more specific detection, allows you to test your detection with actual exploits and the mere fact that it exists tells you that there is a true threat to you if you use Illustrator and .eps files. Adobe did not provide a link to the exploit, which is very useful to organizations with advanced security practices. Adobe did not even disclose the existence of exploit code (which changes the response priority in most organizations).

CASE 3: You discover that a previously unknown vulerability is being exploited in the wild.

RESPONSE:

First, have a meeting and explain to everyone that your company has screwed up. Say it just about in those terms, get people in the mindset that you have an obligation to your customers. Make them understand that there are companies that will be adversely affected, perhaps dramatically, by the vulnerability in the software that your company makes.

Second, notify your dedicated security development team that they will now be earning their money. Overtime as necessary until it is resolved. Notify your QA team that they will have to put on hold whatever they are doing and prep for testing of a patch. Notify your web and infrastructure team that you will be having an out-of-band patch and they need to be ready whenever the patch is ready to publish.

Third, immediately notify your customers that there is an active exploit. Give every possible piece of data you have. Exploit code, sample files, log snippets, IP addresses of known attackers...everything. You owe them every piece of information you can possibly give them while your team slaves away on a patch. Now is not the time to be sly or hide behind your website. Get out there and earn the trust of your customers.

OK...now is Adobe smashing time. Let me tell you the absolute WRONG way to go about it. I'll use Adobe twice, because they've screwed up at least twice this year, and they are dead-center of the client-side bullseye right now.

First, with the JBIG2 problem, it was abundantly clear that they simply didn't have a plan in place for handling active exploitation. The vulnerability, which was very, very easy to take to exploitation was first announced publically sometime around the 19th of February by ShadowServer (http://www.shadowserver.org/wiki/pmwiki.php/Calendar/20090219). Also, it was noted that this vulnerability was actively exploited in the wild. It became apparent that this had been exploited since at least January 1st (we're thinking earlier...which is a bang against anti-virus vendors (us included) for not finding it earlier). We also find out that Adobe was notified by McAfee around January 11th of the problem. Long, sad story short, no patch was issued until March 23rd and then only for one version of Reader and no versions on the Unix platform.

A second, actively exploited 0-day was found in December. This time Adobe released a security advisory with well documented mitigation techniques (find the advisory at http://www.adobe.com/support/security/advisories/apsa09-07.html) including some accurately worded (reduced risk...) information about the role of DEP in the exploit.

Now, I missed this on the first pass through the bug, as I assumed the JavaScript Blacklist Framework technote was their standard blurb about the Blacklist. But the technote is actually where most of the data I would want to have from an ops side is located, including naming the function with the problem (DocMedia.newPlayer). With the information in the technote more accurate mitigation is available. I would have liked to of had that on the front page, but a diligent ops guy would find it. (Find the technote at http://kb2.adobe.com/cps/532/cpsid_53237.html)

The real issue, though, is that the patch would not be available until January 12th. For organizations where the JavaScript Blacklist Framework wasn't an option, such as those groups running non-Windows systems, or Windows users that don't have the ability to fix up all of their end-user registry settings or home users who most likely have no idea there is a threat at all, this time period is one where there is very little in the way of mitigation. Microsoft patches every month and still out-of-band patches when there is a significant 0-day threat. Why can't Adobe break into its quarterly patch cycle for the same reason?

ComputerWorld seemed to think this was a problem also, and had an article entitled "Adobe explains PDF patch delay" (http://www.computerworld.com/s/article/9142479/Adobe_explains_PDF_patch_delay). The first paragraph pretty much sums up Brad Arkin's (Adobe's director for product security and privacy) explanation of the delay:

"Adobe chose to wait until mid-January to patch a critical PDF bug because issuing an emergency update would have disrupted its quarterly security update schedule, the company said today." -- ComputerWeek's Gregg Keizer, December 18, 2009

Unacceptable. Seriously, there is no other way to describe this situation. You cannot justify pushing out an actively exploited 0-day vulnerability because "it would disrupt the quarterly update schedule". I invite you to read the article to get Brad Arkin's exact wording. Part of his justification was that they had deployed the JavaScript Blacklist Framework, which if you were at the correct version, would allow you to lock down JavaScript enterprise-wide.

What is so important in the quarterly patch that it can't be put off? The obvious answer is the Adobe Illustrator bug with public exploit code...maybe that should be rolled into the out-of-band patch, don't you think? If this weren't a blog that represents a group of people that are part of a company, I'd say this more plainly than this, but here is my final thought on Adobe over the past year:

WHAT THE HELL?

Look, Adobe has been doing better lately, but I think they still have a ways to go.

CONCLUSION

If you skipped to the conclusion (and who could blame you, it was almost too long, did not write...), let me lay it out for you:

1) Have a plan

2) If an exploit is underway, give customers every piece of information you have and patch as quickly as you possibly can.

3) If no one knows but you, keep your mouth shut while you patch.

4) Treat your security researchers well, they are doing you a huge service.

5) Know that you have an obligation to your customers to protect them from vulnerabilities in your software, give them every tool and piece of information possible to protect themselves while you code the patch.

Not surprisingly, the VRT is part of the Sourcefire vulnerability response process. We really do try and follow the ideas I've expressed here. Hell, we even credited Neil Mehta (an ISS employee, you know...one of our competitors) for the work he did on the Back Orifice buffer overflow. Which we patched, tested and released within 80 hours of learning of the problem.

We need to demand more from our software vendors (including Sourcefire, keep us honest). Use whatever leverage you have (and it is pitifully small in many cases) to get a better response from your software providers.

As always, I welcome your comments in the comments field below.

Add to Technorati Favorites Digg! This

10 comments:

Andre Gironda said...

Thanks for the post, Matt. There are a few things that I really liked about this blog post.

I think one of my favorite parts was when you said, "Configure your bug handling software to have a security flag that allows viewing only by a dedicated security response development team. If you have an external bug tracking system, remove immediately any information about the bug" (under case 1). You also explained how it was contradictory to your general advice and why. This is advice that I don't hear very often and it can be extremely valuable.

Some of your conclusions were excellent, but I consider them a bit biased. Sourcefire is an IDS/IPS vendor, and some of this advice appears to be catering to only these types of security vendors. For example, the way you mention "partners" and making ISVs do all of your work appears to be biased to me.

I would rather see this in some sort of "security policy" format. I tend to be biased on the secure software development and "build security in" side. I like to see application security policies, especially at major ISVs. Having a FTE "security bugfixer" is a priority over many of the items you mentioned.

I do think it's nearly impossible for many ISVs to just drop what they are doing when an unknown security vulnerability appears (hence my suggestion of prioritization based on the job title of "security bugfixer"). What I do agree with you is that ISVs tend to shove one particular mitigation to their customers -- when instead it should be multiple mitigation ideas followed by more and more specifics. I think the patch is more important than the temporary mitigations in every situation. Many interim and/or "virtual patches" are neither perfect in the short-term or long-term for both defense as well as performance/scalability. The developers should know how their product is going to best "design in" the new security requirements based on the vulnerability and/or working-exploit(s). Operators and marketing people tend to jump too quickly to their "obvious solutions" when really all they do is pollute/confuse the issue. I would definitely put IPS/HIDS/SIM/AV vendors in this category. The last thing the world needs is another permanent port 1434 block filter or equivalent (or a registry value to change on every computer that runs a Windows OS).

There are at least 700 different types of software weaknesses (CWE) that can and will be exploited in thousands of different ways (Metasploit payload variations) in millions of pieces of software (number of ISV products, websites, etc). Vendors are bound to fail in many ways similar to your case studies. Add in the human factor, as well as the recitivism rate of security bugs -- and you have a tall order to try and "solve" the issue of vendor response.

I really enjoyed to hear about your ideas and experiences. Thanks for the post!

Matt Olney said...

Hey Andre! Thanks for the comment. A couple of responses for you.

I can see where you'd call me biased for suggesting the security vendor partnership. But I did call out a number of different technologies and I do indeed mean ALL vendors. Not all of a specific ISV's customers run Sourcefire gear, some run ISS, some run Tipping Point, some run Cisco. Everyone deserves as much coverage as can be provided. I didn't ask for partnerships with just Sourcefire, I truly meant everyone.

I also giggled when you said "making ISVs do all of your work". The wide degree of disparity in the level of technical feedback we get from various vendors is enormous. I'd say the only one who is consistently outstanding with their guidance is Microsoft. A lot of companies just don't have the expertise or infrastructure in place to provide the data that supports our work. The main benefit we get there is a heads up before patch day. Either way, it does improve response time and quality for our customers, who are also the software vendor's customers.

I never suggested that ISVs drop everything for unknown (unless by this you mean 0-day) vulnerabilities. I specifically allowed for the much more common case of responsible disclosure, which allows time for routine triaging and patching. It is only in the very serious cases of active exploitation of an unpatched vulnerability that I feel justified in saying that ISVs do indeed need to drop everything to fix it. I can't imagine asking someone to work on anything besides fixing an actively exploited unpatched vulnerability. But also know that this should be a rare, rare event. The VAST majority of your time should be spent on routine security notifications, especially if you treat security researchers well.

As to your "virtual patch" comment, I've never been a big fan of that line. By engaging as vendors in a range of security technologies you can provide layers of protection. Might they be evadable? Depending on the issue, certainly. But in my experience the vast majority of attacks don't bother with evasion, they don't have to. While I work for an IDS vendor, I know that technology is a small part of large puzzle. It just happens to be the one I'm most intimately familiar with. I actively tell people to use every tool at their disposal. (We don't have a HIDS or SIM product, for example).

Anyways, I really appreciate your feedback. Hopefully others who read this will have similar questions, and you've given me an opportunity to answer them.

Monarch said...

Absolutely wonderful

Rocky said...

You can have the best VRT team in the world, with sub-microsecond patch releases after notification, but if I can't roll the patch out, it doesn't do me any good.

Vendors: Release your patches in a form we can push with common patch management tools.

We simply cannot push an "installer" that requires multiple user clicks, which then downloads something from your web site that is the actual patch. I'm looking at you, Adobe!

And a "patch" is not a 150 megabyte complete software replacement. I'm looking at you, Apple!

We're often ordered to patch something immediately or the world will come to an end. Except we need to roll out a "patch" that requires 10,000 monkeys on rollerskates to install, or requires more Internet bandwidth than Google to download to every single computer. That simply doesn't work.

curtw said...

Hi Matt. What I think it comes down to is economic - the perceived cost and risk of exploitation vs the cost of mitigations. In a widely decentralized environment such as a public .edu with a /16 and no central control, risks are going to be high due to the organizational dynamics, especially involving something like an Adobe 0day bug (for example various exploit kits with adobe exploits having been seen actively owning boxes at an .edu I know of). Inline IPS is a godsend in such a case, assuming that it can parse the file format in a timely manner without melting the CPU. I know of an org that requires Javascript to be enabled within Adobe Reader to process digital signatures. Guess how many of those people actually disabled Javascript? Anyhow, I'm drifting a bit but I appreciate your word to the vendors who have a great deal of influence in such situations. Thanks & happy new year.

Matt Olney said...

Rocky, Curtw:

Both of those are pretty accurate points. The patch management issue is only going to become more important as attackers increasingly target client-side issues. Although as Apple's share in corporate environments increases, I'd really like to see them create something like Microsoft's Server Update Services system.

I love the idea of a common patch manager, but I don't know what it would take to get enough vendors together to get critical mass. The only alternative I've seen is some of the remote-install tools enterprise management systems give you.

Curtw, it really is, in the end, about the dollar. The problem here is that then entity with the greatest ability to fix the problem doesn't have much of an immediate financial driver to fix it. Their customers face an economic toll, both in terms of actual attacks as well as the overhead of managing the issue between exploit release and patch. But they have little to no ability to fix the overarching issue.

The only way to adjust the behavior of the larger vendors is to call them on it. Microsoft went through some very serious pains in this issue. There are still people who have no confidence in Microsoft's management of software security issues. I would disagree, and actually feel that Microsoft is (now) one of the best examples of how to manage security issues. That being said, they only got there after years of missteps.

So if you have an issue with vendor response, call them on it. Every voice helps.

Matt Olney said...

In a similar vein..check this nonsense out:

http://jbrownsec.blogspot.com/2009/12/mozilla-code-sighs.html

Click through to the bug entry for some classic haha:

"However, this is a debugging tool used in specific situations, to read as input
output generated by other programs. So even if it crashes in response to some
other input, I don't see how that's a vulnerability. This tool is not designed
to handle arbitrary input, nor does it need to be."

Brad said...

Hi Matt,

A couple correction regarding your JBIG2 timeline. Adobe learned of the bug on January 16, 2009 and issued the first patch for version 9 on Mac/Win platforms on March 10, 2009. Although this is a tighter range than you mention in your post, we certainly weren't satisfied with this response time. Our investments since then in improving patch turn around time allowed us to get three zero day patches out to users in around two weeks in April, July, and October.

The ship schedule for the patch to the December bug was complicated by a variety of factors, not all of which were covered in the ComputerWorld article. My ASSET blog post provides some details here:

http://blogs.adobe.com/asset/2009/12/background_on_reader_update_sh.html

and a podcast I did with the Threatpost guys provides further insight into what went into the schedule decision:

http://threatpost.com/en_us/blogs/brad-arkin-adobe-reader-zero-day-flaws-and-security-response-121709

Our goal in this incident and every incident is to help protect as many users as possible as quickly as possible against the threats that we are aware of. Happy to talk more if you are interested. I'm @bradarkin on twitter or you can get my mobile number from Matt W. if you'd like to talk on the phone.

Thanks,

Brad Arkin
Director, Product Security & Privacy
Adobe Systems

Rocky said...

Matt,

Apple's Software Update Server:
http://preview.tinyurl.com/AppleSUS

The issue is gigantic software updates from Apple and others, like 473 MB Mac OS X 10.6.2 update. Even using an internal Software Update Server, that's pushing a lot of data around when you need to update a few thousand systems, particularly for slower WAN links.

Patches: Let me restate.

We need patches in commonly used patch formats, like .msi for Windows patches.

Lots of tools can push msi files, but some major vendors use proprietary patch formats.

Matt Olney said...

Rocky,

Thanks for that info...going to need to take a look at that :)

Your original comment sparked a long conversation in the second office (that is, the patio out front where some of the VRT smokes and the rest of us stand around and yell at each other).

I can certainly say no one disagreed with you, and there was a lot of talk about different (currently non-existent) solutions. Just know that the next time we have the ear of some of the larger vendors, we'll pass your concerns on.