To test the new hash set, I created a Windows 7 virtual machine and hashed all of the files on it. I then submitted all of those files to the Kyrus NSRLookup server, with the following results:

$ nsrllookup -K known.txt -U unknown.txt -s nsrl.kyr.us < all.txt &&
wc -l *own.txt
40650 known.txt
7567 unknown.txt
48217 total

Of 48,000 input files, 7,600 were unknown. That’s a lot of unknowns! So I made a custom version of md5deep (which will be published soon), which only hashes Windows executables. I then hashed just the Windows executables on my VM and submitted them to the server:

$ nsrllookup -K known.txt -U unknown.txt -s nsrl.kyr.us < exe.txt &&
wc -l *own.txt
15937 known.txt
291 unknown.txt
16228 total

That's a LOT fewer files overall! We went from a total of 48,000 files to 16,000. That's a dramatic reduction thanks to ignoring non-executable files. Why just executables? Depending on the case type, you may only be looking for executables. In eDiscovery or illicit imagery cases, where the focus is on documents, you are probably better off searching for those file types directly rather than attempting to eliminate everything else. When doing executable analysis, look for executables!

But the real payoff  from this process is that there are only 291 unknown executables! That's actually manageable. Comparing 291 executables against the Malware Hash List is entirely do-able, as is sending the hashes to a service like VirusTotal. (Truth be told, I looked at the filenames of those 291 files, and most of them were part of VMWare.)

If you're champing at the bit to try out searching for executables, you can use another tool I wrote, Miss Identify, http://missidentify.sf.net/, to try it now. As a bonus, that program can generate warning messages when it finds executables which don't have an executable extension.

Finally, like you I have heard the complaints about the NSRL, such as http://ballinyourcourt.wordpress.com/2011/08/31/de-nisting-defective/. I asked the NSRL folks for a comment on the matter and they told me the following:

In December 2011 NIST identified a type of container file they were not recursing into when generating the hash sets. Failure to process these container files led to many hashes being omitted from the data set. The latest hash set, produced in January 2012, contains many more files, especially for those files in Windows 7. The next hash set, version 2.36, scheduled to be released in March 2012, will have even more.

They added, "NSRL appreciates ALL feedback from the community. We endeavor to respond in a timely manner, and we encourage you to contact us directly at nsrl@nist.gov to enable NSRL to turn a solution around within a publication cycle."

How much of your job as a lethal forensicator is spent searching for needles in haystacks? Known File Filters can be helpful in this regard . . . until the filters get too large to be useful. Reducing the data set you have to look through in order to find that precious needle (ouch) is one of those Holy Grail problems forensics people have been trying to address for years.

NIST on the Case

NIST is kind enough to distribute the National Software Reference Library (NSRL). A collection of hashes of known software (usually provided by vendors themselves), it contains over 78 million hashes, 21 million of which are unique. It is arguably the best resource almost no one uses because 78 million is enough hashes to choke almost any forensics tool you’re using. Go ahead and try it. We’ll wait . . .

One Step Beyond

Starting today we are offering (in Beta) the Kyrus NSRL Lookup Service (NSRLookup), which is based on the hard work of Rob Hansen at Red Jack, who coded nsrlquery. You can download a Windows binary or the *nix source code at SourceForge. Once installed, you can use the output of Jesse Kornblum’s md5deep to query the server. The output can be piped directly:

C:\> md5deep -r * | nsrllookup -s nsrl.kyr.us

. . . or a saved file can be used

C:\> md5deep -r * > known.txt
C:\> nsrllookup -s nsrl.kyr.us < known.txt

By default the server responds with the hashes of unknown files. You can get the hashes of known files by adding the -k flag, like this:

C:\> nsrllookup -s nsrl.kyr.us -k < known.txt

To see a help screen and a list of all command line options, use the -h flag:

C:\> nsrllookup -h
nsrllookup for Windows version 1.0.6-1
nsrllookup [-hvukx] [-U FILE] [-K FILE] [-s SERVER] [-p PORT]
-h: display this help message
-v: display version information
-u: show only unknown hashes (default)
-k: show only known hashes
-U FILE: write unknown hashes to FILE
-K FILE: write known hashes to FILE
-s SERVER: connect to a specified nsrlquery server
-p PORT: connect on a specified port

Please report bugs in the Sourceforge discussion forum.

We hope that NSRLookup proves useful to the forensics community. It isn't quite a “find evidence” button, but with people like Rob trying to tackle the problem, its close. If you’d like to provide us with your feedback, please contact us.

From the SANS site:

In one hour these Digital Forensics and Incident Response experts will discuss the coolest techniques and solutions they have discovered in 2011. If you have never been to a lightning talk it is an eye opening experience. Each speaker has 360 seconds (6 minutes) to deliver their message. This format allows SANS to present 10 experts within one hour, instead of the standard one presenter per hour. The compressed format gives you a clear and condensed message eliminating the fluff. If the topic isn't engaging, a new topic is just 6 minutes away.

If you cannot attend in person, you can watch the simulcast by registering at:

https://www.sans.org/webcasts/digital-forensics-incident-response-lightning-talk-%96-live-webcast-94919

It is tough being in cybersecurity. Defense is a cost center, and it's hard to find meaningful metrics to demonstrate success. Interest in security is also cyclical: Major breaches stir action, but as time passes, interest and resources wane, though the threat is still there. Yet the biggest problem with cybersecurity is ourselves. Before we can succeed, all of us must agree to change.

Read the full article here.

http://research.microsoft.com/apps/pubs/default.aspx?id=156020

The gist: I want to build a model of what each CPU instruction does in an unguided (completely automated) manner. we'll do it by having the CPU execute instructions and observing their side effects. this lets us build a "very good" model of what each instruction does.

The problem is how do I have input samples that can let me see everything? i.e. what if add is fine for integers from 0 to 1000000 but at 1000001 there is a problem? "the only way to find out is to exhaust" you say, that is, exhaustively try every possible input value (and every combination of input values).

That sucks. They have an algorithm that lets them do it "smarter". I'm still deciphering it, it's in the paper.

This is interesting to me because you can consider each instruction as a "function" and when you say "oh I have a way to reason about these functions reactions to certain inputs", ears should perk up.

Anyway, some results:

"We also discovered cases where observed behaviors contradict the x86 reference manual (which is unsurprising given the size and complexity of the spec). For instance, we discovered by accident while debugging our template T -ARImain that the overflow OF flag should be set to 0 after executing IMUL[8] with 65 and 254 as inputs according to the Intel spec, while the OF flag is actually set to 1 after the execution of this instruction with those inputs on an Intel XEON3.7 processor.

Moreover, we discovered, again by accident, that the semantics of instruction varies across Intel processors. For instance, on an Intel XEON3.7 or Core2 or i7 M620 processors and in accordance with the x86 spec, executing instructions ROL, SHL or SHR does not set the overflow OF flag if the count argument is not 1. However, on an Intel i7-2620M processor (HP EliteBook 2760p, 2.7Ghz, 8Gb RAM, 64-bit) processor, the OF flag is set to 1 even for certain   cases when the count argument is greater than 1. Our template T -BSf lag is actually unable to capture this behavior, which is why we detected these corner cases.

Finally, and unsurprisingly, we also discovered several errors in previous manually-written x86 instruction handlers used in the whitebox fuzzer SAGE [5]"

We discovered something similar with the shift/rotate instructions while implementing a project of our own (I forget the exact details, but I think it had to do with rotating by more than the register width would "still work" even though that case wasn't covered in the intel documentation). if you can put infrastructure like this together, you can discover interesting things about CPUs.

And of course, if you see some malware doing something "weird seeming" with instructions, you could perhaps infer that they were trying to do something like fingerprint what CPU they were on or flummox static analysis tools doing  instruction-level emulation, and then you could infer that whoever wrote the malware might have had enough time on their hands to come to grips with works like this paper.

The downside to being able to make that inference is I'm sure if you know just that much, you'll be super quick to blame anything "weird looking" on "oh I bet they're doing some super awesome per-cpu heuristic etection thingy" when really it's just some behavior you've never seen before. there has to be a word for that thought trap ...

And of course, there is prior academic work in discovering discrepancies between the docs and the reality, these people did it too: http://www.cl.cam.ac.uk/~pes20/weakmemory/index3.html there are some similarities between the Cambridge approach and the MS research approach, but, the Cambridge model of the CPU is built by hand (as I recall) and the thing the MS research model brings to the table is the automated building of the model.

Regardless of whose numbers you use, the cyber security market is measured in tens of billions of dollars. Cyber security is where the food is, to paraphrase an old Sam Kinison routine, and people are flocking to it. What most of the best practitioners are not prepared to flock to is the paperwork, bureaucracy and (frankly) bulls*** that is par for the course for government work. There is a lot of cool work in the government, some things only the government can do, but the government is not for everyone.

Even if the government could get as many people as it needed to improve security, why do some think that throwing more bodies at the problem is the only way to ‘scale’ to address the issues? Throwing bodies at a problem stopped being a successful strategy back when the attrition warfare model went out of vogue. Better security isn’t going to come with a rise in the size of the ranks; it will come when we have in place models that let security mechanisms scale to match security threats.

On Sept. 22^nd, Microsoft filed for an ex parte temporary restraining order from the U.S. District Court for the Eastern District of Virginia against Dominique Alexander Piatti, dotFREE Group SRO and John Does 1-22. The court granted our request, allowing us to sever the known connections between the Kelihos botnet and the individual “zombie computers” under its control. Immediately following the takedown on Sept. 26^th, we served Dominique Alexander Piatti, who was living and operating his business in the Czech Republic, and dotFREE Group SRO, with notice of the lawsuit and began discussions with Mr. Piatti to determine which of his subdomains were being used for legitimate business, so we could get those customers back online as soon as possible. We are also beginning our efforts to notify the other John Doe defendants in this case, and will be actively continuing our investigation to find out more about the people behind this botnet.

...and we're proud that we had a role in making this happen.

Read the full post at the MS Blog.

Consider the following two functions:

void copyBytes(char *dst, char *src, size_t len) {
    if( len == 0 ) {
        return;
    } else {
        *dst = *src;
        return copyBytes(++dst, ++src, --len);
    }
}

void copyBytes2(char * dst, char *src, size_t len) {
    size_t copied = len;
    char *d = dst;
    char *s = src;

    while( copied > 0 ) {
        *d = *s;
        d++;
        s++;
        copied--;
    }

    return;
}

The question is, what do these functions do? Consider them for a few minutes before continuing (please).

So, they both pretty obviously copy memory. copyBytes performs the copy recursively, while copyBytes2 performs it iteratively. So, lets think about "what do these functions do" in terms of side effects.

Both result in "len" bytes from "src" moving to "dst". Simple enough, case closed, right? They're the same! Wait a minute though, copyBytes works recursively, this means that an additional side effect is for every byte copied, the stack pointer is decremented enough to create a new frame. Stack space is a limited resource, so, if you were copying a lot of data, copyBytes has the potential to exhaust stack space, resulting in the termination of that thread.

So, they're not the same?

Alright, compile it, test it, wait a minute, they are both capable of copying for large values of "len". Whats up?

So lets feed these 2 functions to LLVM, on two different optimization levels, to REALLY dig into this:

copyBytes with no opts:

copyBytes2 with no opts:

copyBytes with -O1 LLVM opts:

copyBytes2 with -O1 LLVM opts:

(LLVM IR is documented here: http://www.llvm.org/docs/LangRef.html but this is short enough and non-weird enough that a lot of it should be pretty clear on its own)

So with no opts, copyBytes demonstrates the recursive behavior we expect. When we look at the IR for the "slightly" optimized version, we see the optimized has recognized how to optimize tail-call recursion into iterative logic, and that the resulting iterative logic is in fact EXACTLY the same as the source code that is explicitly iterative. The kind of interesting thing is that you can line up the two LLVM IR control flow graphs for the optimized functions and prove they are the same by comparing their side effects. The order, width, and count of loads and stores are identical.

So ... what controls whether or not these two functions express different behavior? Compiler optimization levels. So... people who audit source code... what exactly are you auditing? Usually, people are auditing what the compiled code will ACTUALLY DO and not really the code as written, because you really care about what the code actually does. This tells us, pretty conclusively, that to understand what code does you also have to have a very clear understanding of how it is compiled and what goes into its compilation.

So ... everyone who is (or was) focused on automated ways to verify C code ... have fun! When you're evaluating the sanity of some code, you're talking about something more meta-level than C...