First the good news… I’ve upgraded Trac for the Tcpreplay website which now supports built-in spam filtering. This means I’ve opened up the ticketing system to people who haven’t yet registered an account. Hopefully people will find it easier to open tickets for bugs and feature requests.

The bad news is that rewriting the L2/DLT code is even more complicated then I originally thought. Right now I’m spending my time working on the design, trying to make sure it’s flexible and easy to extend over time. Anyways, if you’re reading this, why not go check out that ticket, read the design document (realizing it’s a work in progress) and let me know what you think?

About 15 years ago an audio tape company challenged people to figure out if it was live or if it was Memorex. It seems that in the networking security space we’re at the same place, trying to figure out if live tools like Metasploit can be replaced by replay tools like tcpreplay.

Now, it would seem obvious that the answer, like most things is that “it depends”, but there seems to be people who are convinced that if it’s not live it’s not a real test.

Probably the best argument against replay tools is GIGO (garbage in, garbage out). If you don’t properly vet your test samples (often in libpcap format) then you’re going to have a bad day. Generally speaking, live tools like Metasploit are easier to validate- they either break into the target or they don’t. Replay tools don’t actually attack anything, and thereby require you to trust that the test case is correct or require you to do more strenuous validation.

The second reason which seems to come up against replay tools is that somehow it’s not really like a live test. Nobody has ever actually been able to give me a case where a libpcap capture of an exploit was different from what was actually on the wire or why resending those captured packets created any differences, but people still claim that it is. They’ve argued that replay tools don’t “emulate the state of the client and server” or that “live tools adapt to differences in the server runtime” but both these arguments miss the point. A IDS/IPS doesn’t know the state of the client or server, it only guesses at the state by watching the packets on the wire. Hence, a replay tool doesn’t need to emulate the state of the client/server, just repeat the externally visible process of those state changes. And that’s something that replay tools do very well.

So are there any advantages or situations where replay is better suited? I think so:

1. Comparative tests. If you’re comparing two products (like two IPS’s) then it’s important to use the same test cases. Live tools, due to their dynamic nature tend to auto-magically adapt behind the scenes.
2. Regression tests. The key behind regression tests is that they don’t change.
3. Single solution. While tools like Metasploit provide a good framework for developing a wide range of attacks, they have a limited number of attacks. This means you often need multiple tools with different interfaces to provide the coverage you’re looking for. Replay tools like tcpreplay provide a unified and protocol/attack agnostic approach to any network based attack.
4. Simple environment. Running live tools requires you to have a victim to attack. Often this means running VMware on a 2nd system, installing multiple operating systems and configuring countless applications to attack. Replay tools on the otherhand are virtually self-contained since they emulate both server and client side of the connection.

What do you think?