Well thank you for all showing up here after lunch. I know it's the last day. It's great
to see such an audience. It's been really exciting to be here in Montreal. I've always
wanted to visit and this is my first time. So thanks Recon. It's been a great conference.
So what I'm going to talk about is reverse engineering on OSX. In particular, kind of
how we started with DTrace and then what mess we've got ourselves into. So this work is
done by myself and David Weston. He couldn't make the trip unfortunately. He totally pushed
out and made excuses about moving or something. So here I am swinging the whole deal.
So DTrace is a tool that was created by a couple of engineers, about three of them back
in 2001 I think and was publicly put into Solaris 10 and open sourced under the Sun's
SDDL license. And then recently and Leopard OSX picked it up and they ported it to their
operating system. There's been other adoptions to QNX has a partial implementation and then
just last week this guy John Byeril, I think I'm pronouncing that right, committed a DTrace
port to FreeBSD current. So you should be seeing that in 8. We might also see versions
in OpenBSD and other BSD systems. Linux has kind of done something different because they
don't, like they don't get along at all with the SDDL license. So they've created
a system tap which is intended to clone DTrace except kind of lacks application tracing,
user land tracing which is of course what we want to do a lot of. And interestingly
the FreeBSD port was initially sponsored by Cisco who pulled support and then it was picked
up I think by Yahoo and Juniper so extra juice on that stuff. So DTrace is really used as
sort of a performance monitoring framework, right? So it gives you the ability to instrument
all your applications or instrument library calls, instrument kernel and make measurements
or observations about what's going on in memory space with parameters or with timing.
So the way it works is that we define probes that get triggered during runtime. So this
is all very dynamic, right? We're not talking about static analysis. So what we have here
at the bottom is sort of logical categories of probe types and this is where you're
going to specify a probe is a PID probe meaning it's going to trace the application or it's
a kernel probe or it's some sort of IO probe. This all exists in the kernel, right? So DTrace
is heavily involved with the kernel. It's built into it. And then there's a user land
library called libdtrace that we can interface with. So we write these little DTrace scripts
which get, which will utilize libdtrace. There's also a number of other tools on here that
utilize libdtrace. So things you're used to seeing in Linux like Ptrace no longer exist
in OSX. Instead you have everything implementing something that interfaces with libdtrace. So
like truss or something. So this is what we're really interested in here in the beginning.
It's writing DTrace scripts that allow us to analyze our applications we're interested
in reverse engineering. So first about the language. DTrace is said to be a subset of
C except that that's kind of a joke because it just happens to follow the same syntax.
There's not actually loops or conditional statements. So what kind of language is that,
right? We can't do much useful. But there's good reasons for that. Like if you're gonna
instrument something, if you're gonna hook code in the kernel and you accidentally screw
up and you have an infant loop, what's gonna happen? Your kernel's never gonna get control
back and you're just gone, right? You gotta reboot. So there's reasons that they limit
the language. If you've ever seen awk, the structure of a D looks a lot like that where
you have a condition, like a conditional statement that has to be evaluated and then some block
of other statements that happen after that. So people use this a lot for performance monitoring,
right? Measuring IO, understanding information leaks, stuff like that. Another cool part
about DTrace is they built it so that like a Sun engineer can code some DTrace script,
go into a customer site and analyze a production system without having to restart the processes,
take anything down or have any effect on the production operation of the system at all.
So before you instrument anything with DTrace, there's no performance hit and when you're
doing the instrumentation, there's a small performance hit. So that's a goal, a design
goal, right? Another design goal was to not affect the application. So this is different
than like rewriting a function pointer so that you can hook the code because DTrace
is actually providing that framework in the kernel. Kind of like when we saw that device
driver, I think was it Jason, presented where he's grabbing code from kernel land by hooking
things like U name and system calls. This is similar except it's built in the operating
self system itself. In that sense, it provides a framework for hooking, it's almost like
a friendly root kit, right? You don't have to bother with overwriting pointers, we just
specify these probes. So you'll see many of these things, DTrace, PTrace, Truss and
PTrace using DTrace and OSX. And there's a million different things you can do, right,
with a tracing framework like this. So DTrace provides you the framework to implement those
million things. The real takeaway here is that DTrace combines system performance, statistics,
debugging information, execution analysis into one tight package. So then we get to
figure out, okay, how do we take that debugging tool and use it for reverse engineering? There's
a ton of stuff out there that's already been done. For instance, hooking IO to watch
all the IO that applications do. You can hook the keyboard to log keystrokes, cute stuff
like that. Here's a quick example that shows a DTrace script counting the number of sys
calls that different applications are making. So this is system-wide, right? We're tracing
every possible application on the system at once. And every time we see a sys call entry,
we record the name of it and count the number of times. So here we can see VMware is doing
a lot of sys calls. You'd expect that. What you might do next is zoom in on VMware and
say what sys calls are, or is it making. So basically redo this but only with tracing
VMware. Here's another file open snoop, right? So we run this little script and we can see
all the files that are being opened and by which applications. So you'll see something
like this. Here's the application. VMware is opening dev-random. iChat's opening some
pdes preferences files and so on. So good for investigation. So we've got probes, we've
got providers, we've got predicates and actions. I'm not gonna go over this too much. You
can read the docs. The syntax looks a lot like this. Pretty straightforward. The probe
specifies the point of instrumentation. The predicate is some conditional on which to
fire that probe and the actions are what you wanna have happen when that condition is true.
Usually this is just recording data. Okay, so this is the good part, right? How do we
use this for reverse engineering? Well, it turns out to be extremely versatile, right?
You really do need a good tracer. We can see how applications interact with the rest of
the system or with each other. It really provides us with kind of a tool to help with rapid
development of reverse engineering tasks. One of the interesting parts about it is it
doesn't really attach to the process because it's designed to be non-interfering. It doesn't
set breakpoints in the application. So we get a lot of useful things like people will
go implement tracing tools themselves because they need the tracing ability in their RLOS
that doesn't have something like that and they have to deal with all these things like
how are we gonna do symbol resolution and now I have to write a function that's gonna
print the CPU state and I wanna print something that parses the stack and prints a backtrace.
So DTrace provides a lot of that to us. For instance, here's a quick show of the control
flow of, you know, ADM, right? So you can see which functions are calling which functions
and when they return. Here's a simple stack trace printed when we enter STR copy from
FTP. We get symbol resolution. We also get the offsets within each function so we can
open up stuff in IDA and see the exact instructions being executed. We can also print function
parameters, right? So this is really interesting when you wanna see what is being passed to
some function that you found out is being as vulnerable or crashing. You can also change
values in memory with a copy out if you enable destructive actions. You get CPU state, right?
You can easily print out all your registers, EIP, EAX, stack pointer, stuff like that.
Here's another example of a destructive action, right? Here we're hooking the entry point.
We record where we are, where the argument is it will be returned. Then we hook the return
point for U name, the system call which is supposed to return the system's current
information. And then we can modify it how we want, right? So we've got Windows running
on a PowerPC, 2010, etc. So anytime any application then calls the U name, it's called, that's
what it's gonna get back. Another example of snooping, right? So this is gonna record
everything that any application prints out to the terminal. So this is how you can snoop
on another user's shell.
So hopefully this sparks a lot of ideas with you guys about things you can use this for,
just for fun. Things we're thinking about is like monitoring the stack, right? For detecting
stack smashes, EAP smashes, doing code coverage, integrating that with IDA. We haven't implemented
fuzzer feedback yet but you can do smart fuzzing where you're tracing inputs to certain functions
that you're trying to fuzz as you're fuzzing it. So it's important not to think of Dtrace
as a debugger, right? Because it's actually totally different. It's a tracer. So these
are two separate tools, a debugger and a tracer. And you really need both of them. Tracing
really provides you with a dynamic monitoring of the application at runtime. One of the
really cool things about Dtrace is you can trace kernel space at the same time. So if
you want to see, if you're troubleshooting a kernel bug, you can see where you transfer
from user space into kernel space and then what's going, what's happening in kernel
space in terms of parameters and stuff. Or if it gets tricky if your kernel crashes,
right? Because then you can't do anything about that. So I got a section on kernel debugging
later.
So that's one point is that Dtrace doesn't do exception handling at all because it doesn't
want to interfere or change behavior of the application. So that's something that we
wanted to do and had to do differently with our own debugger. But what you can do is like
trace to a specific spot and execution and under a certain condition stop the application
and then attach a debugger and continue on with doing whatever you want. Like just attach
GDB. There's an interesting part about Apple not wanting you to trace certain applications
and you can imagine why, right? They have this whole digital rights management thing
with iTunes. They don't want you just checking out what's going on in memory there. So they
prevent GDB and Dtrace from tracing things like iTunes by using PT attached deny. So
I'll talk about that later. It was actually discovered in a really interesting way by
someone who was tracing stuff and going where are my probes going? So there's a bunch of
other traces out there too, right? Tross, Ltrace, but they have a more limited scope and Dtrace
is really a framework for implementing your own tracers. There's things like Valgrind,
Ptrace is totally broken on OSX. There's Vtrace as well but there's again some limitations
that we already touched on. The language is very limited so you can't do loops, right?
And that's of course what we want to implement in control structures. Also you can't modify
registers, right? That would be really helpful for us. You can modify memory but not the
registers themselves. So that's the kind of thing you want to do when you're stepping
the IP back on so that you can, you know, run a break point or run an instruction on
the break point that you set. So there's some other issues to be aware of, right? Dtrace
is for performance so if you instrument too many probes it will just drop probes and it'll
tell you that. So you need to tune and scope so that you're not instrumenting too much.
There's also performance options like buffer sizes and buffer swapping rates which you
can tune to make sure that you don't drop these probes. There's also the issues with
exception handling. So this is where Redbug as we call it comes in, right? So first we're
going to touch on retrace which we wrote kind of a wrap around Dtrace with Ruby. So we did
this because we wanted loops, we wanted object oriented programming structures, we wanted
to have, be able to interface with other Ruby libraries out there like the Metasplay project,
MetaASM, IDARub. So what was I looking at? There's this library Ruby Dtrace that's
written by Chris Andrews that allows you to call Ruby from, call Dtrace from within Ruby.
So that's what we're utilizing. There's also IDARub which was presented at the last
Recon which lets Ruby interface with IDA over the network. So this is key in a lot of our
examples is that we have IDA running on a Windows system with the IDARub plug-in and
then we have Ruby running on the OSX system with Dtrace tracing so that way we can trace
stuff and communicate back and forth with IDA. So we found that we can quickly shorten
the time it takes to track down bugs and pinpoint vulnerabilities. So we were presenting this
once before at another conference and some guys came to us and said, hey we found this
iPhoto bug and we want to, but we're stuck. And so over a couple of beers at the bar we
just sat down with them and said, okay here's what you can do. You want to trace through
so many print apps until you hit this exact one and then that's when you want to start
analyzing the stack. So that was a fun time. Another cool part we can do is trace instructions
and then graph code coverage in IDA. So this will show us which basic blocks we're hitting
and then we know what we're not interested in. We can also do it the other way around,
so you trace the application and you monitor for all like, you use the application normally
and it generates all this code you're not interested in. And then you trace the application
and cause the overflow and then you know the delta, right. All this boring stuff happened,
we don't care about that. All the boring stuff happened with some extra things and you can
do the delta and figure out what code parts you're actually interested in. So Redbug
picks up where retrace left off. So it's a programmatic debugger for the mock API on
OSX. It partially exists in some forms already, right. Charlie Miller did a PyDGB port to
make PyMA run on OSX so you could do like file fuzzer parsing. So what he did there
was like, right, a MACDL file that translates everything that PyMA needs into the mock debugging
API equivalence. But that's not, it's not complete in the sense of what everything we
want to do is. So we're re-implementing it. So you can expect to see something like
KnoxGDB, right, programmatic Python debugger where you set breakpoints, you can change
memory values. So this is our plan, right. We use retrace and detrace for tracing. Ida
rub to interface with the disassembler, you could also use something like meta ASM and
then Redbug is our debugger. So where we're at now is implementing C code that wraps around
the mock API and then use Ruby bindings to bind those C functions to Ruby. Then we'll
create a higher level Ruby class that allows you to easily create processes, attach, set
breakpoints, specify handlers for breakpoints, walk memory structures, change permissions
on memory segments so that you can modify them. And also query information from a disassembler
like Ida. So for example, if we consider this iPhotobug format, right, it's a URL handler
bug where you put in some characters and it crashes. So what we want to do is trace through
the code until we see a specific invocation of printf and then we want to pause the application
there and attach our debugger. Then we want to do some analysis with the, with something
from the disassembly information and maybe set some breakpoints or write some memory
or look at values in memory. So the code should look something like this. First we set up
our Ruby detrace script and then we just attach to the process and run it. So that gets us
down to here. Once this hits this spot in memory, it's going to stop and it will stop
the application so then we'll have control here. We can connect to our IdaRuby, figure
out where we are in memory and then get the disassembly information, right. Look for the
return instruction, attach our debugger, set the breakpoint there and then continue until
we hit it. Once we hit that breakpoint, we can inspect our memory registers, we can look
at the memory we tried to overflow, make sure that the value we thought was there is actually
correct. So that's the kind of thing we hope to do with a programmatic debugger. Also,
you know there's something funky about Apple's ASLR implementation, like not everything is
random and it's not clear when things change or get re-randomized. So it would be cool
if we could, you know, load up an application and do a bunch of simple resolution to find
out where all the addresses of certain functions are or certain libraries and then search through
memory to see can we find a value somewhere that references that function or that library
or some value that's relative to that library. And then rerun the application again checking
that same address and see if we can ever find an address in memory that's static, it's
always at the same spot and has a value that's always relative to that library. And in that
way we could be able to locate libraries at runtime based off this static address. So
that kind of analysis we hope to be able to do with debug, right debug.
Unfortunately, it's not out yet. I really wanted to have it done. We've been working
on it all week but we need to do some more testing so we're thinking probably in another
week. So if it's not out by then just bug us, send us email. Okay, so let me make this
all concrete by showing you an example. What we're going to do is trace the stack until
we find an overflow. When we detect the overflow we'll print all the information that's useful
to us. So imagine that we fuzz our application and now we need to pinpoint the vulnerability.
The easiest example, right, is this. Trace all function entries in our application. If
EIP is ever for on, for on, for on we know we have our overflow. So stop the application
and then we do our analysis, right. Okay, this is trivial. This actually isn't really
useful though. We want something much more generic, something that works in all overflow
cases. So it's kind of obvious what to do here, right. We've all seen stack monitors
before. You just, you record the return address that's put on the stack. When you exit the
function you check that value. If it's different, you know something funky happened. If it's
the same, everything's good. That's simple enough but you have to understand how D-Trace
works with tail call optimizations because sometimes the compiler will say, hey, I don't
actually need to create a new stack frame on the stack because what's being returned
from this function is it will not be processed after the next function. So I can explain
that later if you want to come afterwards but tail calls are, they're just tricky
in terms of how you trace them. Like is it considered a new function entry or not because
it's a new function but it's not a new stack frame. There's also problems with functions
without return probes and you have to deal with those. So our first approach didn't
work. We went back and redid it. I won't go over the details of tail calls because
it's actually extremely boring but if you're excited about it please talk to me afterwards.
So our new approach is really to store the return address when we enter a function in
an array and then once we leave that function we compare the return address. We read that
same memory segment and compare. They're not different. We know that some overflow
happened. Now how to deal with the missing probes is simply not to trace them, right?
Dtrace gets tripped up by things like jump tables and other certain functions that just
don't have return probes because they couldn't figure out how to properly implement it. These
are usually functions that you're not interested in so you can just exclude them from the trace.
There's ways to detect which functions are not, which don't have returns. You can simply
print out all the traceable functions, see their entry points and then figure out which
ones don't have return points. You can do that with a dash L flag on Dtrace. You can
also run the application and see, count all the places where it functions entered and
then record all the places where they exited and just note the delta. Then you exclude
them with some predicate like at the bottom there. If the probe function is not equal
to everybody jump then don't trace it. So, okay enough talking. Let me show you a demo.
So here I'm going to start a public QuickTime RTSP exploit, right? There's a ton of these.
This is a server so I'll open up QuickTime and connect to it. So what I'm going to do
is visit the malicious URL, right? Get pwned and then watch my stack trace monitor detect
it. So I just note the PID. I'm going to go ahead and launch retrace and attach to this
process. So here's our menu. You can see some stuff in here like search memory, attached
debuggers. This is already up so you can have access to this, no problem. It's on pophotred.org
well that's the only place it's at right now. So we just launch our monitor stack script.
Now go visit the malicious URL. So in the background you can see everything printed
out. Now this is halted because retrace stopped the application when it detected the overflow.
So we get our message about overflow being detected and then here's all the useful information,
right? We get the module name. It was in the QuickTime streaming library, function name,
our expected return value and what it actually was and then we print out the CPU context.
You can see a number of these are overflowed. So pretty big stack smash there. We also get
a handy call trace which has all the instruction or the addresses, libraries, functions, offsets
and then we just stop the process. So from there we can attach our debugger and continue
on doing our analysis to figure out what happened or if you're just trying to use defensively
you can kill your application. So there's some other fun stuff we can do. We can actually
trace every instruction and application that will allow us to do block level coverage.
We can't do things like trace block entry points only though. I already mentioned some
of this stuff. One of the coolest parts is conditional tracing based on function parameters.
That turns out to be really handy. So let's talk about that. This is actually pretty simple,
right? We just write a D script, a D trace script that traces every instruction in the
application. That can be like pretty thorough. You kind of usually want to do all the instructions
in one library or something like that. I can trace application or I can not like highlight
what's being run in IDA in real time through the IDA root connector if you can catch up,
you know, if you can keep up with that. Here's the D trace script for the process trace in
the main module trace all functions and all instructions. Print out EIP. So that's all
D trace does and then the re-trace part of it parses all those EIPs and then changes
the color of them in IDA. Now there are some things to be aware of, right? If you're
tracing a library at run time, the offsets are different than when you load that binary
in IDA. So you have to deal with that mapping which is easy. A VM map will tell you where
the libraries are loaded at run time. So this allows us to like, you know, fuzz our application
and then see what code we didn't hit and try and inspect to understand how we do get
to that spot in code. We can also visualize code paths real easily and this is the real
takeaway there. So as Craig said, every presentation needs an IDA screenshot. This is a function,
a full function. All the red blocks are code that actually got executed during the run.
So at run time we can see what code we're interested in and what code that never got
run. So that's pretty handy. This all works now. You can also do graphing of dynamic call
traces, right? So similar to, I have to get that application that, I forgot his name already,
Gara had because his graph was way cooler. But this shows tracing at run time so we can
see which functions call which other functions. So if you know there's a vulnerability here
and they patched it but you notice from diffing the binary that they changed only this, you
might be interested in seeing how can I get here to still reach that vulnerability from
some other code path. All right, so stack is kind of trivially right. Let's talk about
monitoring the heap because that's where all the fun stuff happens. This is now sort
of turning into a major attack factor for exploits today. There's a lot of cool work
that's been done on heap exploits already. Defensive mechanisms are being put in place
to make the trivial heap exploits really difficult, right? So we can't just do standard
unlink exploits anymore. There's some good papers explaining these topics and kind of
justifying that by Dave Ito and Wiseman, I think it's Nicholas Wiseman on their presentation
site. So this is a new breed of attack which really requires us to understand what's
going on with heap allocation. So of course with Dtrace we can just hook malloc and see
what's going on there, hook string copy and make the association so that when we do
see mallocs we can wait until we see that memory actually being used and then make sure
it's being used in an appropriate way or check if it's being overflown. So for instance
we can hook the malloc functions in OSX. There's actually a bunch of different ones, right?
And we can find out something like this where mallocs all, it's all being, these are sizes,
right? So each malloc at this address is mallocing a memory space this large or this large or
this large. That's because on OSX there's actually four different heaps of different
sizes. So depending on what size you malloc you get memory space in a separate heap. So
there's like a tiny heap, right? Small heap, large heap and huge heaps. So we have to deal
with monitoring all of these. So I'm skipping ahead, I already mentioned this. There's already,
there's four heaps to monitor. So we hook our standard memory copy functions, right?
And we hook allocs and we make sure they're appropriately used. So let me skip to the
video. So in this case we're going to trace a browser. We'll go through a trivial example
and then we'll trace the browser. So this is your typical bad C program, right? You've
got your buffer, you've got your bad usage here. So we're going to make sure that we
can detect this with our heap smasher. Now we're also running IDA at the same time so
that we can relate what's going on with the overflow to the disassembly code. So this is our program text, right? You see this trend copy in there. So we can see that we have a lot of different
things. We're going to start up our IDA root server. So now we'll run our heap trace, our
heap smash detector launching this application with some bad input. So this is going to
tell us where, not only where the crash happened but also where did the malloc happen that
was eventually abused. What the sizes were, the destination address was 16, the actual
copy size was 30, okay we know what we have to play with. Also when it was detected it
was also marked in IDA. So we said okay this is the instruction where it happens. It's
kind of hard to read that but it says all the same information in here. Heap overflow
probe PID, it specifies the probe and then the addresses and the size and everything.
So same stuff gets marked in your codes. You can imagine fuzzing your application having
IDA hooked up to it and then getting a bunch of these in different locations and then later
going back and analyzing them. So there's a lot more you can do with this like visualizing
the heap space or trying to monitor heap allocation algorithms and understand them so that we
can create tools like Alexander Sotirov's heap lib which does this really cool heap
feng shui thing. So if you're not familiar with that work he's figured out how to from
the browser within JavaScript make allocations of strings or you know call string concatenation
functions that actually translate into process level heap allocations. So and a lot of systems
like where you have a JavaScript library has its own heap. So you're interested in how
does and in what situations does creating this object create a malloc space in the custom
heap versus when does it malloc space in the process heap and is there a way that can I
pinpoint the situation in which it's always mallocing space in the process heap because
then I can do things like a heap spring or more accurately heap feng shui where I can
construct and control the layout of the heap. So that's what you really want to be able
to do and that's where tracing this stuff can really come in. We imagine that you guys
all have a lot of great ideas about how to use this stuff as well maybe for garbage collection
tracing so you can understand when things are free to nod. Maybe there's a class of
attacks there. Okay I'm going to skip this section right using D-trace defensively this
is kind of not the context of recon but there's this paper where they say okay you can fingerprint
your application by recording the syscalls it makes and then if it ever makes a syscall
that's not within that fingerprint something must have gone wrong. So for instance if QuickTime
ever starts changing your DNS settings or calling bin bash maybe you know something's
gone wrong right. So you can write quick D-trace scripts that will check that make sure that
it doesn't ever call that never execs bin bash and if it does you can stop it. Also
worth mentioning is D-trace and root kits. There's a paper out there on the CINAHR
root kit about how to avoid being traced with D-trace. It's actually easy right because
Apple implements this whole PT-attach and I thing so you can just set that in your root
kit code and then D-trace won't be able to trace you. But there's also a kernel module
which of course came out the day after that announcement that allows you to trace things
like iTunes. So this is an interesting area but actually it's kind of easy to trip up
D-trace because it's really sensitive and its goal isn't to do malware analysis. Its
goal is to do production level tracing and not interfere with the application. So you
know you just use a bunch of function pointers and you can really confuse D-trace. But we
did find one piece of malware on OSX right. That's the problem is that there's not
a lot of stuff out there on OSX but there is a simple DNS changer. So we've wrote a
script just to test it if we can trace it and stop it from executing, understand what
it's doing. So in a lot of cases you're interested in the IO that the malware will
make. What functions does it open, what files does it open, what files does it change, stuff
like that. So again the PT-attached-in-I is the mechanism for avoiding D-trace. There's
this kernel, let's see right here is the URL for the kernel extension that disables
PT-attached-in-I by Landon Fuller. Okay so here's another section to talk about kernel
debugging on OSX. So you get something like this right. This is Apple's blue screen
of death or Steve Jobs. We get it in four languages with universal power symbol so a
nice fade and everything. I don't know about you guys but if you ever use a Mac and you
know sometimes it fades out to that media desktop thing, what is that called? I forget
it's like you can play music or view photos or something but it has the same fade effect
that the disk has and it always freaks the crap out of me because I end up activating
it somehow and I think I just blue screen myself. So actually all the useful information gets
put in this place called library logs panic reporter. So this is a dump of that file.
So let me see if I can zoom in here and show you what's going on. That's totally different
on my screen. So you can see this CPU syntax or context there right. We get first of all
the kernel trap page fault. This is tells us what happened right. So some sort of invalid
memory access. Unfortunately we don't have symbols on our stack trace because that's
just the way it works when you don't have a remote debugger. But we can see these three
functions that I have the same, they're in the same area and this one is very different.
It starts with 5C, this is 1.9. Also it says appear kernel trap at 5C, 3.1, blah, blah,
blah. So that's the address where the bug really happened. It turns out the addresses
are just signal handling stuff in the kernel. So this is the one we're really interested
in. So you'll note this address here, CR204. I'm going to point this out on the next slide.
You can't see this also here. They also note which kernel drivers are on the stack at the
same time. So I just pulled this panic log out of my directory before doing this. So
I haven't actually troubleshoot it but this is like a real bug right. So there's some
keyspan drivers, serial to USB connector. It's got some driver interface with it. So it tells
me that in this kernel stack trace, although I don't have symbols, these are related kernel
modules that have addresses in the stack trace. So you can see keyspan, Iokit, the keyspan
USA driver is loaded at this memory address through this memory address. That's where
we saw the 5C earlier. So we know that we crashed within that kernel driver. So we can
actually tell a lot just from this stack trace. Now in the case of a page fault, the CR2 register
tells us the address it tried to access. In this case 4, right, obviously bad. What we
can do is map the runtime address where it crashed, this one right here, to an instruction
address and the disassembly of the keyspan driver. So you go find the keyspan driver,
open it in IDA, you do a little math down here to find out the correct offset and you
go look up what's happening at that instruction. So here it is. We see a move EAX, EAX plus,
this one right here, EAX plus 4. So that's where our crash happened. You can see the
value of EAX actually is 0, CR2 is what we tried to access which is 4, so this all makes
sense right. Now we pinpointed exactly where this crash happens and we haven't even done
any debugging yet. Now the problem is that now you need to do remote debugging because
you have to try and reproduce this problem. We were hoping that with Dtrace because it
can trace kernel calls, you'd be able to somehow make this a lot easier. So we do not
need to set up a remote debugger. Let's just trace the kernel calls, look at the function
parameters and see what's happening. So I tried doing this right. Let me run a kernel
exploit against my machine that's going to DOS it and trace everything that happened
in the kernel. Well, it turns out there's way too much stuff going on in the kernel,
so you try and scope it down but still what's happening in the kernel that you're tracing
never hits the disk before the DOS actually happens. So I haven't found a way to make
this really useful yet. So, but there is a step in kernel exploit development where this
still might be useful and that's where you want to trace the data that you're sending
to the application and see when it enters a particular function what form does it have.
So if you're tracing packets you want to see that, you know, the packet parsing function
gets called and the packet data looks like this and then you change it a little and it
looks like this. You change it a little and it changes in this way so you can try and
figure out what's happening at a higher level. But as soon as you actually DOS the machine
then you're screwed. So if you really want to do kernel debugging what you have to do
is set up a remote debugger, export symbols from your target machine, import them on your
debugging host, set up some networks magic so that they don't forget about each other
and then you can configure the target to enable kernel debugging and it will look for the
other host when something goes wrong. You can also do this with dumping kernel core
dumps so dump them to the remote host. There's a lot of documentation on this on Apple site.
I won't say a lot but what you should look at really is this here because they talk about
the whole process of setting up the remote debugger. All right. So there's also new
work being done to integrate probes like higher level abstract probes into applications. So
this means we can trace abstract actions like a page load rather than the 10 million functions
that get called during an HTML page load or trace how long a DNS request happens. So a
lot of this stuff is for performance monitoring, right? Because we have such complex web 2.0
applications that we want to know where our bottlenecks are. But as reverse engineers
this can be interesting for us because we can hook things like the SQL parse because
we want to, we want to fuzz the application and then monitor what's happening on the,
on the database side. So for instance, we fuzz all our inputs and then we hook the database.
So in this case with MySQL we might hook the dispatch command and then print out the argument.
So this tells us exactly what SQL statements being executed on the server. So then we can
check our parsing routine to see are they escaping quote marks, you know, how is it
responding to our different inputs. So there's a lot more to be done in this area, right?
Once we finish the program active bugger there will be a lot of fun things we can do but
also the automated feedback and fuzzing stuff, if anybody wants to take a look at that. There's
things like writing analysis libraries to, to better understand heap allocation, memory
management, garbage collection, also taking advantage of a lot of these JavaScript probes
or MySQL probes or browser probes or whatever. So there's going to be, there's also probes
in Python for when a, when a Python function happens or when an Objective-C function happens
or when a Ruby function happens. Because if you're just tracing the lower level functions
you're going to see all the stuff the VM is doing and when you're debugging a Ruby app
usually you want to see what the Ruby functions are actually doing. They're also implementing
new probe categories, things like the IP category so you can trace packet receives and packet
sends very easily and TCP events too. That's all in the works right now. So a lot of what
happens is this stuff gets put into Sileris 10 and then later adopted by the OSX developers.
And of course your own ideas, I'm sure you all have your own. I'd love to hear about
them. So in conclusion, Dtrace gives you this ability to like trace things at a very broad
level like across the whole OS or trying to understand the behavior of something or to
zoom in and pinpoint a specific situation in which something happens. It is limited
though, there's only so many things you can do with a tracer. You really want to have
a tracer, a good debugger and a disassembler. So that's what we're aiming for here is
to get retrace, redbug and IDA all together. All right. So that's my presentation. We're
going to have a redbug up soon on popupred.org so check it. If it's not up, bug us. Yeah.
Audience member 2 – How much storage is there in the machine?
We did some metrics with general tracing like how much does Dtrace slow down an application
in general but we didn't do any metrics with comparing it to another heap smash detector.
I wrote those stats down. It seems like in general you'll see about a 63% decrease
in performance or speed with a standard tracer and with Dtrace you see about a 10% drop.
Of course it depends on how many probes you instrument but the performance is actually
very good. And a lot of cases when you try and use a tracer on anything besides notepad
it's just ridiculous. It takes forever and then when you try and use it like on a browser
for instance. But Dtrace lets you scope stuff and it's much better performance in general.
Anything else? Come on guys, I know you want to ask questions. Okay. Thanks.