Making Apache leak behind SSL
Posted 1 year, 3 months ago at 11:01 pm. 6 comments
I’ve been working on a really great product for terminating SSL and encrypting HTTP parameters in a FIPS-140-2 Level 3 hardware security module. This is really handy if you run an internet banking application or payment processing service and want to be able to collect or deliver credit-card related info (such as PINs, PANs, CVVs, etc) in a manner compliant with the hardcore PCI PIN Security Requirement (PCI-PSR). Some simple usages might be the ability to provide customer self-selected PIN numbers, or the ability to replace those “super secure” post office delivered PIN printer envelopes by allowing customers to register and receive their PINs from the internet banking platform.
Unfortunately, any internet banking application which deals with credit-card information is PCI-PSR non-compliant by default, purely because this sensitive information “pops” out in the clear in the web server. SSL obviously does a great job of securing the transportation of the traffic between browser and web server, but once that traffic is decrypted you’re screwed. And this is not purely a theoretical attack either - think malicious administrator schnyfing the HTTP packets off somewhere to be collected later.
So it’s a real problem, and I’d like to prove it by building a tool to grab packets out of apache memory space after SSL decrypt. I was kind of hoping this would be simple, and I started off by just making use of system tools commonly available to most Linux servers, such as strace, ltrace, and python. It turns out that apache is *actually* designed quite well, not allowing decrypted traffic to cross the system boundary, therefore making strace pretty useless. ltrace by default borks apache dead on startup, possibly because it doesn’t follow the fork() / clone() combination used by the worker-mpm threading model, so I gave up on this approach quite quickly. So thats as far as my initial investigation went, and I think I’ve got a much better idea of how to tackle this, the following doors are still open to me:
- Although I don’t get decrypted traffic out of strace, I do get all the ENcrypted traffic, including the SSL handshake / key exchange. I can schnife the key, then do the RC4 decrypt myself.
- Write a custom apache module which hooks into the apache stack and grabs traffic, this is really easy to do, but isn’t as elegant or edgy as I’d like. Some existing apache modules available already give us this, e.g. mod_forensic and mod_security
- Hack openssl to make it bleed, and dynamically overwrite the default reference to the openssl system library (using ldconfig, LD_LIBRARY_PATH, or hacking the new lib path into the apache binary directly). This one sounds like fun, and might be useful for making more than apache leak, but still not as great as I’d like
- Scrape apache memory directly from an external app - this sounds really appealing, but might have some issues. How long does the decrypted stream stay in memory and therefore how reliably can I grab packets? If I were apache I would zero my memory after handling the request, and close this door instantly on me. Also, I’d need to spend many moons researching linux memory management. So a potentially good last-resort approach for me.
And thats all I’ve got so far. Any other ideas?
Dominic White Jun 3rd 2009
Yeah, freeze the ram with liquid nitrogen and use an electron microscope to see if you can get it out of memory, they both about equally likely as an attack vector :P
Tom Jun 3rd 2009
Ha! This is exactly the kind of perception I expect to blow out of the water with this research. I admit that scraping apache’s memory is hard, but I think you might be surprised how close I am. I’m already dumping the SSL handshake + encrypted conversation, and because I’m assuming the machine is compromised, access to and use of the private key is all I need to start decrypting.
haroon Jun 3rd 2009
i might be missing the point here a little, but i wouldnt really expect privacy once the traffic hits the host machine ?
Im guessing turning verbose logging on the httpd will leave the clear text info you want in /var/log/apache ? (or even in the application server logfiles if you are using one?)
Once it hits the app, theres a basquillion places to hook the data (because the app doesnt notice at all..)
if for some reason u still want to:
i took a quick look this evening, and if u run httpd -X (for simplicity for now) and then attach a debugger, breaking on SSL_read() you should be able to find your way to the un-encrypted buff..
if you did this in pydbg you could automate this fairly easily..
if you are running solaris (or OSX) u could also try dtrace (just for the geek fun of it)
trace -n ’syscall:::entry /execname==”httpd”/{self->arg1=arg1;}’ -n ’syscall::write:return /self->arg1/ {printf(”%s(\”%S\”)\n”,probefunc,copyinstr(self->arg1)); }’
on my machine returns:
CPU ID FUNCTION:NAME
1 17719 write:return write(”::1 - - [03/Jun/2009:22:03:44 +0200] \”GET /?moo=blah/ HTTP/1.1\” 304 -\n”)
1 17719 write:return write(”[03/Jun/2009:22:03:44 +0200] ::1 TLSv1 DHE-RSA-AES256-SHA \”GET /?moo=blah/ HTTP/1.1\” -\n”)
(Both are still calls to the log write, but with more messing about you should get what you need?)
Tom Jun 3rd 2009
Hey Haroon!
Thanks for the comment - and yes I’ve gone down this same track, however even with “more messing about” it’s not actually as easy as it sounds. Apache is quite good about not making system calls with the clear HTTP (except for the log file entry), leaving a tool like “strace” pretty much in the dark. Of course I can see the read and writes to the connected client socket, but this data is already encrypted.
Something like ltrace would be ideal, and I was hoping to hook the openssl calls here, but for whatever reason it manages to crash my apache process, rendering it pretty useless (on my environment at least).
Of course a debugger would be great - but I believe it would be unacceptable to halt the process (think malicious web server admin attempting to steal user credentials in a production environment). I haven’t looked at pydbg, if it’s possible to automate this process then that’s certainly an option. Thanks, I’ll check that out.
So (assuming pydbg doesn’t lead me anywhere) my option using strace is to actually break apart the SSL handshake (as I have the private key), steal the symmetric session key and use this to decrypt the conversation. This is cool because from an attack execution point of view it’s as simple as attaching strace to the running apache processes and logging all the data - the decrypt can occur after the fact, and no fancy tools are required (except strace, which is very common on machines I’ve encountered).
btw - I wanted to say hi and bravo on your itweb talk (i think the most entertaining one of the summit, dom’s a close second), but you sp guys are way too sneaky… :)
Ciao,
Tom
haroon Jun 4th 2009
heya..
thanks for the positive comments on the talk.. any talk i manage to give without being called a dumbass, i chalk up as a success..
pydbg is pretty sweet in general as a full featured python debugger.. i’ve kicked it around on windows for some sillyness and just tried the OSX port (http://code.google.com/p/paimei/source/browse/#svn/trunk/pydbg) which seems to work ok..
(it supports a dbg.search_memory(’foo’) call which should yield instant joy, except it doesnt.. i suspect u will have to end up doing your break on SSL_read(), then locating / dumping ur buff)
are u currently using ptrace to get as far as u are ?
Post ur success story when u are done.. :>
david b Jul 8th 2009
thoughts on the google os ?