A couple of short but funny stories from over the summer. I have been learning to weld and do it outside using the MIG fluxcore/FCAW-S process.
Over the summer, I was working on some projects and at one point I started hearing a weird sound. It was very random but seemed close. I took off my welding shirt to take a break from the substantial summer heat for a few minutes. I also looked around for the source of the sound but couldn't quite place it. It was made harder because the Cicadas were buzzing loudly in the nearby woods and a few trees in the yard. I was thinking that it was our water softener in the garage at one point because it tends to make some noise when it runs but I determined that wasn't it. I finally went back to my welding for a while. At a later point, my wife was outside and heard it too - even together we couldn't find the source of the weird sound. We looked around a while and finally gave up. It was time for a break and dinner was ready. I put away my welding shirt and welding supplies and went to the table. As we were sitting there; the sound started up again. The wife and I looked at each other and the kids watched us. I said that sounds like it was coming from near her. She said it sounded like it was coming from the other side of me. The sound would only last a second so we sat there. I turned and looked at the window wondering if something had come inside or was in the window. Nothing. I suddenly wondered if something was on my shirt so I turned my back to her and she didn't see anything. A few more minutes go by and then I hear a sound and feel something on my right shoulder - there was something in my shirt! I screamed like a girl, slapped at my shirt (which resulted in a bad sound from something that didn't like being slapped) as I ran for the door while trying to rip my shirt off. As I got outside and the shirt fly off, a cicada plopped on the deck - very unhappy. Not sure what happened to it in the end but I was happy he was gone. Glad I won't see them again for ~17 years (at least while I am welding).
A bit later in the summer, I was working on a different project and at one point something lightly smacked into my helmet. I thought maybe a bee bumbled into me or a small piece of metal had somehow gotten flung but I didn't see anything. Then I heard a scratching sound. I stopped and looked around - I thought I was scraping my helmet on something but didn't notice anything. I went back to my welding and after I had some welds done I took a break and was starting to get some more pieces clamped up. I was reaching over to put a clamp on when suddenly something moved. I jumped a bit and then realized that it was a preying mantis climbing up the metal frame I was working on. I am pretty certain it had flown and landed on my helmet which is what I felt smack it and the scratching was it crawling around. I am guessing he jumped off when I was taking my helmet off and he decided to check out the frame.
I am thinking that the light from welding was attracting my bug brained friends in both of my above stories. Can't wait to see what "friends" I make next summer.
Sunday, January 19, 2014
Java/Linux practical security
I have needed to delve more deeply into web application security methods in the last year. As part of this, I have come to recognize a lot of short comings in this area. There has been a continuous stream of Java updates from Oracle and a number of security fixes for various Java EE/servlet containers and libraries commonly used in web applications. It is good to keep the technology stack updated with the most current security fixes - no denying that. The problem is that this leaves the potential for too much time between unethical people finding the info needed to exercise an exploit and actually having a security fix in production. And this assumes that a security fix doesn't break your web application which is a real possibility. So this isn't what I would call a first line of security.
So what does one do? I have been doing lots of research and the funny thing is that most of it is what I would call "additive security". What I mean by that is, it is security provided by adding some new mechanism into the technology stack. Things SELinux, AppArmor, Java Security Manager policies, TripWire, fancy application security firewalls, etc. I won't deny that those have the potential to prevent various security flaws from being exercised or at least identifying when it occurs so they can have a benefit but they also add complexity and workload to what is often an already overworked IT staff.
I also looked into some various research in the area of "hardened" run times and such. What I found there was still more on the "additive" side and was more targeted at the typical older exploits - stack smashing, etc. Useful stuff but this didn't really fit the types of problems I was trying solve and there were implications of high overheads in some of the research implementations.
I finally came to the conclusion that a better first step is the disabling or removal of functionality which is most often exploited. This got me thinking a bit out of the box. With Java, it isn't really practical to remove "insecure" features from Java and it's libraries. The use of Java Security Manager can disable the functionality but that turns out to be a nightmare when you start working with lots of Open Source dependencies in large/complex web applications. I really like the idea of disabling items which are commonly exploited though.
As I was thinking about this, it occurred to me that maybe there is a way to disable some items with a minimal overhead. Java is generally implemented in C/C++ and uses things such as the standard C/C++ libraries. The nice thing about this is that features that I "care about" tend to funnel down to a fixed set of functions in the C/C++ libraries. I decided it was not very practical to physically create new versions of glibc, etc though. The good news is that I didn't have to. There is functionality within the Linux loader which makes things fairly easy.
[I had to cut a lot of detail out of this for now; somewhat for time - may be able to include more details later].
A simple use case goes like this; you want to prevent a web server from being able to launch an external process. This is an attack vector which has plagued a number of open source technology on/off. First write a simple Java program which only calls "Runtime.getRuntime.exec()" and have it do something you can verify like adding/appending some data to a file. Next run the test program while prefixing the command with "LD_DEBUG=symbols" - so the command line would look similar to "LD_DEBUG=symbols java p1.p2.C1" . If you start a 'script' before this, you can capture the somewhat substantial output. In the output, I found that the exec() call was using vfork() from the threading library. I determined that my web server really should not need to call vfork() so I chose it as my target for "disablement". So next, write a small C file which implements a dummy vfork() - use the various Unix/Linux man pages to determine what return value should represent an error/failure to the caller and return that in the implementation. In this case, -1 was the appropriate value. Next compile and link that into a shared library. Now, all that remains is to get the Java process to use your vfork() instead of that from the standard threading library. This can be accomplished by starting Java with a "LD_PRELOAD=libyourlib.so" which for testing purposes will include "LD_DEBUG" as well and would look something like: "LD_DEBUG=symbols LD_PRELOAD=libyourlib.so java p1.p2.C1". Again, if you run 'script' before this you can easily review the output and note that the vfork() func was picked from your library or some error in your lib prevented it from loading. Once you get any loading errors resolved (maybe you needed to compile/link with -fPIC, etc) and you see the vfork getting loaded from your library - go back and verify that the Java test program is no longer producing the effect that the Runtime.exec() call was producing.
So at this point you have written a C func for a "standard library" function which hard-codes an error type return value and effectively hides the real implementation of the function.
You can take this general idea as far as you want. Make sure that various IO calls only have access to a predefined list of resources, etc. You are mainly limited by imagination. It is fairly low in overhead but is not a total solution in itself. You could incrementally secure an application over time though which is a nice aspect.
This isn't a silver bullet and won't help everyone all the time. I am sure there are applications which would require a lot more effort to make similar changes to because of an actual need for some of the functionality which does get exploited in web apps.
I plan on keeping this idea in the toolbox for use when it is a good fit. It could also be a good reactive method in a crisis.
[Update 2014/02/25] It seems quite awkward to implement this in a minimally invasive manner with software such as Apache Tomcat because of how the startup scripts work. It may be possible but it will take a lot more scripting research if so. I think the general recommendation I found for a similar need is to simply create a new script with the end result of what the startup script uses to start the Java process. Not a very operational/upgrade friendly method but if you must do it..
[Update 2014/03/17] I am wondering whether JEP178 may provide a better/easier supported way to remove certain limited functionality from the runtime at the process level through some static linking. I think this will need from further review at a later date. This would prevent the need to customize start up scripts and such.
So what does one do? I have been doing lots of research and the funny thing is that most of it is what I would call "additive security". What I mean by that is, it is security provided by adding some new mechanism into the technology stack. Things SELinux, AppArmor, Java Security Manager policies, TripWire, fancy application security firewalls, etc. I won't deny that those have the potential to prevent various security flaws from being exercised or at least identifying when it occurs so they can have a benefit but they also add complexity and workload to what is often an already overworked IT staff.
I also looked into some various research in the area of "hardened" run times and such. What I found there was still more on the "additive" side and was more targeted at the typical older exploits - stack smashing, etc. Useful stuff but this didn't really fit the types of problems I was trying solve and there were implications of high overheads in some of the research implementations.
I finally came to the conclusion that a better first step is the disabling or removal of functionality which is most often exploited. This got me thinking a bit out of the box. With Java, it isn't really practical to remove "insecure" features from Java and it's libraries. The use of Java Security Manager can disable the functionality but that turns out to be a nightmare when you start working with lots of Open Source dependencies in large/complex web applications. I really like the idea of disabling items which are commonly exploited though.
As I was thinking about this, it occurred to me that maybe there is a way to disable some items with a minimal overhead. Java is generally implemented in C/C++ and uses things such as the standard C/C++ libraries. The nice thing about this is that features that I "care about" tend to funnel down to a fixed set of functions in the C/C++ libraries. I decided it was not very practical to physically create new versions of glibc, etc though. The good news is that I didn't have to. There is functionality within the Linux loader which makes things fairly easy.
[I had to cut a lot of detail out of this for now; somewhat for time - may be able to include more details later].
A simple use case goes like this; you want to prevent a web server from being able to launch an external process. This is an attack vector which has plagued a number of open source technology on/off. First write a simple Java program which only calls "Runtime.getRuntime.exec()" and have it do something you can verify like adding/appending some data to a file. Next run the test program while prefixing the command with "LD_DEBUG=symbols" - so the command line would look similar to "LD_DEBUG=symbols java p1.p2.C1" . If you start a 'script' before this, you can capture the somewhat substantial output. In the output, I found that the exec() call was using vfork() from the threading library. I determined that my web server really should not need to call vfork() so I chose it as my target for "disablement". So next, write a small C file which implements a dummy vfork() - use the various Unix/Linux man pages to determine what return value should represent an error/failure to the caller and return that in the implementation. In this case, -1 was the appropriate value. Next compile and link that into a shared library. Now, all that remains is to get the Java process to use your vfork() instead of that from the standard threading library. This can be accomplished by starting Java with a "LD_PRELOAD=libyourlib.so" which for testing purposes will include "LD_DEBUG" as well and would look something like: "LD_DEBUG=symbols LD_PRELOAD=libyourlib.so java p1.p2.C1". Again, if you run 'script' before this you can easily review the output and note that the vfork() func was picked from your library or some error in your lib prevented it from loading. Once you get any loading errors resolved (maybe you needed to compile/link with -fPIC, etc) and you see the vfork getting loaded from your library - go back and verify that the Java test program is no longer producing the effect that the Runtime.exec() call was producing.
So at this point you have written a C func for a "standard library" function which hard-codes an error type return value and effectively hides the real implementation of the function.
You can take this general idea as far as you want. Make sure that various IO calls only have access to a predefined list of resources, etc. You are mainly limited by imagination. It is fairly low in overhead but is not a total solution in itself. You could incrementally secure an application over time though which is a nice aspect.
This isn't a silver bullet and won't help everyone all the time. I am sure there are applications which would require a lot more effort to make similar changes to because of an actual need for some of the functionality which does get exploited in web apps.
I plan on keeping this idea in the toolbox for use when it is a good fit. It could also be a good reactive method in a crisis.
[Update 2014/02/25] It seems quite awkward to implement this in a minimally invasive manner with software such as Apache Tomcat because of how the startup scripts work. It may be possible but it will take a lot more scripting research if so. I think the general recommendation I found for a similar need is to simply create a new script with the end result of what the startup script uses to start the Java process. Not a very operational/upgrade friendly method but if you must do it..
[Update 2014/03/17] I am wondering whether JEP178 may provide a better/easier supported way to remove certain limited functionality from the runtime at the process level through some static linking. I think this will need from further review at a later date. This would prevent the need to customize start up scripts and such.
Subscribe to:
Posts (Atom)