A couple of short but funny stories from over the summer. I have been learning to weld and do it outside using the MIG fluxcore/FCAW-S process.
Over the summer, I was working on some projects and at one point I started hearing a weird sound. It was very random but seemed close. I took off my welding shirt to take a break from the substantial summer heat for a few minutes. I also looked around for the source of the sound but couldn't quite place it. It was made harder because the Cicadas were buzzing loudly in the nearby woods and a few trees in the yard. I was thinking that it was our water softener in the garage at one point because it tends to make some noise when it runs but I determined that wasn't it. I finally went back to my welding for a while. At a later point, my wife was outside and heard it too - even together we couldn't find the source of the weird sound. We looked around a while and finally gave up. It was time for a break and dinner was ready. I put away my welding shirt and welding supplies and went to the table. As we were sitting there; the sound started up again. The wife and I looked at each other and the kids watched us. I said that sounds like it was coming from near her. She said it sounded like it was coming from the other side of me. The sound would only last a second so we sat there. I turned and looked at the window wondering if something had come inside or was in the window. Nothing. I suddenly wondered if something was on my shirt so I turned my back to her and she didn't see anything. A few more minutes go by and then I hear a sound and feel something on my right shoulder - there was something in my shirt! I screamed like a girl, slapped at my shirt (which resulted in a bad sound from something that didn't like being slapped) as I ran for the door while trying to rip my shirt off. As I got outside and the shirt fly off, a cicada plopped on the deck - very unhappy. Not sure what happened to it in the end but I was happy he was gone. Glad I won't see them again for ~17 years (at least while I am welding).
A bit later in the summer, I was working on a different project and at one point something lightly smacked into my helmet. I thought maybe a bee bumbled into me or a small piece of metal had somehow gotten flung but I didn't see anything. Then I heard a scratching sound. I stopped and looked around - I thought I was scraping my helmet on something but didn't notice anything. I went back to my welding and after I had some welds done I took a break and was starting to get some more pieces clamped up. I was reaching over to put a clamp on when suddenly something moved. I jumped a bit and then realized that it was a preying mantis climbing up the metal frame I was working on. I am pretty certain it had flown and landed on my helmet which is what I felt smack it and the scratching was it crawling around. I am guessing he jumped off when I was taking my helmet off and he decided to check out the frame.
I am thinking that the light from welding was attracting my bug brained friends in both of my above stories. Can't wait to see what "friends" I make next summer.
Sunday, January 19, 2014
Java/Linux practical security
I have needed to delve more deeply into web application security methods in the last year. As part of this, I have come to recognize a lot of short comings in this area. There has been a continuous stream of Java updates from Oracle and a number of security fixes for various Java EE/servlet containers and libraries commonly used in web applications. It is good to keep the technology stack updated with the most current security fixes - no denying that. The problem is that this leaves the potential for too much time between unethical people finding the info needed to exercise an exploit and actually having a security fix in production. And this assumes that a security fix doesn't break your web application which is a real possibility. So this isn't what I would call a first line of security.
So what does one do? I have been doing lots of research and the funny thing is that most of it is what I would call "additive security". What I mean by that is, it is security provided by adding some new mechanism into the technology stack. Things SELinux, AppArmor, Java Security Manager policies, TripWire, fancy application security firewalls, etc. I won't deny that those have the potential to prevent various security flaws from being exercised or at least identifying when it occurs so they can have a benefit but they also add complexity and workload to what is often an already overworked IT staff.
I also looked into some various research in the area of "hardened" run times and such. What I found there was still more on the "additive" side and was more targeted at the typical older exploits - stack smashing, etc. Useful stuff but this didn't really fit the types of problems I was trying solve and there were implications of high overheads in some of the research implementations.
I finally came to the conclusion that a better first step is the disabling or removal of functionality which is most often exploited. This got me thinking a bit out of the box. With Java, it isn't really practical to remove "insecure" features from Java and it's libraries. The use of Java Security Manager can disable the functionality but that turns out to be a nightmare when you start working with lots of Open Source dependencies in large/complex web applications. I really like the idea of disabling items which are commonly exploited though.
As I was thinking about this, it occurred to me that maybe there is a way to disable some items with a minimal overhead. Java is generally implemented in C/C++ and uses things such as the standard C/C++ libraries. The nice thing about this is that features that I "care about" tend to funnel down to a fixed set of functions in the C/C++ libraries. I decided it was not very practical to physically create new versions of glibc, etc though. The good news is that I didn't have to. There is functionality within the Linux loader which makes things fairly easy.
[I had to cut a lot of detail out of this for now; somewhat for time - may be able to include more details later].
A simple use case goes like this; you want to prevent a web server from being able to launch an external process. This is an attack vector which has plagued a number of open source technology on/off. First write a simple Java program which only calls "Runtime.getRuntime.exec()" and have it do something you can verify like adding/appending some data to a file. Next run the test program while prefixing the command with "LD_DEBUG=symbols" - so the command line would look similar to "LD_DEBUG=symbols java p1.p2.C1" . If you start a 'script' before this, you can capture the somewhat substantial output. In the output, I found that the exec() call was using vfork() from the threading library. I determined that my web server really should not need to call vfork() so I chose it as my target for "disablement". So next, write a small C file which implements a dummy vfork() - use the various Unix/Linux man pages to determine what return value should represent an error/failure to the caller and return that in the implementation. In this case, -1 was the appropriate value. Next compile and link that into a shared library. Now, all that remains is to get the Java process to use your vfork() instead of that from the standard threading library. This can be accomplished by starting Java with a "LD_PRELOAD=libyourlib.so" which for testing purposes will include "LD_DEBUG" as well and would look something like: "LD_DEBUG=symbols LD_PRELOAD=libyourlib.so java p1.p2.C1". Again, if you run 'script' before this you can easily review the output and note that the vfork() func was picked from your library or some error in your lib prevented it from loading. Once you get any loading errors resolved (maybe you needed to compile/link with -fPIC, etc) and you see the vfork getting loaded from your library - go back and verify that the Java test program is no longer producing the effect that the Runtime.exec() call was producing.
So at this point you have written a C func for a "standard library" function which hard-codes an error type return value and effectively hides the real implementation of the function.
You can take this general idea as far as you want. Make sure that various IO calls only have access to a predefined list of resources, etc. You are mainly limited by imagination. It is fairly low in overhead but is not a total solution in itself. You could incrementally secure an application over time though which is a nice aspect.
This isn't a silver bullet and won't help everyone all the time. I am sure there are applications which would require a lot more effort to make similar changes to because of an actual need for some of the functionality which does get exploited in web apps.
I plan on keeping this idea in the toolbox for use when it is a good fit. It could also be a good reactive method in a crisis.
[Update 2014/02/25] It seems quite awkward to implement this in a minimally invasive manner with software such as Apache Tomcat because of how the startup scripts work. It may be possible but it will take a lot more scripting research if so. I think the general recommendation I found for a similar need is to simply create a new script with the end result of what the startup script uses to start the Java process. Not a very operational/upgrade friendly method but if you must do it..
[Update 2014/03/17] I am wondering whether JEP178 may provide a better/easier supported way to remove certain limited functionality from the runtime at the process level through some static linking. I think this will need from further review at a later date. This would prevent the need to customize start up scripts and such.
So what does one do? I have been doing lots of research and the funny thing is that most of it is what I would call "additive security". What I mean by that is, it is security provided by adding some new mechanism into the technology stack. Things SELinux, AppArmor, Java Security Manager policies, TripWire, fancy application security firewalls, etc. I won't deny that those have the potential to prevent various security flaws from being exercised or at least identifying when it occurs so they can have a benefit but they also add complexity and workload to what is often an already overworked IT staff.
I also looked into some various research in the area of "hardened" run times and such. What I found there was still more on the "additive" side and was more targeted at the typical older exploits - stack smashing, etc. Useful stuff but this didn't really fit the types of problems I was trying solve and there were implications of high overheads in some of the research implementations.
I finally came to the conclusion that a better first step is the disabling or removal of functionality which is most often exploited. This got me thinking a bit out of the box. With Java, it isn't really practical to remove "insecure" features from Java and it's libraries. The use of Java Security Manager can disable the functionality but that turns out to be a nightmare when you start working with lots of Open Source dependencies in large/complex web applications. I really like the idea of disabling items which are commonly exploited though.
As I was thinking about this, it occurred to me that maybe there is a way to disable some items with a minimal overhead. Java is generally implemented in C/C++ and uses things such as the standard C/C++ libraries. The nice thing about this is that features that I "care about" tend to funnel down to a fixed set of functions in the C/C++ libraries. I decided it was not very practical to physically create new versions of glibc, etc though. The good news is that I didn't have to. There is functionality within the Linux loader which makes things fairly easy.
[I had to cut a lot of detail out of this for now; somewhat for time - may be able to include more details later].
A simple use case goes like this; you want to prevent a web server from being able to launch an external process. This is an attack vector which has plagued a number of open source technology on/off. First write a simple Java program which only calls "Runtime.getRuntime.exec()" and have it do something you can verify like adding/appending some data to a file. Next run the test program while prefixing the command with "LD_DEBUG=symbols" - so the command line would look similar to "LD_DEBUG=symbols java p1.p2.C1" . If you start a 'script' before this, you can capture the somewhat substantial output. In the output, I found that the exec() call was using vfork() from the threading library. I determined that my web server really should not need to call vfork() so I chose it as my target for "disablement". So next, write a small C file which implements a dummy vfork() - use the various Unix/Linux man pages to determine what return value should represent an error/failure to the caller and return that in the implementation. In this case, -1 was the appropriate value. Next compile and link that into a shared library. Now, all that remains is to get the Java process to use your vfork() instead of that from the standard threading library. This can be accomplished by starting Java with a "LD_PRELOAD=libyourlib.so" which for testing purposes will include "LD_DEBUG" as well and would look something like: "LD_DEBUG=symbols LD_PRELOAD=libyourlib.so java p1.p2.C1". Again, if you run 'script' before this you can easily review the output and note that the vfork() func was picked from your library or some error in your lib prevented it from loading. Once you get any loading errors resolved (maybe you needed to compile/link with -fPIC, etc) and you see the vfork getting loaded from your library - go back and verify that the Java test program is no longer producing the effect that the Runtime.exec() call was producing.
So at this point you have written a C func for a "standard library" function which hard-codes an error type return value and effectively hides the real implementation of the function.
You can take this general idea as far as you want. Make sure that various IO calls only have access to a predefined list of resources, etc. You are mainly limited by imagination. It is fairly low in overhead but is not a total solution in itself. You could incrementally secure an application over time though which is a nice aspect.
This isn't a silver bullet and won't help everyone all the time. I am sure there are applications which would require a lot more effort to make similar changes to because of an actual need for some of the functionality which does get exploited in web apps.
I plan on keeping this idea in the toolbox for use when it is a good fit. It could also be a good reactive method in a crisis.
[Update 2014/02/25] It seems quite awkward to implement this in a minimally invasive manner with software such as Apache Tomcat because of how the startup scripts work. It may be possible but it will take a lot more scripting research if so. I think the general recommendation I found for a similar need is to simply create a new script with the end result of what the startup script uses to start the Java process. Not a very operational/upgrade friendly method but if you must do it..
[Update 2014/03/17] I am wondering whether JEP178 may provide a better/easier supported way to remove certain limited functionality from the runtime at the process level through some static linking. I think this will need from further review at a later date. This would prevent the need to customize start up scripts and such.
Wednesday, November 13, 2013
Eclipse Memory Analyzer experience
Skipping to the end of the story - the Eclipse Memory Analyzer is awesome!
We have a high profile and very politically entrenched application which is plagued by many issues outside the control of the IT department. Recently a consultant with much history with the application produced changes to support a mobile application. We were told that the changes were minimal - basically bolt-on pieces only. That did not turn out to be the case, the consultants idea of small bolt-on involved replacing some core Struts configuration affecting the original core application as well as the new code. Timing of this could not have been much worse since we ended up needing to upgrade to the Struts 2.5.15.3 library due to security issues at the same time. Anyways, the mobile bolt-on had not been well-tested and neither was the combined application code. This was determined a bit later when IT went back and did some testing of the mobile code related branch code.
Anyways, IT was forced to fix a number of issues before a Struts upgrade can occur while merging the mobile release and producing a build for deployment. Due to a lack of adequate functional user resources assigned to the application, testing is typically inadequate. The build was forced to go into production and then the problems began. The symptoms were basically high heap utilization, increasingly high CPU utilization followed by instability and finally out-of-memory (OOM) errors being logged after which the application is basically brain dead.
After a couple days of analysis, it was still unclear as to what the root cause was. There was never a clear repeating pattern. We ended up implementing multiple daily application restarts to help clear the memory but that was only mildly effective. There were some items which seemed odd while analyzing the application with jvisualvm but the problem was "lost in the trees". Some initial attempts at analyzing the heap dump produced when the OOM error occurred failed because of issues related to the size of the heap dump and a lack of functionality to help identify the root cause.
I had installed the Eclipse Memory Analyzer a while back with the intent of trying it out but never had the time to do so. At this point, it seemed like a good time to try. The first attempt was not successful though since I tried it on my "old" laptop which was running Windows 7 32bit. Fortunately, I had received another laptop a week or so prior with Win 7 x64 which I was in the midst of setting up. After increasing memory in Eclipse upto around 3200mb, I started the import of the heap dump from our prod system and after around 45 minutes it was done processing and offered up to show leak suspects. I gladly selected yes and found that it had one suspect which was taking up more than 300mb of memory. It provided a stack trace which led me to a some particular functionality. The interesting item found was that data in the stack/trace was indicative that the functionality was accessed by a non-logged in user. After some communication with the consultant, functional staff and another developer on the application indicated that the functionality should NOT be accessible to a non-logged in user - further investigation followed. This lead to identifying that a DB call was returning around over 400k rows of data instead of under of 100 rows and was retaining it. It turned out that the consultant had an "OR" condition which explicitly returned all data to a non-logged in user (even though the functionality shouldn't be available when not logged in..). After removing the extraneous OR and having the functional team test for any negative effects - the changes were put into production.
The result of the fix was the application was stable and using under 50% of the average memory in use when unstable and CPU utilization averages mostly in the single digits.
So roughly one hour of work with the Memory Analyzer helped solve a problem which was not anywhere near being solved after multiple days of effort.
Hats off the the people responsible for the Memory Analyzer - you have my gratitude!
Now if we can figure out what the consultant did which is causing a non-logged in behavior.
We have a high profile and very politically entrenched application which is plagued by many issues outside the control of the IT department. Recently a consultant with much history with the application produced changes to support a mobile application. We were told that the changes were minimal - basically bolt-on pieces only. That did not turn out to be the case, the consultants idea of small bolt-on involved replacing some core Struts configuration affecting the original core application as well as the new code. Timing of this could not have been much worse since we ended up needing to upgrade to the Struts 2.5.15.3 library due to security issues at the same time. Anyways, the mobile bolt-on had not been well-tested and neither was the combined application code. This was determined a bit later when IT went back and did some testing of the mobile code related branch code.
Anyways, IT was forced to fix a number of issues before a Struts upgrade can occur while merging the mobile release and producing a build for deployment. Due to a lack of adequate functional user resources assigned to the application, testing is typically inadequate. The build was forced to go into production and then the problems began. The symptoms were basically high heap utilization, increasingly high CPU utilization followed by instability and finally out-of-memory (OOM) errors being logged after which the application is basically brain dead.
After a couple days of analysis, it was still unclear as to what the root cause was. There was never a clear repeating pattern. We ended up implementing multiple daily application restarts to help clear the memory but that was only mildly effective. There were some items which seemed odd while analyzing the application with jvisualvm but the problem was "lost in the trees". Some initial attempts at analyzing the heap dump produced when the OOM error occurred failed because of issues related to the size of the heap dump and a lack of functionality to help identify the root cause.
I had installed the Eclipse Memory Analyzer a while back with the intent of trying it out but never had the time to do so. At this point, it seemed like a good time to try. The first attempt was not successful though since I tried it on my "old" laptop which was running Windows 7 32bit. Fortunately, I had received another laptop a week or so prior with Win 7 x64 which I was in the midst of setting up. After increasing memory in Eclipse upto around 3200mb, I started the import of the heap dump from our prod system and after around 45 minutes it was done processing and offered up to show leak suspects. I gladly selected yes and found that it had one suspect which was taking up more than 300mb of memory. It provided a stack trace which led me to a some particular functionality. The interesting item found was that data in the stack/trace was indicative that the functionality was accessed by a non-logged in user. After some communication with the consultant, functional staff and another developer on the application indicated that the functionality should NOT be accessible to a non-logged in user - further investigation followed. This lead to identifying that a DB call was returning around over 400k rows of data instead of under of 100 rows and was retaining it. It turned out that the consultant had an "OR" condition which explicitly returned all data to a non-logged in user (even though the functionality shouldn't be available when not logged in..). After removing the extraneous OR and having the functional team test for any negative effects - the changes were put into production.
The result of the fix was the application was stable and using under 50% of the average memory in use when unstable and CPU utilization averages mostly in the single digits.
So roughly one hour of work with the Memory Analyzer helped solve a problem which was not anywhere near being solved after multiple days of effort.
Hats off the the people responsible for the Memory Analyzer - you have my gratitude!
Now if we can figure out what the consultant did which is causing a non-logged in behavior.
Subscribe to:
Comments (Atom)