Selecting technology for a software system

Selecting technology for a new software system is challenging. There are very diverse technologies available and opinions on each are equally diverse. You may find some people with the opinion that the selection isn't hard because there is only one good choice (or close to it) - in their opinion given their requirements and assumptions.  

Why do I feel the selection is difficult? There are a number of reasons. 

• Is there a perfect programming language for all needs?
• No; each language has trade-offs.
• In a number of cases, a specific language may be a good choice but you also need to consider different runtime environments for a language and the inherent trade-offs.
• Different business domains / uses may match up better to specific languages or technology stacks.
• A language / runtime isn't a solution on its own; you also need to consider the overall environment in which the software will operate.
• Some software systems require higher security and some technology stacks are more mature in that area.
• Some systems are especially sensitive to the speed of implementation.
• The expected lifetime of a system and maintenance needs affects choices.
• The maturity of the 3rd party ecosystem for a language / tech stack affects choices.
• Another item which tends to be considered but usually without thinking about it is - what languages are your staff skilled with?
• And then you have the choices affecting performance, scalability, reliability, cost, etc.

The above list is a bit abstract in nature.  What are some very real and important examples?

A common language that comes up often as the "golden choice" is Python. Is Python a good language - yes, for appropriate uses/domains.  I won't dive into what are good uses for it, I tend to prefer knowing how to rule out items so I can get to a short list for final selection. In the case of Python, what aspects are worth considering? 

• Memory handling
• Threading / Giant lock
• Build time / dependencies
• Backers of the language

One aspect of memory handling that is problematic (at a minimum for CPython) is that there is no control of limits by the runtime system. In standalone / local applications, this isn't a huge burden but if you are using a containerization / orchestration type systems such as Docker or Kubernetes (K8S) then you run into a dilemma.  Docker / K8S enable you to set memory limits and when those limits are exceeded, generally the container is killed and restarted.  The CPython runtime isn't aware when there is memory pressure and isn't tuned to attempt to prevent over-allocation.  To prevent containers from over allocating memory and being killed - more headroom memory is needed per container instance.  This can reduce the efficient use of memory which increases cost.  If you try to run more CPython processes in a container to improve generally efficiency - you can cause more workloads to restart due to a single over-allocation. The result is poor user experience. Increasing the number of container instances requires adequate memory for each which translates into higher costs.  

CPython also has a giant lock which impedes its multi-threading capabilities (Python GIL / Threading). So managing high request rates in a web application may provide some challenges. You may think it is easily resolved by use of some messaging type system (maybe Kafka, MQ, etc) but if the client libraries don't work-around the inherent issue then you end up with problems that may be hard to diagnose and even harder to fix. 

If you are using a standard interpreted version of Python then you don't really need to worry about build time related to your actual code but you may need to deal with the time required for building any required native dependencies. There are many libraries that work with Python but they are implemented using native libraries which the Python runtime loads and uses. This often means you need C/C++ and / or other compilers to enable building the native libs. This can be slow and error prone when working with containers initially. Given time and effort, you can create a solid build process but this is something that brings the overall complexity of creating, containerizing and deploying a software system to levels similar to compiled languages.

Every popular language remains popular by changing enough to meet new needs and challenges in the software development ecosystem. In some cases, languages are backed by committees (C++) and others have a community/large organization as the primary backing (Java, .Net). For Python, Guido appears to maintain primary control and he has done a wonderful job. The question is - someday, when Guido decides to step away completely - who or what controls the direction of the language and will it diverge much from Guido's current direction?  One link regarding the "no future Python 4" path is: No Python 4.

If you invest heavily in Python and something changes significantly (and quickly) then it could be very costly to change. At the same time, if a language looses popularity after its creator moves on then you may also be in a bad state.

The above coverage of Python is mainly targeted at CPython. The analysis for a different runtime, such as Jython, would need an independent review.  Performance, backing/maintenance and implementation details can be very different.

Another common language stack is NodeJS / JavaScript. It is often touted as high performance and has a large ecosystem of 3rd party libraries.  What aspects of this might affect the consideration for some software system?

Two item that comes up regularly for me are

• The maturity of the 3rd party ecosystem. 
• The expected lifetime of a system.

The problem I tend to run into is that there are many 3rd party libraries but the quality differs drastically and the long term support/maintenance of some frameworks is lacking. If you have a system which you expect to maintain for 5-10 years then you probably want to base it on technology which gets continued incremental maintenance over time.  Relying on unmaintained libraries and frameworks when security is important seems like a poor bet. You also don't want to do massive rewrites of a system every year because some new "great idea" arrived.  I've run into situations where core libraries used in NodeJS applications were abandoned by authors in favor of other completely different solutions.  I'd even agree with the authors decisions to abandon something which was no longer a good fit - but it isn't a good place to be in if you heavily rely on that software in a large or complex system.

Another popular language is Java - what are some considerations for it?

• Speed of system implementation
• Business domain for new system

Java is a good general language but it isn't normally considered as the first choice when "speed to market" is the most critical aspect. If you have a very limited time frame and no other requirements push you towards Java then another language may be more appropriate.  That is also true if your business domain is potentially in a scientific area where there are fewer 3rd party libraries available compared to a language such as Python. 

And if you have requirements which cross a number of these items resulting in no "perfect for this use" selection then you have hard decisions to make. 

This post could go on for many more examples and languages but hopefully provides a useful analysis for some of the challenges involved.

Wishing you the best!

Scott

Application Architecture and Design (and cloud aspects)

 Let's say you work for some organization and there is a need for a new application.  If you are responsible for the architecture / design then 

My initial post was going to end up being a "ancient" IT history book with some extra stuff. I decided that wasn't very helpful - not many people want to read a book in a "blog format".

I'm condensing the content down to the following. 

• Use standard practices for
• project management
• design techniques
• Understand the reasons/basis for soft/hard requirements - This covers why a requirement exist and why a specific requirement was chosen over other similar prospective requirements.
• Examples reasons are;
• development / operational / infrastructure costs
• time to market /  speed of handling requirement changes
• maintainability
• performance / scalability / availability
• Carefully evaluate technology choices.
• Consider the risk of using any technology that could become unsupported before the end of the useful life of your application. New "amazing" technologies may only be around for a short period of time before something new replaces it.
• Make plans to keep 3rd party frameworks, libraries and languages/run-times up-to-date with regard to the most current versions.
• Consider the risk, for example, of using early versions of SpringBoot such as 2.1 which are not maintained now - if a new security flaw comes up, it may be very hard to fix the issue.

With regard to standard practices, use whatever is appropriate for your organization and the project need. Many organizations are trying follow "Agile" practices but that doesn't mean that "Waterfall" is always bad to use. Use what is appropriate for the situation.

Understanding the reasons used for requirement selection helps to keep requirements aligned with each other and provides useful context for handling business changes over time.

Technology comes and goes.  Don't love or hate particular technology - it is a tool.  Use appropriate technology and if needs change then consider more appropriate technology.  Don't select technology just because it is considered "great" by any person or group - consider the positive and negative aspects in-depth for your use-case.  

Small example of considerations - given a service expecting high transaction rates; what technology stacks might be considered and what are the trade-offs?

1. Python
1. Positives
1. Increasingly used in enterprises
2. OpenSource
2. Negatives
1. Library ecosystem isn't as mature as for Java
1. You want a library to help with web requests?  Which option do you pick? What version of Python are you on?
1. https://pypi.org/search/?q=web+requests&page=1
3. Considerations
1. Security
1. https://www.theregister.com/2021/07/28/python_pypi_security/
2. https://medium.com/ochrona/arbitrary-code-execution-during-python-package-installation-3a60990350ef
3. https://snyk.io/blog/python-security-best-practices-cheat-sheet/
2. Some libraries need local compilation
3. Cross-version compatibility
4. Transitive dependency handling
5. Long term maintenance questions
1. https://medium.com/codex/python-4-0-will-never-arrive-3d994dce54f1
6. Runtime aspects
1. Python global lock
1. https://wiki.python.org/moin/GlobalInterpreterLock
2. Memory management - also consider context including containers
1. https://www.askpython.com/python/examples/memory-management-in-python
3. Threading
1. https://docs.python.org/3/library/threading.html
2. Java
1. Positives
1. Heavily used in enterprises
2. Generally has useful features and reasonably performance
3. Many enterprise level libraries and frameworks
2. Negatives
1. New version every 6 months by Oracle can cause some thrashing of work
2. OpenSource vs Oracle licensing
3. Considerations
1. Alternative JDK / JVM / run-times
1. Oracle JRE
2. OpenJDK
3. Azul
4. IBM OpenJ9
5. Eclipse Temurin
6. Amazon Corretto
2. SpringBoot vs JEE
3. Cross-version compatibility
4. Transitive dependency handling

I highly recommend reading about and understanding the runtime aspects of Python. Consider the effects of memory pressure within and across container instances (i.e. Docker, Kubernetes, etc) - especially in a cloud environment and / or where you are cost / latency sensitive. 

Regarding compatibility and transitive dependency handling - I've had people with architect and team lead titles make comments to me along the lines of:

• "Java versions are backward compatible so keeping up with current versions isn't important."
• "We are pinning the SpringBoot version at version 2.3 instead of upgrading to 2.7+ because it improves stability by preventing changes."

I believe these statements reflect some general misunderstandings or lack of knowledge. There are very instances of 100% cross-version compatibility for operating systems or programming languages over significant spans of time. You also have to clarify whether you are talking about things like source code / binary compatibility and forward / backward version compatibility. Try running a bunch of games written for Windows 95 on Windows 11 - how many will work?  In my opinion - likely few.  For operating systems, computer languages and frameworks - features are added and deprecated / removed across releases.  API's change due to needs such as performance, security, new features, etc. If OpenSource is brought up as a solution to these type of issues - I'll claim it isn't reasonable.  There are support matrices for most major OSs, computer languages and library / frameworks - they don't maintain old versions forever. If you are outside a support matrix - you have no guarantee that anything will work and getting help will likely be somewhere between hard and impossible. Needing help for a major security issue or something that breaks a critical system is not the time you want to find out that no one can help you or can't help you quickly. Some common support / compatibility matrices below..

• OpenJDK support matrix from RedHat
• https://access.redhat.com/articles/1299013
• SpringBoot support matrix
• https://spring.io/projects/spring-boot#support
• SpringBoot / SpringCloud compatibility matrix
• https://spring.io/projects/spring-cloud#overview
• Django support matrix / roadmap
• https://www.djangoproject.com/download/
• Nx / NodeJS / TypeScript compatibility matrix
• https://nx.dev/packages/workspace/documents/nx-nodejs-typescript-version-matrix

Thanks,

Scott

Cloud / Containerization / Virtualization Technologies

 This is a bit of a research brain dump regarding a number of technologies and/or projects that are often encountered when looking into cloud/container/virtualization technologies.  

I've condensed this portion to a diagram which is still a work-in-progress.  It tries to organize some common items into "buckets" and tries to hint at some of the relationships between items. As time passes, the differentiation between buckets get cloudy.  Some projects / technologies end up sharing, reworking or supporting other projects in ways that make it hard to distinguish meaningful differences. Other projects start off with lots of excitement and then seem to fade quickly.   

I'm working on another diagram that tries to cover network items deeper along with some security and storage items.  Those are usually not cloud / container / virtualization specific so it made more sense to work on those separately.  That diagram is very messy still - I'll create a new post for it "soonish" with the note that it is very much a work-in-progress.  I'll add a link to it from here though.  

This is all public information from various public projects, tech company public sites, tech "news and information" sites, etc.  There is still a significant amount of change going on and it will be interesting to see what truly new and innovative ideas appear.  

[Edit 2021/01/28] Here is my initial post for network / storage / security focused items:

Software security - is your application secure enough?

Let me start off by saying that there is no such thing as a 100% secure application - unless maybe it is only on a computer that is never turned on and it is the only copy.

I'll qualify that with - I don't have a degree is software security so you may ask why you might pay attention to anything I say? I've been in the software development field for many years and have experienced a number of the failings related to software security. I've worked through mitigating a few instances of those failings and recognize that certain choices make future security failures more likely.

I can't fix your security issues but there is simple advice that *may* help you deliver more secure software. A lot of security advice is pretty standard and I'm hoping to not repeat those items here.  

I'll do a spoiler for this post.

My advice is to at least minimally understand past and current security issues with the technology stack you are considering before fully committing. This doesn't mean read every known possible exploit but you should get to know general types and magnitudes of possible exploits across the stacks you are considering.

What?  Is that all?  How does that help me?

Lets say you have a number of Apache Struts applications as part of your intranet presence.  One day, management decides that there is a need for a great new wiz-bang public facing application.  So far, so good - we'll assume what they are asking for fits a real need and solves some problem.  

You and your team have Apache Struts experience so you decide to just go with Struts as the MVC framework along with other existing and new technology. By templating off of existing internal applications, you produce a nice app very quickly.

Ok, so what is the problem?  When management sees large companies with lots of IT knowledge failing in the area of security - they may ask you what the security implications are for the app. At this point, you become very thoughtful regarding what to say.  You may indicate that you and your developers are following all the best coding practices and therefore the risk is minimal.

This is where I ask - are you missing anything? You may be following all the best coding and deployment practices but in this day and age - nearly every application being built is done so with the help of tons of open source software. Please don't take what I am saying to mean that open source software is bad because it isn't  - but you need to understand each packages' security implications. Most open source software packages contain lots of features you don't need and may never use. Those features may enable attackers a level of unauthorized access to your systems through things like configuration mistakes or oversights. Evaluating current and past security history is a good way to start understanding what general types of things may be found in the future. 

For example, with Apache Struts - if you google for "struts 2 cve" you will likely find a link like:

https://www.cvedetails.com/vulnerability-list/vendor_id-45/product_id-6117/Apache-Struts.html

This site maintains a list of known security vulnerabilities for a large number of software libraries, frameworks, packages, etc.  For Struts you will notice that the list of vulnerabilities is pretty long.

If this was 2015 and you made the decision to use Struts - you would have had the potential to be impacted by the items that WERE found after that - which included quite a few. Since it is now 2018, is Struts secure now? I only noticed 1 or 2 reports in 2018. In the end, you don't know what other security flaws are still unknown but based upon history I would wager that some exist.

Software security management is less about absolutes and more about a level of security and your risk tolerance. Some software is less prone to security issues than other software for various reasons.  Going back to example situation, if instead of struts you had considered JSF and reviewed the CVEs for it:

https://www.cvedetails.com/vulnerability-list/vendor_id-5/product_id-13552/SUN-JSF.html

There is only 1 item - https://www.cvedetails.com/cve/CVE-2008-1285/.  Does that mean that JSF it super secure - not necessarily.  It just means that few security flaws have been reported and show up at the Cvedetails site for this specific Sun JSF implementation.  If you do some googling around further on JSF, you will find additional concerns about JSF implementations regarding view serialization which should be considered. So we can't say that JSF is "secure" but we can say that there is a difference in the known quantity and scope of issues between Struts and JSF.

So what do we get out of this exercise?  Knowing current and past security flaws in potential stacks allows you to select a stack that matches the applications actual security needs. The selection will almost certainly be a trade-off of some sort.  Common trade-offs I have encountered included security, scalability, staff knowledge/experience, ease of use and maintainability.

When someone asks how secure your app is - you can report that the implementation stack and development practices were chosen to meet the actual security need. That doesn't guarantee that an app is secure but hopefully through the choices that were made the app will be less likely to experience a security issue.

In the scenario I described initially - a public facing app, I purposely left out some details to help demonstrate the idea.  If this public facing app was part of a kiosk that runs off a read-only DVD and isn't on a network - do I need to worry hackers on the internet?  No, not really.  Could someone hack a kiosk?  Maybe under some circumstances but is the risk or impact very high?  Doesn't seem like it.  So in this case, something like Struts is probably a fine choice if it meets other functional and soft requirements.  

Now, if the public facing app was related to internet accessible airline reservations with some ties to airline traffic control - I would stick with stacks that have a much smaller list of known security issues and likely significantly different other attributes.

The hard part is making decisions for applications that are in the grey areas. If moderate money, reputation and/or liability is involved - I'd lean towards stacks with fewer security issues unless overridden by stake holders or other requirements. In those cases, document the preferred stack and make note of why something else is chosen. 

Disclaimer: Please note that my example scenarios are intended for demonstration only. 

Thanks for reading.
Scott

Proverbs 19:2 New American Standard Bible (NASB)

Also it is not good for a person to be without knowledge,
And he who hurries [a]his footsteps [b]errs.

Proverbs 14:15 New American Standard Bible (NASB)

The [h]naive believes everything,
But the sensible man considers his steps.

Software security - sad state

Events over the last year have pushed me to investigate software security much closer.

The summary of my thoughts are (somewhat just common sense):

• Developer statements regarding secure coding practices only have real weight if the developers have substantial security training.
• Security is a joke if not part of the initial design.
• Statements from application designers regarding security only have weight if the designer has some security training.
• Systems are at the mercy of their libraries and frameworks.  
• If you use Struts, I think you should be very concerned.
• Libraries and frameworks should be refreshed (or at least reviewed) regularly. 
• It is a bad idea to use extra functionality in libraries "just because you can" versus a realistic need.  I would recommend throw-away prototypes of tools as learning tools instead of high profile production applications.
• Put lots of time into understanding default configurations and any functionality which allows you to disable unneeded functionality. 
• Layers of security are needed.
• Too many people underestimate security needs and don't worry about it unless you end up in the news.  Not being in the news implies that they are doing something right from their perspective - hard to change that. 
• Security is not implement and forget functionality.
• Software companies want to sell you "just one more" layer of security.  Shaped like a silver bullet in worst case.  Other times it indicates it is green product - i.e. much recycled content.
• Security regression tests would be useful.
• Automated security tests, in general, would be a benefit.  Problem is time and resources though. 

I have a few other thoughts I am mulling over;

• The root of many issues is the generalist nature of systems and a hierarchical nature of dependencies. Example:
• Programming frameworks support much more functionality than will be used in individual applications.  Some of that functionality has security implications - such as the ability to launch a new OS process.
• Regarding the previous item, the framework leverages generic functionality in the language runtime to launch a process.
• The  programming language runtime uses OS level support to launch processes.  
• The OS support for launching processes is normally provided in the form of a shared library which provides generic services.
• The shared library services are available to all applications with access to the library.

Restricting access to functionality, such as launching processes, is possible via things such as SELinux but:

1. The complexity is pretty high which also leads to potential errors
2. Harder to provide justification to management for the required time with the ever increasing pace of software development.  
3. This particular solution affects entire servers and most likely many servers.

Other possibilities exist to restrict functionality at the language runtime level (i.e. Java process).  An example of this is implementing a Java Security Manager & appropriate policy.

1. This implies that a large number of activities need to be white listed.  That just isn't reasonable or feasible in some cases due to the amount of code involved and the resources available to maintain the policy.
2. Any change in the environment (i.e. lib or framework version) effectively requires a total security review to prevent inclusion of unused white list entries.

So white listing is difficult because:

1. It is an all or nothing process (i.e. affects an entire server or entire application).  
2. Much of the work is related to 3rd party software (libs and frameworks) for which you would need to figure out the needs from scratch.  

Alternatively, if you are working from a blacklist only standpoint:

1. You may miss items and have unexpected security holes.
2. There are few methods available for blacklisting probably because of issues related to item 1.

If Java flags existed which disabled functionality (process launching), it could be considered a form of blacklisting (functionality).  At the moment, I don't believe any flags like this exist. 

*****
Work in progress.. Rewording some sections needed to clarify the intended thought.  Overall, there has got to be a better way to deal with security (especially in substantially resource constrained shops).

Java/Linux practical security

I have needed to delve more deeply into web application security methods in the last year.  As part of this, I have come to recognize a lot of short comings in this area.  There has been a continuous stream of Java updates from Oracle and a number of security fixes for various Java EE/servlet containers and libraries commonly used in web applications.  It is good to keep the technology stack updated with the most current security fixes - no denying that.  The problem is that this leaves the potential for too much time between unethical people finding the info needed to exercise an exploit and actually having a security fix in production.  And this assumes that a security fix doesn't break your web application which is a real possibility. So this isn't what I would call a first line of security. 

So what does one do?  I have been doing lots of research and the funny thing is that most of it is what I would call "additive security".  What I mean by that is, it is security provided by adding some new mechanism into the technology stack.  Things SELinux, AppArmor, Java Security Manager policies, TripWire, fancy application security firewalls, etc.  I won't deny that those have the potential to prevent various security flaws from being exercised or at least identifying when it occurs so they can have a benefit but they also add complexity and workload to what is often an already overworked IT staff.

I also looked into some various research in the area of "hardened" run times and such.  What I found there was still more on the "additive" side and was more targeted at the typical older exploits - stack smashing, etc.  Useful stuff but this didn't really fit the types of problems I was trying solve and there were implications of high  overheads in some of the research implementations.

I finally came to the conclusion that a better first step is the disabling or removal of functionality which is most often exploited.  This got me thinking a bit out of the box.  With Java, it isn't really practical to remove "insecure" features from Java and it's libraries.  The use of Java Security Manager can disable the functionality but that turns out to be a nightmare when you start working with lots of Open Source dependencies in large/complex web applications.  I really like the idea of disabling items which are commonly exploited though.

As I was thinking about this, it occurred to me that maybe there is a way to disable some items with a minimal overhead. Java is generally implemented in C/C++ and uses things such as the standard C/C++ libraries.  The nice thing about this is that features that I "care about" tend to funnel down to a fixed set of functions in the C/C++ libraries.  I decided it was not very practical to physically create new versions of glibc, etc though.  The good news is that I didn't have to.  There is functionality within the Linux loader which makes things fairly easy.

[I had to cut a lot of detail out of this for now; somewhat for time - may be able to include more details later].

A simple use case goes like this;  you want to prevent a web server from being able to launch an external process.  This is an attack vector which has plagued a number of open source technology on/off.  First write a simple Java program which only calls "Runtime.getRuntime.exec()" and have it do something you can verify like adding/appending some data to a file.  Next run the test program while prefixing the command with "LD_DEBUG=symbols"  - so the command line would look similar to "LD_DEBUG=symbols  java p1.p2.C1"  .  If you start a 'script' before this, you can capture the somewhat substantial output.  In the output, I found that the exec() call was using vfork() from the threading library.  I determined that my web server really should not need to call vfork() so I chose it as my target for "disablement".  So next, write a small C file which implements a dummy vfork() - use the various Unix/Linux man pages to determine what return value should represent an error/failure to the caller and return that in the implementation.  In this case, -1 was the appropriate value.  Next compile and link that into a shared library.  Now, all that remains is to get the Java process to use your vfork() instead of that from the standard threading library.  This can be accomplished by starting Java with a "LD_PRELOAD=libyourlib.so" which for testing purposes will include "LD_DEBUG" as well and would look something like: "LD_DEBUG=symbols  LD_PRELOAD=libyourlib.so  java p1.p2.C1".   Again, if you run 'script' before this you can easily review the output and note that the vfork() func was picked from your library or some error in your lib prevented it from loading.  Once you get any loading errors resolved (maybe you needed to compile/link with -fPIC, etc) and you see the vfork getting loaded from your library - go back and verify that the Java test program is no longer producing the effect that the Runtime.exec() call was producing.

So at this point you have written a C func for a "standard library" function which hard-codes an error type return value and effectively hides the real implementation of the function.

You can take this general idea as far as you want.  Make sure that various IO calls only have access to a predefined list of resources, etc.  You are mainly limited by imagination.  It is fairly low in overhead but is not a total solution in itself.  You could incrementally secure an application over time though which is a nice aspect.

This isn't a silver bullet and won't help everyone all the time.  I am sure there are applications which would require a lot more effort to make similar changes to because of an actual need for some of the functionality which does get exploited in web apps.

I plan on keeping this idea in the toolbox for use when it is a good fit.  It could also be a good reactive method in a crisis.


[Update 2014/02/25] It seems quite awkward to implement this in a minimally invasive manner with software such as Apache Tomcat because of how the startup scripts work.  It may be possible but it will take a lot more scripting research if so.  I think the general recommendation I found for a similar need is to simply create a new script with the end result of what the startup script uses to start the Java process.  Not a very operational/upgrade friendly method but if you must do it..

[Update 2014/03/17] I am wondering whether JEP178 may provide a better/easier supported way to remove certain limited functionality from the runtime at the process level through some static linking.  I think this will need from further review at a later date.  This would prevent the need to customize start up scripts and such.

Java security manager policy mechanism and non-trivial applications

We have a "pest" bothering us and to deal with the issue we decided to implement multiple security mechanisms.  The mechanism I am currently working on revolves around the age old Java security manager policy.  This really should be done at initial design/implementation if it is done at all.  A multitude of open source libraries isn't making this easy.  Not complaining mind you - just wishing that security was part of the initial requirements for all open source libraries.  Unfortunately, there are some pretty popular libraries which are somewhat troublesome and securing them is a bit disconcerting.  I can't comment on what libraries I am referencing out of security concerns and respect for my employer.  I will say that this is making my recommendation for complete removal of the libraries much easier to justify.

The general process to getting the security manager working with an existing application is:
* start application server with security manager enabled
* access application while monitoring logs
* when a failure occurs; update the policy file based on data in the log, clear logs and start this process over again

The app server logging is pretty good; it almost always contains 'denied' in the message and many times the remainder of the content can be cut/pasted nearly as is into the policy file.  There are times that debugging isn't easy; as when there are problems with components which start up early and/or eat exceptions instead of logging.

I would not bother trying to debug the security manager polices by setting

   -Djava.security.debug=all

that just generates way too much useless data.  I think that

   -Djava.security.debug=policy,access

is a better setting which gives a good amount of data to help in resolving failures and understanding what the application is accessing.