FIM management agent w/extension - Disabling AD user accounts

Finally got around to implementing a new FIM management agent (MA) with a custom extension that disables active directory (AD) user accounts.

This has been sitting on my whiteboard list of todo items for some time.  It didn't turn out too difficult but I had to make some changes to another MA which was conflicting with an attribute flow from what I can tell.  That along with some new custom code and MA did the trick.

There was some original (written by a MS consultant) code/setup for disabling users that exist in the FIM management agent (for the portal).  The basic idea that was partially implemented - a custom field in the FIM MA and portal which drives updating the Windows user account control flags (reference) through the Active Directory MA & extension.

The custom field in the portal is updatable. A change to the field ends up in the FIM connector space and propagates to the metaverse (MV).  The MV value flows to the Active Directory (AD) MA which has a custom extension which translates the flag to the appropriate user account control flag value and manages OU changes to the account.  When the custom field indicates to disable a user then logical-or the user account control with 0x2.  When the user is re-enabled then logical-and the user account control flag with a negated 0x2.

We only keep certain users in the FIM MA and portal for performance reasons so the original design wasn't 100% useful and was never fully implemented. It did provide the basic framework for my final solution though.

So what I ended creating is a database feed (new table/views) going to the new MA which has a C# extension.  The database is populated from a utility I created which the local security team has access to.  The new DB feed replaces the custom field in the FIM MA but still updates the same metaverse (MV) data which feeds into the AD MA. So the largest change was removing the existing FIM MA support and replacing it with the new MA.  The remaining changes were more cleanup in nature with some changes needed to simply get things working which had not been fully implemented initially.

At some point, a more general tool will be created to populate the DB table feed and decentralized security teams will have access to it.

FIM R2, Active Directory and the Password Change Notification System - success

Ok, so a project ends up defined different from my expectations and now I need to sync passwords between Active Directory and some external systems where one of the targets is an Oracle database.  After a bit of thought, I decided that the best solution to this was to utilize our FIM R2 environment and implement the MS Password Change Notification Service (PCNS).

This solution is somewhat complicated by the fact that my Windows programming skills are a bit rusty; mostly ending at around MS Visual Studio 6 and prior to having to work with .Net.  At the time, I worked with C/C++ in the Windows environment and C/C++ in several Unix environments as well.  Most of my recent experience is Java.  I decided early on that this solution should use C# to fit in with some existing integrations implemented originally by MS FIM consultants.  The good news is that we had an appropriate version of Visual Studio available and installed where I could use it.  The downside was that after our spring upgrade to FIM R2, I had not taken the time to setup Subversion source control integrations in Visual Studio. 

So before starting any development work; I wanted to make sure I could version control the new stuff.  I had an existing repository for the existing integrations which was good.  I couldn't remember which Subversion Visual Studio integration we used originally but I knew I didn't want to try and use anything requiring a purchase or a license incompatible with our needs.  I ended up using AnkhSVN (which was probably what I did in our original FIM environment).  It was an easy install and compatible license wise.

With my source control prerequisite out of the way; I took a look at some coding resources that a co-worker had found.  One of the resources had some notes, documentation and a code sample/template (found here).  This looked pretty straight forward; so I jumped into the work.

First I needed to create a solution/project; I needed a DLL in this case which is one of the types available.  I renamed the initial c# source file I created to match my intent a bit more.  I was able to add the solution to Subversion pretty easily and tested the check-in of changes in the project to verify it was working ok.  Next, I copied in the template code and then started modifying to our needs.  Some changes I had to make involved externalizing the DB connection info.  I was able to pull some code from a different integration which accessed an XML configuration file for some settings.  I was then able to copy and modify that code to access more appropriate tags containing my DB connection info.  After making a quick copy of the XML configuration file from the other integration and tweaking it to match the expected tag names and connection info; my configuration needs were pretty much done.  After this; I updated the SQL statement per our environment needs. 

At this point; the code was in place but it was not able to compile.  I still had to add some missing assembly references.  Fortunately, with a little trial and error I was able to identify the required assemblies and they were visible in the dialog so I didn't have to go researching where to acquire them or register them, etc. 

Once I got it compiling; there were 2 other items which I identified as needing attention.  I needed to update the project to produce a 64bit artifact and I needed to set the .Net framework to 4.0 to match the existing FIM integrations.

At this point; I had what was a clean but untested build.  I did not have any type of test harness for this so a couple of co-workers setup the AD side of things and performed the FIM side setup which required this DLL.  You have to make sure the DLL is in a particular location for it to be visible in the FIM configuration screen.

Anyways, to make a long story shorter.. after a couple false starts in the FIM/PCNS setup we finally had all the pieces in place and were able to see it work.  A test utility which made password changes against AD was used and we were able to see the effect of the password changes in the destination DB.

There are still some test cases we need to verify because of user on-boarding workflow but the technology is working as expected.  Next, we have some ERP systems to update as well and the basic PCNS work will support that as well.  We will be working these solutions through our test environment on their way to production which should be in the next 4-5 months. 

I do have a few final thoughts on this part of the project.  I am very happy that the technology is working for us.  I do wish that I had a good method for out-of-system testing - hopefully time will permits it as we continue through our test/release process.  As in Java, there are a number of logging libraries for .Net from what I can tell.  In this project, I ended up with the same logging setup as used by our other integrations but the use and setup was not documented [anywhere I have found] by our past consultants.  I feel a bit like I bumbled along with some of this work.  Since we are likely to continue investing in FIM, I'll probably purchase some C#/.Net related books so I can feel a little more comfortable with those technologies.

[Update 2014/12/05] We started to set this up in our test environment but ran into issues.  As it turns out, there are some patch level differences between our FIM environments.  It looks like we need to
update out environments to at least the minimum level of our dev environment which appears to be a unreleased private build from Microsoft.  I suspect our MS guy had used a version available to him which fixed an issue at the time we were working on some other stuff.  We have yet to push some new stuff through the test environment and into prod so we hadn't noticed the difference until now.  There are a few recent public patches so we will likely level-set ourselves to one of those in dev and retest before moving the FIM updates to test and prod.  Wasn't planning on this - never  a dull moment.

Web Services, Service Buses and such

 Problem:
An application sends data updated by users to several other dependent systems (via different mechanisms per target) where each has its own maintenance schedule.  The downtime of dependent systems results in lost updates or delays during manual cleanup.

Solution (attempted):
Use Apache Synapse to mediate between source and target systems.  Setup to queue messages ("dead letter queue") when target system is unavailable.  The queued messages are delivered once the target system becomes available (system retries sending message on a scheduled timer).  A reason this particular solution is attractive is its potential transparency to the application (no code change to client application).

Initial technology involved:  Synapse 2.1.0, CXF 2.4.2 client, MS FIM 2010  & IIS hosted target web service

Solution (implemented):
I ended up implementing several important additions to the client application and some minor changes. Synapse was not used at all. The first addition was the creation of functionality which verifies whether a WSDL URL returns data (no network connection failure and HTTP 200 response). This was integrated into some existing "service checking" code which runs in a regularly scheduled thread and manages the state transitions of some flags which the client application references to determine if services are available without trying to make DB or HTTP calls during each end-user activity.  The second major addition was creation of a queue/web services proxy(implements same interface as existing service) which holds data structures representing the data provided by the caller of the target web service.  The caller was modified to check the WSDL service status flag on each call and if it indicated the service was unavailable, it retrieved a Spring based singleton for the queue/proxy instance.  Also, if the client received a network IO type exception but the service flag indicated the service had been available then the client performs the call again but against the queue/proxy.  The above logic at this point results in either successful calls to the final target web service or  queued up data for future retry calls.  To process any queued up calls, the "service checking" thread (on a transition from WSDL URL down to URL up) will tell the queue based proxy to use the normal client to send the queued messages - so they either success or stay in the queue if there are service/network connectivity issues.  I may document this a bit better here later.

Notes regarding attempt with Synapse:

1. The Synapse docs are not great and a number of classes and settings are poorly named or have various inconsistencies.  There are a good number of examples but they tend to only demonstrate very simple situations.  I found very few third party references to Synapse which makes we wonder how widely it is used in production applications.  I wish there was a greater variety of more complex examples and better documentation.  This may not be a good long term solution overall but could fit a near term tactical need if it worked.
2. The most time consuming aspect was trying to get the CXF client to communicate with Synapse in certain required use cases.  The CXF client utilizes the exposed WSDL and did not like what was being produced by Synapse.  
1. A better detailed explanation is:
1. CXF did not like the default WSDL returned by Synapse - which was  really a modified version of what the live FIM web service returned.
2. The use case requirement was that the CXF client must be able to communicate with the Synapse proxy when the final target web service is not running (implying that the target FIM web service isn't returning data for a WSDL request).
2. The normal Synapse samples appear to require the target service to return WSDL.  There was some documentation/samples mentioning the Synapse way of producing a hard coded WSDL.
3. The solution which was nearest to being successful was to produce a compatible WSDL using wsdl2soap and force Synapse to return that WSDL.  The problem run info though is that some imported schema references were still referencing the FIM/IIS server which would not be available in this use case.  I gave up trying to rework the WSDL to get around parts being imported.  Getting to that point was about 2 days of time and the need for a production solution was only about a week away. 
3. It took 3-4 days to get the basic proxy with in-memory "dead letter" queue support working.  A bit embarrassing that I missed something in the docs and did not spend enough time in the samples which resulted in about 2-3 lost hours fighting an incorrect URL for the proxy in some initial prototyping.  Synapse is using Axis 2 and doesn't allow custom URL's which is somewhat annoying.  During my testing of the target service down use case, I was running into null pointer exceptions in the Synapse code.  I was able to track that down and a minor change to their source and a local build got me farther but then I was getting Synapse errors about configuration in wrong format as it prepared to reprocess messages when I would transition from the target service down to target service up.  After about 5 minutes of looking at the Synapse source producing the error I had to make the call that this was not the way to go at this time.  I may try to submit the couple changes I made back to Synapse (if my management is ok with it).  I wish this had been more straight forward.
4. Some further googling on the general "dead letter" handling in many systems returned lots of similar reports of people trying to something similar and technology used not fully supporting it.  I know I saw some thread which indicated changes had been checked into some project (Synapse, ServiceMix or Camel??) which sounded like it would come closer to handling this use case but I don't think a release is out yet.  Don't remember which one it was at the moment.  Will have to revisit in future - I know this type of need will occur again.

Technology considered:

• Apache ServiceMix
• took too much time going through docs; feature rich but complex
• Not sure we are ready to tackle OSGI
• Plan to review further later
• Apache Camel
• Usable from ServiceMix or directly
• Started looking into this but research incomplete
• Mule ESB
• Need to review license/user agreement further; don't want to get tied up in legal tape
• I think this solution requires attribution to original authors (and I am all for that) but I am not sure of how to get management/legal authorization to make that type of change.  Not sure the solution warrants the pain of trying to work through the red tape. 
• Not sure I want to maintain an Erlang install on servers
• WebMethods
• We own this but are trying to migrate off due to ridiculous cost for only minor benefit
• Reasonably easy to use
• Oracle ESB
• We own this as well but cost is excessive
• Resource intensive & complex
• Forces certain architectural aspects to meet our needs which increases overall cost
• WSO2 ESB
• Need to review license/user agreement; don't want to get tied up in legal tape

Several others were reviewed but don't remember which ones off the top of my head.

Things to consider:

1. At some point, we will likely implement some work-flow solutions so we should make sure that any BPEL type technology integrates cleanly into long term technology selections.  
2.  Long term I expect large amounts of various application functionality to end up being shared.  I expect this sharing to likely be done by exposing it via web services (OK, I'll say SOA).  It doesn't make sense to reimplement the wheel or share only via frameworks. 
3. With a future out-sourced portal in the plans, it makes a lot of sense to use web services locally and only expose the minimal interface to the provider that will host the portal.  This should reduce the security exposure by limiting secure data access to well defined API's which can be secured and audited easily.

MS FIM 2010 - the good and the bad

Identity management is an area which has been growing in importance for some time.  There are numerous commercial products and a few decent open source options.  This post is mainly to document the issues we encountered with MS FIM 2010.

• Support
• There is very little FIM knowledge available.  You should expect to use MS consultants and because of the small number that know FIM you may encounter delays.  
• There is some online information available and I think there is now a course for FIM but there are substantial training needs just to understand the basics. 
• Premier support and MS consulting do try to do the best they can
• Architecture
• It requires SQLServer
• If you are not a SQLServer shop and are not prepared to support it then this can be a substantial issue.
• It is complex 
• Our implementation has various items coded in C# (by a MS consultant), setup in FIM portal and setups in FIM sync service.  I would not consider our identity needs very complex but we do have lots of information to manage.
• Do you need high availability?  Are you prepared to implement/manage/support SQLServer clustering? 
• Certain processes cannot run concurrently and generate various problems if they do collide.  This turns into a juggling act to handle peak times and provide a fast turn-around on additions/changes.  In the end, large implementations likely have to decide on trade-offs whether they want to or not.
• It is a somewhat painful fit into our disaster recovery infrastructure.  The root issue may be how the licensing code works in the overall product. In the end, we are not able to do a real DR test and can only document the steps which we think are required.
• Performance/scalability
• If you have *lots* of users then performance can be an issue
• Initial loading of data for hundreds of thousands to around a million users is pretty slow
• If you experience a high rate of provisioning or updating users then you may have some unpredictably long jobs
•  More server cores != improved performance.  Basically single threaded processing in certain areas of the application from what I can tell.
• We have an ERP as a source which has thousands of tables in the system catalog.  FIM doesn't handle this well at all - likely trying to populate drop down lists and such.  We ended up routing the ERP data through another DB system (provide views to the target tables over links) so the size of the DB catalog didn't cause problems. 
• The normal response to certain issues is "full sync".  This is sort of the "reboot" equivalent to OS problems.  With a large implementation, this will likely result in processing delays.
•  
• Stability/Quality
• We encountered some errors early on which required hacks to get around and required substantial time for MS to provide hot fixes
• We continue to have some issues which have no obvious source in production and have not been reproduced in non-production.  It is possible that currently unapplied updates and hot fixes may remedy the issues but we are in effect a guinea pig for some updates/fixes.
• 
  
• Ease of use.  
• The goal of a GUI is to improve ease of use but when there are items that if selected (or not selected) at the right time can result in massive delays as it tries to process the "mistake" aspect of a request. 
• The reporting capabilities are horrendous and MS knows it.  Maybe it will improv in 2010 R2 but I will wait and see.  If you are trying to research large number of errors you will find it beyond frustrating that you can't export a list of errors.  In the best case, you have to perform a good number of mouse clicks per entry to cut/paste what you need into Excel, etc.  In the worst cased, there are places where all we could do was perform a screen shot or manually transcribe data because you couldn't select it for cut/paste.  Error  messages are poor and misleading in some cases.
• Validation of data is painful.  You are not given (SQLServer) schema information for FIM so if your primary identity sources are a database you cannot simply write utilities to compare sources.  We are implementing custom utilities to try and compare one of our data sources to some data that ends up in active directory.  So we are able to compare the beginning and end but cannot see intermediate results when they don't match.
• Organization/Corporate culture
• If scope is increased above what is initially planned and you have hard deadlines - you likely will regret choices which end up rushed and lack enough forethought.
• If you don't have some (mostly) dedicated staff then expect a lot of unhappiness
• If customers(users of the data) are not fully involved early on to help determine the data needs and validate a data dictionary - expect thrashing later.  

Related commercial tools:

• http://www.oracle.com/us/products/middleware/identity-management/overview/index.html
•  http://www.quest.com/identity-management/
•  http://www.bmc.com/products/offering/Identity-Management.html
•  http://www-01.ibm.com/software/tivoli/products/identity-mgr/
•  https://www.pingidentity.com/products/pingone/
•  http://www.openiam.com
•  

Related open source tools:

• http://www.forgerock.org/
• http://developers.sun.com/identity/
• http://shibboleth.net/
• http://www.jasig.org/cas
• http://www.josso.org
• http://shiro.apache.org/
• http://static.springsource.org/spring-security/site/
• http://openid.net/
• http://www.sourceid.org/

As a note to self, I should keep an eye on the items at forgerock.org and maybe consider a proof of concept in case FIM doesn't work out.