Apache ServiceMix - intial thoughts

So you have a really expensive (purchase price) integration product which has ridiculously high yearly maintenance fees in a public service type organization which isn't out to make money.

What can you do to bring costs down and provide better value to the public/consumers?

In this case, I am evaluating replacing the commercial product with Apache ServiceMix.

My first impression is that there is a lot of value bundled with ServiceMix and the wide assortment of technologies it works with.  The initial major downside appears to be a lack of equivalent tooling for handling the details of various DB trigger based integrations.  The existing commercial tool is more technical-end-user / operations oriented than ServiceMix which is developer oriented.  For this organization, this is a somewhat painful tradeoff but hopefully methods to reduce the pain will be found over time.

Other items to note include there is a general expectation that Maven is used; the ServiceMix examples involve its use.  I decided to include Eclipse in the mix since a number of people will be involved in this and I am hoping that a GUI may help in the transition.  I may regret the Eclipse / Maven integration; it has provided a few pain points in getting started.  If there is a budget for more formal training, it would likely be very worthwhile.

Primary aspects of ServiceMix functionality I want to evaluate:

• OSGI and the impact on the development and operations processes
• CXF web service implementations
• Camel routes to tie some of the functionality together
• ActiveMQ for some internal messaging/high availability/reliability needs
• Activiti for performing some more complex workflows

I'm not an expect (yet!) in any of these but I have been slowly starting to work myself through the details.  I've been spending a little time over the last month or so getting a better grasp of the technology and if/how it may fit into our needs. Below is just my brief overview - I am planning to more in-depth posts as time permits.

OSGI is a lot of things but it all starts with modularity.  In the case of ServiceMix, Karaf is the underlying core OSGi container.  I am focusing on Karaf 2.4.0 (in ServiceMix 5.3.0) for now; it is a recent release which is sort of a bridge between major differences between 2.3.x and 3.0.x.  It supports the OSGi R5 specification and has fairly recent dependencies.  I am not convinced that we really need some of the features in R5+ right now but hopefully this version may ease any later upgrade we need to perform.  There are some supporting technologies which I am looking into as well.  The Apache Cave and Cellar projects are partly what drew me into taking a closer look.  The Cave project is an OSGi bundle repository which I am still figuring out how it fits in compared to a plain Maven repository or repository manager such as Artifactory.  The Cellar project is for clustering Karaf.  This may be a good method of gaining scalability and availability in a production environment.

I have been prototyping a few things but decided to take a chance on a few books to see if they could provide a little extra insight or provide quicker/more clearly.  The books in question are

• Enterprise OSGi in Action - Manning Publishing
• Learning Karaf Cellar -  PACKT Publishing
• Learning Apache Karaf - PACKT Publishing
• Apache Karaf Cookbook - PACKT Publishing

Sort of funny but I am reading them in the above order.

I had picked up Enterprise OSGi in Action quite a while back and had skimmed it but had no time to really put it to use.  I think it is a decent book and was my main initial exposure to much of the terminology.  I do find that there are things that didn't become clearer until reading a few other books/resources. 

I found Learning Karaf Cellar to be a reasonable book.  It did clarify and confirm a few things I was wondering.  It didn't answer all my questions but got me a little further than I was.  One area of clarification includes the need/use of the cellar-eventadmin feature - in my prior testing I had installed it but the book seems to neglect it.  Most of what I was really looking for (at the moment) was in the last 20 pages or so of the book.

The Learning Apache Karaf has a few items I did not know.  There is overlap with the Cellar book but that isn't too terrible.  Since I have been playing with Karaf for a bit over a month now some of the basic command info wasn't needed but if you were starting from nothing would be useful.

I'm just poking around the Cookbook for now.  It tries to point out differences between Karaf 2.x and 3.0 but may be more 3.0 focused.  Some of the items are interesting and I may try to evaluate or leverage.  Will have to really read more in depth before commenting much more on it.

Between my research and prototyping, I am thinking that my desire to use the Distrubuted OSGi (DOSGi) enterprise functionality may be premature (with regard to remote services mainly).  I think just clustering Karaf with Cellar and putting it behind a load balanced will likely provide most of what I am thinking of for an initial deployment.  I also recognize that the default Cellar setup isn't fully capable of meeting my initial expectations - mainly by not persisting some settings/config to disk.  The Cellar book discusses that aspect of their use of Hazelcast a bit which is nice but I will likely have to find some Hazelcast specific documentations to get the details I need.  I am still trying to  work out if/how declarative services should fit in to my plans (versus Spring or Blueprint).

Onto CXF; I have a service and client in prod using this but they are outside of any OSGi environment.  I have prototyped a couple other services in OSGi with CXF and am very pleased with some aspects.  One big plus is with PeopleSoft; I must be able to utilize multiple versions of the PeopleSoft provided API jars for accessing application servers.  OSGi allows me to do that; allowing me to create services targetting multiple PeopleSoft instances with different PeopleTools versions and have those services deployed in the same process.

For Camel, I am still working on how to best utilize it.  I think that creating a number of fine grained services in place of the few but beast like monolithic services is a better way to go.  With those in place, I can use Camel to tie them together to match existing data flows.  Combined with OSGi, there is a distinct dynamism which we don't currently have which hopefully speed up turn-around on changes.

I am still considering ActiveMQ for our environment.  It is a touch call for whether the cost of potential lost transactions is higher than the implementation and operations cost of using ActiveMQ.  I am really torn on this; it would probably benefit our users on likely rare occasions but would incur a cost to manage/maintain.  Still an item of active consideration.

There are a few places where real workflow would just make a lot of sense and using appropriate tools likely will be a big benefit here versus just codifying it.  That is the reason I would like to look into Activiti.  It will not do much without changes to some of our applications and those are likely big changes.  This is a longer term target.

Adopting something like this will have an effect on our development processes/environment, testing environments, production setups and deployment/maintenance methods.  I'm planning on trying to document some of the details of processes and procedures I am going through during my research and planning. 

God Bless!
Scott

FIM R2, Active Directory and the Password Change Notification System - success

Ok, so a project ends up defined different from my expectations and now I need to sync passwords between Active Directory and some external systems where one of the targets is an Oracle database.  After a bit of thought, I decided that the best solution to this was to utilize our FIM R2 environment and implement the MS Password Change Notification Service (PCNS).

This solution is somewhat complicated by the fact that my Windows programming skills are a bit rusty; mostly ending at around MS Visual Studio 6 and prior to having to work with .Net.  At the time, I worked with C/C++ in the Windows environment and C/C++ in several Unix environments as well.  Most of my recent experience is Java.  I decided early on that this solution should use C# to fit in with some existing integrations implemented originally by MS FIM consultants.  The good news is that we had an appropriate version of Visual Studio available and installed where I could use it.  The downside was that after our spring upgrade to FIM R2, I had not taken the time to setup Subversion source control integrations in Visual Studio. 

So before starting any development work; I wanted to make sure I could version control the new stuff.  I had an existing repository for the existing integrations which was good.  I couldn't remember which Subversion Visual Studio integration we used originally but I knew I didn't want to try and use anything requiring a purchase or a license incompatible with our needs.  I ended up using AnkhSVN (which was probably what I did in our original FIM environment).  It was an easy install and compatible license wise.

With my source control prerequisite out of the way; I took a look at some coding resources that a co-worker had found.  One of the resources had some notes, documentation and a code sample/template (found here).  This looked pretty straight forward; so I jumped into the work.

First I needed to create a solution/project; I needed a DLL in this case which is one of the types available.  I renamed the initial c# source file I created to match my intent a bit more.  I was able to add the solution to Subversion pretty easily and tested the check-in of changes in the project to verify it was working ok.  Next, I copied in the template code and then started modifying to our needs.  Some changes I had to make involved externalizing the DB connection info.  I was able to pull some code from a different integration which accessed an XML configuration file for some settings.  I was then able to copy and modify that code to access more appropriate tags containing my DB connection info.  After making a quick copy of the XML configuration file from the other integration and tweaking it to match the expected tag names and connection info; my configuration needs were pretty much done.  After this; I updated the SQL statement per our environment needs. 

At this point; the code was in place but it was not able to compile.  I still had to add some missing assembly references.  Fortunately, with a little trial and error I was able to identify the required assemblies and they were visible in the dialog so I didn't have to go researching where to acquire them or register them, etc. 

Once I got it compiling; there were 2 other items which I identified as needing attention.  I needed to update the project to produce a 64bit artifact and I needed to set the .Net framework to 4.0 to match the existing FIM integrations.

At this point; I had what was a clean but untested build.  I did not have any type of test harness for this so a couple of co-workers setup the AD side of things and performed the FIM side setup which required this DLL.  You have to make sure the DLL is in a particular location for it to be visible in the FIM configuration screen.

Anyways, to make a long story shorter.. after a couple false starts in the FIM/PCNS setup we finally had all the pieces in place and were able to see it work.  A test utility which made password changes against AD was used and we were able to see the effect of the password changes in the destination DB.

There are still some test cases we need to verify because of user on-boarding workflow but the technology is working as expected.  Next, we have some ERP systems to update as well and the basic PCNS work will support that as well.  We will be working these solutions through our test environment on their way to production which should be in the next 4-5 months. 

I do have a few final thoughts on this part of the project.  I am very happy that the technology is working for us.  I do wish that I had a good method for out-of-system testing - hopefully time will permits it as we continue through our test/release process.  As in Java, there are a number of logging libraries for .Net from what I can tell.  In this project, I ended up with the same logging setup as used by our other integrations but the use and setup was not documented [anywhere I have found] by our past consultants.  I feel a bit like I bumbled along with some of this work.  Since we are likely to continue investing in FIM, I'll probably purchase some C#/.Net related books so I can feel a little more comfortable with those technologies.

[Update 2014/12/05] We started to set this up in our test environment but ran into issues.  As it turns out, there are some patch level differences between our FIM environments.  It looks like we need to
update out environments to at least the minimum level of our dev environment which appears to be a unreleased private build from Microsoft.  I suspect our MS guy had used a version available to him which fixed an issue at the time we were working on some other stuff.  We have yet to push some new stuff through the test environment and into prod so we hadn't noticed the difference until now.  There are a few recent public patches so we will likely level-set ourselves to one of those in dev and retest before moving the FIM updates to test and prod.  Wasn't planning on this - never  a dull moment.

About Me - work & goals

 My view of employment

• Work is a gift from God.  I feel too many people treat it as a curse.
• I believe in doing the best I possibly can; even if it isn't something enjoyable at various times.
• I believe that my work should benefit my employer.
• To excel at what I do requires continuous learning; 
• books, research papers, prototyping, self-learning, organizations like the ACM, Google and software related sites are required resources both during work and outside of work to gain and maintain skills
• Learning from others is wise; only relying on other peers is laziness.  You cannot improve at something without actually spending your time doing the work. 
• Don't just give up when problems get difficult; brainstorm with others, research alternative solutions, redefine the problem to provide a way to a different solution space, rethink the box.
• Work ethic is important to me; don't cut corners simply to take the easy way out.  If an employer requires cutting corners; make the implications known and either do your best or find an employer with compatible work ethics depending on the circumstances.
• Admit mistakes, learn from them and move on.  
• Remember to treat others as I want to be treated; abrasive personalities make this difficult but do the best to readjust self-attitude before making comments which cannot be taken back.
• Help others in need whenever possible.
• A sense of humor can help everyone during stressful time
• I should provide an honest assessment of both things that I can do and things that I am  not (overly) familiar with.  If you don't know something say so; even if you plan to figure it out.
• I should provide as realistic estimates as possible; yes a fudge factor for the unknown may be warranted.  If an organization has little/no emphasis on managing overall employee workload and the project on-boarding is a high-speed ramp then estimations probably have little value though.

My personal hope and expectations of an employer

• Respect for me and my family. 
• Respect for employees as a whole.  Actions speak much louder than words.
• Appropriate support so I may fully perform my duties to the best of my abilities 
• Reasonably clear and attainable goals
• Long hours on occasion are an expected part of the job; normal planning using 45+ hour weeks is not respectful of anyone or their families IF not stated during interviews and paid appropriately for the overtime.  In an interview, stating "occasional overtime required" but expecting normal 42+ hour weeks is deceitful.  I expect an honest assessment of the organizations needs. 
• Pay which matches industry expectations for the job function and area.
• The ability to use annual leave without a guilt trip.  i.e. having 6 months out of the year which are "too busy to use leave" is not very respectful unless it is made clear during interviews. 
• A day off should not have the normal meaning/expectation of "work from home".
• I expect managers not to use fear and intimidation as management tactics.

Hope this provides some insight into me for anyone interested.

For any readers of this that are interested in some of the biblical backing, here are some references from the New American Standard Bible of things I try to live by (but fail at all to often).

• Colossians 3:23 - Whatever you do, do your work heartily, as for the Lord rather than for men.
• Proverbs 14:23 - In all labor there is profit, but mere talk  leads only to poverty.
• Acts 20:35 - In everything I showed you that by working hard in this manner you must help the weak and remember the words of the Lord Jesus , that He Himself said, 'It is more blessed to give than to receive.'
• 1 Peter 4:10-11 - As each one has received a special gift, employ it in serving one another as good stewards of the manifold grace of God.
• 1 Timothy 5:8 - But if anyone does not provide for his own, and especially for those of his household, he has denied the faith and is worse than an unbeliever.

Welding table/cart - shielding gas tank holder

I finally got around to purchasing some shielding gas.  I ended up getting it from Arcet in Richmond.  I had intended to lease a largish tank (from a hobby perspective) of around 120-150 cf but they talked me into doing a purchase of a 80 cf tank.  It easy to move the tank around at least.

Anyways, I had intended to strap the large tank along the wall but with the smaller tank I decided to mount it to my welding cart/table.  I had intentionally left a spot on the rear of it to do this someday and now it is time. 

For now, I purchased a tank of Argon/CO2 (C25).  I would like to try out some Tig welding but am going to save up a bit of money and do some further research before selecting a torch/lead and an additional tank of straight argon.  With this in mind, I am planning on how to mount 2 tanks.  I'll just repeat what I did here for the other tank at some point.

I did a couple drawings and decided on a basic idea I wanted to try.  The basic idea involved adding a bracket to the cart to which I could mount a small piece of open ended square tubing mounted vertically.  I would also weld a small length of round bar to a large pipe clamp.  The pipe clamp is slid over the tank and the round bar is slipped into the top of the square tube.  The pipe clamp is then tightened a bit to keep it from moving around. 

Comments/Lessons learned:

• It is hard to keep skills up when you only weld every few months.  My welds are ugly.. :)
• For the pin to clamp weld, I had a good amount of trouble.  I tried the technique of using settings closer to needed for the pin and only slipping briefly onto the clamp but the heat was still WAY TOO high.  It blew through the clamp significantly on one side.  For the other side, I split the difference roughly between that needed for clamp thickness and the pin which was a little better but still well on the high side.  For what I am doing the clamp will work for now but I may do it over with different(lower) settings.
• I may add an additional clamp  higher up for better support or just wrap a chain around for "just in case".
• A minor challenge for this was that I have a folding side extension on the cart and when it folds down, part of the hinge bracket was going to rub on the tank.  A minute with the angle grinder allowed me to trim off the (unneeded) corner of the bracket which was rubbing.

Here is a rear view of the cart.  The new bracket is on the left side and you can see the pin in the square tube.

Here are 2 side views of without/with the tank.  You can see the square tube which accepts the pin.

 

Here is the pipe clamp with the pin welded on - this is shown upside down from how it is used.

 

Hope you found this interesting.

Have a blessed day!
Scott

Hobby Metal working on a budget and lessons learned - conclusion

In my previous post, I talked about some useful tools and a project where I tried out my Harbor Freight bench top drill.  In this post, I am simply going list out what I think is a useful progression of tools and some of the reasoning/needs.  I originally had a goal of producing a post with a spreadsheet which could help in determining a rough tooling cost but that  turned out to be too complex.  There are too many variations in needs to do what I wanted.  Someone who just wants to some arc welding can probably get by with < $1000 in equipment; maybe a touch less if you find deals/used items.  Someone wanting to welding + precision metal working on larger projects might be looking at $10,000-$50,000 in equipment depending on the details (but still talking about hobby scale).

Some of this is based on my own limited experience.  Some is based on research I did on various manufacturers, suppliers, youtube videos and numerous blogs.  I also have a few books which have been extremely helpful.

• Machine Shop Essentials - Questions & Answers
• Second Edition, Frank Marlow, PE
• Welding Essentials - Questions & Answers
• Expanded first-edition, William Galvery, Frank Marlow
• Welding Know-How - Tips & Techniques of Master Welders
• Frank Marlow, PE with P.J. Tallman

I highly recommend those books, especially if you are new to welding or metal working.  Overall there is a huge amount of great content.  You will find some parts that are pretty much shared as-is between the books - mainly between the welding books and a lesser extent with the machine shop book.  There is still interesting and/or useful content in each book.  I also think that there are a few mistakes in the books but I have not gone to check and see if there was existing errata covering the items I found. The writing is generally pretty clear.  There are a few items I wish could be expressed in more detail to make sure I truly understand.  If I had to pick the top 2 books it would be the first and last in my list.

Regarding this post, one observation I want to make is that some tools usefulness is dependent upon the types of projects you intend to work on.  It very hard to make general recommendations due to the number and/or type of variables involved - things from user skill/training, material differences, availability of supporting materials/parts, storage/usage space, portability, etc.  I am trying to focus on a more hobbyist view but I admit that several items listed are very unlikely to be found in a hobbyist workshop.

My relative cost info is very rough.  Since many of these technologies/items I don't own, it is harder to provide accurate estimates without lots of extra research.  I am saying these are best guesses / educated guesses depending. Feel free to disagree or disregard; I'm sure many folks have much more experience with most of the items.

• Items in green I use now - a general progression once I bought a welder and likely what many others start out with.
• Items in orange I am evaluating with the hope of picking one as being the most useful.
• Items in yellow I deem highly usable but due to cost need to save up for. I may have a specific vendor/model in mind - will comment if known.
• Items in purple are on a wish list but likely unobtainable unless I can move from hobby to business. 

Another useful list with a small business slant:starting a metal working business

Cutting action
Saw/slice	Abrasive	Non/low-contact	Initial cost	Ongoing cost	Cut Quality (general use)	Best Use	Notes
Hacksaw			$	$	low/medium	Small finishing cuts, parting off, straight line	Get a good arm workout
Metal snips			$	$	low	Getting forearm workout	Maybe ok for some short/easy cuts but I found much to dislike otherwise
	3" High Speed Cut-Off Tool (pneumatic)		$	$$	low	Non-cosmetic cuts, not overly thick; ok on flat stock	Need decent air compressor.  I tend to destroy blades with under-supported work (pinching).
	4 ½ angle grinder		$	$$	low	non-cosmetic cuts, not overly thick; ok on flat stock	I tend to destroy blades with under-supported work (pinching).
	14” abrasive chop saw		$$	$$	low/medium	straight/small miter cuts on bar/tube/angle
Electric shear			$	$	low	~18+ gauge sheet metal	Tons better than snips. Variety of vendors/capability/cost
Jig saw			$$	$	low/medium	flat material was ok; must be WELL secured.	I found thick angle iron painful to cut this way. Make sure you are not setup with any type orbital action.
Metal cutting circular saw			$$$	$$	low/medium	plate/sheet, relatively straight
Dry cut saw			$$$	$$	medium	straight cuts;bar /tube /angle	user skill and maybe other factors appear to play a good sized part in how well this works.
Metal cutting band saw			$$$	$	medium	General use, small curves possible depending on material/thickness/saw style/blade type-size	Larger foot print is a downside.  Vertical/horizontal or combo models available.
		wet SMAW rods or specialized rods	$	$	very poor	making ugly holes/cuts	emergency use
		Oxygen lance/burning bar	?	?	poor	very thick materials	burning bar can penetrate concrete; high oxygen consumption
		Air Carbon Arc cutting	$$	$$	low	sever items; great for gouging	Need decent air compressor
		Oxygen / Acetylene cutting torch	$$$	$$	medium	Best above a minimum thickness; curves/shapes fairly simple; can cut very thick material	Not for aluminum/stainless steel; completely portable
		Plasma Cutter	$$$$	$	medium/high	Hand held < ~1.5” or so;easy curves/shapes	Any electrically conductive metal/material; need a decent air compressor.  Need very dry air supply.
		Wire EDM	$$$$$	$$$	very high	higher end, professional detailed cuts/shaping	It is possible that this can be done at a hobby level but would take some decent skills to build.
		Water jet	$$$$$$	$$$$	very high	higher end, professional detailed cuts/shaping	The high pressures make this pretty infeasible for hobby use

Regarding an abrasive cutoff saw, dry cut saw and band saw; I am torn on that mainly due to space.  I think a band saw would provide the most flexibility but I am running out of space fast. I have a couple other items which I would love to acquire at some point which may require the remaining space so I am just letting the decision sit for now (& save for plasma cutter).  Maybe I will find a job which will provide the spare cash for a larger garage :)  Or maybe some folks will want to put some money together for a shared off-site workshop;  maybe start something like this TechShop.

Regarding a plasma cutter; I have been considering both the Thermal Dynamics Cutmaster 42 and the Hypertherm Powermax 30xp.  It is a hard decision from the cost differential.  There is a large amount of pro-Hypertherm user reviews out there and it seems like they have their act together (Jim Colt from Hypertherm posts lots of very useful info in various forums).  From a hobbyist perspective though, is the initial cost differential worth the longer consumable life (it seems) and maybe a bit better support?  I'm certain that the commercial users get great support from Hypertherm but I am less certain from a hobbyist perspective of the support differences - I don't expect them to show up in my garage to help me figure out some need/problem I have.  At the same time, I do know that Hypertherm is made in USA.  I have seen numerous folks comment on finding a used item but at the same time - people indicate that many good quality used items retain their value so you pay near retail but may not have any warranty available.  Tough decision and I don't know anyone with units so that I can determine what is best for me.  Additionally, I am still in the "experimenting" stage so "best for me" is not easily defined (if at all) - I know I want to clean cut 1/4" mild steel and maybe stainless and aluminum - but not sure how much thicker I would go (probably not much..).

[Update] Ok, as I review the Powermax 30xp further, I lean more toward it.  There seem to be reports of repeatable good consumable lifetimes and few problems.  I don't see as many reports on the TD equipment.  I will have to see what my local welding supply places sell - I would like to know there is a local repair option for some of the more expensive equipment.  If I ever build a CNC plasma table, it sounds like I could still use the Powermax 30xp (just no machine torch and capacity pretty low for CNC work).

Here are the hole making and forming type items.

Item	Notes	My take	Cost
Hand held drill	Pretty much a standard for anyone doing any type of home/hobby work. Different size chuck and styles; corded/cordless options as well. Better for thin or softer material.	Consider more of a given than a requirement. Complements a drill press.	$
Drill Press	fast/flexible setup; fast turn-around; relatively small foot print	To get nice plumb holes, a good quality drill press really comes in handy. I just got a 50-60 year old drill press from my father and the quality is much higher than a Harbor freight version. Craigslist is your friend – reasonable deals can be found.	$-$$
Hydraulic press	Bending, pressing. Various extras allow creating box pans and such. Some cross over with an Iron worker but is generally more of a complement.	Presses can be found at reasonable prices on craigslist sometimes; quality, capability and features can vary significantly. Tooling may add a lot to total cost depending on what you are doing. Could use for ornamental art, vehicle maint, help with creating drawers and carts, etc. I'd recommend 20-50 ton capacity.	$-$$
Milling machine	Ability to do more than just make holes (compared to to drill press). Groove/slot and larger cuts. Slower setup and turn-around. Complements a lathes functions.	A vertical knee type Bridgeport clone is likely the most flexible machine you can choose. Many options; DRO, power down feed, power axis, full CNC. I would enjoy a Lagun but I think I have a better chance of finding an affordable Sharp or Wells-Index. There are a number of other good brands but these are the ones which seem to be a good value and seem somewhat common. Tooling is a very large cost to account for and should not be underestimated.	$$$
Metal Lathe	Turning, threading, complements a mills functions.	I don't really have any particular need for a metal lathe at this time. It would likely take a larger/longe item to push me to need a lathe.	$$$
Iron Worker	shear, press, punch, notch, bend. Typically flats, angle, rounds, tubing. For some operations, this is much faster than using a drill press or mill.	Part of me would really like to get an iron worker. I can envision using it to work with ornamental art and lot of other things. The cost and size are a deterrent though.	$$$

Here is a quick list of lots of useful "little" stuff - lots of it depends on what you are doing.  Low precision (maybe metal artwork) versus high precision (such as engine parts) makes a large difference in selecting items.  Browse an online site like use-enco to see a huge selection of items available.

Cleaning	Coating	Hammers	Work Holding /control	Measuring /alignment	Punch/ marking	Holes /threading	Misc
Sand paper	Foam brushes	Welding	C-clamps	Tape measures	Center punches	Tap and die set	Extra Lighting
Acetone	Paint brushes	Ball pein	Bar clamps	Calipers	Transfer punches	135 degree drill bits; HSS/ Cobalt /etc	120v & 220/240v 1 phase outlets
Mineral spirits	Spray cans	Standard	Spring clamps	Micrometers	Pin punches		Extension cords
Files	HVLP sprayers	Mini-sledge	4+” vice	Wiggler tool	Cold chisels		Storage carts
Dremel w/burrs			Vise grips	DTI's	Layout fluid		tool boxes
Die grinder			Angle clamps	sine bar	Soap stone		cutting fluid
Angle grinder w/variety of wheels – dedicated purposes			Toggle clamps	Machinist square	Markers		safety equipment
6” Grinder			Work tables	Height gauge			screw drivers
Belt Sander			Angle plates	Feeler gauge			wrenches
Sandblaster			Parallels	Gauge blocks			metal snips
Parts washer			Saw horses	Thread gauge
Cleanup magnet				Straight edges
				1-2-3 blocks
				Rotary table

This is really just a very short survey of stuff related to welding/machining. I really enjoy working with this type of stuff but it is a major black hole for $.  Definitely need to set limits.  There are many days I wish I could find work that combined my software development skills with metal working skills.  Can keep wishing but won't hold my breath - for now this makes a nice stress reliever from the existing day job.

Hoping someone find something helpful or entertaining in all this.

Wishing you a blessed day.
Scott

Jenkins and Groovy - ERP data quality and validation

Many organizations have an ERP and they may manage large amounts of data.  I won't bother debating the tradeoffs of commercial ERP systems but I want to mention some ways I am dealing with certain data issues.

Our ERP is one that has some pretty old roots (and they show their grey) - with the changes in technology, schema, functionality, etc there are plenty of issues in data validation by the system.  Each new vendor maintenance release is like rolling the dice in regards to what validations change (for better or worse).  On top of that, the vendor process for major upgrades doesn't really handle conversion and/or reporting of some data and it is usually tables which really need extra work.  There are also business process and training issues within the distributed organization.  All of this results in many data messes.

Data issues on their own are bad enough but the way the problems tend to manifest for us provide a very bad experience for our customers.  In our case, the vendor technology manages data in a way where some data which is not actively being used (i.e. by an end user) is still operated on and validated as part of some transaction.  The problem is that when the actively used/accessed data is saved, the system also performs validations on the non-actively used data.  If that non-actively used data is invalid then the system throws a tantrum which usually has no useful message for the end-user since the data at fault is not in anyway related to what they did.  This is especially true when you have a custom application accessing the system - useful error responses from the source system are a rarity.

So what do you do?

I am slowly putting Jenkins jobs which perform checks and validations for some of the common problems.  I write the jobs to identify bad data, form a report of some sort and email it to particular users whom I hope are able to direct it to an appropriate person for correction/mitigation, etc.

The solution I am documenting here is the use of Jenkins with Groovy using JDBC to perform a check, report generation and emailing of any failure result.

I am modifying things somewhat to take organization identifying type information out.

import groovy.text.SimpleTemplateEngine
import javax.naming.*
import javax.sql.DataSource
import javax.mail.internet.InternetAddress
import javax.mail.internet.MimeMessage
import javax.mail.*
import javax.activation.*
import groovy.sql.Sql
import static java.util.Calendar.*
 
class PlanValidations
{ 
    // Produce a connection to the database
    def connect()
    {
          Context initCtx = new InitialContext();
          DataSource sqlDS = initCtx.lookup("jdbc/connid");
          def sql = new groovy.sql.Sql(sqlDS);
          return sql;
    }

        // An array to contain the column names returned by the query
    def colnames = []
 
    def checkPlanInactivation(sql)
    {
            // The query is looking for a misconfiguration which is typically a failure to follow the proper business process.
            // The correct process for disabling the data item is add a new row with a current effective date which states which
            //  period the data should be disabled starting at.  The query finds each max effective item plus several "lagging" fields for which an outer query compares to determine if the proper setup was used.
         
      def badInactivePlanSQL = "

SELECT *
FROM
  (SELECT INSTITUTION,
    FIRST_PERIOD_VALID,
    PROG,
    PLAN,
    LAST_PERIOD,
    LAG(LAST_PERIOD, 1, ' ')
      OVER (PARTITION BY INSTITUTION, PROG, PLAN ORDER BY EFFDT ASC)
        PREV_LAST_PERIOD,
    EFF_STATUS,
    LAG(eff_status, 1, ' ')
      OVER (PARTITION BY INSTITUTION, PROG, PLAN ORDER BY EFFDT ASC)
        PREV_eff_status,
    effdt,
    LAG(effdt, 1, NULL)
      OVER (PARTITION BY INSTITUTION, PROG, PLAN ORDER BY EFFDT ASC)
        PREV_eff_dt
  FROM PLAN_TBL
  ORDER BY INSTITUTION,
    PROG,
    plan,
    effdt
  )
WHERE EFF_STATUS    = 'I'
AND prev_eff_status = 'A'
AND prev_PERIOD = ' '
AND EFFDT           > SYSDATE 

           ";

   
         // This will be a map of lists  {inst -> [rows]}
      def institutionBadSetup = [:].withDefault {[]} 
            
      sql.eachRow(badInactivePlanSQL,
                  // keep track of the column names returned by the query
         { meta ->
           (1..meta.columnCount).each { colnames << meta.getColumnName( it ).toLowerCase() }
         },
                 // save off the bad data by institution
         {
             institutionBadSetup[it.INSTITUTION] << it.toRowResult()
         }
      )
      return institutionBadSetup
    }

    def doChkInactivations()
    {
         def conn = connect();        
         def invalidPlanSetups = checkPlanInactivation(conn)

                   // take all the results and send off an enail per institution

         invalidPlanSetups.each() {
           institution, lst -> sendEmail(getEmailTo(institution, conn), getSubject(institution), getMsg(institution, lst))
         }     
        
         conn.close();        
    }

        //  Produce an html message with description of problem, related data and hints at fixing.
    def getMsg(institutionId, plans)
      {
       def html = """\
         <html>
         <head></head>
         <body>
         There is a problem with the setup of following plans for $institutionId.<br/><br/> 
       
          *** Provide a message here which describes the problem and provides help in correcting. ***
        <br/><br/>
         """
     
            // use the saved off column names and match up with the field data while constructing the simple report
       plans.each() { plan ->
          (0..colnames.size()-1).each {
            println plan.PLAN + " " + it
            html += "${colnames[it]}[${plan[it]}] "
          }
         html +="<br/><br/>"
       }
     
       html += """\
        </body>
        </html>""";
     
        return html
      }
 
    // Produce a subject for the email
    def getSubject(institutionId)
      {
      def env = System.getenv()
      String environ = env['ENV_TYPE']
      String hostname = env['HOSTNAME']
     
      return "invalid INACTIVE plan setup - institution(${institutionId}) <system name>(${environ}) server(${hostname})"
      }
 
      // Use an existing table of users to provide i
      def getEmailTo(institutionId, sql)
      {
      // hard coded a user [me] as email "to" during R&D
      def emailSql = "select 'me@someplace.org' as cntct_eml from inst_misc where institution = '${institutionId}'"
      def emailAdr = sql.firstRow(emailSql)
      return emailAdr.vx_oa_admcntct_eml
      }
 
    def sendEmail(receiver_email, email_subject, msgtext)
    {       
        def reportdt = String.format('%tY/%<tm/%<td  %<tH:%<tM:%<tS', Calendar.instance);       
 
        println "$msgtext";

        Properties mprops = new Properties();
        mprops.setProperty("mail.transport.protocol", "smtp");
        mprops.setProperty("mail.host", "smtp.somewhere.org");
        mprops.setProperty("mail.smtp.port", "25");
        Session lSession = Session.getDefaultInstance(mprops, null);
        MimeMessage msg = new MimeMessage(lSession);
       
        msg.setText(msgtext, "utf-8", "html");
       
        def me = "sender@somewhere.org";
        StringTokenizer tok = new StringTokenizer(receiver_email, ";");
        
        ArrayList emailTos = new ArrayList();
        while(tok.hasMoreElements())
        {
           emailTos.add(new InternetAddress(tok.nextElement().toString()));
        }
        
        InternetAddress[] to = new InternetAddress[emailTos.size()];
        to = (InternetAddress[]) emailTos.toArray(to);
        msg.setRecipients(MimeMessage.RecipientType.TO,to);
        
        msg.setSubject(email_subject);
        msg.setFrom(new InternetAddress(me));
       
        //Send the message via our own SMTP server, but don't include the envelope header.
        Transport transporter = lSession.getTransport("smtp");
        transporter.connect();
        transporter.send(msg);
     }
}

validationChk = new PlanValidations();
validationChk.doChkInactivations();