Jenkins CI - promotion processes

I recently got around to implementing some Jenkins promotion processes for one application.  This is the first time I setup promotions for any of our builds.  It has been on my to-investigate list for a long time though.

So far, it is working out nicely.  We are on an older Jenkins version for now and I think the 2.x version has some significant workflow additions which I hope to try out sometime (sooner than later). Even with the older Jenkins/promotion setup, it is nice having some workflow in place and manageable from one place.  For various reasons, most of the processes are manually initiated except the first one (which results in deployment to a dev server after a build which is initiated when SCM changes are found during polling).

One thing that might be nice is a way to schedule a promotion once criteria are met.  With some effort, this is probably possible with some custom code and/or more jobs but having a more direct way would more convenient. My intent with this would be to use promotion to perform deployments, outside of normal use hours, to systems used for testing/training so as to not disrupt activities. I guess another option is to change the deployment processes so that promotion simply queues up the artifacts that a separate deployment process then handles. I'll have to think that one through further; for now the recent changes are at least a positive step forward.

Another item of research is whether the promotion process definitions can be simplified/shared somehow. I didn't put time into simplifying it upfront, instead I just wanted to make sure it was correct.  As long as the maintenance cost is lower than the benefits I won't complain but reducing non-development time wasters is always a plus so I will hope for continued improvement.

Jenkins - groovy and programmatic job dependency

I really like Jenkins but it certainly isn't a full fledged general job scheduler.  One aspect of various commercial / general schedulers that I miss is the ability to chain jobs together based upon job status.  A scheduler like CA Autosys handles this pretty well.

I can't really justify the cost of a commercial scheduler and we use Jenkins so what can be done?

We'll I ran into a need and decided to find out.  I need to verify the status of an Apache ServiceMix system.  On occasion (still trying to determine root cause), the connectivity with an ERP drops and isn't reacquired.  When that happens, the only solution that worked so far was to restart the ServiceMix process.

What I did is define a Jenkins job which watches the queue of input data for failures.  When a failure occurs I wanted to restart ServiceMix. It took a bit of searching but I found a way to run a second job on identifying our failure condition.  This second job is responsible for performing the restart and the job is initiated programmatically from the first job.  Below is a chunk of code to get, execute and wait for a job to complete.  Not a lot of error handling here but it hopefully provides enough of an example to successfully copy and reuse. 

   def restartWsiServer()  
   {  
    def job = Hudson.instance.getJob('SM RESTART')  
    def anotherBuild  
    try{  
     def future = job.scheduleBuild2(0, new Cause.UpstreamCause(build))  
     out.println "Waiting for the completion of " + HyperlinkNote.encodeTo('/' + job.url, job.fullDisplayName)  
     anotherBuild = future.get()  
    } catch(CancellationException e)  
    {throw new AbortException("${job.fullDisplayName} aborted.")}  
    out.println HyperlinkNote.encodeTo('/' + anotherBuild.url, anotherBuild.fullDisplayName) + " completed."  
   }  

Hope someone finds this useful.

Jenkins and Groovy - ERP data quality and validation

Many organizations have an ERP and they may manage large amounts of data.  I won't bother debating the tradeoffs of commercial ERP systems but I want to mention some ways I am dealing with certain data issues.

Our ERP is one that has some pretty old roots (and they show their grey) - with the changes in technology, schema, functionality, etc there are plenty of issues in data validation by the system.  Each new vendor maintenance release is like rolling the dice in regards to what validations change (for better or worse).  On top of that, the vendor process for major upgrades doesn't really handle conversion and/or reporting of some data and it is usually tables which really need extra work.  There are also business process and training issues within the distributed organization.  All of this results in many data messes.

Data issues on their own are bad enough but the way the problems tend to manifest for us provide a very bad experience for our customers.  In our case, the vendor technology manages data in a way where some data which is not actively being used (i.e. by an end user) is still operated on and validated as part of some transaction.  The problem is that when the actively used/accessed data is saved, the system also performs validations on the non-actively used data.  If that non-actively used data is invalid then the system throws a tantrum which usually has no useful message for the end-user since the data at fault is not in anyway related to what they did.  This is especially true when you have a custom application accessing the system - useful error responses from the source system are a rarity.

So what do you do?

I am slowly putting Jenkins jobs which perform checks and validations for some of the common problems.  I write the jobs to identify bad data, form a report of some sort and email it to particular users whom I hope are able to direct it to an appropriate person for correction/mitigation, etc.

The solution I am documenting here is the use of Jenkins with Groovy using JDBC to perform a check, report generation and emailing of any failure result.

I am modifying things somewhat to take organization identifying type information out.

import groovy.text.SimpleTemplateEngine
import javax.naming.*
import javax.sql.DataSource
import javax.mail.internet.InternetAddress
import javax.mail.internet.MimeMessage
import javax.mail.*
import javax.activation.*
import groovy.sql.Sql
import static java.util.Calendar.*
 
class PlanValidations
{ 
    // Produce a connection to the database
    def connect()
    {
          Context initCtx = new InitialContext();
          DataSource sqlDS = initCtx.lookup("jdbc/connid");
          def sql = new groovy.sql.Sql(sqlDS);
          return sql;
    }

        // An array to contain the column names returned by the query
    def colnames = []
 
    def checkPlanInactivation(sql)
    {
            // The query is looking for a misconfiguration which is typically a failure to follow the proper business process.
            // The correct process for disabling the data item is add a new row with a current effective date which states which
            //  period the data should be disabled starting at.  The query finds each max effective item plus several "lagging" fields for which an outer query compares to determine if the proper setup was used.
         
      def badInactivePlanSQL = "

SELECT *
FROM
  (SELECT INSTITUTION,
    FIRST_PERIOD_VALID,
    PROG,
    PLAN,
    LAST_PERIOD,
    LAG(LAST_PERIOD, 1, ' ')
      OVER (PARTITION BY INSTITUTION, PROG, PLAN ORDER BY EFFDT ASC)
        PREV_LAST_PERIOD,
    EFF_STATUS,
    LAG(eff_status, 1, ' ')
      OVER (PARTITION BY INSTITUTION, PROG, PLAN ORDER BY EFFDT ASC)
        PREV_eff_status,
    effdt,
    LAG(effdt, 1, NULL)
      OVER (PARTITION BY INSTITUTION, PROG, PLAN ORDER BY EFFDT ASC)
        PREV_eff_dt
  FROM PLAN_TBL
  ORDER BY INSTITUTION,
    PROG,
    plan,
    effdt
  )
WHERE EFF_STATUS    = 'I'
AND prev_eff_status = 'A'
AND prev_PERIOD = ' '
AND EFFDT           > SYSDATE 

           ";

   
         // This will be a map of lists  {inst -> [rows]}
      def institutionBadSetup = [:].withDefault {[]} 
            
      sql.eachRow(badInactivePlanSQL,
                  // keep track of the column names returned by the query
         { meta ->
           (1..meta.columnCount).each { colnames << meta.getColumnName( it ).toLowerCase() }
         },
                 // save off the bad data by institution
         {
             institutionBadSetup[it.INSTITUTION] << it.toRowResult()
         }
      )
      return institutionBadSetup
    }

    def doChkInactivations()
    {
         def conn = connect();        
         def invalidPlanSetups = checkPlanInactivation(conn)

                   // take all the results and send off an enail per institution

         invalidPlanSetups.each() {
           institution, lst -> sendEmail(getEmailTo(institution, conn), getSubject(institution), getMsg(institution, lst))
         }     
        
         conn.close();        
    }

        //  Produce an html message with description of problem, related data and hints at fixing.
    def getMsg(institutionId, plans)
      {
       def html = """\
         <html>
         <head></head>
         <body>
         There is a problem with the setup of following plans for $institutionId.<br/><br/> 
       
          *** Provide a message here which describes the problem and provides help in correcting. ***
        <br/><br/>
         """
     
            // use the saved off column names and match up with the field data while constructing the simple report
       plans.each() { plan ->
          (0..colnames.size()-1).each {
            println plan.PLAN + " " + it
            html += "${colnames[it]}[${plan[it]}] "
          }
         html +="<br/><br/>"
       }
     
       html += """\
        </body>
        </html>""";
     
        return html
      }
 
    // Produce a subject for the email
    def getSubject(institutionId)
      {
      def env = System.getenv()
      String environ = env['ENV_TYPE']
      String hostname = env['HOSTNAME']
     
      return "invalid INACTIVE plan setup - institution(${institutionId}) <system name>(${environ}) server(${hostname})"
      }
 
      // Use an existing table of users to provide i
      def getEmailTo(institutionId, sql)
      {
      // hard coded a user [me] as email "to" during R&D
      def emailSql = "select 'me@someplace.org' as cntct_eml from inst_misc where institution = '${institutionId}'"
      def emailAdr = sql.firstRow(emailSql)
      return emailAdr.vx_oa_admcntct_eml
      }
 
    def sendEmail(receiver_email, email_subject, msgtext)
    {       
        def reportdt = String.format('%tY/%<tm/%<td  %<tH:%<tM:%<tS', Calendar.instance);       
 
        println "$msgtext";

        Properties mprops = new Properties();
        mprops.setProperty("mail.transport.protocol", "smtp");
        mprops.setProperty("mail.host", "smtp.somewhere.org");
        mprops.setProperty("mail.smtp.port", "25");
        Session lSession = Session.getDefaultInstance(mprops, null);
        MimeMessage msg = new MimeMessage(lSession);
       
        msg.setText(msgtext, "utf-8", "html");
       
        def me = "sender@somewhere.org";
        StringTokenizer tok = new StringTokenizer(receiver_email, ";");
        
        ArrayList emailTos = new ArrayList();
        while(tok.hasMoreElements())
        {
           emailTos.add(new InternetAddress(tok.nextElement().toString()));
        }
        
        InternetAddress[] to = new InternetAddress[emailTos.size()];
        to = (InternetAddress[]) emailTos.toArray(to);
        msg.setRecipients(MimeMessage.RecipientType.TO,to);
        
        msg.setSubject(email_subject);
        msg.setFrom(new InternetAddress(me));
       
        //Send the message via our own SMTP server, but don't include the envelope header.
        Transport transporter = lSession.getTransport("smtp");
        transporter.connect();
        transporter.send(msg);
     }
}

validationChk = new PlanValidations();
validationChk.doChkInactivations();

Jenkins jobs and subversion access via certificate and batch related stuff

Jenkins is a very nice solution to a number needs outside of automated builds.  Use as a general batch solution is feasible at smaller scales.  This has further improved recently with the credentials plugin.  This makes it pretty easy to setup jobs which use certificates to access subversion instead of requiring a specific persons credentials which require a much more regular change.  Of course, this works best when an organization has the infrastructure to securely manage certificates.

[2014/05/20] Works!  It seems easiest to setup 2 separate Apache Virtual servers though; one with Basic Auth and the other with Certificate auth.  

Related to general batch processing, the Elastic Axis plugin is a great solution.  This is a perfect fit for when you have load balanced servers implementing an application and you need to run a job on any one but only one of them.  You can take an application node offline and Jenkins/Elastic Axis will happily pick an available node out of the remaining configured nodes.

My wish list for Jenkins includes a few enhancements which would improve some things to a large extent - this mainly revolves around new/improved support for high availability, multiple masters with fail-over and a clear/clean clustering solution.  A database backed job store might be a big plus depending on how the previously mentioned features were implemented.  I would love to be able to easily integrate Jenkins into our DR environment but as it stands it is a more manual integration.

Jenkins CI SQLTool Integration idea

I find that Jenkins works well for simple batch tasks but it would certainly be nice if there was an integration with the HSQL SQLTool so that a "SQLTool" job could be defined and maybe it could get connections from JNDI.  Instead of writing Groovy code to execute queries, we could then define the job as a SQLTool compatible script (either define script directly in Jenkins or specify file).  This would cover a good number of our utility batch jobs.

No time at the moment though.  Someday.

A Groovy time with Oracle and Hudson/Jenkins - poor mans batch

This is an example of a relatively simple solution to automating batch processes using tools at hand.  I hope someone else finds this useful.

The tools:

• Hudson/Jenkins with plugins
• Hudson Groovy Builder
• Time stamper
• Glassfish 3
• Groovy 
• Oracle
• Linux
• VMWare

You may ask why someone would do this?

• Leverage existing staff skills and tools
• Provides some security features (project based matrix security) to help provide limited access to a variety of staff
• Provides an easy to use web interface for running and checking job status
• Provide log of job output
• Relatively easy to implement disaster recovery
•  

 What are the downsides?

• Development/testing of Groovy jobs is challenging if not done in an IDE which supports Groovy or if leveraging Hudson specifics.  Making changes directly in a Hudson job and rerunning it is time consuming and error prone.
• Additional development challenges if using J2EE container/JNDI to provide DB connections.
• This is not a true enterprise scheduler so handling dependencies and other complicated work flows is difficult at best.
• This example was only intended to run on the master node - no attempt was made to support running jobs on remote nodes.
• This example puts the Groovy code directly in the job definition; it may be better to define the job as a script under source control and have Hudson/Jenkins check out the script for execution.  The trade offs need evaluation.

This doesn't cover every detail but will pull in a number of details which took some research and trial/error.  I am fairly new to Groovy so this was a bit of an experiment. As an experiment I try to demonstrate a Groovy variation on some other activities which I have not seen in Groovy along with some other more common items documented on other Groovy relates sites.

The starting point for this was a Linux server running as a VMWare guest and a recent Sun JDK installed.  Glassfish was then installed.  The Oracle ojdbc6.jar file was then added to the $GLASSFISH_HOME/glassfish/lib directory.  A copy of Groovy was installed/extracted on the server.  Follow Glassfish install instructions and then start server.  The Hudson/Jenkins war file was then deployed by copying to the Glassfish autodeploy directory. 

Next configure Glassfish; select "Enable Security", define the security realm and pick the Project-based Matrix Authorization Strategy.  Make sure you either initially give "Anonymous" full rights or add yourself with all rights before saving changes else you risk not being able to further configure things.
Next add the plugin "Hudson Groovy Builder plugin" and restart Glassfish (depending on versions of various things you may be able to simply tell Hudson/Jenkins to restart).  In the Hudson/Jenkins configuration page, add a Groovy installation and provide the path where you extracted Groovy.  


In the Glassfish admin console (or using the command line tools), define a connection pool named "MYDB" of resource type "javax.sql.DataSource" and DS classname of "oracle.jdbc.pool.OracleDataSource".  Set the pool size to something reasonable.  I provided an idle timeout of 300s and max wait time of 60000ms.  No transaction settings are selected. On the next conn pool tab, select "wrap jdbc objects" and "Pooling".  I have "validate at most once" set to 60s and a leak timeout of  600s.  The rest of the settings on the tab are defaults. On the "Additional properties" tab, I updated the list so only the properties "User", "Password" and "URL" exist and the URL is of the general form: jdbc:oracle:thin:@<server>:<port>:<service>.  If on the first tab you have "Ping" enabled - you can click the "Ping" button and should get a success message if the pool is configured close to correct (so the system can at least find and communicate with the DB).



Now create a JDBC Resource (i.e.associate pool to JNDI).  Fill in the name with "jdbc/mydb" and make sure the pool name is that of the pool you just created.  Status should be "enabled".



Let's create an example job now.  In Hudson/Jenkins, create a new job (pick free-style software project) -and call it something like GroovyBatchTest.  Configure the job; select "Enable project-based security" and add yourself/others with reasonable rights.  Pick "Build periodically" and set schedule to whatever is appropriate (uses cron style schedule definition). I also select "Add timestamps to the console output" which I find somewhat helpful in analyzing/debugging things.

Select Execute system Groovy Script, Groovy Command.   Please note this code was just hacked together as a demonstration only and has roots in a number of other blogs/websites - there is little original content on my part.  I will try to locate and reference some of the original websites which I pulled some of this general information from.  I will clean up the code when time permits.  In this example, I translated some stuff into Groovy which allows getting Oracle DBMS_OUTPUT.PUT_LINE data back via JDBC and also demonstrate some remote procedure calls and other Groovy/JDBC access methods. 

import javax.naming.*
import javax.sql.DataSource
import groovy.sql.Sql

class DbmsOutput
{
      private groovy.sql.Sql sql = null;
      public DbmsOutput(groovy.sql.Sql conn)
      { 
           sql = conn;
       }

       public void enable(int size)
       {
           sql.call("begin dbmss_output.enable(?); end;", [size]);
        }

       public void disable()
       {
             sql.call("begin dbms_output.disable; end;");
        }

        public String show()
        {
              def stmt = "declare "+
                   " l_line varchar2(255); " + 
                   " l_done number; " +
                    " l_buffer long; " +
              "begin " +
                "   loop " +
                 "      exit when length(l_buffer) + 255 > ? or l_done = 1; " +
                  "      dbms_output.get_line(l_line, l_done); " +
                  "      l_buffer := l_buffer || l_line || char(10);  " +
                 "   end loop; " +
                  " ? :=  l_done; " +
                  " ? := l_buffer; " +
               "end; " ;

              def done = 0;
              def dta = "";

               for (;;)
               {
                      sql.call(stmt, [32000, Sql.INTEGER, Sql.VARCHAR]) { d,txt ->
                           done = d;
                           dta = dta + txt + "\n";
                       }
                      if (done == 1)
                        {
                              break;
                        }

                       return dta;
                   }
              }
 
           DbmsOutput dbout = null
           groovy.sql.Sql sql = null

            try
             {
                     Context initCtx = InitialContext();
                      DataSource sqlDS = initCtx.lookup("jdbc/mydb");
                      sql = new groovy.sql.Sql(sqlDS);

                      def cur = sql.firstRow("select id, cnt from cur_process_tbl where ts = (select max(ts) from curr_process_tbl)")

                      dbout = new DbmsOutput(sql);
                       dbout.enable(4000);

                        sql.call("  begin " +
                                            "  mypkg.proc1(?, ?); " +
                                     "end;", [cur.id, cur.cnt])

                       curOutput = dbout.show()
                        println "data:" curOutput

                        dbout.disable()
                
                         sql.executeUpdate("update out_tbl set output = ? where id = ?", [curOutput, cur.id])
                    
                         sql.commit();
                  }
                  catch(Exception e)
                    {
                        if (sql != null)
                          sql.rollback()
                         println "error occurred:" + e
                   }
                 finally
                {
                         if (sql != null)
                              sql.close()
      }


In this example, my Oracle package "mypkg" with procedure "proc1(?, ?)" would contain DBMS_OUTPUT.PUT_LINE calls which produce data which is dumped in this example

If you now save the job and do a "build now", if all was done correctly you should see your DBMS_OUTPUT.PUT_LINE dumped to the console output of your jobs build-run.

Regarding disaster recovery(DR), there are a good number of implementation variations.  A simple way to handle a mostly warm setup where all DR resources are separate and distinct from production is to have a matching DR instance of the Hudson/Jenkins server preconfigured the same as production except for resources such as JDBC/JNDI resources and it should probably reference a DR source control server if one is in use. In the case were DR databases are also warm and distinct from production, you can preconfigure JDBC/JNDI resources in Glassfish to refer to the DR database.  If you keep job definitions in sync between production and DR (or put most of job in script in source control) and the job only acquires DB resources via JNDI then the unmodified job will work in both production and DR environments.  The main caveat is regarding jobs running on a schedule - if you don't want DR jobs running when no DR event occurred then you must disable jobs or alter the schedule.  Note that server virtualization and NAS storage were very helpful in handling some of the details.