Jitterbit Logging and Error Notifications #Boring #SomeoneHasToDoIt

I got a question from Veronica in the comments of the Jitterbit Best Practices post asking about specifics on logging and error handling.

Out-of-the-box Jitterbit actually provides decent default logging and notifications, but the details can be really lacking. It does require a bit of tinkering to get exactly the information you want recorded or sent out exactly how you want it.

The basic strategy is to pull the information we want to see into a variable, so that we can then either use a function to write it out to log, or reference it by name in an email template.

Side note:  all this info (and my previous Jitterbit post) relate to the classic, self-hosted, on-premises version only.  I haven’t yet had a chance to play with the hosted “cloud” version of Jitterbit.

What to Log

This is obviously the first place to start. What extra info would we want to see in our error messages? Here are a few I’ve used:

  • Jitterbit server – self explanatory, the server running the job
  • Jitterbit environment (DEV/QA/UAT/PRODUCTION) – useful to quickly differentiate an error message from development from one from production.
  • The operation start time
  • Any server or database connection information
  • The name of the operation that failed
  • The error message

How to Log

For informational messages, the WriteToOperationLog() JitterScript function is your friend. This is great for recording info like start time, end time, and number of records processed in the logs so you can check them out when there hasn’t been a failure.

You can log individual records in a transformation by adding a condition to a target structure and inserting appropriate script statements there, but be aware if you have thousands of records going across, writing a log entry for each will overflow what Jitterbit can store per operation. A better option, I’ve found, is to simply log counts instead of specifics.


    $company.count = $company.count + 1
    $company.meetsRequirements = // some If() statement to determine if this record is included
    If($company.meetsRequirements,
        $company.includedCount = $company.includedCount + 1;
        true; // include
    ,
        false; // don't include
    );        

Now we have the $company.count and $company.includedCount variables to use in our logs, or in an email. A good place to record these is in a final post-execution “clean-up” script, which is very easy to do if you have a Master Operation setup (see-below).

Multiple Operations – Using a Master Operation

A related topic is how to structure multiple operations in Jitterbit. It’s very common in my experience that you’ll have a need to load several different items in sequence as part of one larger operation. It is possible to chain operations using the On Success built-in functionality of Jitterbit, so that as each finished successfully, the next is started. I find this visually confusing and prefer to use scripts to execute sub-operations of a multiple step load. We can use a simple script to execute our sub-operations:


    // always prefix variables with the company name to avoid collisions with built-in Jitterbit variables
    $company.operationName = 'Load Widget Data';
    WriteToOperationLog($company.operationName + " started.");
    If(!RunOperation("Load Widget Data"), // returns false if the operation run fails
        CancelOperationChain($company.operationName + " operation failed.");
        $company.fatalErrorMesage = $company.fatalErrorMesage + "\n\n" + GetLastError();
        RaiseError(GetLastError());
    );
    WriteToOperationLog($company.operationName + " complete.");

In this way we can have a Master Operation that simply executes one or more of these scripts, and if any sub-operation fails, the information will bubble up. The Master Operation then gets assigned the failure email message, using the default Jitterbit On Failure functionality. The Master Operation also gives you a nice place to run a preparation script, which could do things like record the start date/time, as well as clean-up scripts which could handle any final tasks like recording the end date/time and writing success information out to the operation log.

Jitterbit logging multi-operation setup

Email Template

Once you’ve defined the scripts that build your variables, the email templates are actually quite simple. I recommend, again, using jitterbit.conf-defined variables for the template settings, including the From and To addresses, in order to make sure you don’t have to make updates when deploying a Jitterbit package between environments. Here is an example email body:

There has been an operation failure for a Jitterbit job.  

Server Name : [company.jitterbit.serverName]
Project Name : [company.projectName]
Operation Name : [company.operationName]
Operation Start Date/Time : [company.startDateTime]
Source Database : [company.widgetSource.server] - [company.widgeSource.database]
Target Database : [company.widgetTarget.server] - [company.widgeTarget.database]

Error Message
[company.fatalErrorMesage]

And here is what the subject line could look like:

[company.environmentName] - [company.projectName] Operation Failure

Conclusion

That’s about it. A little bit more information can make handling Jitterbit issues a lot simpler and quicker. Good luck!

11 thoughts on “Jitterbit Logging and Error Notifications #Boring #SomeoneHasToDoIt

  1. Ok, this is my new favorite blog. Thank you so much for writing this up, it’s super helpful!

  2. Hi,

    I have been seeing this blog for a while and has find some good blogs on jitterbit. Since I am learning jitterbit and there is very less documentation, I have some questions to ask-:

    1) First I want to do incremental load from salesforce to a flatfile, in which there should be only new and updated records from salesforce everytime the operation is run.

    2) How to pass a datetimestamp into a flatfile as a variable & how it can be retrieved later in the operation.

    1. Hey Hemant,

      I might do a post on it if I get time, but in short here is how you could do it:

      1) Create a “Salesforce Query” operation using the wizards as normal for whatever object you’re interested in.
      2) In the operation, add a script before the first step in the query. In this script, populate your $dateTimeStamp variable somehow – via a database call, or calling another operation to load from a file, or whatever.
      3) In a second script after that one, or in the same script, create a $whereClause variable to use in the SOQL query using the $dateTimeStamp. We need to format the date time to match Salesforce-style : $whereClause = ” WHERE LastModifiedDate >= ” + FormatDate($dateTimeStamp, “yyyy-mm-ddTHH:MM:SS.000Z”);
      4) Under your Jitterbit Connect Salesforce Queries folder, open the query, and add [whereClause{}] to the end after the “FROM WhateverObject”. Note that the square brackets will cause Jitterbit to look for a variable, and the curly brackets give a default value of blank (so it’s ignored during testing when the script hasn’t run).

      If you’re paying for Jitterbit licenses, you could also shoot their support an email, they might have a better way that isn’t in the documentation. Good luck!

      -Jeremy

  3. Hey Orgo,
    Thank you for the nice blog posts.

    I assume these are for the Jitterbit Harmony product. Any specific version?

    Regards,
    Sajid

    1. Sajid, you’re welcome! The articles were written using an older version of Jitterbit – 5.2 “on-premises”, not Harmony. Once we get our jobs migrated over to the latest Harmony version, I may update the articles with any necessary changes. Thanks!

  4. Hey Ogro,
    Please do continue your posts when you move to Harmony!

    Where did you define “company”?

    Regards,
    Sajid

    1. Hey Sajid,

      Wherever I have company before a variable name, you can just replace that with whatever name you’d like. It’s just a safe-guard to make sure you don’t accidentally overwrite any existing Jitterbit variables. So if your company name is Acme, you could use $Acme.projectName or $acmeProjectName, or $acme_project_name, whatever is your preference as long as it’s clear and distinct.

  5. Got it. Thanks, Orgo. Please do add blog posts on some advanced topic – Scripting in Jitterbit itself comes to my mind. Also, I am interested to get your thoughts on change management and deployments in Jitterbit Harmony, please.

  6. Hi Ogro,
    Thanks for the above post however my requirement is bit different for doing error handing.

    I want to generate the error file for the records that I want to push to salesforce BUT on the same time I want them to insert in salesforce. The basic reason is my client want the error notification of which records holds not good so that he can go in salesforce and fix them.
    My Question is
    1) Where to store the daily error file. Should I strore on the FTP server( this is my source to jitter bit
    2) Do I need to write the script which will have all the conditions for error out records(email to be sent to user) but on the same time inserts it into salesforce?

    Kindly share your view as this is very critical for my business.

    Thanks
    Mohit

  7. Hi Ogro,

    I am looking to setup a daily notification of all processes run with simply Operation Name, Total Records, Successes, and Failures for each one. I would imagine appending each operation to a custom log file or even a daily log file may work then have a script pickup the log file and send that as the body.

    My issue is I am not sure where/how to start this process in Jitterbit. I don’t see the “Admin Console” or where I would enter this in. Are you on a different edition of Jitterbit? I currently have the free Jitterbit Data Loader Is this possible for me?

    Thanks,
    Josh

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s