After years as a consultant building both internal and public facing web sites I had the opinion that all batch processing that any given organization had in place was something left over from the dark ages or put in place by a technology Luddite that did not "get it".
Two situations I've run into recently have given me a wider perspective on the topic of batch processing.
First I had a customer replace an entire CICS system with a shiny new J2EE based Core Transaction Processing application.
This was my Nirvana, a new flexible application that would allow the IT governor to be removed, enabling the business to innovate. Well of course it was not that simple but having a new system with up to date documentation, that did not have years of changes and hacks applied to it, that could be modified without side-effects popping up throughout the system would be fabulous.
One thing that was hoped for from the new system was the removal of the reliance on batch jobs. The new application was designed to receive real-time updates via JMS, resulting in up to the minute status changes and business processing occurring in real-time. Wow, things would be great and they were, for a while.
The system was working surprisingly well until the first fiscal month end hit. That was when all hell broke loose. The replaced CICS system with it's finely tuned and orchestrated set of batch processing happily running overnight was now a storm of JMS messages pounding the new system throughout the business day. Process states were changing while human users were trying to access records resulting in confusion. Manual processes had to be applied to fix information or roll things back. The combination of "real-time batch" messages and operational loads pushed the system to the breaking point and a 16 node J2EE cluster buckled and crashed. OUCH!!!!
This could be attributed to just needing to work the kinks out of the system, which was partially true. When the 2nd month end rolled around, although the user confusion was no longer a factor due to bug fixes and operator training, the processing load from applying the month end processing during the day still resulted in slowdowns and performance problems. Sigh....
Ultimately the problem will be solved with new dedicated hardware to handle JMS processing and by ironically holding the messages for overnight processing. A little piece of me died.
Was this just a case of the business not being ready for the technology, why have month end processing at all? Well I guess this might be a legitimate point, a truly real-time business might not require any periodic business processes, but even if such a business did exist the problem of daytime load would still hamper the idea of a 100% real time processing IT architecture.
I hit this one square in the face when working with another client where we were discussing rolling out a new enterprise ETL tool that will result in dramatic decreases in time to build a data warehouse. The new ETL software had the ability to trickle feed information to the warehouse or to run large batches. I of course was thinking real-time, which would allow the warehouse to provide more timely information to the business.
However I hit another snag. The organization had carefully balanced its IT resource usage between daytime operational use and nighttime batch use. Daytime data warehouse feeds would require additional hardware. As I sat in the meeting I could not help think of all the recent articles in the media about making more efficient use of resources. Charge your electric car overnight, run the dishwasher after 9PM, use geothermal heating to try and level out heating and cooling spikes. It hit me that batch jobs are Green! They enable efficient balanced usage of IT infrastructure and actually save resources.
While these points won't sway me from my preference for real-time IT systems with immediate propagation of data it was a bit humbling to realize that the "Luddites" might have had some very valid reasons for the creation of batch jobs in organizations. With all the hype about Green Computing from hardware and software vendors the humble batch job was truly ahead of its time.
