Category — Business Transaction Management

I’d like to share my vision and definition for what I feel best describes Business Transaction Management (BTM). I’ve started a new page to track this emerging area, value proposition, marketplace, vendors and technology approaches. I’m planning a series of BTM oriented guest authors and podcasts as well as deeper dives into BTM as it relates to my emerging BSM Value Proposition model for 2009. Feel free to join in the conversation!

First off, let’s talk about just what a transaction is. Here’s my current working definition.

Transactions are one of, if not the most, critical components of how IT supports the business in meeting business goals and objectives. They are the simple and complex entities that get work done. They are the “movers and shakers” within the IT environment that are actively responding to client or business requests via the business services and applications that IT delivers. They move key data and information between infrastructure components that make up complex end-to-end business services and applications that exist in nearly every company today.

There are many common types of transactions and transactional architectures deployed by IT to support getting things done within companies today. These include the most common n-Tier (web, appsvr, db) transactions, the much hyped Service Oriented Architecture (SOA) approach, the tried and true mainstay of Mainframe (CICS, IMS) based transactions and I’m sure many others could be added to this list in the EAI, Complex Application, Web, J2EE, Mainframe or Network Service Provider areas.

I also like to bundle into this area something that’s likely not often thought of when we discuss types of transactions or transactional architectures and that’s the area of transactional based Workload Management (Batch Job/CRON) and Managed File Transfer (FTP/SFTP/SCP) type solutions. These technology areas have a very transactional based nature and are often responsible for getting some of the most high value units of work done within a typical enterprise. They should not be forgotten from any modern day or leading edge BTM solution.

What is Business Transaction Management (BTM)?

My views for BTM span many areas and focus on the following types of BTM activities. The BTM marketplace has vendors focused on one or more of these areas. Hopefully we can establish a uniform way to classify vendors and their technology, products and solutions into these or other categories.

Transaction Discovery and Inventory: Techniques for discovering transactions whether active, passive, configured, scheduled, or in-flight, for the purposes of establishing relationships between dependent systems, applications, databases, components, etc. based upon this transaction discovery as well as establishing an inventory or repository of transaction information. This would be used to extend a configuration management system, database (CMDB), asset management system or other. Ideally, we’d use this capability to establish the finite business relationships each transaction has, how it supports a business service, application or other unit of work, the impact if the transaction fails, blocks, queues, is slow, etc.

Transaction Tracing: Heavily focused on in-flight transactions, flows and paths, transaction tuning and optimization, transaction troubleshooting, complex relationships with other systems applications databases. This section includes transaction latency tracking, timing and profiling.

Transaction Monitoring: Focused on monitoring every aspect of the transaction, performance, availability, capacity, SLA, timings, etc. This section includes latency monitoring, an area of critical importance in many financial and trading environments.

Transaction Intelligence and Analytics: The application of sophisticated analytics to transactions to learn, monitor and managed transactions with applied intelligence. This is often seen in the financial and telecommunications industry as anti-money laundering solutions, credit card or other fraud or abuse of services/networks or similar solutions aimed at watching for patterns of activity or behavior as seen in transactional activity. The Business Activity Monitoring (BAM) and Business Process Management (BPM) areas are also very focused on using transaction intelligence and analytics to drive the business, make better decisions and respond to changing conditions in the business environment.

Transaction Management: General term, focused on all of the above areas. I consider a transaction management solution as one that provides capabilities across all of these areas rather than just one or two.

Are there and other areas?

Why does Business Transaction Management (BTM) matter?

BTM is a key component of my “BSM Value Proposition” model and a maturing BSM strategy for any company. I’m tempted to say that if you’re not incorporating BTM into your BSM strategy than you’re likely to never get real value from your BSM solution.

I think BTM matters because the transactions are the things that get stuff done in business. It doesn’t matter if you have vendor x, y or z software, application, database, etc. they could all be black boxes. It’s the transactions that flow in and out of those things, those finite units of work that are the backbone of every business.

The future will be filled with clouds or generic compute capability full of loosely coupled services, interfaces, and transaction pipelines to get work done. Things we know today as web servers, application servers, database servers, etc. will all be commodity black boxes in the future and we’ll program against API’s or other loosely coupled interfaces to accomplish work or move data and information between the IT environment components.

Understanding the performance, availability, capacity, reliability, and impact of these transactions on IT and the business is critical just as it has been for the traditional IT infrastructure that enables them. This level of understanding and impact must not be limited that of the typical IT organization, application or integration architect, developer or administrator. Transactions and their impact must be easily understood by the folks on the front line, within IT management and the line of business. Every transaction’s contribution to the bottom line or impact on a higher or lower level system, interface or business goal or objective must be thoroughly understood.

Folks, it’s simple. Transactions that don’t complete, fail, slow down, queue up, hit a bottleneck, etc. will impact the business in significant ways. Batch jobs or other file transfers that don’t complete can cost companies in many industries large fines and penalties, lost revenue, botched orders, and ultimately upset customers or business partners. Managing end-to-end transaction flow and user experience could easily be said to be much more important than anything else you do in the network/systems/application management & monitoring area.

How important are transactions in your environment? What are you doing to increase your ability to think, operate and respond differently by deploying transaction management within your organization?

What’s the state of operationalizing Service Oriented Architecture (SOA) monitoring and management tools? Are the typical network, systems, enterprise operations/management centers (NOC/SOC/EOC/EMC) up to speed on how to manage, monitor, triage, troubleshoot and in general understand how SOA is being used in companies that are adopting it? Should the operations center care that they have an event from something related to SOA infrastructure and respond differently than they would for a non-SOA event? Have SOA events, incidents, problems, process and workflow been thoroughly implemented in such a way that “it just works” like traditional enterprise monitoring and management? Or, are these fancy SOA monitoring and management solutions really reserved for those applications experts responsible for complex application support and development?

If a client continues to struggle with fundamental e2e service monitoring and management, transaction monitoring and management or even batch job monitoring and management, what will their chances of success be for SOA monitoring and management? Could SOA and associated “service or transaction oriented monitoring” be a catalyst to shore up these other areas? Should one be tackled/improved before starting on another? At a minimum, instituting a “service oriented” organizational structure and mentality is certainly something I’d recommend for anyone adopting broad based SOA principles.

Eric Roch offers some solid advice on SOA Monitoring and Management which highlights that there’s more need for doing the fundamentals of systems, application and service management and monitoring really well as a foundation for SOA Monitoring and Management.

Others (and my preferred focus area) feel that monitoring SOA should really be more closely related to monitoring what this SOA initiative and deployment’s all about - the business. Business Service Management (BSM), Business Activity Monitoring (BAM) and Business Process Management (BPM) all play a key role in helping understand how IT infrastructure, systems, applications, etc. support and impact the business’s goals and objectives. The fairly new buzzword Business Transaction Management (BTM) spearheaded by Correlsense and OpTier really speaks to the desired need here.

I feel that it’s got to be a focus on both of these areas, but with a strong preference to the “B” buzzword set since most IT organizations are likely using the “improve/standardize/reuse/efficiency/time-to-market” spin to aide in business support and justification for their SOA initiatives. That said, you’d BETTER focus on the things that the business cares about and show them tangible evidence that your SOA initiative is making things better for them. This is very possible by adopting a BSM, BAM, BPM, BTM (or a preferred combination) strategy that focuses on providing the right level of business visibility into the SOA environment and more importantly the e2e business services, applications, transactions, processes and activities. It ultimately all ties back to the service level agreements delivered to the business anyway right?

What’s on the market these days for SOA Monitoring and Management? Should you get your monitoring and management tooling from your core SOA platform vendor or should you take a third party, “best of breed” approach? Are there true “vendor neutral” solutions out there? Are clients implementing SOA architectures based on multiple vendor’s technology, solutions and products?

• Progress/Actional
• Amberpoint
• Avicode
• CA Wily
• Correlsense
• HP
• IBM Tivoli ITCAM for SOA
• Manged Methods JaxView Suite
• MuleSource Galaxy and HQ
• Nastel Autopilot
• OpTier
• Oracle/BEA
• Progress (Actional, Sonic, Mindreef)
• SOA Software
• Symphoniq
• Tidal Software
• Truviso

Some additional content on some of these vendor solutions is available here.

Who might the “market leader” be of these SOA specific solutions? What makes them a leader? What capabilities, features, functions would be considered “best of breed”, differentiator, must have, core, desired, nice to have, etc.?

What’s “really” needed for SOA monitoring and management?

1. Web Services
2. ESB
3. Transaction Performance
4. Transaction Availability
5. Transaction State/Status
6. SOA Registry
7. SOA Security
8. Service Discovery and Relationship/Dependency Mapping
9. Transaction Discovery and Mapping

Anything else missing here? What here needs to be specialized in its own product versus just extending the investments clients have already made?

Please do share your thoughts here. There are folks lurking who really need help in figuring this stuff out and/or improving products and capabilities on the market today!

In an repost of an article from a couple years back, Robin Bloor provides some updated color on the state of BPM and SoA. It’s apparent that some of the other “B” buzz words have the same challenges that exist on the BSM front.

Things to ponder…

• How can these projects with such touted value to the business or IT be successfully implemented?
• Where are vendors falling short in helping “solve the organizational problems” that often cause these “B” projects to fail?
• Is throwing technology/product at the problem the best place to start?
• What should a next generation organizational structure look like? Can IT and Business organize around end-to-end business service/process delivery and support?
• How can organizations be incented, encouraged, mandated to have an end-to-end business service/process focus?
• Where success stories for BPM, SOA, BSM, BAM, etc. exist, how have these technologies been operationalized, organizations changed, workflow/process/procedure modified to reap the benefits?
• Is it foolish to think that any of these organizational challenges can ever be solved or at least minimized?
• Do we have generational issues here that will change as Baby Boomers retire and Gen X/Y/Z move up the ranks in IT and Business?

Give the post a read, I found these two very applicable to all of the things I’m seeing with BPM, BSM, BAM, etc.

Question 5: What are the most difficult steps within a BPM project – and what makes a SOA project tedious?

“B” is our middle name. We have “B” scattered throughout everything that we do. At times we fight over who owns the “B” word. I’m in search of a unified “B” story and solution. IMO, if we had this, it’d be tough to compete with us in any of the “B” acronyms.

The “B” Business TLA’s: Business Service Management (BSM), Business Process Management (BPM), Business Performance Management (BPM), Business Activity Monitoring (BAM), Business Transaction Management (BTM), Business Intelligence (BI) and I’m sure there are others.

What’s it going to take to have a unified “B” story and solution? Sure, we’ve probably got mentions in individual roadmaps and presentations of how we’ll integrate with this, share data what that, use Cognos here or there, send events from one tool to the other, etc. but what about a real “B” solution? IMO, these approaches just prolong client value and significantly delay any real innovation in core products.

What’s the cost of “forking” and creating a new solution entirely? One that focuses on becoming best of class in all of the “B” areas (ok, at least do all of them pretty darn good)? One that can be implemented and managed by one team free from (well, probably not) the organizational politics that’d exist if it was a “solution by integration” solution. One that has the best possible chance of truly aligning business and IT. Ok, this is probably cost prohibitive, but its GOT TO BE THE END GOAL!

This is where the politics come in unfortunately…where would you start? Which “B” is the most important “B”? Is it Business Service Management - my preference is here of course. Our friends in other organizations would see it other ways for sure. We must find the right way to develop the “B” story and solutions in ways that are most beneficial to the client. We must include content in each others products that “treads” on each others turf. We must have joint releases that build towards the unified “B” story and solution. When we release a new business process management suite (BPMS) we must include dashboards, models and integrations that provide value OUT OF THE BOX inside our BSM product. This must be backed up with the business and services consultants who have consultative based skills to guide our clients through the process because this isn’t about the product as much as it is about working through the organizational problems and politics.

A unified “B” story and solution may sound like a pipe dream, but it’s what clients really want to strive towards and our competitors are making giant strides in this direction. What would your ideal “B” solution look like? If you were king for a day …

That one day, a typical American tourist from Atlanta, GA would be walking along the Great Wall of China and be able to send Tweets along the way? I’m not impressed by the technology, but just the fact that I was updating Twitter from the GREAT WALL OF CHINA! My perceptions of China have changed forever and I believe the future of the WORLD will change significantly based on what China does.

The leading mobile provider China Mobile has more customers than the entire population of the USA, well north of 375M. The last I heard, the largest mobile provider in the USA had less than 50M. Everyone is using a mobile phone here in China. I rode on a boat through the city of Zhouzhuang and the “granny” who was the [insert the word for boat operators like in Venice] on her mobile heading back up stream for more tourists. There is a mass collision of hyper growth and significant poverty in the areas that I have seen over the past two weeks. That hyper growth bleeds into poverty stricken areas throughout Shanghai and Beijing. Cellular phones are everywhere. Handset technology that is very “cool and hip” with the younger generations today blows ours away.

I challenged my IBM host yesterday to think about what China would be like when his two year old son turns 15 or 20. The largest bank in China (ICBC - 1984) and the largest mobile provider in China (China Mobile - 2000) were founded in radically brief timeframes compared to similar US companies. The Chinese government is considering establishing a market for start up companies to get funding, build their businesses, IPO, etc. What will China look like in 15 years? What will China have to do to operate successful companies at this hyper scale that is very unfamiliar to most US companies? How will technology have to adapt? What technology needs to be created? What software will need to be developed to manage 500M - 1B mobile users, mobile content, mobile applications (banking, etc)?

I don’t think that the acceptable answer will be segmentation, regionalization, geo-localization of technology. This is the answer for technology that can’t scale. If you’re following John Willis as he dives into cloud computing, might this be a possible answer? Maybe we need to think in terms of how we all learned about the solar system here and orbiting planets around the sun. If the utility compute cloud is the core of the hyper scale architecture of the future, each orbiting “planet” around the compute core is more compute clouds for hyper scale applications, and then more for hyper scale transactions, and hyper scale services. Additional compute cloud “moons” would be hyper scale resources for managing at mass scale. Sometimes there are multiple “moons” required to provide more fine grained mass scale management, additional services, functions, etc. Wow, I’m rambling here but my mind is in some sort of hyper mode thinking about all of this stuff. Can you name a US company that could on-board (acct mgmt, provisioning, billing, etc.) over 7M new customers in a single month as China Mobile did in January 2008?

China is big. China will be bigger than we can ever imagine. I think we need to be seriously reconsidering how we do things today to prepare for what China will require in 3, 5, 10 + years. China needs progressive, abstract thought leaders who can challenge the status quo to help them continue to meet hyper growth requirements. China needs highly skilled practitioners and business partners who can help show them the way. From there, they’ll dedicate their own resources to carry things forward. (what’s possible when you have THOUSANDS of skilled resources in development centers…ANYTHING).

I think Ryan’s monthly wrap up posts are really nice and I’m going to start these as well.

Hot Posts for January

In Where is Quest Software’s BSM Play? I ask what’s taking Quest Software so long to roll out a new BSM story based on their Magnum Technologies acquisition. Lots of good comments here from Quest Software clients who have had both success and challenges with Foglight. Quest Software’s Foglight Product Manager chimes in for some commentary! Don’t be a stranger Brad!

In Will Compuware 2.0 Include a Clear BSM Story and Viable Solution? I question if Compuware can really reinvent itself and compete in the BSM space. Still no response here from Compuware. Where’s the 2.0?

In So You Want to do BSM? and several articles that EMA has put out in the trade rags this month highlighting the results of their BSM/SLM Market Forecast for 2008-2012 they have painted a bright future for Business Service Management. They also are emphasizing the challenges with the traditional ways of implementing it, something that I’m in complete agreement with and have been dealing with for years. Still looking for a final version of this market forcast report. More here and here.

DevCampTivoli Thought Provoking Series

In Is Your Tivoli Monitoring, Netcool/OMNIbus or TBSM Organization Structure a Barrier to BSM Success? to propose new approaches for IT organization structure to focus on end-to-end service management ownership.

In My ITM 6.x BSM Profile should include a BSM Descriptor File I propose an approach for every managed system to provide key information needed for BSM.

In All I want for the New Year is a BSM Profile for ITM 6.x I propose a concept for specific and purpose built instrumentation of managed systems using a BSM Profile.

WYNTK on TBSM Series

In WYNTK on TBSM Design Patterns: Architectural Model for COTS and Composite Applications I introduce TBSM design patterns for modeling COTS and Composite Applications.

Industry Highlights

BarCampESM was a success. Check out the OMC site for all the follow on activity and blip.tv for all the sessions. My presentation and video are available here. Take 30 minutes and watch the video. Let me know what you think? Am I way out in left field here? Too passionate?

Digital Fuel issued a couple PR’s announcing some impressive wins within very large telecommunications companies. Me thinks they’re ripe for an acquisition.

I’ve started to watch a bunch of new vendors this month. These all fit into various niche areas of the management and monitoring space and are very critical to a maturing Business Service Management deployment. True value oriented and powerful BSM can’t be done without capabilities offered by vendors such as these.

Integrien looks like the newest player in this “monitoring analytics” area. Netuitive and the former ProactiveNet (BMC) also play in this area. Integrien released version 6.0 of their product and apparently has addressed some of their scalability challenges. I was very impressed with what I saw in terms of their presentation layer, but haven’t seen much other than that. Steve Henning (VP Products, Integrien and ex-IBM Tivoli Security guy) joins the conversation and shares some insight as do many others who are very familiar with all three vendors in this space. I’ve invited Steve to guest author on the blog so maybe we can get some more insight into this much needed space!

I’ve always been pretty close to the user experience, user performance, synthetic/real-user monitoring segment. Three new vendors crossed my wires this month to join the others I’m pretty keen on (Keynote, Gomez, Tealeaf, Coradiant, IBM, HP, Quest, Compuware). Mature BSM deployments absolutely depend on the perspectives that vendors like this provide. It’s absolutely required for successful BSM and is the “glue” that joins the end-to-end service delivery chain together as the end user sees it.

Check out Knoa Software, Symphoniq and Aternity. If you have any information or experience with these vendors, I’d love to hear about it!

Almost as important to knowing how critical business services and applications are performing from the end user perspective, trying to really understand in instrument the ultra-critical transactions flowing across end-to-end services and applications is a sign of a very mature BSM deployment. In Two to Watch in Transaction Management and Monitoring Space” I call out Correlix and Correlsense as two that should be considered. I’m also looking for anyone with personal experience or information on these vendors and their technology. Correlsense’s CTO/Founder is an ex-IBMer and has a great start to a blog with teeth!

On the To-Do List for February

• The Next Generation of Business Service Management
• Hey! You got your monitoring in my RIA!
• New IBM Tivoli developerWorks collaboration sites
• More WYNTK on TBSM Design Patterns
• More in the DevCampTivoli Thought Provoking Series

What do you want to hear about?

One of the keys of a maturing Business Service Management deployment and solution rests heavily on insight into the key critical business service and application transactions. These are the things that businesses live and die by. They’re what get things done. They’re the “movers and shakers” that flow through the IT environment and when things go wrong, are often the most difficult to identify, isolate and repair. I had no fewer than five tools helping in this area when I ran the tools group at EarthLink. The mature BSM deployment will leverage insight and visibility into these business transactions, understand the impact on the business when one fails or performs poorly and ultimately have an intimate understanding of how the broader IT environment supports and enables business transactions to get things done.

Keep an eye on these up and comers!

Correlsense: Business Transaction Management - beautifully explained here. “The goal is to track every transaction end to end and correlate to the information collected from the infrastructure. Such an end-to-end view enables to quickly isolate and troubleshoot the root cause of performance problems and start tuning proactively. This application-centric information base enables a group of professionals working together to “speak” the same language and focus on facts, rather than guesswork.” Looks like they have a good pitch to start off with. Yet another agent though. Their CTO’s blog (ex-IBMer) has a nice intro post as well!

Correlix: “Business Transactions not performing as expected, affecting your customers, partners and revenues, and wondering how to resolve the problem? Correlix delivers a transaction performance management platform enabling companies to automatically discover, trace and optimize transactions across all layers of the enterprise data center. Correlix’s network based CorrelixTM, understands the heterogeneous nature of today’s complex data center and uniquely supports a wide variety of technologies agnostic to operating systems, applications, protocols deployed without interfering with production operations.” No product information on the website yet.