Category — BPM 2.0

What’s the state of operationalizing Service Oriented Architecture (SOA) monitoring and management tools? Are the typical network, systems, enterprise operations/management centers (NOC/SOC/EOC/EMC) up to speed on how to manage, monitor, triage, troubleshoot and in general understand how SOA is being used in companies that are adopting it? Should the operations center care that they have an event from something related to SOA infrastructure and respond differently than they would for a non-SOA event? Have SOA events, incidents, problems, process and workflow been thoroughly implemented in such a way that “it just works” like traditional enterprise monitoring and management? Or, are these fancy SOA monitoring and management solutions really reserved for those applications experts responsible for complex application support and development?

If a client continues to struggle with fundamental e2e service monitoring and management, transaction monitoring and management or even batch job monitoring and management, what will their chances of success be for SOA monitoring and management? Could SOA and associated “service or transaction oriented monitoring” be a catalyst to shore up these other areas? Should one be tackled/improved before starting on another? At a minimum, instituting a “service oriented” organizational structure and mentality is certainly something I’d recommend for anyone adopting broad based SOA principles.

Eric Roch offers some solid advice on SOA Monitoring and Management which highlights that there’s more need for doing the fundamentals of systems, application and service management and monitoring really well as a foundation for SOA Monitoring and Management.

Others (and my preferred focus area) feel that monitoring SOA should really be more closely related to monitoring what this SOA initiative and deployment’s all about - the business. Business Service Management (BSM), Business Activity Monitoring (BAM) and Business Process Management (BPM) all play a key role in helping understand how IT infrastructure, systems, applications, etc. support and impact the business’s goals and objectives. The fairly new buzzword Business Transaction Management (BTM) spearheaded by Correlsense and OpTier really speaks to the desired need here.

I feel that it’s got to be a focus on both of these areas, but with a strong preference to the “B” buzzword set since most IT organizations are likely using the “improve/standardize/reuse/efficiency/time-to-market” spin to aide in business support and justification for their SOA initiatives. That said, you’d BETTER focus on the things that the business cares about and show them tangible evidence that your SOA initiative is making things better for them. This is very possible by adopting a BSM, BAM, BPM, BTM (or a preferred combination) strategy that focuses on providing the right level of business visibility into the SOA environment and more importantly the e2e business services, applications, transactions, processes and activities. It ultimately all ties back to the service level agreements delivered to the business anyway right?

What’s on the market these days for SOA Monitoring and Management? Should you get your monitoring and management tooling from your core SOA platform vendor or should you take a third party, “best of breed” approach? Are there true “vendor neutral” solutions out there? Are clients implementing SOA architectures based on multiple vendor’s technology, solutions and products?

• Progress/Actional
• Amberpoint
• Avicode
• CA Wily
• Correlsense
• HP
• IBM Tivoli ITCAM for SOA
• Manged Methods JaxView Suite
• MuleSource Galaxy and HQ
• Nastel Autopilot
• OpTier
• Oracle/BEA
• Progress (Actional, Sonic, Mindreef)
• SOA Software
• Symphoniq
• Tidal Software
• Truviso

Some additional content on some of these vendor solutions is available here.

Who might the “market leader” be of these SOA specific solutions? What makes them a leader? What capabilities, features, functions would be considered “best of breed”, differentiator, must have, core, desired, nice to have, etc.?

What’s “really” needed for SOA monitoring and management?

1. Web Services
2. ESB
3. Transaction Performance
4. Transaction Availability
5. Transaction State/Status
6. SOA Registry
7. SOA Security
8. Service Discovery and Relationship/Dependency Mapping
9. Transaction Discovery and Mapping

Anything else missing here? What here needs to be specialized in its own product versus just extending the investments clients have already made?

Please do share your thoughts here. There are folks lurking who really need help in figuring this stuff out and/or improving products and capabilities on the market today!

“B” is our middle name. We have “B” scattered throughout everything that we do. At times we fight over who owns the “B” word. I’m in search of a unified “B” story and solution. IMO, if we had this, it’d be tough to compete with us in any of the “B” acronyms.

The “B” Business TLA’s: Business Service Management (BSM), Business Process Management (BPM), Business Performance Management (BPM), Business Activity Monitoring (BAM), Business Transaction Management (BTM), Business Intelligence (BI) and I’m sure there are others.

What’s it going to take to have a unified “B” story and solution? Sure, we’ve probably got mentions in individual roadmaps and presentations of how we’ll integrate with this, share data what that, use Cognos here or there, send events from one tool to the other, etc. but what about a real “B” solution? IMO, these approaches just prolong client value and significantly delay any real innovation in core products.

What’s the cost of “forking” and creating a new solution entirely? One that focuses on becoming best of class in all of the “B” areas (ok, at least do all of them pretty darn good)? One that can be implemented and managed by one team free from (well, probably not) the organizational politics that’d exist if it was a “solution by integration” solution. One that has the best possible chance of truly aligning business and IT. Ok, this is probably cost prohibitive, but its GOT TO BE THE END GOAL!

This is where the politics come in unfortunately…where would you start? Which “B” is the most important “B”? Is it Business Service Management - my preference is here of course. Our friends in other organizations would see it other ways for sure. We must find the right way to develop the “B” story and solutions in ways that are most beneficial to the client. We must include content in each others products that “treads” on each others turf. We must have joint releases that build towards the unified “B” story and solution. When we release a new business process management suite (BPMS) we must include dashboards, models and integrations that provide value OUT OF THE BOX inside our BSM product. This must be backed up with the business and services consultants who have consultative based skills to guide our clients through the process because this isn’t about the product as much as it is about working through the organizational problems and politics.

A unified “B” story and solution may sound like a pipe dream, but it’s what clients really want to strive towards and our competitors are making giant strides in this direction. What would your ideal “B” solution look like? If you were king for a day …

Is there such a product or capability in a product that could do enterprise wide scheduling and calendaring? What I’m interested in is not related to people, meeting, conference room scheduling, but more along the lines of enterprise planning, BI, etc. where one could input into such a solution that ‘between the hours of 12pm and 2pm’ are the most critical hours of the day for this business service, application, transaction. Or something that could be the repository for killer business impact information based on BI type data feeds, etc. Or something that bleeds into service catalog/IT cost & usage that could track costs/impacts/revenues for use of business and IT services during a certain period, from a certain location, etc.?

Not sure if this is an SLA type tool as SLA’s and OLA’s may not always necessarily be involved. Something more along the lines of an uber-business-intelligence knowledgebase that’s business service, application, process, transaction, flow, etc. aware.

Just curious…

webMethods’ has put out a new whitepaper titled “Business Activity Monitoring (BAM): The New Face of BPM” that’s a good read. For those new to the acronym, check out the BAM basics section in the document or my references on the side panel. I need to dig more into their interesting “BAM Fingerprint” concept.

They’re also hosting a webinar on August 16th.

Bruce and Ismael have a bet on over whether a top down approach to BPM has ever worked. Anyone out there working on BPM or BPMS should check out Bruce’s post here and help him out!

Ismael talks about how clients are asking for more content out of the box from their BPM vendors.

It’s certainly a similar problem in other vendor product and technology areas as well - and a big reason I came onboard Micromuse / IBM Tivoli was to contribute in this area. Will we be everything for everyone, of course not. Are we on the right track, I think so. I’ve been in more converstations recently that all sounded very similar - more out of the box, less “do it ourselves”, more vendor support and quicker.

Resources will always be challenged and budgets tight. We’ve got to rally the troops (clients, practitioners, partners, vendors) and work togther on some of this stuff - in accordance with open standards and practices. A set of ITIL templates straight from the books would go a long way towards giving people that 80% they need to start. Then allow them to EASILY customize, integrate and operationalize into their own environment. The Open Management Consortium may be the right place to start. Ismael, have you considered joining in this effort? Your technology would be a good fit!

I’ve got a huge list of things I’d like to do to enable our clients and partners via OPAL, ITUP and other similar IBM Tivoli outlets. The ITSM story and CCMDB come to life in June and I think we’ll be positioned very well with capabilities much like Ismael talks about. It’s neat to see the sum of all the parts come together to enable our clients with the potential to really make changes in their IT environments. Stay tuned for sure!

The folks over at RSSBus have taken a few more covers off of thier RSSBus product. I’ve played around with the RSSBus Desktop Server some now and continue to believe in the potential it has in many of the areas I write about in this blog, especially enabling the “average person” to publish events for consumption by business rules, event and visualization solutions. I’m very excited about the ability to suck metrics, kpi/kpm, etc. out of all those BASS out there!

Check out RSSBus here and download their public beta here. Read the whitepaper, it’s really good.

They’re keeping a blog here where you can follow the product’s progress. I’ve had private email exhanges with their CEO Gent Hito about their plans for the product. They plan to keep parts of it free and are considering open source for parts. Reach out and encourage them to consider this!

I’ve taken you through the trenches of the organization and IT environment to find and capture what’s important to your audience. If you need to catch up, don your safari hat, some boots and check out this page. The next part in this series is one of my favorite areas and probably the most important. This is the part where you’ll show off the fruits of your labor, where the rubber meets the road in terms of how valuable your work and solutions will be for the business and your audiences. This is also the part that everything you do can come into question, be challenged, or simply blown off as garbage, eye candy or a waste of time and money.

Visualization of data and information is an art in itself. There have been many books written on the subject. See the Dashboards page for a list of references. Our goal here is simple. Take everything you’ve done to this point and present the message in the most meaningful, efficient and effective way possible for your audience’s consumption. Your challenge is to figure out what works best for your audience and to ensure that the message can be consumed and have the desired effects of prompting action, decision making, etc.

Iteration is key in getting the visualization right. Allow for a considerable amount of time in your project plans for work in this area. I strongly recommend mockups and prototyping in Microsoft PowerPoint, Excel, Visio or your favorite graphics program such as Adobe Illustrator, Photoshop or Fireworks. A tip here is to look at the dashboard references or vendor products/presentations and cut/copy/paste the widgets (dials, gauges, charts, etc.) into your mockups and prototypes. It’ll help to be as close to what your capabilities are as you review with your various audiences. Keep on this task until you get buy in and a sense that this will work for them. Seek the 80% rule here.

I encourage you to ask your vendor for examples of successful dashboard deployments. See if you can speak to referencable customers and then really dive in with them about how they’ve visualized messages within their environments. A new blog is available that has been collecting examples of dashboards and visualizations called Dashboard Spy. I encourage you to take a look at what has been captured here for ideas.

There are references and links on the Dashboards page that will help you with all the right and wrong uses of gauges, stoplights, sliders, stoplights, charts, etc. I won’t go into those because I don’t necessarily have an opinion one way or another on what they are saying. I do know that every audience will be different. There will likely not be a one size fits all representation of your message. You may be able to get it to look similar, but I promise there will be someone who prefers a gauge or dial instead of stoplight or chart.

Once you’ve got the mocked up visualization of your message complete, it’s time to start implementing it within your solution. I’ll defer from speaking in detail on how to do this, but we’ve previously talked about how to generate events on what’s important and the message. Your solution should offer an easy way to extract this information from an event, database, or any other datasource for alignment and mapping into the visualization components that you will be using. It should be capable of processing these events, metrics, etc. in large quantities in real-time from a large number of distinct sources. You should be able to apply analytical logic, rules, calculations and statistical evaluations, timers, counters, etc. to any single piece of data or any group or collection of data. I’d be happy to recommend a very flexible solution for accomplishing this within your environment off-line.

Release your visualization into a controlled production environment and let it run over the course of the normal cycles associated with your message, what’s important and your data sources. Make sure you’ve also built up enough of the contextual references that may be needed. If you feel the visualization is at a point where it closely resembles your mockups using your solution and real data, it’s time to review and level set with your various audiences.

Get their feedback. Show them how it will work in production. Ask them if they “get” the message. Can they tell you what needs to be done or the state of the business? Will it work? Do they believe in it? Stand behind it? Iterate here until you get to this point. Go back and review everything you’ve done up to this point - discovery interviews, audience needs, what’s important, the message and make sure you’ve got everything covered.

When you’re 100% there, release into your production environment and place under your normal change, configuration and document controls. Establish a monthly or quarterly (at a minimum) review cycle to sit back down with the various audiences and review the solution with them. Talk with them. See how they use it. Capture metrics associated with any improvements, value, savings, etc. that can be attributed to the solution. Get them to vouch for these accomplishments. Don’t walk away from this review meeting without knowing what’s working, not working or needs to be changed or updated. There is nothing worse than a solution that’s not used or is ignored because it’s out of date or providing no value.

Here are some general guidelines I’ve picked up over the years will help ensure your success. Check the Dashboards page for more ideas and starting points. There are a lot of really good things out there from the BI, BPM and analytics folks!

Consumability

• The message (what’s important) should be communicated in seven (7) seconds or less (one page/screen of information)
• Choose 3-5 key messages, themes or topics to communicate for each audience or each level (see “Determining Your Audience” and “Determining Your Message“)
• Keep things aggregated, correlated and presented in summarized views that prompt action
  (Is the ship on the right path? Do I need to take some action to steer around the iceberg? , How quickly do I need to take action?)
• Try to convey a sense of movement or flow in a uniform manner for visualizations that represent activity, processes, workflow, transactions, etc. Keep them top to bottom or left to right as much as possible.
• Try to draw your audience’s eyes to the most important parts of the message. Don’t let these get lost on the page.
• Think Web2.0 - white space, rounded corners, smoothness, etc.
• Don’t use wild color schemes. Avoid eye candy, all black backgrounds, etc. for more executive and non-technical audiences.

Freshness

• Ideally one hour updates or more frequently
• No more than one week’s data points on a dashboard (just enough to have some context on what’s happening)
• Goal is to manage where the business is going “real-time” (using GPS) versus where we were yesterday (looking in rear view mirror)

Provide Context Where Relevant

• Historical view/info providing as needed context for making decisions
• Aggregate daily, weekly snapshot
• Provide comparables - Hour of day compared to hour of day, Day of week compared to day of week, Week in month compared to week in month
• Link out to or reference other sources that may provide context - avoid replicating data

Ease of Use

• Should support drill down from any click-point or metric and maintain context through every click through
• A common display panel is desirable for maintaining context. Clicking on a metric or indicator causes the results to be displayed in the common window.
• Double clicking or right clicking would cause drill down
• Hover displays are also useful for this approach, but not for key metric or indicator display (you want those immediately visible)
• -“Breadcrumbs” should be used to help understand where in the navigation drilldown someone is and how they can get back to upper layers

Organizational Politics

• Anticipate questions that may be asked
• Have your ducks in a row - what’s important and why, your metrics catalog, source quality, etc.
• Avoid overlap and “competition” with data warehouse, business intelligence/analysis or enterprise reporting groups
• If results make an organization, department, group or person look bad, seek them out in advance to review and prepare them as needed

Stay tuned for my next planned topic in the “You’ve Got Events, Now What? series where I’ll focus on “Managing in Real-Time”.

Catch up with the “You’ve Got Events, Now What?” series here.