Category — Business Assurance

What’s the state of operationalizing Service Oriented Architecture (SOA) monitoring and management tools? Are the typical network, systems, enterprise operations/management centers (NOC/SOC/EOC/EMC) up to speed on how to manage, monitor, triage, troubleshoot and in general understand how SOA is being used in companies that are adopting it? Should the operations center care that they have an event from something related to SOA infrastructure and respond differently than they would for a non-SOA event? Have SOA events, incidents, problems, process and workflow been thoroughly implemented in such a way that “it just works” like traditional enterprise monitoring and management? Or, are these fancy SOA monitoring and management solutions really reserved for those applications experts responsible for complex application support and development?

If a client continues to struggle with fundamental e2e service monitoring and management, transaction monitoring and management or even batch job monitoring and management, what will their chances of success be for SOA monitoring and management? Could SOA and associated “service or transaction oriented monitoring” be a catalyst to shore up these other areas? Should one be tackled/improved before starting on another? At a minimum, instituting a “service oriented” organizational structure and mentality is certainly something I’d recommend for anyone adopting broad based SOA principles.

Eric Roch offers some solid advice on SOA Monitoring and Management which highlights that there’s more need for doing the fundamentals of systems, application and service management and monitoring really well as a foundation for SOA Monitoring and Management.

Others (and my preferred focus area) feel that monitoring SOA should really be more closely related to monitoring what this SOA initiative and deployment’s all about - the business. Business Service Management (BSM), Business Activity Monitoring (BAM) and Business Process Management (BPM) all play a key role in helping understand how IT infrastructure, systems, applications, etc. support and impact the business’s goals and objectives. The fairly new buzzword Business Transaction Management (BTM) spearheaded by Correlsense and OpTier really speaks to the desired need here.

I feel that it’s got to be a focus on both of these areas, but with a strong preference to the “B” buzzword set since most IT organizations are likely using the “improve/standardize/reuse/efficiency/time-to-market” spin to aide in business support and justification for their SOA initiatives. That said, you’d BETTER focus on the things that the business cares about and show them tangible evidence that your SOA initiative is making things better for them. This is very possible by adopting a BSM, BAM, BPM, BTM (or a preferred combination) strategy that focuses on providing the right level of business visibility into the SOA environment and more importantly the e2e business services, applications, transactions, processes and activities. It ultimately all ties back to the service level agreements delivered to the business anyway right?

What’s on the market these days for SOA Monitoring and Management? Should you get your monitoring and management tooling from your core SOA platform vendor or should you take a third party, “best of breed” approach? Are there true “vendor neutral” solutions out there? Are clients implementing SOA architectures based on multiple vendor’s technology, solutions and products?

• Progress/Actional
• Amberpoint
• Avicode
• CA Wily
• Correlsense
• HP
• IBM Tivoli ITCAM for SOA
• Manged Methods JaxView Suite
• MuleSource Galaxy and HQ
• Nastel Autopilot
• OpTier
• Oracle/BEA
• Progress (Actional, Sonic, Mindreef)
• SOA Software
• Symphoniq
• Tidal Software
• Truviso

Some additional content on some of these vendor solutions is available here.

Who might the “market leader” be of these SOA specific solutions? What makes them a leader? What capabilities, features, functions would be considered “best of breed”, differentiator, must have, core, desired, nice to have, etc.?

What’s “really” needed for SOA monitoring and management?

1. Web Services
2. ESB
3. Transaction Performance
4. Transaction Availability
5. Transaction State/Status
6. SOA Registry
7. SOA Security
8. Service Discovery and Relationship/Dependency Mapping
9. Transaction Discovery and Mapping

Anything else missing here? What here needs to be specialized in its own product versus just extending the investments clients have already made?

Please do share your thoughts here. There are folks lurking who really need help in figuring this stuff out and/or improving products and capabilities on the market today!

In an repost of an article from a couple years back, Robin Bloor provides some updated color on the state of BPM and SoA. It’s apparent that some of the other “B” buzz words have the same challenges that exist on the BSM front.

Things to ponder…

• How can these projects with such touted value to the business or IT be successfully implemented?
• Where are vendors falling short in helping “solve the organizational problems” that often cause these “B” projects to fail?
• Is throwing technology/product at the problem the best place to start?
• What should a next generation organizational structure look like? Can IT and Business organize around end-to-end business service/process delivery and support?
• How can organizations be incented, encouraged, mandated to have an end-to-end business service/process focus?
• Where success stories for BPM, SOA, BSM, BAM, etc. exist, how have these technologies been operationalized, organizations changed, workflow/process/procedure modified to reap the benefits?
• Is it foolish to think that any of these organizational challenges can ever be solved or at least minimized?
• Do we have generational issues here that will change as Baby Boomers retire and Gen X/Y/Z move up the ranks in IT and Business?

Give the post a read, I found these two very applicable to all of the things I’m seeing with BPM, BSM, BAM, etc.

Question 5: What are the most difficult steps within a BPM project – and what makes a SOA project tedious?

“B” is our middle name. We have “B” scattered throughout everything that we do. At times we fight over who owns the “B” word. I’m in search of a unified “B” story and solution. IMO, if we had this, it’d be tough to compete with us in any of the “B” acronyms.

The “B” Business TLA’s: Business Service Management (BSM), Business Process Management (BPM), Business Performance Management (BPM), Business Activity Monitoring (BAM), Business Transaction Management (BTM), Business Intelligence (BI) and I’m sure there are others.

What’s it going to take to have a unified “B” story and solution? Sure, we’ve probably got mentions in individual roadmaps and presentations of how we’ll integrate with this, share data what that, use Cognos here or there, send events from one tool to the other, etc. but what about a real “B” solution? IMO, these approaches just prolong client value and significantly delay any real innovation in core products.

What’s the cost of “forking” and creating a new solution entirely? One that focuses on becoming best of class in all of the “B” areas (ok, at least do all of them pretty darn good)? One that can be implemented and managed by one team free from (well, probably not) the organizational politics that’d exist if it was a “solution by integration” solution. One that has the best possible chance of truly aligning business and IT. Ok, this is probably cost prohibitive, but its GOT TO BE THE END GOAL!

This is where the politics come in unfortunately…where would you start? Which “B” is the most important “B”? Is it Business Service Management - my preference is here of course. Our friends in other organizations would see it other ways for sure. We must find the right way to develop the “B” story and solutions in ways that are most beneficial to the client. We must include content in each others products that “treads” on each others turf. We must have joint releases that build towards the unified “B” story and solution. When we release a new business process management suite (BPMS) we must include dashboards, models and integrations that provide value OUT OF THE BOX inside our BSM product. This must be backed up with the business and services consultants who have consultative based skills to guide our clients through the process because this isn’t about the product as much as it is about working through the organizational problems and politics.

A unified “B” story and solution may sound like a pipe dream, but it’s what clients really want to strive towards and our competitors are making giant strides in this direction. What would your ideal “B” solution look like? If you were king for a day …

Is there such a product or capability in a product that could do enterprise wide scheduling and calendaring? What I’m interested in is not related to people, meeting, conference room scheduling, but more along the lines of enterprise planning, BI, etc. where one could input into such a solution that ‘between the hours of 12pm and 2pm’ are the most critical hours of the day for this business service, application, transaction. Or something that could be the repository for killer business impact information based on BI type data feeds, etc. Or something that bleeds into service catalog/IT cost & usage that could track costs/impacts/revenues for use of business and IT services during a certain period, from a certain location, etc.?

Not sure if this is an SLA type tool as SLA’s and OLA’s may not always necessarily be involved. Something more along the lines of an uber-business-intelligence knowledgebase that’s business service, application, process, transaction, flow, etc. aware.

Just curious…

I worked up a pretty slick demo for a client this week who expressed an interest in Business Activity Monitoring (BAM) as part of their Service Oriented Architecture (SOA), Enterprise Service Bus (ESB) and Websphere MQ/Message Broker/ESB evaluations. I’ve had many thoughts on this topic from my past, but the work on this demo and discussions with the client affirmed that leveraging the Tivoli Netcool/RAD platform and enabling technolgies/products as an entry point into Business Service Management (BSM) or Business Activity Monitoring (BAM) is viable. Maybe it’s just the way my brain works, but bringing a Service Oriented Architecture and associated services to life with the Tivoli Netcool/RAD Business Service Management (BSM) / Business Activity Monitoring (BAM) solution was really quite easy.

I continue to be amazed by the depth of assets and resources that we have here at IBM. The things that I’m seeing come from our Service Oriented Architecture (SOA) efforts are incredible enablers for creation of the Business Service Management (BSM) and Business Activity Monitoring (BAM) practice. The BSM, BAM, BPM, B-I-N-G-O acronym soup of technologies, products and concepts will benefit greatly should the concepts of service orientation take hold in the mainstream and we can make it easy for clients to grasp these concepts, give them straight forward and easy ways to change from the legacy ways of enterprise and service provider architecture and design, and support them with solutions that help them manage, monitor and leverage these investments for the betterment of their business goals and objectives.

Lots of exciting times in the future for sure!

Let’s face it, here in North America most IT organizations are structured around functional silos of expertise. It’s not uncommon to see the Network Group, Windows Server Group, Unix Server Group, Mainframe Group, Application Group, Operations Group (NOC/OCC/ECC, etc), Tech Support/Help Desk, etc. These functional silos support everything and anything the IT organization delivers for the business. Organizing this way seems to be the norm here in the United States as it allows for certain economies of scale to be achieved in the functional silo areas. I’ve seen weekly/monthly reports highlighting measures such as “server admins per 100 servers” or “network engineers per 100 interfaces” as the key contribution from within the functional silo. The “we can do more with less” battle cry was born of the days of controlling costs and headcount. This process of organizing to achieve economies of scale is generally fairly broad at the lower levels of the IT infrastructure and tapers as you progress up the stack. There are fewer experts in the application areas, fewer database, web and application server administrators, etc. About the only time you see someone who claims ownership for a key business or IT service may be found in a “Product Manager” type role serving as a bridge (roadblock) between IT and the business.

Let’s look at this from the operations perspective. The typical Network Operations Center (NOC), Operations Center, Enterprise Command Center (ECC) or Help Desk are the “ones” who get “stuck” with some level of responsibility for how the business, clients, customers, etc. use the sum of all the parts developed and delivered by those functional IT silos. They get the calls that the service is unavailable, that transactions are failing, that the business is being impacted by poorly performing IT services, applications, etc. They resemble the responsible party, but generally aren’t given the responsibility, access, authority or tools to be anything more than ticket and triage experts who know how to follow the run book to escalate the tickets before the timer for the SLA is violated. Fair enough, this capability is needed.

But who really owns the service? Who owns the business experience? Who owns and is responsible for poorly performing services that impact business goals and objectives? Who doesn’t get their bonus paid out in full? Who gets a poor performance review? Who will stand up and take responsibility for the end-to-end service instead of pointing a finger towards another functional silo? Who will work with the business directly, in partnership, to manage the service as a whole, top to bottom, across all technologies, integration points, applications, and functional IT silos?

You’re all very well aware of how these organizations work. There is constant focus and incentives on silo specific tasks, projects and activities over end-to-end service delivery goals and objectives. Finger pointing, territorial mud slinging, gnashing of teeth, etc. are the water cooler conversations after the traditional “all hands on deck troubleshooting crisis conference bridge” session is complete. Do the goals and objectives of Business Service Management (BSM) have a chance becoming reality in these environments? I don’t think so. If these groups can’t check their emotions and egos at the door and realize that they exist for the business to meet business goals and objectives, Business Service Management will never mature to a high level in the Business Service Management maturity model. Yes, Business Service Management projects often start within these organizations. Yes, Business Service Management projects may be viewed as successful in these environments. But moving up the maturity model to a point where quantifiable value and direct business improvement can be measured will be a significant challenge in these organizations.

I don’t think this end-to-end service owner is the VP or Director of IT, MIS, or other common IT management roles or functions. They certainly play a role, but I have a feeling they’re just too distracted with the other pressures of running IT organizations today. I don’t think this is the Operations Center or Help Desk. These folks are often in the trenches, fighting fires, and dreaming of getting in front of problems rather than reacting to them. They do a great job managing the lifecycle of outages and incidents in the IT environment, escalating, communicating and manning the phones and this is a needed function as well. I’m not sure if this needs to be a new organization that breaks down those functional silos into functional service silos. It may make sense in smaller organizations to consider this, but you do need to have some economies of scale to manage larger environments. I see the early stages of Service Managers here in the USA but these tend to be too closely aligned with ITIL and IT process improvement efforts. They don’t appear to have enough responsibility to do anything other than define SLA/OLA/UPC’s related to service delivery and support.

Does a service management/ownership matrix overlay organization make sense? A matrix overlay that has named senior level ownership responsibility for each service where this senior level owner may be from the IT side or the business side of the organization. One that joins together named functional SME’s across the service delivery, service support silos AND from the business side to form a cross-functional, cross-organizational team that can step up to the plate in terms of service delivery and support ownership. One that has complete ownership of the service, associated transactions, processes, activities and flows that deliver IT and business services. Their goals and objectives would be aligned to service delivery and support that is in complete alignment with business goals and objectives.

Expanding the tenants of Business Service Management and IT - Business Alignment across an organizational structure similar to this would be a huge step forward in realizing the true and measurable value of Business Service Management. Aligning technology components with the business services and applications they deliver and support is an initial step, but aligning people, process, goals and objectives with the same is HUGE. I’d be very interested to hear your thoughts and comments on these ideas. What works, what doesn’t work, are organizations thinking like this? Am I way off base thinking like this???

Additional food for thought:

• New roles of the service aligned organization: Business Service Management (BSM) Analyst, Business Service Management (BSM) Architect, Business Service Architect, Service Modeling Architect, Transaction Management Engineer, Process Management Engineer
• Traditional IT organization management evolves into the emerging Enterprise Architecture function
• Shared IT Services, Virtualization, Grid and Utility Computing could be a key component to underpin this service aligned organization
• How to sell / deploy BSM into the “traditional” IT organization
• How to sell / deploy BSM into an emerging and evolving IT organization

The next release of the Netcool/RAD product is now generally available. The Netcool/RAD 3.0 relesase includes significant improvements over the Netcool/RAD 2.0 release (highlights below). If creating dynamic Business Service Management (BSM), Business Activity Monitoring (BAM) and IT Service Management (ITSM) solutions is in your future, I encourage you to investigate Netcool/RAD 3.0.

The flexibility that the Netcool/RAD 3.0 platform offers is tough to be matched by competing solutions. The ability to design, develop and release value-added business and IT service management and monitoring solutions (dashboards, scorecards, visualizations, complex service modeling and monitoring, process and workflow visualization and monitoring) is just a few mouse clicks away in many environments.

Release highlights include:

• Numerical Data Support: Incoming Status (event-mapping) rules, Aggregation (Dependency) rules, Formula (Composite) rules
• Wizards: Many point-click-create wizards to guide you through steps to create integrations, policies, visualizations
• Data Preview: Very useful for modeling integration of data into models, policies, dashboards, etc.
• Improved GUIs for Administration, Editors, Canvas/Dashboards, ESDAs, Custom Policies
• Active Service Navigation (Realtime Scorecards): Absolutely cool capability for creating rich scorecards quickly
• External Service Dependency Adapters (ESDA): Powerful integration mechanism for bringing in external service relationship and dependency information from other sources such as a CMDB, Asset or Inventory Database, NMS Solution, etc.
• Data Fetchers for SQL Databases: “Database Pollers” for integrating any data, metric, KPI, etc. into policies, rules, models, and dashboards
• Automatic provisioning of Netcool/ISMs and configuration for Netcool/ISM server connections
• Expanded RAD Shell command-line interface functions (if you can do it in the GUI, you can do it via the API), custom template and service property support, GIS map support
• Upgrade to Impact 3.1 architecture and Impact 3.1 expression language: Virtually limitless possibilities for policy creation, evaluation, integration, etc.
• Some of the best documentation I’ve seen for any of the Netcool products - great job Richard & team!

More information and documentation will be available for Netcool/RAD 3.0 on the IBM Tivoli websites later this month. If you’re an existing Micromuse client, you can access the Netcool/RAD 3.0 software and documentation via the legacy Micromuse support website here. The legacy location for Micromuse Netcool/BSM is located here.

Think you’ve got what it takes to turn complex data into something easily understood by an executive? Got a special knack for creating dashboards, reports, charts or graphs? Do this in your day job or as a (weird) hobby? The Business Intelligence Network announced its 2006 Data Visualization Competition. Download the spreadsheet and take a look at this contest based on five different business scenarios.

I like scenario #4 and may give it a go:

“You are a consultant who has been hired by a U.S. commercial airlines to design a dashboard for its executives. The information that the executive team wants to monitor has been identified and now its your job to create the dashboard’s visual design. You must try to display all of this information in some manner on a single screen such that the executives will be able to quickly identify anything that needs their attention and then have the means to discern enough about the situation to decide if they can ignore it for now or must perhaps take some action. It is up to you to determine the appropriate manner, level of detail, and means to display each piece of information.”

Fore more information, visit here.