Category — Event Processing

Just some rambling thoughts here…feel free to join in.

Is another tool what’s really required here? What should/could be done in domain specific resource monitoring solutions that addresses the problems at the edge? Should I really be monitoring everything that comes out of the box in a default configuration? Why do I have all of these profiles, situations, thresholds, events, etc. in the first place? Do I even now what I’m monitoring and why?

What if I have a multi-vendor, multi-sourced environment where I may or may not have visibility? What if I don’t have a CMDB or other source of topology, relationships and dependencies? What if I don’t even know the state and status of the applications, databases or services to begin with? What will I be able to do with investments into these technologies?

What if I have adopted a “manager of managers” concept where I have a consolidated operations eventing environment with feeds from across the entire business environment (facilities, plant, IT, datacenter, logistics, telephony, manufacturing, contact centers, etc.)? Shouldn’t this dynamic “learning” and “thresholding” concept be really applied at this level for some sort of “intelligent event management” free from manual intervention, policies, codebooks, etc? How about the context of the business calendar and schedule merged with the IT operations calendar and schedule? I doubt that this can all be “learned” magically.

If I invest in a BMC ProactiveNet, Netuitive or Integrien (or other fundamental dynamic “learning” or “trending” tool - my favorite was a company called Premonitia - now defunct, based on research from accoustic modelling of whales and shrimp IIRC), how will I recognize and measure the value from that investment? How should the operations environment change to adopt the promises of the “secret sauce” within these emerging technology areas? Will IT operations and second/third tier support teams need to change the ways they work today? If so, how? Does IT operations know how to respond to a future state that hasn’t occurred or someone stating that a service is “slow”? I think most operations and support teams are still in their infancy here.

I’m all for emerging technologies that speak towards making the lives of the folks on the front line better and for sensing, isolating and resolving issues within complex IT environments before they impact the business services, but will investing in these tools really improve the status quo within the typical operations environment? The Next Generation Operations Center, Command Center, Service Management Center or whatever we want to call it must be enabled with these types of technology, but also must prepared to think, operate and respond differently than they do today.

How are you changing? Will you change? Where’s your value proposition? Is it at the front line, second/third line of the support process, at the LoB? Is it about efficiencies in workflow? Do more, with less? Automation? Availability? Becoming proactive? Do you know the real root causes prompting your interests in this technology? What are your vendors doing about it? What is your monitoring tools group doing about it? Should they be doing something different?

Please share your thoughts on how best to operationalize and really recognize value from your investments into these technologies or what you’re doing to address the real root causes of the symptoms this technology addresses.

In this series of thought provoking posts I’ve asked for the ability to instrument for Business Service Management (BSM) at the managed system source and introduced a concept for an ITM 6.x BSM Profile and BSM Descriptor File. I’ve also proposed new organizational concepts that would establish end-to-end ownership for BSM within the typical monitoring tools group. As I peel the layers of this ITM 6.x product back, I’m now in search of the capability to create purpose built BSM Situations and BSM Events directly from ITM 6.x.

The first level of maturity in Business Service Management (BSM) is achievable by ensuring that a solid foundation in the fundamentals of network, system, application and service management and monitoring is in place. Where we’re failing our clients is not providing the necessary BSM best practices to help them use ITM 6.x with BSM as the end state. If clients have a well instrumented IT environment and if ITM 6.x has this capability, all events generated from ITM 6.x monitoring should include core BSM contextual information that establishes the most basic level of IT – Business alignment.

There’s no reason this needs to happen using more complex technology or products such as Netcool/Impact, TADDM or a CMDB. Sure, it may help make things easier, but the fact of the matter is that not all clients will have the ideal Tivoli environment with all of our enabling technology and products. Every client I’ve been to in the last two years has a heterogeneous environment with core products from all key vendors. If we don’t think about enabling fundamental BSM capabilities in ALL of our core products, we’re letting our clients down.

This BSM Situation and BSM Event concept would enable ITM 6.x clients to build BSM Situations that generate purpose built BSM Events. The key here is that every situation within ITM 6.x needs to allow for a purpose built BSM Situation with its own BSM contextual information, policies, thresholds, business calendars, etc. to be associated with it. This would then enable key BSM Event field information to be mapped into the core event emitted. BSM Situations and BSM Events may stand on their own and never be seen by the traditional NOC/EOC or support operator. Think of certain information such as common system information, metrics, KPIs, performance or capacity data that simply flows northbound to build or drive the BSM models and scorecards within TBSM.

Some of my initial questions:

• What capabilities do we have to do something like this?
• Could a BSM Situation be triggered by another situation and map in key BSM information into the generated event?
• What attributes can be mapped in?
• Is there a limit?
• Can attributes read from a file (the BSM Descriptor file) on the managed system?
• Can there be custom attributes defined in the BSM Situation?
• How many?
• How and where does information get mapped into the event format?
• How can every field of a generated event be controlled, overwritten or customized? (message summary, custom fields, etc.)
• Can I create custom slots/fields in the outgoing event?
• How many?

My initial queries to the experts and skimming of our manuals and other internal training materials leads me to the conclusion that these fundamental systems management capabilities do not exist in ITM 6.x. I hope I am wrong. I hope there is some way to do this. My end state objective here is that I get events flowing northbound from ITM 6.x monitoring that convey the critical BSM information within the event such as business services, applications, transactions, LoB, Clients, server OS, location, support group, compliance/risk classification, business impact, etc. I do not want to have to add this upstream unless it’s absolutely necessary.

In an effort to collaborate on how to generate powerful events that convey the most fundamental IT - Business alignment and help clients reach the first phase of Business Service Management, DevCampTivoli has been created. The theme for this event is “Collaborative Development of End-to-End BSM Solutions”. The desired outcome is to come up various approaches for developing a BSM Situations and BSM Events from ITM 6.x and the necessary configurations within the Tivoli EIF probe, Netcool/OMNIbus and TBSM 4.x that can be easily customized and implemented at any client. Whatever the DevCampTivoli outcome is it will be freely available to anyone to take, modify and use to improve their BSM deployments.

Take a few minutes to visit DevCampTivoli. This event will be the May 17-18, 2008 which is the weekend before the annual IBM Tivoli User Conference Pulse 2008 in Orlando, FL. The thought and hope is that SME’s and practitioners in ITM, Netcool/OMNIbus and TBSM will already be coming to Pulse 2008 and will be able to come in a couple days earlier to participate.

More to follow…

I’m always skeptical by what I see in a demo until I can dig into what’s under the covers, but what I saw in the Integrien Alive demo impressed me. It looks like what could be a solid foundation for Business Service Management (BSM) in the future with focus by Integrien in key areas such as dashboard visualization, modeling and alignment to business services and applications.

It looks like Integrien competes firmly with Netuitive and the former ProactiveNet (now BMC), maybe Firescope and Managed Objects to some degree.

Effective, trusted and value oriented Business Service Management absolutely depends on an accurate data stream whether it be events, metrics, KPIs, etc. Taking the default out of the box configurations and thresholds with your monitoring tools and poor monitoring and event management lifecycles has led to the development of solution such as Integrien’s to “take back control” and give you back a trusted insight into IT infrastructure.

I’d love to see or hear more about Integrien technology. Anyone have any first hand experience? IMO, we have a gap in the IBM Tivoli portfolio in this technology and capability area.

I’ve heard some pretty cool stuff can be done with this product. Haven’t had the chance to play with it yet, but it sounds like it’s the “swiss army knife” tool much like Netcool/Impact is.

Creating EIF events with Tivoli Directory Integrator for Tivoli Netcool/OMNIbus and Tivoli Enterprise Console Redpaper draft available here.

—snip—

This Redpaper describes an integration solution developed for IBM Tivoli Directory Integrator’s with IBM’s event management offering products IBM Tivoli Netcool/OMNIbus and IBM Tivoli Enterprise Console.

This integration solution illustrates an integration scenario aimed towards improving Tivoli Directory Integrator’s integration capabilities and leveraging these capabilities with IBM’s event management offering products. Along with some examples we discuss the architecture behind this approach.

This document is divided into several sections. For those readers who are not familiar with the IBM products covered in this Redpaper, we provide a brief overview of Tivoli Directory Integrator, Tivoli Netcool/OMNIbus, and Tivoli Enterprise Console. We then cover the integration with Netcool/OMNIbus and describe an architectural overview, and the implementation, installation and configuration for Tivoli Directory Integrator’s integration with Netcool/OMNIbus. Similarly, we discuss the integration with Tivoli Enterprise Console and describe an architectural overview, and the implementation, installation and configuration for Tivoli Directory Integrator’s integration with the Tivoli Enterprise Console. We discuss further details about the EIF EventSender component because it represents a key component developed as part of this Redpaper integration. Finally, the Miscellaneous section documents the files shipped along with this Redpaper and links to various official documentation.

Our third Netcool/Impact OPAL contribution!

Available here.

-snip-

This integration will send IBM Tivoli Monitoring Situation Events to Netcool/Impact for event enrichment, advanced data analysis and correlation, and for notifications and escalations.

The IBM Tivoli Monitoring Situation Events are sent to Netcool/Impact via Web Services. Upon reception of the event Netcool/Impact will run one or more Netcool/Impact policies. These policies make use of Netcool/Impacts wide array of Data Source Adapters and Netcool/Impact Policy Language to perform event enrichment, advanced data analysis, and to perform notifications and escalations.

After the policy is finished, Netcool/Impact can write the results back to the IBM Tivoli Monitoring Universal Message Console where the results can be viewed by support staff.

Douglas “Dougie” Stevenson has initiated a Sourceforge project focused on developing a high performance, ultra scalable polling, correlation and event management engine. If you don’t know Dougie, he’s got a tremendous history and knowledge in this space. He’s built some powerful tools like this before and I’m sure he’ll do wonders with this new project. He’s the ultimate geek, coder, hacker, bits and bytes twiddler that is driven by taking the concepts and ideas he’s laid out below to places nobody has imagined they could go before.

I encourage you to check out the project page and get in touch with Dougie if you’d like to participate. It’d be great to see a modular approach taken here where the efforts of the many other great OSS projects can all come together to form that ecosphere I’ve mentioned before in the blog. (something for inventory/discovery like ZipTie, visualizations/dashboards/wiki/mashups/knowledgebase/enrichment like ??, rules/workflow/bpel/bpms like Intalio, integration/esb/glue with Mule, etc.)

-snip-

In getting this project off the ground, I’m putting together the requirements, features, and use cases for the different functions of DICE.

I’m also soliciting input, thoughts, and ideas of what could be put into a World class Correlation Engine. So, if you want to contribute, I’m all Ears!

Within the basic requirements, the needs include:

• Scaling to handle greater than 1 million events a minute.
• Be able to dynamically add and subtract handler components on the fly.
• Be able to accomodate a single Control port to the entire distributed application such that any component can be controlled, queried, and monitored via this control port.
• Be able to take raw data feeds from Syslog, various log files in differing file formats, SNMP Trap data, SOAP Services, and assign handlers and process these as a function of workflow.
• Be able to log and track event records throughout the process.
• Be able to enable administrative users to build and modify filtering, parsing, and processing rules as part of a web based build process.
• Be able to add, remove, or schedule changes related to built and tested filters, parsers, and processing functions without causing application downtime.
• Be able to display Objects and states via a Tabular type display.
• Be able to provide state information to an iconic, canvas based map sort of display. This map display ought to be exportable to Visio.
• Be able to enact state based polling via SNMP, TCP connections, and SSH.
• Be able to provide for an intelligent MIB Compiler and analysis function. MIBs should be loadable in any order and be able to be verified and tested against devices in the infrastructure. In effect, the MIB compiler function needs to be able to document differences between a published MIB Structure and real attributes from a given devices’s agent.
• Workflow and states mechanisms should be documented within the product in BPEL format as well as Excel Spreadsheet formats.

Looks like an innovative client and the folks over at Splunk got togther and put together a nice way to leverage Splunk with Netcool/OMNIbus. Details can be found here.

Giving operations and support folks a capability to work with events in an easier way that’s relevent to their jobs and real time daily responsibilities is crucial. The traditional sort, filter, this but not that approach to working with events is past its time. Leading edge event management techniques incorporating direct linkages with all other IT resource, application and service information (Splunk), instant collaboration (RSS Event Feeds, Event Wikis, Event/Incident IM Channels), trending/reporting/analytics, situational management mashups (shift based event/incident management tear-aways, in-flight rules/analytics, workflow, etc.) and other contextual management applications which are developed and managed by the individual users instead of the tools group is the future.

Thinking outside the box with a focus on how to do things better, faster (agile, less-code) and cheaper such as this is an area ripe for the OSS community to create an ecosphere of new tools, applications and add-ons that can greatly complement and enhance the traditional network, systems, application and servivce management and monitoring vendor solutions. The front line operations and support groups and the tools group would be two key IT organizations that would greatly benefit from initiatives such as this.

-snip-

Integrated IT Data Search with IBM Netcool and Splunk

Splunk-2-Netcool is an integrated module that provides seamless workflow and data integration between Splunk Professional and IBM Tivoli Netcool. It allows Netcool customers to launch Splunk directly from the Netcool/Webtop and Netcool/OMNIbus Event List. It also configures Splunk to seamlessly index events from any Netcool ObjectServer, to provide the ability to search Netcool events alongside other kinds of IT data, such as logs and configuration files from servers and applications. Finally, it allows Live Splunk alerts to be forwarded to a Netcool ObjectServer for notification and correlation.

Looks like we’ve got some Netcool/OMNIbus contributions on OPAL now. This is at least a start. There’s a ton of stuff that needs to migrate from the legacy Netcool Tools & Utilities section, GAT, etc. as those legacy sites are shutting down soon. Hopefully this will continue to grow as things get “bluewashed” as we say.

Story on Tivoli Beat here.