Category — Event Processing

Douglas “Dougie” Stevenson has initiated a Sourceforge project focused on developing a high performance, ultra scalable polling, correlation and event management engine. If you don’t know Dougie, he’s got a tremendous history and knowledge in this space. He’s built some powerful tools like this before and I’m sure he’ll do wonders with this new project. He’s the ultimate geek, coder, hacker, bits and bytes twiddler that is driven by taking the concepts and ideas he’s laid out below to places nobody has imagined they could go before.

I encourage you to check out the project page and get in touch with Dougie if you’d like to participate. It’d be great to see a modular approach taken here where the efforts of the many other great OSS projects can all come together to form that ecosphere I’ve mentioned before in the blog. (something for inventory/discovery like ZipTie, visualizations/dashboards/wiki/mashups/knowledgebase/enrichment like ??, rules/workflow/bpel/bpms like Intalio, integration/esb/glue with Mule, etc.)

-snip-

In getting this project off the ground, I’m putting together the requirements, features, and use cases for the different functions of DICE.

I’m also soliciting input, thoughts, and ideas of what could be put into a World class Correlation Engine. So, if you want to contribute, I’m all Ears!

Within the basic requirements, the needs include:

• Scaling to handle greater than 1 million events a minute.
• Be able to dynamically add and subtract handler components on the fly.
• Be able to accomodate a single Control port to the entire distributed application such that any component can be controlled, queried, and monitored via this control port.
• Be able to take raw data feeds from Syslog, various log files in differing file formats, SNMP Trap data, SOAP Services, and assign handlers and process these as a function of workflow.
• Be able to log and track event records throughout the process.
• Be able to enable administrative users to build and modify filtering, parsing, and processing rules as part of a web based build process.
• Be able to add, remove, or schedule changes related to built and tested filters, parsers, and processing functions without causing application downtime.
• Be able to display Objects and states via a Tabular type display.
• Be able to provide state information to an iconic, canvas based map sort of display. This map display ought to be exportable to Visio.
• Be able to enact state based polling via SNMP, TCP connections, and SSH.
• Be able to provide for an intelligent MIB Compiler and analysis function. MIBs should be loadable in any order and be able to be verified and tested against devices in the infrastructure. In effect, the MIB compiler function needs to be able to document differences between a published MIB Structure and real attributes from a given devices’s agent.
• Workflow and states mechanisms should be documented within the product in BPEL format as well as Excel Spreadsheet formats.

Looks like an innovative client and the folks over at Splunk got togther and put together a nice way to leverage Splunk with Netcool/OMNIbus. Details can be found here.

Giving operations and support folks a capability to work with events in an easier way that’s relevent to their jobs and real time daily responsibilities is crucial. The traditional sort, filter, this but not that approach to working with events is past its time. Leading edge event management techniques incorporating direct linkages with all other IT resource, application and service information (Splunk), instant collaboration (RSS Event Feeds, Event Wikis, Event/Incident IM Channels), trending/reporting/analytics, situational management mashups (shift based event/incident management tear-aways, in-flight rules/analytics, workflow, etc.) and other contextual management applications which are developed and managed by the individual users instead of the tools group is the future.

Thinking outside the box with a focus on how to do things better, faster (agile, less-code) and cheaper such as this is an area ripe for the OSS community to create an ecosphere of new tools, applications and add-ons that can greatly complement and enhance the traditional network, systems, application and servivce management and monitoring vendor solutions. The front line operations and support groups and the tools group would be two key IT organizations that would greatly benefit from initiatives such as this.

-snip-

Integrated IT Data Search with IBM Netcool and Splunk

Splunk-2-Netcool is an integrated module that provides seamless workflow and data integration between Splunk Professional and IBM Tivoli Netcool. It allows Netcool customers to launch Splunk directly from the Netcool/Webtop and Netcool/OMNIbus Event List. It also configures Splunk to seamlessly index events from any Netcool ObjectServer, to provide the ability to search Netcool events alongside other kinds of IT data, such as logs and configuration files from servers and applications. Finally, it allows Live Splunk alerts to be forwarded to a Netcool ObjectServer for notification and correlation.

Looks like we’ve got some Netcool/OMNIbus contributions on OPAL now. This is at least a start. There’s a ton of stuff that needs to migrate from the legacy Netcool Tools & Utilities section, GAT, etc. as those legacy sites are shutting down soon. Hopefully this will continue to grow as things get “bluewashed” as we say.

Story on Tivoli Beat here.

Link to a presentation given on EDA by K. Mani Chandy of Cal Tech: http://complexevents.com/?p=93

He introduces a strategy he calls BAM++ which makes a lot of sense:

-snip-

“Here’s a strategy that will work for many of your enterprises. I call it a BAM++ strategy. The idea is to start with BAM and then add a function: determine if reality deviates from expectation.

The value proposition is that instead of having the business user, say the CFO, continuously monitor the portal, the EDA system will monitor the portal for the user. When something significant happens then (1) alert the appropriate people, and (2) bring links to the appropriate tools into the portal with links to the appropriate data, so that the business user can immediately respond to the threat or opportunity.

The value proposition here is the attention amplifier. This helps the business user amplify his/her attention, and respond rapidly with appropriate tools when a situation arises.

The advantage of the BAM application is that the sensor data — the data that identifies reality — is already present; it’s sending data to the portal. So, you don’t need to connect to new data sources. Secondly, the issue of error is already understood. If the data shows up in the portal, it is sufficiently accurate to be useful. Thirdly, improving BAM seems less radical than developing an event-driven application.”

-snip-

The folks over at RSSBus have taken a few more covers off of thier RSSBus product. I’ve played around with the RSSBus Desktop Server some now and continue to believe in the potential it has in many of the areas I write about in this blog, especially enabling the “average person” to publish events for consumption by business rules, event and visualization solutions. I’m very excited about the ability to suck metrics, kpi/kpm, etc. out of all those BASS out there!

Check out RSSBus here and download their public beta here. Read the whitepaper, it’s really good.

They’re keeping a blog here where you can follow the product’s progress. I’ve had private email exhanges with their CEO Gent Hito about their plans for the product. They plan to keep parts of it free and are considering open source for parts. Reach out and encourage them to consider this!

I’ve taken you through the trenches of the organization and IT environment to find and capture what’s important to your audience. If you need to catch up, don your safari hat, some boots and check out this page. The next part in this series is one of my favorite areas and probably the most important. This is the part where you’ll show off the fruits of your labor, where the rubber meets the road in terms of how valuable your work and solutions will be for the business and your audiences. This is also the part that everything you do can come into question, be challenged, or simply blown off as garbage, eye candy or a waste of time and money.

Visualization of data and information is an art in itself. There have been many books written on the subject. See the Dashboards page for a list of references. Our goal here is simple. Take everything you’ve done to this point and present the message in the most meaningful, efficient and effective way possible for your audience’s consumption. Your challenge is to figure out what works best for your audience and to ensure that the message can be consumed and have the desired effects of prompting action, decision making, etc.

Iteration is key in getting the visualization right. Allow for a considerable amount of time in your project plans for work in this area. I strongly recommend mockups and prototyping in Microsoft PowerPoint, Excel, Visio or your favorite graphics program such as Adobe Illustrator, Photoshop or Fireworks. A tip here is to look at the dashboard references or vendor products/presentations and cut/copy/paste the widgets (dials, gauges, charts, etc.) into your mockups and prototypes. It’ll help to be as close to what your capabilities are as you review with your various audiences. Keep on this task until you get buy in and a sense that this will work for them. Seek the 80% rule here.

I encourage you to ask your vendor for examples of successful dashboard deployments. See if you can speak to referencable customers and then really dive in with them about how they’ve visualized messages within their environments. A new blog is available that has been collecting examples of dashboards and visualizations called Dashboard Spy. I encourage you to take a look at what has been captured here for ideas.

There are references and links on the Dashboards page that will help you with all the right and wrong uses of gauges, stoplights, sliders, stoplights, charts, etc. I won’t go into those because I don’t necessarily have an opinion one way or another on what they are saying. I do know that every audience will be different. There will likely not be a one size fits all representation of your message. You may be able to get it to look similar, but I promise there will be someone who prefers a gauge or dial instead of stoplight or chart.

Once you’ve got the mocked up visualization of your message complete, it’s time to start implementing it within your solution. I’ll defer from speaking in detail on how to do this, but we’ve previously talked about how to generate events on what’s important and the message. Your solution should offer an easy way to extract this information from an event, database, or any other datasource for alignment and mapping into the visualization components that you will be using. It should be capable of processing these events, metrics, etc. in large quantities in real-time from a large number of distinct sources. You should be able to apply analytical logic, rules, calculations and statistical evaluations, timers, counters, etc. to any single piece of data or any group or collection of data. I’d be happy to recommend a very flexible solution for accomplishing this within your environment off-line.

Release your visualization into a controlled production environment and let it run over the course of the normal cycles associated with your message, what’s important and your data sources. Make sure you’ve also built up enough of the contextual references that may be needed. If you feel the visualization is at a point where it closely resembles your mockups using your solution and real data, it’s time to review and level set with your various audiences.

Get their feedback. Show them how it will work in production. Ask them if they “get” the message. Can they tell you what needs to be done or the state of the business? Will it work? Do they believe in it? Stand behind it? Iterate here until you get to this point. Go back and review everything you’ve done up to this point - discovery interviews, audience needs, what’s important, the message and make sure you’ve got everything covered.

When you’re 100% there, release into your production environment and place under your normal change, configuration and document controls. Establish a monthly or quarterly (at a minimum) review cycle to sit back down with the various audiences and review the solution with them. Talk with them. See how they use it. Capture metrics associated with any improvements, value, savings, etc. that can be attributed to the solution. Get them to vouch for these accomplishments. Don’t walk away from this review meeting without knowing what’s working, not working or needs to be changed or updated. There is nothing worse than a solution that’s not used or is ignored because it’s out of date or providing no value.

Here are some general guidelines I’ve picked up over the years will help ensure your success. Check the Dashboards page for more ideas and starting points. There are a lot of really good things out there from the BI, BPM and analytics folks!

Consumability

• The message (what’s important) should be communicated in seven (7) seconds or less (one page/screen of information)
• Choose 3-5 key messages, themes or topics to communicate for each audience or each level (see “Determining Your Audience” and “Determining Your Message“)
• Keep things aggregated, correlated and presented in summarized views that prompt action
  (Is the ship on the right path? Do I need to take some action to steer around the iceberg? , How quickly do I need to take action?)
• Try to convey a sense of movement or flow in a uniform manner for visualizations that represent activity, processes, workflow, transactions, etc. Keep them top to bottom or left to right as much as possible.
• Try to draw your audience’s eyes to the most important parts of the message. Don’t let these get lost on the page.
• Think Web2.0 - white space, rounded corners, smoothness, etc.
• Don’t use wild color schemes. Avoid eye candy, all black backgrounds, etc. for more executive and non-technical audiences.

Freshness

• Ideally one hour updates or more frequently
• No more than one week’s data points on a dashboard (just enough to have some context on what’s happening)
• Goal is to manage where the business is going “real-time” (using GPS) versus where we were yesterday (looking in rear view mirror)

Provide Context Where Relevant

• Historical view/info providing as needed context for making decisions
• Aggregate daily, weekly snapshot
• Provide comparables - Hour of day compared to hour of day, Day of week compared to day of week, Week in month compared to week in month
• Link out to or reference other sources that may provide context - avoid replicating data

Ease of Use

• Should support drill down from any click-point or metric and maintain context through every click through
• A common display panel is desirable for maintaining context. Clicking on a metric or indicator causes the results to be displayed in the common window.
• Double clicking or right clicking would cause drill down
• Hover displays are also useful for this approach, but not for key metric or indicator display (you want those immediately visible)
• -“Breadcrumbs” should be used to help understand where in the navigation drilldown someone is and how they can get back to upper layers

Organizational Politics

• Anticipate questions that may be asked
• Have your ducks in a row - what’s important and why, your metrics catalog, source quality, etc.
• Avoid overlap and “competition” with data warehouse, business intelligence/analysis or enterprise reporting groups
• If results make an organization, department, group or person look bad, seek them out in advance to review and prepare them as needed

Stay tuned for my next planned topic in the “You’ve Got Events, Now What? series where I’ll focus on “Managing in Real-Time”.

Catch up with the “You’ve Got Events, Now What?” series here.

Regardles of what kind of IT Operations shop you work in, if you’re interested in any of the best practices frameworks, ITSM, IT Operations, continuous improvement, network and systems monitoring and management, etc. you should look at what Microsoft is talking about in their new Dynamic Systems Initiative (DSI).

Get past the “dynamic” buzz word and what the trade magazines and analysts will talk about in the next week with new and renamed products. Look into this and absorb the information. Start with the DSI Core Principles and the DSI whitepaper. Should other vendors be thinking the same way? (I know we are) Does it make sense? Is it believable? Implementable?

There’s some great content in some of their initial documents. See these: System Definition Model, Model Based Management and Health Modeling.

I know it’s possible to implement similar things with other tools or internally develop new capabilities natively. Some of the stuff in the Health Modeling document reminded me of what I did in the past in OpenService NerveCenter with state models and with a home grown CMDB of sorts for service management.

How much should we as vendors be providing out of the box versus expecting you to develop it yourself? Should every system, application, router, firewall, etc. come with a higher layer management and operations model that plugs into these automated, dynamic frameworks? Should we be federating and integrating at that level instead of selling you just another specialized GUI or tool just for that component or solution? Could it be as easy as consuming some web service, XML document, etc. in an SOA environment??

Stop Doug, stop….the wife wants to watch Survivor…

Funny how this blogging stuff works. The minute you post something, soon after I usually find something similar or something that enhances or detracts from what I was writing about. Fortunately, this one may greatly enhance my post!

I talked about having an arsenal full of instrumentation, data and information collecting tools in yesterday’s posting YGE, NW? Part IV: Mapping Events to What’s Important and Your Message. I mentioned using the normal NMS/EMS/OSS/BSS tools, logfiles, scripts, database triggers and stored procedures, etc. to help collect metrics and KPI/KPM and turn them into events for processing upstream.

I came across another potentially useful approach that may make this instrumentation and collection process significantly easier in the future by using Really Simple Synidcation (RSS) to create a service bus (not ESB). Their goals is to accomplish what has been reserved in the past large companies with large IT staff and large IT budgets - easy integration and sharing of data between applications, services, etc.

The company, RSSBus, is in pre-release mode still and has a white paper available discussing their approach aimed at greatly simplifying integration, access and sharing of information.

I think this has great potential for enabling “the rest of us” to instrument the business and use that important data and information to create rich dashboards and portals and maybe even powerful BSM/BAM/BPM implementations. Imagine subscribing to dashboard feeds, business activity monitoring feeds, etc. Something like Pageflakes could become the enterprise BSM dashboard portal fed by numerous business, technology, people, process and operations feeds. Could this be the start to Web2.0 solutions in these areas?

Some highlights from the whitepaper:

“With RSSBus, our goal is to offer a simple, easy alternative for the small organization with little to no IT assets, little to no professional development tools, and no professional programmers to use them.”

“What we are building is something different, a service platform for the rest of us, the nonacronym-speaking crowd. If you have bits of pieces of data that you would like to quickly exchange with and/or connect to other systems, if simplicity and ease of use is your most important consideration, please read on.”

“With RSSBus, our goal is to build general purpose software that connects or has the ability to easily connect to every system, data, or information source of any significance. Our core focus is to enable connectivity as simply and as easily as possible, and we believe our experience building networking software components and connectivity toolkits for the past decade, and the software assets we have created in the process, give us a unique advantage.”

I’m keeping these guys on my radar to see how their ideas and products develop. No indications as to availability, costs (open source?), etc. yet.