Category — Complex Events

I found this sales and marketing deck over at OSS Professionals that gives a good glimpse into what RiverMuse is up to. I love the logo!

(Mike, if you have a more recent one please send and I’ll update).

This open source start up is unveiling their exciting message and pre-release web site for what could be an industry changing tipping point that firmly places open source as a viable alternative to the “Big4″ and the “Other 6″ within any sized company in any industry.

RiverMuse has launched their website and has plans for initial software availability in early November. RiverMuse (Riversoft and Micromuse) is the brainchild of the founders of the Micromuse and the industry recognized Netcool/OMNIbus solution.

• Chief Science Officer: Philip Tee – Co-Founder of Micromuse/CTO, Founder of RiverSoft/CTO & Chairman, Early software designer Avantgarde (Boole & Babbage/BMC Event Manager)
• Chief Technology Officer: Predrag [Fred] Mutavdzic – Architect Netcool Mediation Technologies, Micromuse
• Executive Director: Mike Silvey – Co-Founder of Micromuse/SVP Marketing and Business Development, VP Business Development and Marketing at RiverSoft.

Here’s a snip from their website - clearly positioning their product at those who’ve made significant investments in or are considering Netcool/OMNIbus technology with promises of a brighter future, improved architecture and a roadmap that if delivered would easily place this open source alternative in the leader’s quadrant of any analyst’s market assessment.

Their plans for putting the administrators first is AWESOME. They get the Total Cost of Ownership (TCO) issue with the Big4 and Other6. They’re putting that first OVER any current buzzword bingo (ITIL, ISM, SOA, Green, and yes even BSM - Mike and I need to have more heart to heart talks on that!). Run the numbers in any decent sized monitoring shop and look at the staff and maintenance costs (HW and SW) and you’ll see that something has to be done in the next decade of IT management and monitoring. Do more with less, smarter, cheaper (free) tools, products and solutions as a competitive differentiator (and job security).

That is, if they can deliver. Some that I’ve talked to advised me that “they’d believe it when they see it”. I spoke with Mike a couple months back and took away the sense of a solid vision and plan to execute against. I’d love to hear about some big wins, replacements or other success (benchmarks against Netcool/OMNIbus, OpenNMS, HPOV, EMC/SMARTS, BMC, etc.). I’ve signed up for the software and look forward to kicking the tires!

RiverMuse for IBM Tivoli Netcool Owners

(IBM Tivoli Netcool Omnibus / Micromuse Netcool Omnibus, Cisco InfoCenter)

What a great product – we think so, we originally conceived, designed and built Netcool as an antidote to the offerings of the day. However, we never finished it and, neither did the people who inherited Micromuse after we left, nor have [or will] IBM. The issue is Netcool’s discombobulated configuration methods that lead to an ownership Tax on you, the customer.

Although Netcool is undoubtedly the best-of-the-best Legacy Event Management system, having invented:

* the Exclusive event paradigm
* automatic repeat filtering ‘de-duplication’
* drag and drop correlation, and
* simplified event enrichment

Netcool hobbles around on a major Achilles Heel. Namely, the more filtering and correlation, the more embedded complexity in the platform since Netcool has three different configuration programming languages that have no configuration integrity. Consequently, the more you use Netcool the higher the Total Cost of Ownership gets.

RiverMuse offers the same out of the box functionality as Netcool, however with a thoroughly modern architecture, configuration is easier to perform and maintain offering a significantly lower total cost of ownership. Oh, and did we tell you the core RiverMuse FreeCool is free?

RiverMuse will gradually introduce migration tools for Netcool customers, initially we’ll enable our customers to consuming Netcool Probe events, and in the future, RiverMuse will launch a ‘Netcool Configuration Conversion’ tool to simplify migrations of Probe Rules and ObjectServer Triggers and Actions.

Link to a presentation given on EDA by K. Mani Chandy of Cal Tech: http://complexevents.com/?p=93

He introduces a strategy he calls BAM++ which makes a lot of sense:

-snip-

“Here’s a strategy that will work for many of your enterprises. I call it a BAM++ strategy. The idea is to start with BAM and then add a function: determine if reality deviates from expectation.

The value proposition is that instead of having the business user, say the CFO, continuously monitor the portal, the EDA system will monitor the portal for the user. When something significant happens then (1) alert the appropriate people, and (2) bring links to the appropriate tools into the portal with links to the appropriate data, so that the business user can immediately respond to the threat or opportunity.

The value proposition here is the attention amplifier. This helps the business user amplify his/her attention, and respond rapidly with appropriate tools when a situation arises.

The advantage of the BAM application is that the sensor data — the data that identifies reality — is already present; it’s sending data to the portal. So, you don’t need to connect to new data sources. Secondly, the issue of error is already understood. If the data shows up in the portal, it is sufficiently accurate to be useful. Thirdly, improving BAM seems less radical than developing an event-driven application.”

-snip-

The folks over at RSSBus have taken a few more covers off of thier RSSBus product. I’ve played around with the RSSBus Desktop Server some now and continue to believe in the potential it has in many of the areas I write about in this blog, especially enabling the “average person” to publish events for consumption by business rules, event and visualization solutions. I’m very excited about the ability to suck metrics, kpi/kpm, etc. out of all those BASS out there!

Check out RSSBus here and download their public beta here. Read the whitepaper, it’s really good.

They’re keeping a blog here where you can follow the product’s progress. I’ve had private email exhanges with their CEO Gent Hito about their plans for the product. They plan to keep parts of it free and are considering open source for parts. Reach out and encourage them to consider this!

I’ve taken you through the trenches of the organization and IT environment to find and capture what’s important to your audience. If you need to catch up, don your safari hat, some boots and check out this page. The next part in this series is one of my favorite areas and probably the most important. This is the part where you’ll show off the fruits of your labor, where the rubber meets the road in terms of how valuable your work and solutions will be for the business and your audiences. This is also the part that everything you do can come into question, be challenged, or simply blown off as garbage, eye candy or a waste of time and money.

Visualization of data and information is an art in itself. There have been many books written on the subject. See the Dashboards page for a list of references. Our goal here is simple. Take everything you’ve done to this point and present the message in the most meaningful, efficient and effective way possible for your audience’s consumption. Your challenge is to figure out what works best for your audience and to ensure that the message can be consumed and have the desired effects of prompting action, decision making, etc.

Iteration is key in getting the visualization right. Allow for a considerable amount of time in your project plans for work in this area. I strongly recommend mockups and prototyping in Microsoft PowerPoint, Excel, Visio or your favorite graphics program such as Adobe Illustrator, Photoshop or Fireworks. A tip here is to look at the dashboard references or vendor products/presentations and cut/copy/paste the widgets (dials, gauges, charts, etc.) into your mockups and prototypes. It’ll help to be as close to what your capabilities are as you review with your various audiences. Keep on this task until you get buy in and a sense that this will work for them. Seek the 80% rule here.

I encourage you to ask your vendor for examples of successful dashboard deployments. See if you can speak to referencable customers and then really dive in with them about how they’ve visualized messages within their environments. A new blog is available that has been collecting examples of dashboards and visualizations called Dashboard Spy. I encourage you to take a look at what has been captured here for ideas.

There are references and links on the Dashboards page that will help you with all the right and wrong uses of gauges, stoplights, sliders, stoplights, charts, etc. I won’t go into those because I don’t necessarily have an opinion one way or another on what they are saying. I do know that every audience will be different. There will likely not be a one size fits all representation of your message. You may be able to get it to look similar, but I promise there will be someone who prefers a gauge or dial instead of stoplight or chart.

Once you’ve got the mocked up visualization of your message complete, it’s time to start implementing it within your solution. I’ll defer from speaking in detail on how to do this, but we’ve previously talked about how to generate events on what’s important and the message. Your solution should offer an easy way to extract this information from an event, database, or any other datasource for alignment and mapping into the visualization components that you will be using. It should be capable of processing these events, metrics, etc. in large quantities in real-time from a large number of distinct sources. You should be able to apply analytical logic, rules, calculations and statistical evaluations, timers, counters, etc. to any single piece of data or any group or collection of data. I’d be happy to recommend a very flexible solution for accomplishing this within your environment off-line.

Release your visualization into a controlled production environment and let it run over the course of the normal cycles associated with your message, what’s important and your data sources. Make sure you’ve also built up enough of the contextual references that may be needed. If you feel the visualization is at a point where it closely resembles your mockups using your solution and real data, it’s time to review and level set with your various audiences.

Get their feedback. Show them how it will work in production. Ask them if they “get” the message. Can they tell you what needs to be done or the state of the business? Will it work? Do they believe in it? Stand behind it? Iterate here until you get to this point. Go back and review everything you’ve done up to this point - discovery interviews, audience needs, what’s important, the message and make sure you’ve got everything covered.

When you’re 100% there, release into your production environment and place under your normal change, configuration and document controls. Establish a monthly or quarterly (at a minimum) review cycle to sit back down with the various audiences and review the solution with them. Talk with them. See how they use it. Capture metrics associated with any improvements, value, savings, etc. that can be attributed to the solution. Get them to vouch for these accomplishments. Don’t walk away from this review meeting without knowing what’s working, not working or needs to be changed or updated. There is nothing worse than a solution that’s not used or is ignored because it’s out of date or providing no value.

Here are some general guidelines I’ve picked up over the years will help ensure your success. Check the Dashboards page for more ideas and starting points. There are a lot of really good things out there from the BI, BPM and analytics folks!

Consumability

• The message (what’s important) should be communicated in seven (7) seconds or less (one page/screen of information)
• Choose 3-5 key messages, themes or topics to communicate for each audience or each level (see “Determining Your Audience” and “Determining Your Message“)
• Keep things aggregated, correlated and presented in summarized views that prompt action
  (Is the ship on the right path? Do I need to take some action to steer around the iceberg? , How quickly do I need to take action?)
• Try to convey a sense of movement or flow in a uniform manner for visualizations that represent activity, processes, workflow, transactions, etc. Keep them top to bottom or left to right as much as possible.
• Try to draw your audience’s eyes to the most important parts of the message. Don’t let these get lost on the page.
• Think Web2.0 - white space, rounded corners, smoothness, etc.
• Don’t use wild color schemes. Avoid eye candy, all black backgrounds, etc. for more executive and non-technical audiences.

Freshness

• Ideally one hour updates or more frequently
• No more than one week’s data points on a dashboard (just enough to have some context on what’s happening)
• Goal is to manage where the business is going “real-time” (using GPS) versus where we were yesterday (looking in rear view mirror)

Provide Context Where Relevant

• Historical view/info providing as needed context for making decisions
• Aggregate daily, weekly snapshot
• Provide comparables - Hour of day compared to hour of day, Day of week compared to day of week, Week in month compared to week in month
• Link out to or reference other sources that may provide context - avoid replicating data

Ease of Use

• Should support drill down from any click-point or metric and maintain context through every click through
• A common display panel is desirable for maintaining context. Clicking on a metric or indicator causes the results to be displayed in the common window.
• Double clicking or right clicking would cause drill down
• Hover displays are also useful for this approach, but not for key metric or indicator display (you want those immediately visible)
• -“Breadcrumbs” should be used to help understand where in the navigation drilldown someone is and how they can get back to upper layers

Organizational Politics

• Anticipate questions that may be asked
• Have your ducks in a row - what’s important and why, your metrics catalog, source quality, etc.
• Avoid overlap and “competition” with data warehouse, business intelligence/analysis or enterprise reporting groups
• If results make an organization, department, group or person look bad, seek them out in advance to review and prepare them as needed

Stay tuned for my next planned topic in the “You’ve Got Events, Now What? series where I’ll focus on “Managing in Real-Time”.

Catch up with the “You’ve Got Events, Now What?” series here.

Funny how this blogging stuff works. The minute you post something, soon after I usually find something similar or something that enhances or detracts from what I was writing about. Fortunately, this one may greatly enhance my post!

I talked about having an arsenal full of instrumentation, data and information collecting tools in yesterday’s posting YGE, NW? Part IV: Mapping Events to What’s Important and Your Message. I mentioned using the normal NMS/EMS/OSS/BSS tools, logfiles, scripts, database triggers and stored procedures, etc. to help collect metrics and KPI/KPM and turn them into events for processing upstream.

I came across another potentially useful approach that may make this instrumentation and collection process significantly easier in the future by using Really Simple Synidcation (RSS) to create a service bus (not ESB). Their goals is to accomplish what has been reserved in the past large companies with large IT staff and large IT budgets - easy integration and sharing of data between applications, services, etc.

The company, RSSBus, is in pre-release mode still and has a white paper available discussing their approach aimed at greatly simplifying integration, access and sharing of information.

I think this has great potential for enabling “the rest of us” to instrument the business and use that important data and information to create rich dashboards and portals and maybe even powerful BSM/BAM/BPM implementations. Imagine subscribing to dashboard feeds, business activity monitoring feeds, etc. Something like Pageflakes could become the enterprise BSM dashboard portal fed by numerous business, technology, people, process and operations feeds. Could this be the start to Web2.0 solutions in these areas?

Some highlights from the whitepaper:

“With RSSBus, our goal is to offer a simple, easy alternative for the small organization with little to no IT assets, little to no professional development tools, and no professional programmers to use them.”

“What we are building is something different, a service platform for the rest of us, the nonacronym-speaking crowd. If you have bits of pieces of data that you would like to quickly exchange with and/or connect to other systems, if simplicity and ease of use is your most important consideration, please read on.”

“With RSSBus, our goal is to build general purpose software that connects or has the ability to easily connect to every system, data, or information source of any significance. Our core focus is to enable connectivity as simply and as easily as possible, and we believe our experience building networking software components and connectivity toolkits for the past decade, and the software assets we have created in the process, give us a unique advantage.”

I’m keeping these guys on my radar to see how their ideas and products develop. No indications as to availability, costs (open source?), etc. yet.

Now that you’ve identified the sources of what’s important within your environment and crafted that data and information into messages that prompt action and decision making, it’s time to think about getting this data and information into a manageable format for processing and visualization.

I’ve discussed what events are and shared some initial thoughts on building events for BSM here and included references to complex event processing (CEP), event driven architecture (EDA) and event stream processing (ESP) here. I still plan on diving in deep to the topic of building events and the idea of the Common Base Event and Common Event Format. I also want to introduce the Event Data Dictionary / Event Catalog which will be useful for capturing information about what events exist in your environment and why. Every event that’s generated should be done so for a purpose. There’s nothing that will turn your NOC or IT support group against you quicker than if you’re collecting data and generating events just for the sake of doing so because they may be available via and SNMP MIB or agent. They don’t need any more “noise” to deal with during the day.

There may be many ways to incorporate this data and information into the messages you’re planning to communicate. The approaches and their ease of use are going to be entirely up to the tools, applications and solutions you’re using. You may be able to establish direct connections with the datasource, perform screen scrapes, import spreadsheets, or even perform queries against the source. The general concept of this series of articles has been around the assumption that you have the ability within your environment to generate events. Generating events usually comes through some form of instrumentation, collection and evaluation against a threshold, state, rule, etc.

What I want to talk about here is instrumenting those sources of important information, data and metrics within your environment you’ve identified as you completed your Metrics Catalog. Some of these sources may be outside the comfort zone or capabilities of the average IT Operations group normally used to operating with SNMP, server and application monitoring agents.

Since you’ve identified the source of the important information or data, how frequently it gets updated, and how to access it you’re half way there. The next task is to identify the person(s) or group(s) responsible for that information source. This may be the owner, administrator or support group for the application, tool, file, spreadsheet, database, server, etc. that produces, evaluates, communicates or makes available that information or data. The task here is to establish the business need with the owner to instrument that source so that the important data or information is provided in a way that can be easily processed upstream.

Once you’ve established the business need, you can have a discussion about the best way to instrument the information source and generate those events. Discuss the various tools in your event generating arsenal with the owner and their technical staff. Cover the normal EMS/NMS/OSS/BSS solutions and their capabilities for collecting information and generating events. Discuss more generic approaches such as log files (application, system, etc.), scripts, XML/SOAP/WebServices, etc. Scripts can be written to parse logs or collect other information from applications, GUIs, command lines, etc. and pass those off to an event generation function. If you’ve been able to consolidate information into a database or corporate data warehouse, consider leveraging database triggers and stored procedures to collect, format and generate an event. There are certainly more sophisticated methods available here if your organization leverages an EAI or ESB technology. Just keep in mind that the goal is to keep it simple, efficient and effective. You don’t want to be blamed for causing a performance slowdown or outage to that important business application!

You’ll want to map the events you’re generating into the appropriate format of your internal systems that will process them. Be sure to capture the relationship between these event types and their purpose for communicating an important metric, KPI/KPM, etc. At a minimum, one of the fields in the event format should be the Metric ID from the Metrics Catalog. This will be critical in linking the events to their purpose. The more thought and planning you put into how you build these events the better. Consider the use of an enumeration schema to capture information. This can be parsed and evaluated later by other solutions such as dashboard, BSM, BAM, BPM, rules or workflow solutions. An example may be populating a field in an event like this: “A1-2-3″ which may represent Metric Source = A1 (CRM System), Metric ID = 2 (Customer Count) and Metric Update = 3 (Daily). The sky’s the limit here but do consider the impact these may have on your internal event processing solutions or those that will need to parse and evaluate the enumeration schema you create.

Spend some time testing and evaluating the effectiveness of the new instrumentation you’ve done. Follow up with the owners you identified and the business to make sure that the data, information, metrics, etc. you’re now collecting passes their “sniff tests”. They’ll have a fairly good understanding of what’s good or bad - they always seem to have a sixth sense about this. If you get the sense that this information isn’t accurate, useful or otherwise have them excited, immediately start to evaluate why and do whatever you can to remedy it. You absolutely do not want to be presenting bad information later!

Now that we’ve got these important bits of data, information, metrics, etc. being collected and processed by our internal systems and tools automatically, it’s time to think about visualizing our message effectively for our various audiences. Stay tuned for that topic in “You’ve Got Events, Now What? Part V: Visualizing the Message.

Catch up with the “You’ve Got Events, Now What?” series here.

Hopefully by now you’ve had the opportunity to determine what was important in your environment. Now we need to figure out how to leverage that information. If you haven’t read the other posts in this series, please visit here to catch up and contribute in the discussion.

Ask yourself what is more important in your environment, the metrics or the message? I’d say the message is much more important than communicating a bunch of metrics without proper context. The message is made up of metrics, but it’s how you group, organize, visualize and present these metrics in the right context that communicates a message. Our next steps will be to take a look at all of this information and boil it down into something more manageable.

Metrics Demystified - Creating a Metrics Data Dictionary or Metrics Catalog

Stare at your sources of data - really - lay them all out on a white board or on little yellow Post-It Notes. For each metric, capture the following information:

• Metric ID- the Metric ID should be a unique identifier assigned to each metric.
• Metric Name- this should be the common name for the metric, understandable to the intended audience.
• Source- this is the source of the metric. It should provide enough information to adequately describe where this metric information resides. System name, IP address, Database name, database table, spreadsheet name and location, application name, etc.
• Freshness- this should describe how frequent the metric is updated such as Real Time, X Minutes, Hourly, Daily, Weekly, etc.
• Access Method- this should provide details on how to access this metric from the identified source. This could be login/password information, database table name, SQL query, screen scrape information, etc.
• Message Category- this should capture the message topic/group this metric aligns to? Find your short list of topics or groups you can assign metrics to (availability, performance, quality, revenue, inventory, sales, etc.) Ask yourself what this metric communicates to someone who see it?
• Importance to Business- this should capture how important this metric is to the Business on a scale from 1-10, with 10 being the most important.
• Importance to IT- this should capture how important this metric is to IT on a scale from 1-10, with 10 being the most important.
• Audience- this should capture who the intended audience is for the metric. Is it for executives, management, and technical staff? The Business? CIO? CEO? Who will use this metric to make a decision about something?
• Measure- this should capture what this metric is measuring. Health, status, volume, trend, quantity, speed/velocity, etc. It should also capture the units of measure.
• Use- this should capture information about how this metric is being used. What reports, graphs, charts, dashboards, presentations, etc. is this feeding into?
• Compliance- this should capture and highlight how this metric may be used in compliance reporting and monitoring efforts for Sarbanes-Oxley (SOX), HIPAA, BASEL II, etc. These metrics and the systems, applications, tools, etc. that generate them may need to be under the appropriate control measures.
• Notes- additional notes and information.

The output of this process is a database, document or spreadsheet that captures all of the above information. Try organizing the metrics by source, message, and importance to business, audience and measure. This data dictionary for your metrics will be used (especially the Metric ID) when creating and mapping events to metrics for monitoring, dashboards, reporting, etc. which I will discuss in part four of this series.

Metrics Data Dictionary or Metrics Catalog

Crafting the Message

Since you’ve already determined what was important, it should be easier to use this metrics catalog to craft and communicate a message in-line with what you determined. Think about what you want the message to accomplish. Provide visibility? Prompt action? Show accomplishments? Communicate status or overall health of something?

An earlier posting on a topic called emotional metrics should be revisited here. Don Page of the Marval Group presented on a topic related to what he called “Emotional Metrics”. These metrics are both positive and negative. They are focused on outcome, action and encourage decision making. They aren’t to be used to create a pretty picture on the wall or screen in the NOC.

Again, know your audience and business. Really think here about the business you’re in and the real objectives of any business - control costs and make money. How do you do that? What makes you money? What costs need to be controlled? What happens if you’re not making money? What happens if costs are out of control? What are the positive/negative outcomes? Whose bonus, commission or compensation is impacted? Those people need the right message communicated to them that prompts them to make a decision on what to do next (or at least we hope so).

Look at your metrics catalog and try to generalize into three to five categories that best communicate the message for each audience. There’s more to talk about here in a future posting on visualizing the message, but most people can’t process a ton of information. The message should be interpreted in less than 10 seconds by a competent person familiar with the environment, terminology, business, etc.

Examples may be something similar to below. These would be the key message categories important to the business. In each of those categories, metrics would be mapped into them to communicate something about each category (see above).

• IT focused message: Availability, Performance, Reliability, Customer Experience, etc.
• Business focused message: Sales, Inventory, Distribution, Customer Satisfaction, etc.

Double check here – are you providing visibility or prompting action? Are you showing accomplishments or business health? Is it nice to know or very important? Can the metrics you selected prompt someone to ask at least two questions? (Is this Good/Bad? Is this Getting Better/Worse? On-Target/Off-Target? Why? Cause? )

Next in the series: You’ve Got Events, Now What? Part IV: Mapping Events to What’s Important and Your Message.

Previous posts in the series available here