Category — Event Driven Architecture

Funny how this blogging stuff works. The minute you post something, soon after I usually find something similar or something that enhances or detracts from what I was writing about. Fortunately, this one may greatly enhance my post!

I talked about having an arsenal full of instrumentation, data and information collecting tools in yesterday’s posting YGE, NW? Part IV: Mapping Events to What’s Important and Your Message. I mentioned using the normal NMS/EMS/OSS/BSS tools, logfiles, scripts, database triggers and stored procedures, etc. to help collect metrics and KPI/KPM and turn them into events for processing upstream.

I came across another potentially useful approach that may make this instrumentation and collection process significantly easier in the future by using Really Simple Synidcation (RSS) to create a service bus (not ESB). Their goals is to accomplish what has been reserved in the past large companies with large IT staff and large IT budgets - easy integration and sharing of data between applications, services, etc.

The company, RSSBus, is in pre-release mode still and has a white paper available discussing their approach aimed at greatly simplifying integration, access and sharing of information.

I think this has great potential for enabling “the rest of us” to instrument the business and use that important data and information to create rich dashboards and portals and maybe even powerful BSM/BAM/BPM implementations. Imagine subscribing to dashboard feeds, business activity monitoring feeds, etc. Something like Pageflakes could become the enterprise BSM dashboard portal fed by numerous business, technology, people, process and operations feeds. Could this be the start to Web2.0 solutions in these areas?

Some highlights from the whitepaper:

“With RSSBus, our goal is to offer a simple, easy alternative for the small organization with little to no IT assets, little to no professional development tools, and no professional programmers to use them.”

“What we are building is something different, a service platform for the rest of us, the nonacronym-speaking crowd. If you have bits of pieces of data that you would like to quickly exchange with and/or connect to other systems, if simplicity and ease of use is your most important consideration, please read on.”

“With RSSBus, our goal is to build general purpose software that connects or has the ability to easily connect to every system, data, or information source of any significance. Our core focus is to enable connectivity as simply and as easily as possible, and we believe our experience building networking software components and connectivity toolkits for the past decade, and the software assets we have created in the process, give us a unique advantage.”

I’m keeping these guys on my radar to see how their ideas and products develop. No indications as to availability, costs (open source?), etc. yet.

Now that you’ve identified the sources of what’s important within your environment and crafted that data and information into messages that prompt action and decision making, it’s time to think about getting this data and information into a manageable format for processing and visualization.

I’ve discussed what events are and shared some initial thoughts on building events for BSM here and included references to complex event processing (CEP), event driven architecture (EDA) and event stream processing (ESP) here. I still plan on diving in deep to the topic of building events and the idea of the Common Base Event and Common Event Format. I also want to introduce the Event Data Dictionary / Event Catalog which will be useful for capturing information about what events exist in your environment and why. Every event that’s generated should be done so for a purpose. There’s nothing that will turn your NOC or IT support group against you quicker than if you’re collecting data and generating events just for the sake of doing so because they may be available via and SNMP MIB or agent. They don’t need any more “noise” to deal with during the day.

There may be many ways to incorporate this data and information into the messages you’re planning to communicate. The approaches and their ease of use are going to be entirely up to the tools, applications and solutions you’re using. You may be able to establish direct connections with the datasource, perform screen scrapes, import spreadsheets, or even perform queries against the source. The general concept of this series of articles has been around the assumption that you have the ability within your environment to generate events. Generating events usually comes through some form of instrumentation, collection and evaluation against a threshold, state, rule, etc.

What I want to talk about here is instrumenting those sources of important information, data and metrics within your environment you’ve identified as you completed your Metrics Catalog. Some of these sources may be outside the comfort zone or capabilities of the average IT Operations group normally used to operating with SNMP, server and application monitoring agents.

Since you’ve identified the source of the important information or data, how frequently it gets updated, and how to access it you’re half way there. The next task is to identify the person(s) or group(s) responsible for that information source. This may be the owner, administrator or support group for the application, tool, file, spreadsheet, database, server, etc. that produces, evaluates, communicates or makes available that information or data. The task here is to establish the business need with the owner to instrument that source so that the important data or information is provided in a way that can be easily processed upstream.

Once you’ve established the business need, you can have a discussion about the best way to instrument the information source and generate those events. Discuss the various tools in your event generating arsenal with the owner and their technical staff. Cover the normal EMS/NMS/OSS/BSS solutions and their capabilities for collecting information and generating events. Discuss more generic approaches such as log files (application, system, etc.), scripts, XML/SOAP/WebServices, etc. Scripts can be written to parse logs or collect other information from applications, GUIs, command lines, etc. and pass those off to an event generation function. If you’ve been able to consolidate information into a database or corporate data warehouse, consider leveraging database triggers and stored procedures to collect, format and generate an event. There are certainly more sophisticated methods available here if your organization leverages an EAI or ESB technology. Just keep in mind that the goal is to keep it simple, efficient and effective. You don’t want to be blamed for causing a performance slowdown or outage to that important business application!

You’ll want to map the events you’re generating into the appropriate format of your internal systems that will process them. Be sure to capture the relationship between these event types and their purpose for communicating an important metric, KPI/KPM, etc. At a minimum, one of the fields in the event format should be the Metric ID from the Metrics Catalog. This will be critical in linking the events to their purpose. The more thought and planning you put into how you build these events the better. Consider the use of an enumeration schema to capture information. This can be parsed and evaluated later by other solutions such as dashboard, BSM, BAM, BPM, rules or workflow solutions. An example may be populating a field in an event like this: “A1-2-3″ which may represent Metric Source = A1 (CRM System), Metric ID = 2 (Customer Count) and Metric Update = 3 (Daily). The sky’s the limit here but do consider the impact these may have on your internal event processing solutions or those that will need to parse and evaluate the enumeration schema you create.

Spend some time testing and evaluating the effectiveness of the new instrumentation you’ve done. Follow up with the owners you identified and the business to make sure that the data, information, metrics, etc. you’re now collecting passes their “sniff tests”. They’ll have a fairly good understanding of what’s good or bad - they always seem to have a sixth sense about this. If you get the sense that this information isn’t accurate, useful or otherwise have them excited, immediately start to evaluate why and do whatever you can to remedy it. You absolutely do not want to be presenting bad information later!

Now that we’ve got these important bits of data, information, metrics, etc. being collected and processed by our internal systems and tools automatically, it’s time to think about visualizing our message effectively for our various audiences. Stay tuned for that topic in “You’ve Got Events, Now What? Part V: Visualizing the Message.

Catch up with the “You’ve Got Events, Now What?” series here.

Hopefully by now you’ve had the opportunity to determine what was important in your environment. Now we need to figure out how to leverage that information. If you haven’t read the other posts in this series, please visit here to catch up and contribute in the discussion.

Ask yourself what is more important in your environment, the metrics or the message? I’d say the message is much more important than communicating a bunch of metrics without proper context. The message is made up of metrics, but it’s how you group, organize, visualize and present these metrics in the right context that communicates a message. Our next steps will be to take a look at all of this information and boil it down into something more manageable.

Metrics Demystified - Creating a Metrics Data Dictionary or Metrics Catalog

Stare at your sources of data - really - lay them all out on a white board or on little yellow Post-It Notes. For each metric, capture the following information:

• Metric ID- the Metric ID should be a unique identifier assigned to each metric.
• Metric Name- this should be the common name for the metric, understandable to the intended audience.
• Source- this is the source of the metric. It should provide enough information to adequately describe where this metric information resides. System name, IP address, Database name, database table, spreadsheet name and location, application name, etc.
• Freshness- this should describe how frequent the metric is updated such as Real Time, X Minutes, Hourly, Daily, Weekly, etc.
• Access Method- this should provide details on how to access this metric from the identified source. This could be login/password information, database table name, SQL query, screen scrape information, etc.
• Message Category- this should capture the message topic/group this metric aligns to? Find your short list of topics or groups you can assign metrics to (availability, performance, quality, revenue, inventory, sales, etc.) Ask yourself what this metric communicates to someone who see it?
• Importance to Business- this should capture how important this metric is to the Business on a scale from 1-10, with 10 being the most important.
• Importance to IT- this should capture how important this metric is to IT on a scale from 1-10, with 10 being the most important.
• Audience- this should capture who the intended audience is for the metric. Is it for executives, management, and technical staff? The Business? CIO? CEO? Who will use this metric to make a decision about something?
• Measure- this should capture what this metric is measuring. Health, status, volume, trend, quantity, speed/velocity, etc. It should also capture the units of measure.
• Use- this should capture information about how this metric is being used. What reports, graphs, charts, dashboards, presentations, etc. is this feeding into?
• Compliance- this should capture and highlight how this metric may be used in compliance reporting and monitoring efforts for Sarbanes-Oxley (SOX), HIPAA, BASEL II, etc. These metrics and the systems, applications, tools, etc. that generate them may need to be under the appropriate control measures.
• Notes- additional notes and information.

The output of this process is a database, document or spreadsheet that captures all of the above information. Try organizing the metrics by source, message, and importance to business, audience and measure. This data dictionary for your metrics will be used (especially the Metric ID) when creating and mapping events to metrics for monitoring, dashboards, reporting, etc. which I will discuss in part four of this series.

Metrics Data Dictionary or Metrics Catalog

Crafting the Message

Since you’ve already determined what was important, it should be easier to use this metrics catalog to craft and communicate a message in-line with what you determined. Think about what you want the message to accomplish. Provide visibility? Prompt action? Show accomplishments? Communicate status or overall health of something?

An earlier posting on a topic called emotional metrics should be revisited here. Don Page of the Marval Group presented on a topic related to what he called “Emotional Metrics”. These metrics are both positive and negative. They are focused on outcome, action and encourage decision making. They aren’t to be used to create a pretty picture on the wall or screen in the NOC.

Again, know your audience and business. Really think here about the business you’re in and the real objectives of any business - control costs and make money. How do you do that? What makes you money? What costs need to be controlled? What happens if you’re not making money? What happens if costs are out of control? What are the positive/negative outcomes? Whose bonus, commission or compensation is impacted? Those people need the right message communicated to them that prompts them to make a decision on what to do next (or at least we hope so).

Look at your metrics catalog and try to generalize into three to five categories that best communicate the message for each audience. There’s more to talk about here in a future posting on visualizing the message, but most people can’t process a ton of information. The message should be interpreted in less than 10 seconds by a competent person familiar with the environment, terminology, business, etc.

Examples may be something similar to below. These would be the key message categories important to the business. In each of those categories, metrics would be mapped into them to communicate something about each category (see above).

• IT focused message: Availability, Performance, Reliability, Customer Experience, etc.
• Business focused message: Sales, Inventory, Distribution, Customer Satisfaction, etc.

Double check here – are you providing visibility or prompting action? Are you showing accomplishments or business health? Is it nice to know or very important? Can the metrics you selected prompt someone to ask at least two questions? (Is this Good/Bad? Is this Getting Better/Worse? On-Target/Off-Target? Why? Cause? )

Next in the series: You’ve Got Events, Now What? Part IV: Mapping Events to What’s Important and Your Message.

Previous posts in the series available here

In part one of my series on the role of events, Building the Right Event Foundation for BSM introduced some simple concepts for building useful events for upstream processing in BSM and BAM solutions. This was followed up by a posting about how important it is to determine the audience for your events in You’ve Got Events, Now What? Part I: Determining Your Audience. This is a continuation of my thoughts on creating the right events in your environment for driving BSM and BAM solutions, dashboards, reports or other presentations.

MarkG. posted in the Darth.Net blog a follow up to the excellent article on the Event-Driven Architecture by Brenda Michelson. He provides additional insight and observations into the implementation challenges of EDA.

A great read on events and application of them within an Event-Driven Architecture by Brenda Michelson in her Elemental Links blog. I agree with everything she’s laid out in her posting. Events, complex event processing (CEP) and the associated business rules build the foundation for managing the business in real-time with business service management (BSM), business activity monitoring (BAM) and similar concepts and solutions.

It’s critical for those within IT Operations (NOC, help desk, network and systems management fields, etc.) to learn how to think outside of the box when it comes to events. The common stereotype for most in IT Operations is that an event is associated with somthing bad happening. This is just plain wrong. Events can and should be used to communicate all kinds of messages, whether related to the infrastructure you’re managing or to somthing completely unrelated such as a process, activity, or anything important within the business environment. Use the investments your company has made in creative ways. Events don’t have to only originate from an SNMP agent, system agent or other infrastructure element. Work with non-traditional groups within your organization to find ways that they can generate events that can be processed and help communicate various messages within the organization, drive a dashboard, create a help desk ticket, automate a process, etc. The use of events to manage the business in real-time will separate the successful companies from the unsuccessful (or inefficient) ones in the future!

This is a must read posting! I look forward to reading the rest of her site and future postings on this topic and the Business-Driven Architecture!

Most sales, marketing and analyst messaging positions BSM solutions for those in the corner office. In reality, this may be far from the truth. I tend to boil down the key audiences of a BSM presentation to one of these four: Business/Non-Technical, IT Executive, IT Management and Technical. Every audience and consumer of a BSM presentation has a different need for in-context information to be successful in their job specific functions or role. They each have a different perspective and need for this information in different “time modes” (real-time, near real-time, not real-time (daily, weekly), etc.).

Often, a BSM solution’s audience and implementation is based on the understanding of what BSM is by the technical IT staff implementing it and their unique understanding of the IT environment. Another influencing factor guiding BSM deployments may be the objectives, expectations and positioning of the person who wrote the business case for a BSM project and solution (or who signed the check). Worse, it may be completely determined by the vendor or integrator’s perspectives and positioning with little involvement of the proper customer staff.

The key message that most executives want to know centers on these three main questions.

• What are my problems?
• Where are they at?
• Are they impacting my business? If yes, how?

These three questions form the basis for any good BSM solution. They should be adapted into your specific environment and put in context to your audience’s needs and expectations. You must know both your IT and business environments, how they align, and how they operate 24/7/365 to properly profile and characterize the impact IT problems may have on the business (or business changes on IT).

The audience for your BSM engagement may initially be top down. Ultimately, it will work its way to the bottom levels in the organization. The audience or consumers of BSM information must be determined as early on as possible in the engagement process. If it appears to be very top down focused, the initial deployment may be kept very focused with high-level metrics and be status/performance focused. Simple synthetic testing of business services and applications or simple “binary state” (good/bad, up/down, events/no-events) presentations may be good enough to start. (note “to start” - you will have to do more work at some point, see below)

Technology oriented executives, management and staff may want more technical information than other business/non-technical audiences. If the needs appear to be very technical, be prepared to discuss the approaches and capabilities required for this level of data integration, collection and modeling. Discuss options for full navigation and drill down (or drill up) to the lowest levels of individual IT components, metric data sources, etc. Ultimately, this level of preparation, instrumentation, and consolidation is needed to ensure believablility of the information presented at every level. There’s usually an inherent “sixth-sense” within a company about what “feels right” and what passes the “sniff test”. If the messages you’re presenting in the BSM solution don’t pass this test, it’s the fastest way to be discredited and be labeled unfavorably.

Don’t forget to engage with the business / non-technical side. Ask about who the key business sponsors or consumers may be, if any. It’s more than likely that at some point someone outside of IT will see the BSM solution and ask for access. The message presented to this audience may need to be even more summarized and generalized content for someone outside of IT. The messages here MUST BE IN THEIR LANGUAGE, using THEIR TERMINOLOGY, and THEIR METRICS. You must know the business, how it operates, how it’s managed and how to align with it.

Here are some of my tips for success during the BSM engagement process. Always keep these non-traditional BSM concepts in mind when engaging in information collection with a customer. I’ll create a much more detailed discovery interview template soon that builds upon these questions.

• Ask every person or group identified as an audience or consumer what BSM means to them. (you’ll be surprised by the varying differences in understanding)
• How do they “align” IT infrastructure components to the business or the business to IT? (you’ll be surprised by lack of understanding here)
• How can their “intimate service knowledge” be leveraged above an beyond “this server supports this application for this business unit”? (find ways to leverage what people know (Knowledge Management) in your BSM engagement and you’ll be extremely successful)
• How do they see or understand how their roles, actions, activities, workflow/processes align to what the business needs/objectives are? (don’t forget about the people and process side of the three legged stool - technology isn’t the only thing that can impact business)

You’ve Got Events, Now What? Part Two: Determining What’s Important and the Message(s) to Communicate will be the next part in this series.

Whether you’re following a route, path, or some other plan to find value with Business Service Management, you’ve got to start someplace. I am a “always begin with the end in mind” type of guy so I often think from way out and decompose backwards from there to map out how I want to get there.

A key component existing within nearly every organization regardless of industry, vertical, country or otherwise is the notion of an ‘event’. Some may call this a message, indicator, notification or something similar.

Events simply communicate a message to something. The message communicated can be good, bad or indifferent, true or false, on or off, verbose or terse. The something on the receiving end of an event can be a human, system, or other application, code or similar logic. A programming focus area called event-driven programming exists if you’d like to investigate the nitty-gritty details.

There are many points of view from which we can have a discussion on events. I’d like to focus initially on the importance of creating the right events with the right data (message) that can be used for building a foundation for BSM (among other very important IT Operations things).

Sources

If the goal of BSM is to align IT to the business, we must identify all of the sources capable of enabling this. Nearly every IT organization has some form of network, server, application or other technology monitoring and management tool. These may be home grown, open source or COTS tools. Inventory what you have available that can help provide information about all components in the organization. Don’t forget to venture outside of the organizational silo that you may be in. I guarantee that there are other groups that have something valuable that they may be able to contribute.

A few of the common sources include the network (router, switch, LAN/WAN, etc.), security (firewall, VPN, etc.), server (hardware, operating system, storage, etc.), and application (database, webserver, appserver, etc.). Sources you may not immediately think of are your change management application or your help desk application. It’s extremely valuable to have events generated for each change request that communicate the lifecycle of a change request for any IT infrastructure component.

One of the key sources that may not yet exist in your IT organization is one that simply relates the detailed information from each source to the functional role or purpose that it plays. This could be a simple asset inventory database or a complete configuration management database (CMDB). This source ideally captures information about every single component within the IT organization and what role it plays in delivering IT and business services.

Perspectives

The idea here is that we need to collect from sources that are representative of all the different perspectives. This could be from the customer’s view, business view, third party view, application view, service view, inside/outside the datacenter view, etc. A good example of this is the business unit’s perspective of the CRM solution is that it’s a portal with all of the interfaces and tools used in one place. The IT group’s perspective is that the CRM solution is made up of a dozen servers, databases, application servers and network components. Each group has a different perspective on what the CRM solution is. As we identify our event sources, be cognizant of the different perspectives and identify events representative of them.

Raw Events

Everything generated from any of your sources should be treated as a raw event. A raw event is simply one that communicates a single piece of data which is usually very precise and in context to what generated it. For example, a server is capable of generating events about its CPU utilization. The event generated when CPU utilization is above the desired threshold communicates a discreet datapoint about the CPU on that server. By default, it doesn’t communicate how that event impacts other things on that server such as the application that runs on that server or the key business process which is enabled by that application running on that server. Raw events require further processing as they move northbound to increase their value and usefulness.

Normalization

The first step in processing raw events into more useful events should include passing them through some sort of normalization routine. The goal here is to develop a uniform event structure where every event, regardless of source or perspective, contains the same fundamental makeup. You will no doubt quickly determine that every event source you identified has some variation in the quality and quantity of events it provides.

Here’s a simple scenario. Let’s assume that you have a useful and functional naming scheme for the components within your datacenter (such as those recommended by Sun Microsystems here). Let’s say that you name a server ERP-APP-01.dmz.mycompany.com. This information would obviously be present in the event as the source or node name that generated the event. This may not be very useful in the long run. One approach would be to ensure that in my normalization routine (which is mapped to my event structure) that I parse out key information and add this into the more useful event. I know based on my naming standard that ‘ERP’ is actually the functional application that runs on this server. This would get mapped into a previously empty field in the event reserved for the application type. The next section in the server name is “APP”. This refers to the type of server in our environment and in this case the server is an application server. This would get mapped into a field reserved for server function or type. You should get the picture.

Node/Source: ERP-APP-01.dmz.mycompany.com | Summary: CPU Utilization High

becomes

Node/Source: ERP-APP-01.dmz.mycompany.com | Application: SAP ERP | Server Type: Application Server | Server Number: 1 | Network Location: DMZ | Summary: CPU Utilization High

The event and the information it now communicates is now much more useful since I know my Problem Management team will want to investigate how many incidents have occurred in my environment with our application servers.

You should put considerable thought and planning into what your normalization rules and routines will be. This is where you should really think out into the future with the end in mind. What do you want to use these events for? What process, workflow, or activity will your events enable? Incident Management? Problem Management? Historical Trend Reporting?

I will write on this topic extensively later and offer more detailed recommendations on event structure and normalization practices.

Enriched Events

I think there is a distinct difference between normalization and enrichment. Normalization as described above can generally take place with data and information you already have. The data and information used in the normalization process is generally static in nature and doesn’t change frequently.

Enrichment on the other hand is the process of leveraging a portion of a normalized event as a trigger for further enrichment or “advanced dynamic normalization”. Enrichment only occurs on normalized events and should be reserved for cases where the event couldn’t be normalized with information already known or that was static in nature. Enrichment often involves some form of integration with other data sources, tools or interfaces. It may even require some form of manual intervention.

An example of enrichment based on our scenario above could be as follows. We now know that a CPU Utilization event was generated from one of our application servers. The help desk application is the official repository for on-call support information for our IT organizations. An enrichment routine has been developed that is triggered based on the missing on-call support group information in our event. Because we know the Application Server group is responsible for this event, the enrichment routine fetches on-call information from the help desk application for the Application Server group and adds this to the previously blank support fields.

Node/Source: ERP-APP-01.dmz.mycompany.com | Application: SAP ERP | Server Type: Application Server | Server Number: 1 | Network Location: DMZ | Summary: CPU Utilization High

becomes

Node/Source: ERP-APP-01.dmz.mycompany.com | Application: SAP ERP | Server Type: Application Server | Server Number: 1 | Network Location: DMZ | Summary: CPU Utilization High | Support Group: Team A | Support Engineer: John Doe | Support Shift: 9AM - 5PM EST

The enrichment routine is used to fetch information that changes frequently based on the type of event, time of day, etc. This information is dynamic in nature and is usually stored in an authoritative source location such as the help desk solution.

Messages

The goal of transforming raw events into normalized, enriched events is to aide in communicating the right message. The message intended for communication may be very detailed information about a single specific component in the IT infrastructure or it may be a summarized message that communicates a message for all of the IT infrastructure components supporting a complex application or service.

Remember the goals of BSM are to clearly communicate a message about how the business may be affected by events within the IT environment. At the very least, every event should clearly communicate what business or customer entity, application, service, process, etc. that it most directly affects. Keep in mind the following bigger picture. This is the alignment desired with BSM in such a way that the most junior systems administrator or help desk technician clearly understands.

The CPU Utilization High event caused the Application Server to slow down. The Application Server is a key component of the SAP ERP application. The SAP ERP application hosts the Customer Billing application for MyCompany.Com. Because of the slow down in the Customer Billing application, the nightly transaction job failed to complete causing a $1M loss in revenue recognition for the month.

To summarize, the business impact of the CPU Utilization High event on this server was directly responsible for $1M loss to the business for the month. The events and messages should communicate the impact to the business in terms that everyone understands, convey the severity of the problem and prompt the appropriate actions.

Topics for Future Discussion:

• Event Construction
• Normalization in Detail
• Enumeration
• Naming Schemas
• Incident Classifications (Category, Type, Item, etc.)
• Severity/Priority/Impact Classifications