Extending Netcool Operations Insight Event Search

The initial offering of Netcool Operations Insight (NOI) v1.1 provides a good starting point with some neat event search use cases for front line IT Ops support teams and some basic event analysis support for the Netcool Administrator. The NOI-SCALA integration provides a primary entry point for searching across events via the event list tool menu. In my opinion, the ability to search and interact with events is far more valuable using the search and apps concepts within SCALA. After I talk about some plumbing foundations, I’ll share some of my Event Analysis apps that can get you jump started with event analysis and reduction use cases in no time.

Another area for improvement is with the firehose approach of sending all events by default from the active Netcool integration point towards a single SCALA datasource. What this leads to is a single, all in one datasource in SCALA that can easily grow to be very large in size and lead to slow search performance as well as challenges with keeping the indexed event data pruned when using the SCALA delete utility available today. It’s my best practice to intelligently analyze and route events to unique datasources by key fields such as service or application name, functional technology type or role, etc. so more efficient search and apps can be created.

Another common scenario is when you have established a Netcool historical event archive and you’d like to incorporate some specific historical events within your SCALA environment for analysis. The current NOI solution doesn’t incorporate an easy or flexible way to fold in these historical events within SCALA so more realistic event search and analysis use cases can be developed. I’ll share some thoughts on intelligent event processing and routing for getting the events you need into SCALA for search and analysis in the most efficient way possible.

Fortunately, SCALA comes with the awesome logstash toolkit which can be used to simplify and extend your NOI offering through its wide array of inputs, filters and codecs and our SCALA output plugin! I’ll start this series by updating some past blog posts based on using the latest logstash v1.4 and the SCALA v1.2 release. I’m not sure when the SCALA content team will get around to updating the toolkit to install out of the box support for v1.4 so here’s what you have to do to use this much improved logstash version with SCALA v1.2.

Prepare to use logstash v1.4 and SCALA logstash toolkit

• Download logstash 1.4.x here (zip, tar.gz, deb, rpm)
• Unzip into a new directory where you will run it ($LOGSTASH140-DIR)
• Download SCALA logstash toolkit from here
• Explode it on your laptop or system
• From the SCALA logstash toolkit, copy $MYDIR/LogstashIntegrationToolkit_v1.1.0.1/lstoolkit/LogstashLogAnalysis_v1.1.0.1/logstash/outputs/scala_custom_eif.rb and the unity/ directory to your logstash directory $LOGSTASH140-DIR/lib/logstash/outputs/
• From the SCALA logstash toolkit, copy LogstashIntegrationToolkit_v1.1.0.1\lstoolkit\LogstashLogAnalysis_v1.1.0.1\logstash\outputs\eif.conf to your logstash directory $LOGSTASH140-DIR/lib/logstash/outputs/ and name it something like eif-scala-IPADDR.conf so you know which SCALA server it is configured for (one config file for each unique scala_custom_eif output)

Configure the SCALA logstsah toolit ouput plugin

You can have multiple SCALA logstash outputs enabling you to stream events to multiple SCALA systems. In order to do this, you’ll need multiple EIF configuration files with each one mapped to a specific SCALA system. Name each one uniquely with the SCALA target IP address so you know which one to use.

• Configure the eif-scala-IPADDR.conf file
• Set the BufEvtPath to something unique if you have multiple SCALA outputs in your config
• Set the LogFileName to something unique if you have multiple SCALA outputs in your config
• Set the ServerLocation to your SCALA destination
• Set the ServerPort to your SCALA EIF Receiver port (5529 by default)

Build a configuration for logstash and fire it up

I’ve provided a simple logstash config here (MyLogstash.conf) to start with for this blog series. Feel free to use/extend one you may already have by adding the new scala_custom_eif output and the desired ‘scalaFields’ which are at the heart of how this scala_custom_eif output works in conjunction with the SCALA DSV toolkit.

I’ll continue to blog about using the socket gateway with logstash to stream in events to SCALA. From my experience, it’s tremendously easier than setting up the XML GW used with NOI. I’m sure you could also just configure another end point for that XML GW and route events through logstash that way, I’ve just not experimented with that.

I’m starting this series with a simple logstash configuration file which gets us up and running with a simple ‘send all events’ type flow. As I alluded to above, we really don’t want this generic set up. We’ll build upon this by using more of the powerful logstash options to address our use cases.

To start logstash : $LOGSTASH140-DIR/bin/logstash agent -f /opt/logstash-1.4.0/MyLogstash.conf –verbose &

Building a SCALA DSV pack for your events

To use this simple logstash starting config, we’ll need a SCALA DSV pack to match. The SCALA logstash output module will pre-format every field named ‘scalaFields’ into a CSV formatted message bundle and stream it outbound to the SCALA EIF receiver. The SCALA DSV pack simply breaks apart the CSV message and posts fields properly into the index.

• Create a simple DSV header file (MyEventHeader.csv) made up of the 17 field names from alerts.status that you’re sending across the Socket GW. Reference this post to get help setting up the Netcool Socket GW.
• From the DSVToolkit directory ($SCALAHOME/unity_content/DSVToolkit_v1.1.0.2), run the following command: python primeProps.py MyEventDSVPack.props 17 -f MyEventHeader.csv Verify that you see “The properties file was successfully edited.” returned.
• Edit the MyEventDSVPack.props file
• Set scalaHome to reflect your installation location (default: scalaHome: $HOME/IBM/LogAnalysis)
• Name the DSV Pack by setting this vield to something more intuitive: aqlModuleName: MyEventDSVPack (default: aqlModuleName: dsv17Column)
• Change the name of the LastOccurrence field to timestamp < -- this will become the main indexed timestamp within SCALA. If you want to use FirstOccurrence, then name that one timestamp instead
• Change the dataType for the timestamp field as well as the FirstOccurrence field to: dataType: DATE
• Add this field to the timestamp and FirstOccurrence field: dateFormat: yyyy-MM-dd’T’HH:mm:ssz < -- this must match what's come from OMNIBus Socket GW and through any logstash processing you may have done
• Review all of your field sections and determine which fields you’d like to be able to search, sort and filter on. I generally make them all ‘true’ realizing that there could be performance trade offs on this for your search results.
• Deploy the DSV Pack using this command: python dsvGen.py MyEventDSVPack.props -d -f -u unityadmin -p unityadmin
• Verify that you see one or more ‘BUILD SUCCESSFUL’ responses in your terminal screen.

Final tasks and seeing results

The final SCALA task is to create a new datasource that uses the new DSV pack we just deployed. The key part of setting this up is to use the same host and path names you set in your logstash config. Create a new SCALA datasource that uses this new MyEventDSVPack Type and Collection. Set the host and path to ‘mynetcoolevents‘ based on our sample config. Name the datasource ‘My Netcool Events’ or similar.

Start everything up (Netcool GW and logstash) and verify events are successfully indexed within SCALA by watching the GenericReceiver.log and/or by running searches within SCALA.

We’ll build on this in my next post by laying down my event analysis apps and then getting into how to intelligently route events to specific datasources within SCALA.