Event Analysis using SmartCloud Analytics Log Analysis (SCALA) v1103 – Deploying Logstash

by doug on November 20, 2013

in Event Analytics, Event Management, Events, IBM Log Analytics, IT Operations Analytics, Log Analytics, Logstash, Netcool/OMNIbus

Now that I’m done with what felt like months of work for our big demo at IBM’s IOD show last week, let me get this series done! Next up we’ll walk through the use of Logstash to serve as the collection and mediation tool for streaming in events from Netcool/OMNIbus and getting them indexed within SCALA v1103. We’re still using Logstash v113 here. We’ll have support for Logstash v1.2.x in our next release very soon. NOTE: With SCALA v1103 now available, that will be what I mention moving forward.

To catch up, check out part 1 and part 2.

On a separate system if at all possible, prepare for installation of Logstash v113 and the SCALA Logstash toolkit.

Download logstsah v1.1.13 from here
Create a new directory for the logtash environment. I generally create /opt/logstash.
Copy the SCALA Logstash Toolkit to this directory
Review the SCALA Logstash Toolkit installation steps
Explode the SCALA Logstash Toolkit
Copy the logstash-1.1.13-flatjar.jar package to this /opt/logstash/lstoolkit directory
Update the install configuration file install-scala-logstash.conf
Update the eif.conf file
Run the ./install-scala-logstash.sh script.

The lstoolkit directory contains the following files:

/opt/logstash/lstoolkit/ - LogstashLogAnalysis_v1.1.0.0.zip - install-scala-logstash.conf - startlogstash-scala.sh - install-scala-logstash.sh - logstash-1.1.13-flatjar.jar - start-logstash.conf - logstash/

/opt/logstash/lstoolkit/logstash/ - conf/ -- logstash-scala.conf - outputs/ -- eif-10.10.10.1.conf -- scala_custom_eif.rb - unity/

Next, we need to make a few simple configurations in the Logstash configuration file to get us up and running. In this simple scenario, the following configuration file for Logstash should be updated with a configuration similar to this:

input { #Create your TCP input which your Netcool/OMNIbus socket gateway will connect to


	tcp

	{

       	type=> "netcool"

	format=> "plain"

	port=> 1234

	data_timeout=> -1

    	}
} #End of Inputs
filter

{

#Use the Mutate filter to set the hostname and log path to anything you want. This is used in the SCALA LogSource definition.
	mutate

	{

       	type=> "netcool"

	replace=>["@source_host","MYOMNIBUSNAME","@source_path","Netcool"]

    	}
#Have some events you want to drop out? I used the Grep filter type to filter out some poorly formatted events whose summary message included commas which broke SCALA DSV processing
	grep

	{

       	type=> "netcool"

	match=>[ "@message",".*WAS_YN_WebAppNoActivity_W.* | .*WAS_YN_WebAppActivity_H.*" ]

	negate=> true

    	}
} #End of Filters
output

{

    	#Create a simple output file of all your raw CSV delimited events for future use, replay, etc. 
        file

	{

       	type=> "netcool"

	message_format=> "%{@message}"

	path=> "/opt/logstash/raw-events-csv.log"

    	}
	#Create one or more ouputs to spray events to as many SCALA boxes as you'd like
	scala_custom_eif

	{

        eif_config=> "logstash/outputs/eif-10.10.10.1.conf"

	debug_log=> "/tmp/scala/scala-logstash-10.10.10.1.log"

	debug_level=> "debug"

    	}

} #End of Outputs

Note: If you have multiple SCALA systems, you can spray events to each of them by having more than one output stanza for the scala_custom_eif plugin. Each one must have its own unique eif_config and debug_log configurations. I just put in the IP address of my end points to easily identify each one.

To start up Logstash, use the ./startlogstash-scala.sh script. You may wish to update this to send Logstash to the background when starting up. To stop Logstash, use ps aux | grep logstash and kill the Logstash process.

When we complete the next series of tasks in Netcool/OMNIbus we can peek at the output file we created via Logstash, we can see the raw CSV events that resemble the example below. This is what’s sent across the socket gateway.

INSERT: "WAS_YN_EJBConNoActivity_W:syswasslesNode01:syswassles:KYNS::ITM_EJB_Containers", 2013-09-27T13: 46: 44EDT, 2013-09-27T13: 46: 44EDT, "syswasslesNode01:syswassles:KYNS", "syswasslesNode01:syswassles:KYNS", "WAS_YN_EJBConNoActivity_W[(Method_Invocation_Rate=0.000 ) ON syswasslesNode01:syswassles:KYNS (Method_Invocation_Rate=0 )]", 1, "tivoli_eif probe on systbsmsles", "ITM", "ITM_EJB_Containers", "WAS_YN_EJBConNoActivity_W", 20, 2, 6601, 1, "", "", "~", "09/27/2013 08:29:45.000", "sysitm.poc.ibm.com", "S", "TEMS", "", "WAS_YN_EJBConNoActivity_W", "", "syswasslesNode01:syswassles:KYNS", "", "", "", "", "", "", "", "", "", "", "", 0, "", ""

This is the event passed in from the TCP Input and through the filters to the scala_custom_eif output:

D, [ 2013-09-27T13: 46: 42.601000#21554 ]DEBUG--: scala_custom_eif: Receivedevent: #"tcp://10.10.10.1:52074/", "@tags"=>[


    ],

    "@fields"=>{

}, "@timestamp"=>"2013-09-27T17:46:42.588Z", "@source_host"=>"s3systbsmsles", "@source_path"=>"Netcool", "@message"=>"INSERT: \"WAS_YN_EJBConNoActivity_W:syswasslesNode01:syswassles:KYNS::ITM_EJB_Containers\",2013-09-27T13:46:44EDT,2013-09-27T13:46:44EDT,\"syswasslesNode01:syswassles:KYNS\",\"syswasslesNode01:syswassles:KYNS\",\"WAS_YN_EJBConNoActivity_W[(Method_Invocation_Rate=0.000 ) ON syswasslesNode01:syswassles:KYNS (Method_Invocation_Rate=0 )]\",1,\"tivoli_eif probe on systbsmsles\",\"ITM\",\"ITM_EJB_Containers\",\"WAS_YN_EJBConNoActivity_W\",20,2,6601,1,\"\",\"\",\"~\",\"09/27/2013 08:29:45.000\",\"sysitm.poc.ibm.com\",\"S\",\"TEMS\",\"\",\"WAS_YN_EJBConNoActivity_W\",\"\",\"syswasslesNode01:syswassles:KYNS\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",0,\"\",\"\"\n", "@type"=>"netcool" }>

This is the event sent out of the scala_custom_eif output in the IBM Event Integration Framework (EIF) format fit for consumption by the SCALA EIF Receiver.

D, [ 2013-09-27T13: 46: 42.602000#21554 ]DEBUG--: scala_custom_eif: Sendingtecevent: AllRecords;hostname='s3systbsmsles';RemoteHost='';text='INSERT: "WAS_YN_EJBConNoActivity_W:syswasslesNode01:syswassles:KYNS::ITM_EJB_Containers", 2013-09-27T13: 46: 44EDT, 2013-09-27T13: 46: 44EDT, "syswasslesNode01:syswassles:KYNS", "syswasslesNode01:syswassles:KYNS", "WAS_YN_EJBConNoActivity_W[(Method_Invocation_Rate=0.000 ) ON syswasslesNode01:syswassles:KYNS (Method_Invocation_Rate=0 )]", 1, "tivoli_eif probe on systbsmsles", "ITM", "ITM_EJB_Containers", "WAS_YN_EJBConNoActivity_W", 20, 2, 6601, 1, "", "", "~", "09/27/2013 08:29:45.000", "sysitm.poc.ibm.com", "S", "TEMS", "", "WAS_YN_EJBConNoActivity_W", "", "syswasslesNode01:syswassles:KYNS", "", "", "", "", "", "", "", "", "", "", "", 0, "", ""';logpath='Netcool';END

Logstash is far more powerful than what I’ve showed in this very simple example. I’d encourage you to investigate its capabilities further by reading the website, user group or IRC.

Up next, we’ll walk through the configuration of Netcool/OMNIbus and get our events flowing towards Logstash and SCALA.

Next post: Event Analysis using SmartCloud Analytics Log Analysis (SCALA) v1103 – Setting up Netcool/OMNIbus

Previous post: A Glimpse into Loggly’s Scale Out “Logging as a Service” Architecture