≡ Menu


thoughts on business, service and technology operations and management in a big data and analytics world

Here we are again with another quarterly release of IBM’s Log Analysis solution. About three months ago or so we released version 1.3.0 and this week we’re releaseing version 1.3.1 full of all kinds of log ninja goodness. Let me get some basics out in this first post.

ITOA Community and GitHub

Our new (and hopefully final!) community was launched during the IBM Interconnect conference and is available here. It’s here you’ll find blogs, wikis, forums and our resources catalog full of all kinds of things to help you get started with our ITOA portfolio. What’s even cooler is our GitHub community linked in here where our SMEs, partners and clients can share their own best practices, configurations and code. Check that out here.

Keep a close eye here as the catalog of available content packs rapidly grows. We expect upwards of 100 new content packs to be available over the coming year.


It’s always fun to try and find the docs isn’t it? Here’s some quick links to the online pubs.

v1.3.1: Pubs
v1.3.1: Release Notes

v1.3.0: Pubs
v1.3.0 Release Notes

I’m not a mainframe guy myself, but if that’s your cup of tea, here’s the docs for our mainframe log analysis release. This release is based off of the v1.3.0 mentioned above.

v2.1.0 Release Notes

Here are some of the significant features in the v1.3.x release train so far. I’ll dive deeper into some of these with some how-to’s in follow on posts.

  • Real Time Alerting, Alert Management GUI
  • “Anomaly” Detection
  • Logstash 1.4.2 Bundling with custom Logstash Output Plugin for LA
  • New OOTB Insight Packs – IBM MQ Series and IBM MQ Broker/Integration Bus
  • Ticket Analytics – IBM Control Desk, Service Now and BMC Remedy
  • Hadoop/HDFS Integrations – IBM Big Insights 3.x and Cloudera CDH 5.x
  • Role Based Access Control (RBAC) Phase 1
  • Auditing
  • Additional statistical functions for search facets (for use in visualizations and dashboards
  • Dashboard Auto-Refresh
  • Globalization for ten languages
  • Currency for Apache Solr, RHEL and browsers

If any of this makes you feel warm and fuzzy, why not grab the free trial version (software or VM) and play around in your environment. The LA 1.3.0 version is available here. I hope they get the LA 1.3.1 version up there soon, so keep an eye out for it at the same link.

Don’t hesitate to reach out to me directly if you have any questions or need some help!


Logging_in_Finnish_LaplandTo catch up on this series, start back here.

We started with a simple deployment planning activity – get some servers built so we can get the necessary software installed and then and start hacking some configurations together to allow sample data to be collected and indexed and some simple searches and visualizations from that data to be available. Enter fun activity #1 – SoftLayer and associated VPNs. I would do anything for some way to stay logged into my SoftLayer VPN for longer than a day! Nothing pains me more than having 20 more more putty sessions up and then getting kicked off the VPN when the 24 hour timer ends. Please tell me how I can change this behavior or have some magical way to reconnect the VPN and all those putty sessions!

My partner Hao on the Dev/Ops team sent me some sample logs from a handful of Cloud Foundry (CF) components including the Cloud Controller and DEA components. Turns out each CF component type and end point system (CCI – SoftLayer Cloud Computing Instance) could have upwards of 10-20 different log types. I started out as I do with any project like this and began to immerse myself in the log samples. Before I was asked to start, Hao started kicking the tires on our Log Analysis solution on his own and hit some of the fundamental challenges in this space in terms of how to get logs in. He specifically called out logs that were wrapped in JSON, logs with timestamps in Epoch format, logs without timestamps, etc. as a few of his key challenges.

I think most of us in this space always start out in a common way – find the timestamps, find the message boundaries, find the unique message patterns that might exist within each log type and then think about what meaningful data should be extracted from each log type to enable problem isolation and resolution activities. I talk to clients about taking an approach for finding a sweet spot in their log analysis architecture between simple consolidation and archival of everything, to enabling everything for search to gaining high value insights from log data. Just because you have all kinds of logs doesn’t mean you must send them all into your log solution or invest a lot of time trying to parse and annotate every possible log message into unique and detailed fields. Finding the right balance of effort put into integration, parsing and annotation of logs, frequency of use and the value to the primary persona (Dev/Ops, IT Ops, App/Dev, etc.) are all dimensions and trade offs to consider before starting to boil the ocean. Simple indexing and search can go a long way before deep parsing and annotation is really required across all log types.

What this means to us for our first few milestones is that there are a lot of log types and we don’t have any firm requirements for what’s exactly needed yet. We don’t know which logs will prove to be most valuable to the global BlueMix Fabric Dev/Ops team. Obviously, some of these log types will be more useful in problem isolation and resolution activities. We want to bring everything in initially in a simple manner and then get that iterative feedback on which logs provide most useful insights and work on parsing and annotating those for high value search, analysis and apps. Out of two dozen unique log sources in the CF environment, we started out by taking an approach that kept things as simple as possible. We wanted to get as much data in as quick as possible so we could get feedback from the global Dev/Ops team on which log types were the most useful to them.

From the beginning of my discussions with the Dev/Ops team, we needed to keep in line with their deployment model, base images and automation approaches as much as possible. We didn’t want to deploy the Tivoli Log File Agent (actually not supported on Ubuntu anyway) or install anything else to move the log data off the end point systems. We decided to make use of the standard installation of rsyslog 4.2 (ancient!) on the Ubuntu 10.4 virtual cloud computing instances (CCIs) used within the BlueMix/CF/Softlayer environment.

We’re using the standard rsyslog imfile module to map in the 10-20 different log files per CCI into the standard rsyslog message format for shipping to a centralized rsyslog server. Each file we ship using imfile gets a functional tag assigned (eg DEA, CCNG, etc) which is useful downstream for filtering and parsing. On the centralized rsyslog server we’re using a custom template to create a simple CSV output format which we send to logstash for parsing and annotation. The point here is that we took a simple approach and normalized all of the CF logs to a standard rsyslog format which gives us a number of standardized, well formatted slots single including one slot containing the entire original message content to use downstream. I’ll spend an number of posts on rsyslog and logstash later!

Here’s an example of the typical imfile configuration:

### warden.log
$InputFileName /var/vcap/sys/log/warden/warden.log
$InputFileTag DEA_WARDEN_warden
$InputFileStateFile stat-DEA_WARDEN_warden
$InputFileSeverity debug
$InputFileFacility local3

This is the template we use to take the incoming rsyslog stream from all of the CCIs and turn it into a simple CSV formatted message structure. In reality I could even simplify this a bit more by removing a couple of the fields I’m ultimately not indexing in the Log Analysis solution.

template(name="scalaLogFormatDSV" type="list") {
property(name="timestamp" dateFormat="rfc3339" position.from="1" position.to="19")
property(name="msg" )

This gives a simple output message format like this for each of the 24 or more log types in this environment.

2014-08-07T14:42:17Z,localhost,10.x.x.x,DEA_WARDEN_warden,DEA_WARDEN_warden,-,local3,debug,DEA_WARDEN_warden, {"timestamp":1407422537.8179543,"message":"info (took 0.065380)","log_level":"debug","source":"Warden::Container::Linux","data":{"handle":"123foobar","request":{"handle":"123foobar"},"response":{"state":"active","events":[],"host_ip":"10.x.x.x","container_ip":"10.x.x.x","container_path":"/var/vcap/data/warden/depot/123foobar","memory_stat":"#","cpu_stat":"#","disk_stat":"#","bandwidth_stat":"#","job_ids":[123foobar]}},"thread_id":123foobar,"fiber_id":123foobar,"process_id":123foobar,"file":"/var/vcap/data/packages/warden/43.2/warden/lib/warden/container/base.rb","lineno":300,"method":"dispatch"}

So much like the picture in this blog post, we’ve got lots of logs in all shapes and sizes. Some are huge, some are small and we’re getting them into a uniform format (eg uniform size/length for the log truck) in preparation for getting the most value from them (eg lumber). Up next, shipping an aggregated stream to logstash for parsing these normalized messages

1 comment

SCALA Log Analysis Tuning App v2 As part of this BlueMix Fabric Log Solution project, getting visibility into everything in my log solution architecture is pretty important. I’ve got a lot of instrumentation across the end-to-end pipeline so metrics are overflowing in my environment. I started to work on a simple app to pull all of this together so I can trend and visualize it over time and be able to see the impacts of tuning activities.

This is a first cut at pulling some of the metrics out of the Distributed EIF Receiver / Unity Generic Receiver logs. I’m shipping them with the logstash-forwarder, parsing them in logstash and sending them to the internal elasticsearch server for easy search and visualization using kibana. ELK at its finest!

I’ll update this as I go for other SCALA logs as well as others I’m using frequently such as rsyslog impstats.

Ping me or check my github soon for the configurations.


The_deep_blue_sea_(6834127561) A few months back, I was asked to help deploy our Log Analysis solution for our BlueMix Fabric Dev/Ops team. Their pain point – getting value and insights from massive amounts of Cloud Foundry (CF) log data across multiple development, staging and production environments in order to provide a highly available BlueMix offering. No problem I thought. A log is a log is a log. I’d done this a number of times for various applications or technologies using our Log Analysis solution. Three months later as we move this into an environment most closely resembling our production environment, things have gotten very interesting to say the least in terms of designing for a scale out log solution supporting 100′s to 1,000′s of GB of log data each day.

I want to share my journey here on my blog so others may benefit who chose to do something similar in their own Cloud Foundry environment or within their other very large application or technology environments using our Log Analysis solution. Parts of this are certainly reusable with other similar log collection, consolidation, search and visualization solutions available today and are not all dependent on use of the IBM Log Analysis solution. The overall architecture and design approach, decisions and many of the configurations are reusable for anyone desiring to design, build and deploy a log management solution using modern products, tools and techniques.

Within most highly dynamic and growth projects, start-ups, etc., management and monitoring stuff is often an afterthought, a “we’ll get to it later” kind of thing. There were no firm business or technical requirements guiding us as we began this project. I think everyone on the global Dev/Ops team knew it should be done and many were trying to attack the problem with scripts and one-off tools to help them keep their heads above water and deal with daily problems. What we need, expect and desire from the solution has evolved over each milestone of this project and will continue to as more of the global Dev/Ops team begins to use the solution on a daily basis. There are however, a few fundamental architecture and design goals that I’ve anchored my work on this project around based on our early experiences in the project:

Architecture Design Goals - We didn’t start with these from day zero, but they quickly became the focus of our work as we discovered the operational characteristics of each BlueMix Cloud Foundry environment.

  • Support a sustained message volume of XX MB|GB/s < -- not hiding numbers here, we just haven't set a target yet!
  • Message delivery quality of XX %
  • End to end source to search availability in XX minutes – when a record is written, when is it available for search?
  • Absorb a sustained burst in message volume of XX MB|GB/s over XX minutes
  • Process all rsyslog disk assist cache and/or buffers from burst within XX minutes

Need to Understand - With anything new, there are lots of unanswered questions and concerns from various parts of the Dev/Ops team. We need to work towards being able to answer some fundamental questions such as these.

  • Total daily message volume (GB/TB), messages/sec, network utilization
  • Total message volume by CF component type (eg Cloud Controller) day/week/month
  • Total message volume by end point CCI (with a given CF component)
  • Retention period (eg 30 days) – system resources required, pruning frequency
  • Find the high value log types and/or messages needed for problem isolation/resolution and determine parsing/annotation requirements
  • Find the lower value log types and/or messages and enable filtering and the edge or consolidation elsewhere

Need to Answer - We need to know how to scale up the solution as the overall BlueMix offering grows.

  • How must the log solution architecture scale with expected BlueMix growth?
  • What is the impact on current solution resources when new environments, components or end points are added?
  • How to scale architecture, control costs and provide good UX across a global deployment of BlueMix environments (datacenters)?

That’s a good intro to what I’ve been up to lately along with all of the normal customer and development activity! A lot more to come for sure. Up next, The path to milestone 1 – Sample Cloud Controller and DEA Logs


These are my links for March 25th through June 18th:

  • OpenStack LumberJack – Part 1 rsyslog | Professional OpenStack – Logging for OpenStack has come quite a ways. What I’m going to attempt to do over a few posts, is recreate and expand a bit on what was discussed at this last OpenStack Summit with regard to Log Management and Mining in OpenStack. For now, that means installing rsyslogd and setting it up to accept remote connections.
  • rsyslog.conf file
  • FailoverSyslogServer – rsyslog wiki
  • How to configure failover for rsyslog in Red Hat Enterprise Linux 6? – Red Hat Customer Portal
  • Introducing the Solr Scale Toolkit | SearchHub | Lucene/Solr Open Source Search
  • Highly Available ELK (Elasticsearch, Logstash and Kibana) Setup | Everything Should Be Virtual
  • Logstash configuration dissection
  • Splunk Introduces Splunk Enterprise 6.1 – Enabling the Mission-critical Enterprise Multi-site Clustering: Delivers continuous availability for Splunk Enterprise deployments that span multiple sites, countries or continents by replicating raw and indexed data in a clustered configuration. Search Affinity: Provides a performance increase when using multi-site clustering by routing search and analytics requests to the nearest cluster, increasing performance and decreasing network usage. zLinux Forwarder: Allows for application and platform data from IBM mainframes to be easily collected and indexed by Splunk Enterprise. Data Preview with Structured Inputs: Enables previewing of massive data files to verify alignment of fields and headers before indexing to improve data quality and the time it takes to discover critical insights.
  • Streamlining application logs collection on AWS Elastic Beanstalk with logstash – part 1 | Mob in Tech – However, we like to experiment things, so I decided to try the home made solution for the backend of our new upcoming mobile game. Our backend is a homebrewed Java REST webservices application hosted in an Elastic Beanstalk container, in the us-east-1 region. The final goal is to gather logs from all instances of the Java application into a local (Paris) Elastic Search database, in a scalable manner. In this case, scalable means for us: every single step of the data pipeline has to be horizontally scalable, meaning we can speed up the process by adding additional capacity at each step independently.
  • How to Pre-Process Logs with Logstash: Part III of “Scalable and Robust Logging for Web Applications” ← #workHard / partyHard – This article is an introduction on how to pre-process logs from multiple sources in logstash before storing them in a data store or analyze them in real time. Some common use cases are unifying time formats across different log sources, anonymizing data, extracting only interesting information from the logs as well as tagging and selective distribution.
  • Building an Activity Feed System with Storm – Programming – O’Reilly Media – Problem You want to build an activity stream processing system to filter and aggregate the raw event data generated by the users of your application. Solution Streams are a dominant metaphor for presenting information to users of the modern Internet. Used on sites like Facebook and Twitter and mobile apps like Instagram and Tinder, streams are an elegant tool for giving users a window into the deluge of information generated by the applications they use every day.
  • Wirbelsturm: 1-Click Deployments of Storm and Kafka clusters with Vagrant and Puppet – Michael G. Noll – I am happy to announce the first public release of Wirbelsturm, a Vagrant and Puppet based tool to perform 1-click local and remote deployments, with a focus on big data related infrastructure. Wirbelsturm’s goal is to make tasks such as “I want to deploy a multi-node Storm cluster” simple, easy, and fun. In this post I will introduce you to Wirbelsturm, talk a bit about its history, and show you how to launch a multi-node Storm (or Kafka or …) cluster faster than you can brew an espresso.
  • RapidEngines Application Analytics – We provide the worlds fastest, most flexible and most scalable time series data platform. Delivered as software or a cloud service to help you visualize and detect application performance events before they impact your business.
  • SevOne Acquires Log Analytics Provider RapidEngines | Business Wire – SevOne, the leader of scalable performance monitoring solutions to the world’s most connected companies, today announced it has acquired RapidEngines, a leading provider of highly scalable log analytics software for IT enterprises, service providers and application developers. The acquisition is the first from SevOne since closing the $150M investment from Bain Capital which remains one of the largest venture financings of 2013. SevOne’s large customer base will now have access to RapidEngines’ log analytics software granting users the benefit of automatically collecting and organizing log data to better provide a detailed picture of user and machine behavior.
  • Google Cloud Platform Blog: A New Logs Viewer for Google Cloud Platform – Today we are excited to announce a significantly updated Logs Viewer for App Engine users. Logs from all your instances can be viewed together in near real time, with greatly improved filtering, searching and browsing capabilities. This release includes UI and functional improvements. We’ve added features that simplify navigation and make it easier to find the logs data you’re looking for.
  • About | LOGSEARCH – What started out as an internal development project from within City Index was soon after released as an open source project for all to benefit. City Index realised the potential value of the information available to them in the log files and required a flexible solution to not only view the log files but rather to view and cross analyse them.
  • Approaches to Indexing Multiple Logs File Types in Solr and Setting up a Multi Node, Multi Core Solr Cloud – Apache Solr is a widely used open source search platform that internally uses Apache Lucene based indexing. Solr is very popular and provides a database to store indexed data and is a very high scalable, capable search solution for the enterprise platform. This article provides a basic vision for a single and multi-core approach to indexing and querying multiple log file types in Solr. Solr indexes the log files generated by the servers and allows searching the logs for troubleshooting. It has the capability to scale to work in a multi-node cluster set up in a distributed and fault tolerant manner. These capabilities are collectively called SolrCloud. Solr uses Zookeeper for working in a distributed manner
  • Introducing Morphlines: The Easy Way to Build and Integrate ETL Apps for Hadoop | Cloudera Developer Blog – Morphlines can be seen as an evolution of Unix pipelines where the data model is generalized to work with streams of generic records, including arbitrary binary payloads. A morphline is an efficient way to consume records (e.g. Flume events, HDFS files, RDBMS tables, or Apache Avro objects), turn them into a stream of records, and pipe the stream of records through a set of easily configurable transformations on the way to a target application such as Solr, for example as outlined in the following figure: In this figure, a Flume Source receives syslog events and sends them to a Flume Morphline Sink, which converts each Flume event to a record and pipes it into a readLine command. The readLine command extracts the log line and pipes it into a grok command. The grok command uses regular expression pattern matching to extract some substrings of the line. It pipes the resulting structured record into the loadSolr command. Finally, the loadSolr command loads the record into Solr, typically a SolrCloud. In the process, raw data or semi-structured data is transformed into structured data according to application modelling requirements.
  • Pivotal CF 1.1 Advances Enterprise PaaS with New Capabilities | Pivotal P.O.V. – What’s new in Pivotal CF 1.1:

    Improved app event log aggregation – developers can now go to a unified log stream for full application event visibility (Watch) and drain logs to a 3rd party tool like Splunk for analysis (Watch)

  • elasticsearch-curator 1.0.0 : Python Package Index – Tending your time-series indices in Elasticsearch

The initial offering of Netcool Operations Insight (NOI) v1.1 provides a good starting point with some neat event search use cases for front line IT Ops support teams and some basic event analysis support for the Netcool Administrator. The NOI-SCALA integration provides a primary entry point for searching across events via the event list tool menu. In my opinion, the ability to search and interact with events is far more valuable using the search and apps concepts within SCALA. After I talk about some plumbing foundations, I’ll share some of my Event Analysis apps that can get you jump started with event analysis and reduction use cases in no time.

Another area for improvement is with the firehose approach of sending all events by default from the active Netcool integration point towards a single SCALA datasource. What this leads to is a single, all in one datasource in SCALA that can easily grow to be very large in size and lead to slow search performance as well as challenges with keeping the indexed event data pruned when using the SCALA delete utility available today. It’s my best practice to intelligently analyze and route events to unique datasources by key fields such as service or application name, functional technology type or role, etc. so more efficient search and apps can be created.

Another common scenario is when you have established a Netcool historical event archive and you’d like to incorporate some specific historical events within your SCALA environment for analysis. The current NOI solution doesn’t incorporate an easy or flexible way to fold in these historical events within SCALA so more realistic event search and analysis use cases can be developed. I’ll share some thoughts on intelligent event processing and routing for getting the events you need into SCALA for search and analysis in the most efficient way possible.

Fortunately, SCALA comes with the awesome logstash toolkit which can be used to simplify and extend your NOI offering through its wide array of inputs, filters and codecs and our SCALA output plugin! I’ll start this series by updating some past blog posts based on using the latest logstash v1.4 and the SCALA v1.2 release. I’m not sure when the SCALA content team will get around to updating the toolkit to install out of the box support for v1.4 so here’s what you have to do to use this much improved logstash version with SCALA v1.2.

Prepare to use logstash v1.4 and SCALA logstash toolkit

  • Download logstash 1.4.x here (zip, tar.gz, deb, rpm)
  • Unzip into a new directory where you will run it ($LOGSTASH140-DIR)
  • Download SCALA logstash toolkit from here
  • Explode it on your laptop or system
  • From the SCALA logstash toolkit, copy $MYDIR/LogstashIntegrationToolkit_v1.1.0.1/lstoolkit/LogstashLogAnalysis_v1.1.0.1/logstash/outputs/scala_custom_eif.rb and the unity/ directory to your logstash directory $LOGSTASH140-DIR/lib/logstash/outputs/
  • From the SCALA logstash toolkit, copy LogstashIntegrationToolkit_v1.1.0.1\lstoolkit\LogstashLogAnalysis_v1.1.0.1\logstash\outputs\eif.conf to your logstash directory $LOGSTASH140-DIR/lib/logstash/outputs/ and name it something like eif-scala-IPADDR.conf so you know which SCALA server it is configured for (one config file for each unique scala_custom_eif output)

Configure the SCALA logstsah toolit ouput plugin

You can have multiple SCALA logstash outputs enabling you to stream events to multiple SCALA systems. In order to do this, you’ll need multiple EIF configuration files with each one mapped to a specific SCALA system. Name each one uniquely with the SCALA target IP address so you know which one to use.

  • Configure the eif-scala-IPADDR.conf file
  • Set the BufEvtPath to something unique if you have multiple SCALA outputs in your config
  • Set the LogFileName to something unique if you have multiple SCALA outputs in your config
  • Set the ServerLocation to your SCALA destination
  • Set the ServerPort to your SCALA EIF Receiver port (5529 by default)

Build a configuration for logstash and fire it up

I’ve provided a simple logstash config here (MyLogstash.conf) to start with for this blog series. Feel free to use/extend one you may already have by adding the new scala_custom_eif output and the desired ‘scalaFields’ which are at the heart of how this scala_custom_eif output works in conjunction with the SCALA DSV toolkit.

I’ll continue to blog about using the socket gateway with logstash to stream in events to SCALA. From my experience, it’s tremendously easier than setting up the XML GW used with NOI. I’m sure you could also just configure another end point for that XML GW and route events through logstash that way, I’ve just not experimented with that.

I’m starting this series with a simple logstash configuration file which gets us up and running with a simple ‘send all events’ type flow. As I alluded to above, we really don’t want this generic set up. We’ll build upon this by using more of the powerful logstash options to address our use cases.

To start logstash : $LOGSTASH140-DIR/bin/logstash agent -f /opt/logstash-1.4.0/MyLogstash.conf –verbose &

Building a SCALA DSV pack for your events

To use this simple logstash starting config, we’ll need a SCALA DSV pack to match. The SCALA logstash output module will pre-format every field named ‘scalaFields’ into a CSV formatted message bundle and stream it outbound to the SCALA EIF receiver. The SCALA DSV pack simply breaks apart the CSV message and posts fields properly into the index.

  • Create a simple DSV header file (MyEventHeader.csv) made up of the 17 field names from alerts.status that you’re sending across the Socket GW. Reference this post to get help setting up the Netcool Socket GW.
  • From the DSVToolkit directory ($SCALAHOME/unity_content/DSVToolkit_v1.1.0.2), run the following command: python primeProps.py MyEventDSVPack.props 17 -f MyEventHeader.csv Verify that you see “The properties file was successfully edited.” returned.
  • Edit the MyEventDSVPack.props file
  • Set scalaHome to reflect your installation location (default: scalaHome: $HOME/IBM/LogAnalysis)
  • Name the DSV Pack by setting this vield to something more intuitive: aqlModuleName: MyEventDSVPack (default: aqlModuleName: dsv17Column)
  • Change the name of the LastOccurrence field to timestamp < -- this will become the main indexed timestamp within SCALA. If you want to use FirstOccurrence, then name that one timestamp instead
  • Change the dataType for the timestamp field as well as the FirstOccurrence field to: dataType: DATE
  • Add this field to the timestamp and FirstOccurrence field: dateFormat: yyyy-MM-dd’T’HH:mm:ssz < -- this must match what's come from OMNIBus Socket GW and through any logstash processing you may have done
  • Review all of your field sections and determine which fields you’d like to be able to search, sort and filter on. I generally make them all ‘true’ realizing that there could be performance trade offs on this for your search results.
  • Deploy the DSV Pack using this command: python dsvGen.py MyEventDSVPack.props -d -f -u unityadmin -p unityadmin
  • Verify that you see one or more ‘BUILD SUCCESSFUL’ responses in your terminal screen.

Final tasks and seeing results

The final SCALA task is to create a new datasource that uses the new DSV pack we just deployed. The key part of setting this up is to use the same host and path names you set in your logstash config. Create a new SCALA datasource that uses this new MyEventDSVPack Type and Collection. Set the host and path to ‘mynetcoolevents‘ based on our sample config. Name the datasource ‘My Netcool Events’ or similar.

Start everything up (Netcool GW and logstash) and verify events are successfully indexed within SCALA by watching the GenericReceiver.log and/or by running searches within SCALA.

We’ll build on this in my next post by laying down my event analysis apps and then getting into how to intelligently route events to specific datasources within SCALA.


These are my links for March 3rd through March 25th:

  • Numenta Releases Grok for IT Analytics on AWS | Numenta – Grok anomaly detection leverages sophisticated machine intelligence algorithms to enable new insights into critical IT systems. Grok automatically learns complex patterns and then highlights unusual behavior. As software topologies and usage patterns change, Grok continuously learns and adapts, eliminating the need for frequent resetting of thresholds. Visualization of Grok output is displayed on a constantly updated mobile device, enabling IT professionals to assess the health of their systems anytime, anywhere. Using Grok, IT operators can better prevent business downtime while reducing false positives.

    Grok is the first commercial application of Numenta’s groundbreaking Cortical Learning Algorithm (CLA), biologically inspired algorithms for machine intelligence. The core CLA technology is ideal for large-scale analysis of continuously streaming datasets and excels at modeling and predicting patterns in data.

    “Grok provides an early warning system to IT professionals to give them real-time insights into their system performance,” said Numenta CEO Donna Dubinsky. “Grok anticipates problems before they happen, reduces false positives, and lowers engineering costs through automated modeling and continuous learning.”

    Grok features include:

    Monitoring of performance and health of AWS environments or other systems
    Automatic modeling to determine normal patterns
    Automatic identification and ranking of unusual patterns
    Continuous learning of new patterns as environments evolve – no need for manual threshold setting
    Notification to user when an anomaly occurs
    Output displayed graphically on an Android mobile device
    Simple setup via a web-based or command-line interface
    Support for AWS auto-scaling groups and logical clusters

  • LMAO if you don’t logstash | by Paul Czarkowski | @pczarkowski
  • Elasticsearch.org Kibana 3.0.0 GA Is Now Available! | Blog | Elasticsearch – Today is a big day for Elasticsearch and the Kibana team. After 5 milestone releases and over 1000 commits, we’re happy to announce the release of Kibana 3.0.0 GA. Over the last year, Kibana has moved from a simple interface to search logs to a fully featured, interactive analysis and dashboard system for any type of data. Everyday, we’re incredibly inspired by the people who tell us they’ve solved major problems, optimized their existing deployments and found insights in places they never imagined.
  • Apache Solr vs ElasticSearch – the Feature Smackdown! – The Feature Smackdown
  • SiLK: enterprise-grade log analysis solution | LucidWorks | LucidWorks – LucidWorks Solr integration with LogStash and Kibana (SiLK) is an enterprise-grade log analysis solution that enables the ad-hoc search and analysis of billions of events and transactions across multiple applications, servers and devices.
  • Advanced Web Analytics for Big Data & Hadoop – Alpine Data Labs
  • wise.io | Machine Learning as a Service & Big Data Analytics – Our state-of-the-art machine learning technology reveals hidden value in your data. Our applications integrate seamlessly into your business.
  • Home | Skytree – Machine Learning on Big Data for Predictive Analytics – Machine Learning is the modern science of discovering patterns and making predictions from complex data.
  • UCI Machine Learning Repository – We currently maintain 273 data sets as a service to the machine learning community. You may view all data sets through our searchable interface. Our old web site is still available, for those who prefer the old format. For a general overview of the Repository, please visit our About page. For information about citing data sets in publications, please read our citation policy. If you wish to donate a data set, please consult our donation policy. For any other questions, feel free to contact the Repository librarians. We have also set up a mirror site for the Repository.
  • Log.io – Real-time log monitoring in your browser
  • doubaokun/node-ab – A command tool to test the performance of HTTP services.
  • zanchin/node-http-perf – Node HTTP Server Performance Tool
  • Uptime by fzaninotto – A remote monitoring application using Node.js, MongoDB, and Twitter Bootstrap.
  • Cloud Foundry and Logstash – Scott Frederick’s humble blog – The Cloud Foundry Loggregator component formats logs according to the syslog standard as defined in RFC5424. The logstash cookbook includes an example configuration for syslog consumption, but that configuration follows an older RFC3164 syslog standard.

    Here is a logstash configuration that works with RFC5424 output, with some additional changes for Cloud Foundry:

  • Using your historical data for analytic usage
  • Using R for Educational Research: An Introductory Workshop to Break the Learning Curve – R_intro_SERA_2012.pdf
  • Welcome to a Little Book of R for Time Series! — Time Series 0.2 documentation – This is a simple introduction to time series analysis using the R statistics software.
  • BestFirstRTutorial.pdf
  • Introducing R
  • Introduction to R Seminar – UCLA Institute for Digital Research and Education

These are my links for December 11th through March 3rd:

  • Output to Elasticsearch in Logstash format (Kibana-friendly) – In this post you’ll see how you can take your logs with rsyslog and ship them directly to Elasticsearch (running on your own servers, or the one behind Logsene’s Elasticsearch API) in a format that plays nicely with Logstash. So you can use Kibana to search, analyze and make pretty graphs out of them.

    This is especially useful when you have a lot of servers logging [a lot of data] to their syslog daemons and you want a way to search them quickly or do statistics on the logs. You can use rsyslog’s Elasticsearch output to get your logs into Elasticsearch, and Kibana to visualize them. The only challenge is to get your rsyslog configuration right, so your logs end up where Kibana is expecting them. And this is exactly what we’re doing here.

  • GraphLab Notebook | GraphLab – The power of GraphLab with the ease of Python, running in the Cloud.
  • Prelert Introduces Push Button Machine Learning in Anomaly Detective 3.1 – Prelert, the first vendor to package data science into downloadable applications for everyday users, today announced the release of Anomaly Detective 3.1, which introduces the ability to deploy powerful machine learning tools at the push of a button.

    Anomaly Detective is a deeply integrated app for Splunk Enterprise that helps identify and resolve performance and security issues, and their causes, as they develop. It provides a solution to one of the major problems inherent in working with Big Data – gaining valuable insights from otherwise overwhelming volumes of data in real-time.

  • DataLoop.io – Cloud Server Monitoring for DevOps & Operations Teams – Dataloop.IO is a new start-up in the IT Infrastructure Monitoring space, focused on building a new monitoring tool for DevOps/Operations teams that run Cloud services at scale.

    Our Cloud service significantly reduces the time required to setup and deploy your monitoring. It reduces the friction of writing and deploying new monitoring scripts so your team can ensure full coverage regardless of how quickly your environment is changing.

  • Predictions For 2014: Technology Monitoring | Forrester Blogs – Further development of pattern analytics to complement log-file analytics. For the last five years, log-file analytics has been a major focus area in the area of IT operational analytics. During 2014 we expect further development with pattern analytics or features that can make insights based on data in-stream or in-flight on the network.

    Re-emergence of business service management (BSM) features. Increasing technology innovation is leading to greater complexity in business service architecture. This means that any features that simplify the management of complex business services become a must. Hence why we predict the re-emergence of BSM features that will be more successful than previous attempts, as these new BSM approaches will have automated discovery and mapping of technology to business services.

  • Data Mining Map – An Introduction to Data Mining
  • Actian Analytics Platform™ | Accelerating Big Data 2.0™ | Actian – Actian transforms big data into business value for any organization – not just the privileged few. Our next generation Actian Analytics Platform™ software delivers extreme performance, scalability, and agility on off-the-shelf hardware, overcoming key technical and economic barriers to broad adoption of big data, delivering Big Data for the Rest of Us™.
  • Visual Intelligence for your web application – COSCALE – The COSCALE Application Performance Analyzer provides swift and accessible visual intelligence for your web-application through smart correlations of any application and infrastructure metric
  • Qubole | Big Data as a Service – Switch your data infrastructure to auto-pilot using our award-winning, auto-scaling Hadoop cluster, built-in data connectors and an intuitive graphical editor for all things Big Da
  • Altiscale Hadoop as a Service – Altiscale’s offering is ideally suited for today’s data science needs. Features for data science include permanent HDFS volumes, access to the latest tools, resource sharing without conflict, job-level monitoring and support, and pricing plans that eliminate unpleasant surprises.
  • Boundary Surpasses 400% YoY Growth in Processing of Massive IT Operations Performance Analytics in the Cloud – The Boundary service is processing an average of 1.5 trillion application and infrastructure performance metrics per day on behalf of its clients and has computed occasional daily bursts of over 2 trillion metrics.
  • To Log or Not to Log: Proven Best Practices for Instrumentation – Innovation Insights – To log or not to log? This is an age-old question for developers. Logging everything can be great because you have plenty of data to work from when you have a problem. But it’s not so great if you have to grep and inspect it all yourself. In my mind, developers should instead be thinking about logging the right events in the right format for the right consumer.
  • IT Operations Analytics (ITOA) Landscape – Say goodbye to years of chronic IT Operations pains. IT Operations Analytics (ITOA) is here, and gaining strong momentum. You Are a Leader – Seize The Opportunity.
  • Zoomdata – Next Generation Big Data Analytics

    - Built for the Big Data Revolution
    - Connected to the World in Real-Time
    - Designed for the Touch Generation
    - Fuses Data into a Single Experience
    - Easy & Powerful Interface

  • Enterprise Management Services Enterprise Event Management – Enterprise Event Management Trying to manage a modern IT environment without a consolidated view of operations is like trying to drive a car at 100 mph while looking at six different dashboards. The proliferation of development, build, and operations tools has made it increasingly difficult to stay in control of IT and reduce downtime. Too often, developers and administrators have been left with two unappealing alternatives: Either they have to try and write their own event consolidator or they struggle with legacy products from a different era. Boundary is the industry’s leading SaaS-based enterprise event management offering, enabling you to track and optimize your modern, rapidly changing application infrastructures. With Boundary, you can consolidate, standardize, prioritize, enrich and correlate events and notifications from hundreds of systems into a single console.
  • Cubism.js – Cubism.js is a D3 plugin for visualizing time series. Use Cubism to construct better realtime dashboards, pulling data from Graphite, Cube and other sources. Cubism is available under the Apache License on GitHub.
  • Crosslet – Crosslet is a free small (22k without dependencies) JavaScript widget for interactive visualisation and analysis of geostatistical datasets. You can also use it for visualizing and comparing multivariate datasets. It is a combination of three very powerful JavaScript libraries: Leaflet, an elegant and beautiful mapping solution, and Crossfilter, a library for exploring large multivariate datasets in the browser. D3, a data driven way of manipulating objects. Crosslet also supports TopoJSON, a GeoJSON extension that allows to present geometry in a highly compact way. Crosslet is written in CoffeeScript and uses less for styling.
  • Charts, Graphs and Images – CodeProject
  • dc.js – Dimensional Charting Javascript Library – dc.js is a javascript charting library with native crossfilter support and allowing highly efficient exploration on large multi-dimensional dataset (inspired by crossfilter's demo). It leverages d3 engine to render charts in css friendly svg format. Charts rendered using dc.js are naturally data driven and reactive therefore providing instant feedback on user's interaction. The main objective of this project is to provide an easy yet powerful javascript library which can be utilized to perform data visualization and analysis in browser as well as on mobile device.
  • SharePoint Development Lab by @avishnyakov » Go Cloud – A better logging for SharePoint Online/Office365/Azure apps – It seems that cloud based products and services have a significant impact on how we design, write, debug, trace and deliver our applications. The way we think about this is not the same anymore; there might be no need to have SharePoint on-premises and SharePoint Online/O365 might be a better choice, there might be no reason to host a web application on dedicated hardware/hosting provider, but Azure could bring more benefits. All these trends cannot be simple ignored, and it is a good thing to see how new services and offerings might be used in you applications.