As previously mentioned in the first blog post in this series, one of the goals for the first release of Smart Cloud Analytics for Applications (SCAA) is to make it as simple to download, install and use as possible. One way I’ve found to accelerate this is to start using the tools found within the application development, support and release dev/ops end user community as they are the target end user personas for the first release of SCAA.

Objective

The use and excitement surrounding Vagrant is rapidly increasing and a vibrant community has emerged allowing for rapid development and support across many platforms. This is enabling application developers, support and release dev/ops teams to quickly create and work within application environments that most closely resembles what might be found in production. What works well for the latest release of a trendy web/social/mobile application can certainly work well for those who work with application management and monitoring tools like SCAA.

This follow on blog post advances the use of Vagrant and SCAA to allow the demonstration of features in a more realistic scenario based manner. You’ll be able to install SCAA in either a stand alone manner or along with another system that has WebSphere Liberty 8.5.next installed with a sample application. All of this has been integrated so logs are flowing in real time, indexed and ready for searching within the SCAA web interface.

Install Virtualbox & Vagrant

We’ll create the base of our development and testing environment by using Virtualbox for our virtualized system platform. This will allow us to easily spin up CentOS 6.4 (or other) based systems for laying down the software in our evaluation scenarios for SCAA. I’ll share how to create a RedHat 6.4 based box in a follow on blog post.

On top of the Virtualbox foundation we use Vagrant. Vagrant is the magic sauce that allows command line level automation for spinning up the base CentOS boxes then automatically provisioning base package requirements, installing and configuring the SCAA OpenBeta software drivers.

Please refer to the previous blog post here if you need a basic install and use overview or the Virtualbox and Vagrant websites for deeper guidance.

Decide on SCAA OpenBeta Driver 1 Deployment Topology

While I’m still trying to figure out the best mechanics for releasing incremental scenarios through git, I’ll release this now and solicit your feedback for improvement, bugs, etc. as I feel it’s working well on my laptop environment for now. I’m getting good feedback from some users so hopefully I’ll become better with the use of git and how to do releases, versioning, branching, etc. there.

For now, this release allows for two deployment topologies, standalone and what I’m calling scenario1. Standalone is nothing more than SCAA OpenBeta driver 1 now installed on CentOS 6.4 versus CentOS 5.8 as previously blogged about. This option will spin up an image and fully deploy driver 1 and provision the DayTrader sample scenario demo as shown below.

The other option is called scenario1 and is a multi-box scenario featuring the above described SCAA OpenBeta driver 1 box1 plus a new system box2 with WebSphere Liberty 8.5.next installed using a sample Online Polling application. This new box2 will also have SCAA OpenBeta driver 1 LFA installed to send the application logs to SCAA. On the SCAA box1, the appropriate collections, log sources and quick searches are automatically configured allowing immediate search and interaction within SCAA’s main console.

Scenario 1 (shown below) is a great way to see how SCAA will work in your application environment. You’ll get to interact with the data processed via the WebSphere and DB2 Insight Packs as well as log data that doesn’t have an Insight Pack via our Generic Annotation support.

Review the read me on my git hub here for the latest installation procedure. ** Note ** with the help of C835722, we’re refactoring the repository and process to make it much simpler. I’ll update this blog post with those details when I return to the states. For now – the readme on the repo here has the details.

Launch WebSphere Liberty Sample Application

To generate some application logs within the WebSphere Liberty messages and trace logs, launch the Online Polling Sample server app in your browser at http://10.10.10.3:9080/OnlinePollingSampleWeb/VoteServlet.

You’ll be presented with a few options to initialize the Online Polling Sample server app. Select one and press the submit button.

Cast some votes in the sample app. One vote submission will easily create at least 50 log records to be generated based on the level of logging in place in the WebSphere Liberty OnlinePollingSample server.xml file. Feel free to customize this for more or less logging as desired!

Launch SCAA OpenBeta driver 1 UI

To begin exploring the SCAA OpenBeta driver 1, launch your Firefox 17 browser and point it at http://10.10.10.2:9988/Unity. You will log in with the username of ‘unityadmin’ and password of ‘unityadmin’.

Examine Logs

After logging into the SCAA console, you’ll notice in the upper left corner four quick searches have been created for you. In the SCAA browser window, click on the ‘Liberty_Trace_Log’ Quick Search to see all of the Websphere Liberty OnlinePollingSample application logs within SCAA. Feel free to interact with the results, run wildcard searches across the various log sources, etc.

By default, the returned search results appear in a “raw record” like display. Clicking on the grid icon in the upper left corner above the search results will switch you into a grid (or spreadsheet) like view where the first real interaction with search results becomes possible.

Discovered Patterns

Shortly after a quick search or free form search is run, the Configured Patterns or Discovered Patterns sections will begin to populate with key facets found across returned log records. If you clicked on one of the Liberty quick searches, take a look at the ‘Discovered Patterns’ that are automatically created through analysis of the log records.

Discovered patterns are automatically generated through the application of text analytics techniques (e.g. regular expressions, pattern inspection, noise removal) to find those things that are of interest in common log files. In the open beta driver1 release we’re generating discovered patterns in two areas – Concepts and Key Values. I’ll go into more detail in a follow on blog post. For now, just think of this as our smart way of finding things you might want to use as a directed search input to further drill down and around the log data you’re collecting.

We refer to this as our “Generic Annotation” support where we can consume any log source that has a log record which begins with a supported timestamp format. We call this “generic” in comparison to what we develop (and show in the Configured Patterns panel) using Insight Packs and break out into very specific parsed elements for a specific log source and log record format.

If you see a > next to a Discovered Pattern entry, this will be a Key Value Pair that we’ve discovered. You can expand the > to reveal all of the specific value tokens we’ve found in the search results for a given key token.

Clicking once on a value token will place that token into the search bar. You can add as many as you’d like by this single click action to construct a specific search pattern of interest. Click on search to execute the search.

Here’s a close up view of the matched key value pair in the log record:

If you don’t see the > icon next to an entry in the Discovered Patterns panel, then this is a Concept. In the example below, I’ve clicked on the Concept ‘OnlinePollingSampleServer’. With close inspection of the actual log record, you can see that this text appeared in the path to a specific file used in this application.

We’d love your feedback on how we can better tune our Discovered Patterns feature. If you have things we should have detected, should be filtering out (noise), etc. please let us know! This feature will continue to be improved over time!

Configured Patterns

Let’s take a look at the Configured Patterns feature now and compare it to what we just walked through for Discovered Patterns. We’ll be using the DayTrader Sample Scenario which includes a sample from a WebSphere Application Server’s SystemOut.log and from a DB2 database’s db2diag log for which we have developed Insight Pack support.

Click on the ‘DT_WAS_SystemOut’ Quick Search. Review and interact with the results. Look at the ‘Configured Patterns’ panel to see the key fields we’re extracting as part of the WebSphere Insight Packs. While the differences might not readily appear to the untrained eye, what’s happening behind the scenes is quite different.

Switch over to the grid view by clicking on the grid icon in the upper left corner above the search results. You should now see (in comparison to what you saw when doing the same for the Liberty logs) that we have a number of very specific columns in our grid.

Explore the directed search facets in the Configured Patterns panel. Similarly to the key value pairs in the Discovered Patterns panel, the difference here is that within the Insight Pack we’ve designed the splitting, parsing and annotation logic based on the well known pattern for the SystemOut.log file. Each of the common fields found in a SystemOut.log log record becomes a group within the Configured Patterns panel and all of the discovered values are populated beneath those field names.

The same can be done for the DB2 logs!

Interacting with Search Results in the Grid View

When you’re in the grid (or spreadsheet) view, you can start to interact with the returned search results. I call this “Analytics Level 1” where you’re able to move beyond the basic search and results display type interactions expected of any product in this space. The most basic interactions that can be done here include the ability to select which columns are displayed (UNFORTUNATELY – at this time we can’t change the order of columns, size of the columns, display name of the columns, etc. I HOPE that we can fix this as this is my #1 annoyance and usability issue!!), perform ascending/descending sort operations across a column and perform column plot analytics (shown below).

To analyze the contents of a column, you can select a column of interest such as the ‘msgclassifier’ by clicking on the column name to highlight/select its contents. If column plot analytics are available across that column, the pie chart icon will become illuminated with color in the upper right corner above the search results panel.

Click on this icon and a pie chart will appear and be pinned to a panel along the right hand side of the SCAA web console. This chart shows the distribution percentage of all of the different message ID’s found for the returned search results.

You can pin as many of these as you’d like to help you evaluate the returned search results and decide where to focus further problem isolation and problem determination actions.

Have Fun and Provide Feedback!!

Have fun playing around with this demo scenario. I look forward to hearing how it works for you and the feedback you have!

Don’t forget about these Vagrant commands!

vagrant up – spin up the box and provision as specified in the Vagrantfile
vagrant up –no-provision – spin up the box but don’t execute any provisioners
vagrant suspend – suspend the box as is, allows for quick start up again (consumes more host resources)
vagrant halt – gracefully shut down the box
vagrant destroy -f – wipes out the box – no part of the base box image will remain
vagrant destroy -f && vagrant up – my favorite – kill all and start fresh!

Many application development, support and operations environments are undergoing significant transformation from programs and initiatives such as DevOps, Agile/Lean, Virtualization and Cloud Computing. Over the past few years, companies, communities and people have rapidly sprouted up and created entirely new categories of technologies, tools and operational practices that are enabling them to leap frog traditional slow to transform incumbents. Terms like “born on Internet” or “born on Cloud” nicely describes the differences here.

There are dozens of exciting new technologies and tools emerging from the DevOps community that are now being widely adopted by not only those “born on Internet/Cloud” type companies but traditional enterprises as they mature through their technology transformation programs like Cloud. Vendors like IBM are now openly embracing the same to radically simplify and accelerate application delivery, support and operations by adopting those same tools and techniques enabling broader integration and adoption. Many of these are now being seen through IBM’s participation in industry groups like OpenStack or OSLC and exposed in open, transparent beta programs like Jazz for Service Management, Cloud Consumer Monitoring and the new IBM Log Analytics open beta programs.

With that, I’ve been spending a lot of time in this area getting close to application developer, support and operations teams to really understand how they work today in both traditional and emerging technology environments. I’ve started exploring and applying these same tools and technologies with the simple goal of how to introduce a repeatable process they can use for exploring IBM Log Analytics using tools and techniques they’re likely already familiar with. I want to remove any barriers to participation in our open beta program. For more information on IBM Log Analytics and the open beta program, please visit this webpage.

Vagrant is one of these new tools. From the project website, “Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team. To achieve its magic, Vagrant stands on the shoulders of giants. Machines are provisioned on top of VirtualBox, VMware, AWS, or any other provider. Then, industry-standard provisioning tools such as shell scripts, Chef, or Puppet, can be used to automatically install and configure software on the machine.”

The following blog post describes how I have used Vagrant and VirtualBox to go from a bare bones image to fully installed, ready to use instance of IBM Log Analytics open beta milestone driver in just about 15 minutes right on my laptop. If you are one of our targeted end users in an application development, support or operations groups or a subject matter expert on a WebSphere or DB2 team this approach will allow you to quickly get started in the IBM Log Analytics open beta program. By the way, you do not need to be an IBM customer – just sign up for a My IBM account here (upper right corner).

Lets get started!

Install VirtualBox

Vagrant uses a provider concept which is simply the back end virtualization platform providing key image or “box” administration and operation capabilities. VirtualBox looks pretty similar to VMWare Workstation to me so I was pretty familiar with using it. Support for the use of VirtualBox is provided with Vagrant so we’ll use this as our base virtualization capability.

Swing your browser over to the Virtual Box website located here. I installed the Windows version but it’s also supported on Linux and Mac so all of the common application developer and support engineer bases should be covered.

Install Vagrant

The Vagrant installation is quick and installation packages are available here for all of the preferred application developer and support engineer OS platforms. In my scenario, I chose the Windows package. Vagrant is a command line based interface for creating and managing boxes so you won’t find a GUI for Vagrant. You’ll be working in your terminal window when using Vagrant along with your favorite editor when creating your Vagrantfile and provisioning scripts.

Create and Initialize a Vagrant Project Directory

To get started with VirtualBox and Vagrant we begin by creating a new Vagrant project directory and running the vagrant init command within that directory. This initializes the current directory to be a Vagrant environment and creates a default Vagrantfile within that directory.

Customize Project Vagrantfile

The beauty of Vagrant comes from the ability to describe our virtualized system (“box”) and how we want it built and provisioned with desired environment, packages and software within the Vagrantfile and associated provisioners. Some refer to this as “infrastructure as code” as we’re automating every aspect of building our development, test and operations environment in a configuration file or script.

I’ve placed the Vagrantfile that I’ve used on my git repository here.

Specify a Box

There are a number of publicly available “boxes” available on the Internet that you can use to bootstrap your Vagrant environment. At this time, IBM Log Analytics is only supported on Red Hat 5 & 6 64bit systems. If your organization actively uses Red Hat you likely have the required licenses and support agreements in place and you can create Red Hat images for Vagrant. If you don’t, then an unofficial alternative is to use CentOS 5 & 6 64bit.

For those who may not know about CentOS here’s a brief introduction from the CentOS website.

CentOS is an Enterprise-class Linux Distribution derived from sources freely provided to the public by a prominent North American Enterprise Linux vendor. CentOS conforms fully with the upstream vendor’s redistribution policy and aims to be 100% binary compatible. (CentOS mainly changes packages to remove upstream vendor branding and artwork.) CentOS is free.

In most cases, all that’s required to successfully use CentOS instead of Red Hat is to spoof the installation manager by modifying the /etc/redhat-release file to reflect a supported Red Hat version.

If you want to create your own Vagrant “box” based on Red Hat or CentOS, I recommend installing veewee and following any of the great tutorials on the Internet. For the purposes of my experiments, I’m using a CentOS 5.8 64bit box provided by Puppetlabs. Other boxes are available here.

Add the following line to your Vagrantfile to specify the name of the box to be used.

config.vm.box = "Puppetlabs CentOS 5.8 x86_64"

Specify Box Location

As I’m using a publicly hosted base box, I need to specify where Vagrant will download the box from. When Vagrant is started using the up command for the first time, this base box is downloaded and stored in a location outside of your specified project directory. The box image is always kept whole and is now available for any number of Vagrant projects you decide to spin up. Every project that I create and use the box name from above will simply import the previously downloaded box and use a copy of that.

Add the following line to your Vagrantfile to specify where to download the box.

config.vm.box_url = "hhttp://puppet-vagrant-boxes.puppetlabs.com/centos-58-x64.box"

Configure Port Forwarding

By default, Vagrant will spin up the box with basic localhost (127.0.0.1) networking support. To access the guest image we need to configure port forwarding from the host to the guest for any of the ports we may need access to. SSH forwarding to port 22 is set up automatically. We’ll just need to setup port forwarding for the main IBM Log Analytics port 9988.

If you get hooked on Vagrant and spin up multiple boxes like I have been, you’re going to run into port collisions. Vagrant will automatically detect port collisions and set new ports for you nicely telling you during start up what the new ports are. You can always change these in your Vagrantfile or in the VirtualBox admin GUI.

Add the following line to your Vagrantfile to configure port forwarding.

config.vm.network :forwarded_port, guest: 9988, host: 9988

Customize the Box

Every Vagrant base box is preconfigured with a specific number of CPUs, memory, disk and other typical system specifics. The base image I’m working with comes with 1 CPU and 384MB of memory which isn’t practical for doing much with IBM Log Analytics. At a minimum you should provision your base box to start with 2 CPUs and 4GB of memory. Depending on what you plan to do and what system resources you have on your host system, adding more will not hurt! For more base box customization options, review this page.

Add the following lines to your Vagrant file to provision the base box for this project.

config.vm.provider :virtualbox do |vb|
vb.customize ["modifyvm", :id, "--name", "scla-driver1", "--memory", "4000", "--cpus", "2"]
end


Enable Provisioning

Another powerful capability of Vagrant is to define exactly what should happen after the base box is spun up. This is typically the configuration of common system environment settings, hosts files, profiles, etc. as well as running an update program such as ‘yum update’ to ensure the environment is patched to the current level. Beyond the common system level provisioning, the automated installation of packaged software, applications and configuration of those is something Vagrant allows for within the Vagrantfile through the provisioner capability. For my initial use case, a simple bash script provisioner was used. I look forward to learning more about Puppet/Chef in the near future as an alternative to simple script provisioners, but for now I’m keeping it pretty simple.

Add the following line to configure the shell script that will be executed to provision our IBM Log Analytics testing environment. The script is described in more detail in the next section.

config.vm.provision :shell, :path => "vagrant_scla_provision.sh"

Vagrant Provisioning Script

Let me paint a picture here of the value I see from using Vagrant during the open beta program as well as within the application developer or support engineer’s toolbox for supporting individual business applications.

Over the course of the IBM Log Analytics open beta program, multiple milestone drivers will be offered. Each driver will require some level of installation and configuration before it can be used, tested and evaluated in your environment. At this time, there isn’t an export/import/migrate capability so you’ll be running through the same install and configure steps manually for each driver you test. Using Vagrant and provisioning scripts this process can be radically simplified and automated allowing you to quickly get to value expected from using the software and not spending hours or days trying to get a system built, old software uninstalled, new software installed, configured and integrated. Removing barriers to participation is the goal here!

Beyond the open beta program, using Vagrant with the IBM Log Analytics Workgroup Edition as an application developer or support engineer you can quickly stand up and provision an instance of IBM Log Analytics for business application #1 and spin up another for business application #2 in a matter of minutes. Each box would have the necessary software installed, configured and integrated to consume each business application’s logs automatically, with little to no action beyond executing the ‘vagrant up’ command. Now that’s POWERFUL!

I wrote a very simple bash script that mimics the workflow that I took to manually install the IBM Log Analytics milestone driver #1 on this CentOS 5.8 base box. It’s rough and dirty, but functionally works. The following are the key steps taken in the script. If anyone has suggestions for improving this script shoot them my way! The script is available on my git repo here.

• Capture username and password for installation user
• Capture ILA driver file name
• Turn off SELINUX within the selected base box (it’s on in the CentOS box I used)
• Modify /etc/redhat-release to spoof installation manager
• Install a compatibility library for the ITM LFA
• Create installation user and group
• Create installation directories
• Copy ILA driver to installation directory and explode tarball
• Change ownership of all directories and files to the install user:group
• Install the ILA driver using silent install method
• Install the sample scenario data

ILA Driver Silent Installation File

Included in the IBM Log Analytics milestone driver package is an example silent install response file (sample_smcl_silent_install.xml). I’ve completed the configuration of this file by following the docs here. This will allow us to install the IBM Log Analytics package in a completely automated manner using our Vagrant provisioning script.

The silent install response file I used is available on my git repo here.

Copy Files to Vagrant Project Directory

Before we spin up the box all of the files we will use for provisioning must be placed in our project’s root directory. This is the same directory where you issued the ‘vagrant init’ command earlier.

Copy the following files into the Vagrant project directory:

• ILA driver tarball (e.g. IBM_LogAnalyticsWE_Beta_20130321_1359.tar.gz)
• Replacement selinux file
• Replacement redhat-release file
• vagrant_scla_provision.sh file
• ILA driver silent install file (e.g. vagrant_smcl_silent_install.xml)

Initialize and Provision Box

We’re now ready to spin up and provision our initial version of IBM Log Analytics and begin testing the first driver for our open beta! From your Vagrant project directory, simply issue the ‘vagrant up’ command. You may want to make sure you have enough buffer in your terminal window so you can scroll back and see all of the things that took place during this process.

Issue ‘vagrant up’ now!

I recorded my terminal window so you can watch the entire sequence. It took about 15 minutes to complete on my system (I edited out the longer pauses of the build process). With a nice beefy box and ample CPU/MEM on the base box image I’m sure it would be quicker.

Verbose Terminal Window of Vagrant Spin Up and Provisioning

Browser Support

The current supported browser at this time is the Firefox 17 ESR version. You may have your preference of browser by choice or by corporate policy but one trick I’m a huge fan of is to make use of Portable versions of the Firefox browser. It seems that I need to have a number of versions of Firefox to achieve the best user experience and performance with various products I use frequently. Portable apps are versions of common applications that can run independently from a standalone folder instead of actually being installed in your system. I actually run up to three versions of Firefox at the same time for some of my testing work.

To download the supported version of Firefox for this driver, visit this link to find Firefox 17.0.x ESR.

Accessing IBM Log Analytics

Fire up your browser and point it to the address you configured for your box. I’ve left this to the default network so I just point my browser to http://127.0.0.1:2200/Unity and I’m presented with the login page. Note the use of 2200 as the local port which is forwarded to 9988 to reach the main IBM Log Analytics webserver port. If you’ve set your forwarding up for 9988 -> 9988 then point your browser to http://127.0.0.1:9988/Unity to connect. This should redirect you to http://127.0.0.1:9988/Unity/jsp/login.jsp to actually log in.

The default passwords are unity/unity or unityadmin/unityadmin.

Here are a few example screenshots:

Login Page

Upon successful login, I’m able to immediately begin to use IBM Log Analytics because I’ve automatically loaded the sample scenario data. I’m able to immediately begin executing searches (saved or freeform) across the sample data. Here are a few example screenshots.

Saved Search

Directed Search Using Configured Patterns

Administration Page

Vagrant Box Teardown

Now that we’ve build and provisioned a fully functioning instance of IBM Log Analytics we have a few options on what we can do at the end of the day. Vagrant offers three options on what to do with the box and the work done during provisioning and customizations you may have made afterwards.

The Vagrant documentation explains these options perfectly below.

• Suspending the virtual machine by calling vagrant suspend will save the current running state of the machine and stop it. When you’re ready to begin working again, just run vagrant up, and it will be resumed from where you left off. The main benefit of this method is that it is super fast, usually taking only 5 to 10 seconds to stop and start your work. The downside is that the virtual machine still eats up your disk space, and requires even more disk space to store all the state of the virtual machine RAM on disk.
• Halting the virtual machine will gracefully shut down the guest operating system and power down the guest machine. You can use vagrant up when you’re ready to boot it again. The benefit of this method is that it will cleanly shut down your machine, preserving the contents of disk, and allowing it to be cleanly started again. The downside is that it’ll take some extra time to start from a cold boot, and the guest machine still consumes disk space.
• Destroying the virtual machine will remove all traces of the guest machine from your system. It’ll stop the guest machine, power it down, and remove all of the guest hard disks. Again, when you’re ready to work again, just issue a vagrant up. The benefit of this is that no cruft is left on your machine. The disk space and RAM consumed by the guest machine is reclaimed and your host machine is left clean. The downside is that vagrant up to get working again will take some extra time since it has to reimport the machine and reprovision it.

Starting or Resuming a Previously Provisioned Box

If you’ve suspended or halted your Vagrant box, you’ll want to be sure to start it back up using an additional flag to avoid having the provisioning script run again. To prevent re-provisioning a box after you’ve done it once, use the –no-provision flag

vagrant up --no-provision

Summary

One of the goals for the open beta program is to make IBM Log Analytics as accessible and simple to use as possible for the targeted application developer, application support engineer or middleware subject matter expert. Using Vagrant, VirtualBox and automated build and provisioning capabilities can help remove all barriers from participating in the IBM Log Analytics open beta program.

Over the next few months I’ll write more about how Vagrant can be used to build out an IBM Log Analytics architecture that can be used in your application development, test and production environments. I’ll expand on the topic by introducing additional log shipping, collection, consolidation and analytics products, integrations, automations using emerging open source technologies alongside commercial software such as IBM Log Analytics.

* Updated 4/8/13 after making changes to my Git repo, URLs now point to a folder versus individual files.