{"id":1691,"date":"2009-02-23T11:45:23","date_gmt":"2009-02-23T16:45:23","guid":{"rendered":"http:\/\/dougmcclure.net\/blog\/?p=1691"},"modified":"2009-02-23T11:45:23","modified_gmt":"2009-02-23T16:45:23","slug":"evaluating-a-bsm-solution-measuring-effectiveness","status":"publish","type":"post","link":"https:\/\/dougmcclure.net\/blog\/2009\/02\/evaluating-a-bsm-solution-measuring-effectiveness\/","title":{"rendered":"Evaluating a BSM Solution: Measuring Effectiveness"},"content":{"rendered":"

In my first post, we talked about what is wrong with current solutions<\/a> followed by a post of sharing my experience of making BSM happen (realizing\/implementing it)<\/a>. Then I side tracked for a post to share a really great research invention by folks at IBM and its relevance in BSM (Strategic Capability Network)<\/a>.<\/p>\n

In this post, I intend to share insights from my experience of evaluating BSM\/SQM for clients to gauge effectiveness, and performance of the solution . \u00a0I am sure most consultants on the ground might have encountered this situation when they were hired to evaluate someone else’s BSM solution and recommend changes to make it WORK!!<\/strong><\/p>\n

Measuring effectiveness of a BSM solution is not easily quantifiable as it involves multiple factors which are not just statistical but are also related to\u00a0organization structure, architectural implications, rational behind decisions, culture, process, usability analysis and ecosystem of the company. Guess what,\u00a0to do all the aforementioned –\u00a0 I was given\u00a04 weeks + 1 week for planning. The planning week\u00a0was\u00a0 the most challenging with debates on what factors\/indicators to include and which ones to leave out. Eventually the following were the priorities: measure the usability<\/strong>, effectiveness<\/strong>, completeness <\/strong>(coverage)\u00a0and accuracy.<\/strong><\/p>\n

After researching endlessly on how to accomplish this WE <\/strong>came to an agreement on using the following approach to measure holistic performance of the BSM solution:<\/p>\n

Performance = Complexity <\/strong>Process <\/strong><\/sup>* \u00a0Team\u00a0* \u00a0Tools\u00a0 [1]<\/sup><\/strong><\/p>\n

Let us take these terms one at a time, I have explained these factors with an real examples and the lesson I learnt from these incidents:<\/p>\n

>> Complexity<\/strong>:\u00a0 Does an executive really care about memory on server displayed on executive dashboards?\u00a0 Are the indicators really accurate and reliable? If yes, How much? These are some of the indicators which are measured very seldom.\u00a0 Complexity is also driven by context and environment we deal with; for this we measured utilization, ease of information accessibility for stakeholders, number of influenced decisions\/quarter, time to address issues (before vs. after) and some other subjective quantitative indicators.<\/strong><\/p>\n

Real incident:<\/em> While evaluating\u00a0 BSM built by this great Service Assurance team, we found the dashboards for a production support teams\u00a0(of\u00a0various\u00a0silos’) had fault management metrics which made no sense to the users. Of Course, no one used it!! <\/strong><\/em>Only change we did to make this dashboard a hit was changing the metric terms and status aggregation pattern(auto-population logic and SLA rules). In this case, accuracy and reliability really contributed to the complexity to\u00a0the users who were too skeptic about using an interface which did not even use the language they understood to check on the applications they supported.\u00a0 This change was not a big\u00a0development effort, it was only adapting to the environment and reducing the accidental complexity by streamlining the process of displaying domain driven language.\u00a0<\/p>\n

Lesson learnt:<\/em> Well defined processes will reduce the planned and accidental complexity; measure the effectiveness with the organizational awareness of how to use the solution.\u00a0\u00a0<\/p>\n

>> Team:\u00a0 <\/strong>How much information is easily accessible to the stakeholder?\u00a0Is every category of stakeholder considered in the solution? Does everyone think this “Dashboard” is of any value or Do they prefer some other medium to achieve the same objective? In all the above cases, we need to adapt to the environment and put forward a balanced approach.<\/p>\n

Real incident:<\/em> One\u00a0enviornment where I\u00a0was working on a solution Executives had\u00a0<\/strong>imposed Netcool for an Operations team which was used to custom built tools and situation\u00a0was that of a RIOT!! Users complained for months that Netcool did not show accurate information on device status which they used to get out of the old custom tool.\u00a0 Everyone in the Service Assurance team shooed them away \ud83d\ude42 After talking to them, I realized that they had a point. The old tool used to report after pinging the server but also when the server came up, it would check for sysuptime and if the report if the server was unavailable due to power outage or some other reason. Poor users did not know the logic or the details behind the homemade tool.<\/p>\n

Lesson learnt:<\/em>If they (users)\u00a0are using it, their is a valid reason, look for it!! Hammer will take you only so far. Balance personalization with layering and tiering the solution so that everyone gets what (information) they need, the way they need it, and when they need it<\/strong>. Most importantly, BSM is not about changing the organization 180 degrees, its about increasing productivity and reporting the information for making the best business decisions.<\/p>\n

>> Tools <\/strong>are not only critical to task accomplishment but are also related to the overall organization productivity.\u00a0 Caution: Imposition of tools is not BSM!! Personalization is the only way BSM can really be a successful offering.\u00a0In my experience, implementations where a team selects what suits them the best and communicates information upstream to the enterprise instance have been much more successful and used.\u00a0 Ample experiences are already out there for tools but the lesson <\/strong>that\u00a0I learnt out of it was that, we should not look for silver bullets when evaluating tools. It is best left to the users as to which tool they are comfortable with.<\/p>\n

>> And finally, Performance: <\/strong>Although some of my friends will argue that performance is not a holistic term; we took a objective approach rather than a subjective one to ensure that WE had statistics to back our results.\u00a0 This\u00a0helped us\u00a0immensely!!\u00a0<\/p>\n

All and all, evaluating a BSM was much more challenging than building it because of the merging\/conflicting visions and principles followed while original implementation of the solution. I think underscores the need for standards and guidelines for BSM solutions. (Remember: Only when X.733 was put in, we knew how to define events in a standardized way. ) I am not lobbying for enforcement (via standards)\u00a0but the Industry really needs at least some vendor neutral guidelines to retain the value, vision and\u00a0capabilities\u00a0 for Business Service Management Solution.<\/strong><\/p>\n

References:<\/p>\n

[1]\u00a0\u00a0Grady Booch<\/a> has used the definition of performance in his famous speech at 9th Annul Turings Lecture<\/a>\u00a0 :<\/p>\n","protected":false},"excerpt":{"rendered":"

In my first post, we talked about what is wrong with current solutions followed by a post of sharing my experience of making BSM happen (realizing\/implementing it). Then I side tracked for a post to share a really great research invention by folks at IBM and its relevance in BSM (Strategic Capability Network). In this […]<\/p>\n","protected":false},"author":10,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[116,424,46,47,1],"tags":[756,770,755,771,980],"class_list":{"0":"post-1691","1":"post","2":"type-post","3":"status-publish","4":"format-standard","6":"category-bsm","7":"category-guest-authors","8":"category-metrics","9":"category-metrics-catalog","10":"category-uncategorized","11":"tag-effectiveness-of-bsm","12":"tag-evaluate-bsm","13":"tag-evaluating-bsm","14":"tag-performance-of-bsm","15":"tag-usability"},"_links":{"self":[{"href":"https:\/\/dougmcclure.net\/blog\/wp-json\/wp\/v2\/posts\/1691","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dougmcclure.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dougmcclure.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dougmcclure.net\/blog\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/dougmcclure.net\/blog\/wp-json\/wp\/v2\/comments?post=1691"}],"version-history":[{"count":23,"href":"https:\/\/dougmcclure.net\/blog\/wp-json\/wp\/v2\/posts\/1691\/revisions"}],"predecessor-version":[{"id":1761,"href":"https:\/\/dougmcclure.net\/blog\/wp-json\/wp\/v2\/posts\/1691\/revisions\/1761"}],"wp:attachment":[{"href":"https:\/\/dougmcclure.net\/blog\/wp-json\/wp\/v2\/media?parent=1691"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dougmcclure.net\/blog\/wp-json\/wp\/v2\/categories?post=1691"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dougmcclure.net\/blog\/wp-json\/wp\/v2\/tags?post=1691"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}