![\"\"](\"https:\/\/upload.wikimedia.org\/wikipedia\/en\/9\/95\/Crooked_house_dudley.jpg\")
When the sum of all the PagerDuty parts converge in a best practice configuration, PagerDuty\u2019s platform capabilities ensure people (responder, team lead, manager, exec, etc) receive notifications with the right context at the right time so the appropriate response can be taken.<\/p>\n\n\n\n
If the context conveyed via a PagerDuty service and incoming events is super generalized or named after a monitoring tool like “Nagios Service”, the ability to respond with the right urgency, understanding (context) and then take the appropriate action can be significantly impacted.<\/p>\n\n\n\n
- For example, if an on-call responder is paged at 3 AM for a problem with the \u201cNagios Service\u201d, what\u2019s the appropriate response?\u00a0Does\u00a0the\u00a0“SERVICE_DESC”\u00a0in\u00a0your\u00a0Nagios\u00a0Alerts\u00a0prompt\u00a0the\u00a0desired\u00a0response?<\/em><\/li>
- If the MTTA\/R is increasing for the \u201cNagios Service\u201d, what is the root cause? Is it due to a single server or systemic problem across all the things, certain teams?<\/em><\/li>
- There can only be one Escalation Policy (EP) for that \u201cMonitoring Service\u201d. This means all events from Nagios into your “Nagios Service” go to the same responder(s) or schedule(s)!\u00a0If\u00a0you\u00a0own\u00a0it\u00a0all,\u00a0great,\u00a0but\u00a0chances\u00a0are\u00a0you’ve\u00a0got\u00a0many\u00a0responsible\u00a0groups\u00a0to\u00a0deal\u00a0with.<\/em><\/li>
- There can only be one automated<\/strong> Response Play for that \u201cMonitoring Service\u201d. Mature operations teams seek to automate operational response where seconds count using automated responses with very specific Response Plays for applications, functional technology types, specific teams or responders.\u00a0 This isn\u2019t possible due to the limitations of a single automated Response Play for your \u201cMonitoring Service\u201d.\u00a0Don’t hit the big red panic button for 60% disk full events! (* Multiple\u00a0Response\u00a0Plays\u00a0can\u00a0be configured\u00a0and launched\u00a0manually\u00a0via\u00a0the\u00a0Incident UI or Mobile App.)<\/em><\/li>
- Responder Notification (Urgency) is broadly applied (High Urgency by default – aka “Wake You Up at 3 AM Setting”) to everything that may be coming in rather than specifically applied based on the required response.\u00a0Maybe you want to use PagerDuty’s Dynamic Notifications on that “Monitoring Service” but do\u00a0you\u00a0‘trust’\u00a0that\u00a0incoming\u00a0events\u00a0have\u00a0a\u00a0severity\u00a0that\u00a0accurately\u00a0maps\u00a0to\u00a0the\u00a0needed\u00a0urgency\u00a0of\u00a0an on-call responder’s response?\u00a0I’ll\u00a0bet\u00a0that\u00a0you’re\u00a0probably\u00a0sending\u00a0in\u00a0everything\u00a0as\u00a0‘CRITICAL’ anyway.\u00a0If\u00a01\u00a0of\u00a020\u00a0servers\u00a0in\u00a0your\u00a0web\u00a0tier\u00a0has\u00a0a\u00a0‘CRITICAL’\u00a0disk\u00a0failure\u00a0–\u00a0does\u00a0that\u00a0warrant\u00a0a\u00a0high\u00a0urgency\u00a0page\u00a0at\u00a03\u00a0AM?<\/em><\/li><\/ul>\n\n\n\n
Sometimes Things [ Are | Are NOT ] Better Together!<\/em><\/strong><\/p>\n\n\n\n
![\"\"](\"https:\/\/data.whicdn.com\/images\/251841015\/superthumb.jpg?t=1469721452\")
mmmm bacon…<\/figcaption><\/figure><\/div>\n\n\n\n One of our better practices is to use Alert Grouping as a means of controlling the noise from poorly configured thresholds or alerting logic, or just plain old \u201cSHTF\u201d situations and alert storms that happen in any ops environment. Without the use of something like PagerDuty’s Time-Based Alert Grouping (TBAG), every single incoming event results in a unique incident, which sends notifications to the on-call responder(s), rinse and repeat for every…single…event.<\/p>\n\n\n\n
The situation that many customers fear when talking about “smart stuff” that is supposed to do whiz-bang grouping, correlating or other AI-ML-EIEIO magic is that the wrong alerts are grouped and things get missed. <\/p>\n\n\n\n
PagerDuty TBAG is a hard, time-based approach to group things so if a Network Link 5% Packet Loss event (BFD!) happens in the same time window as a MySQL Process Failure event (Oh, shit!), those things likely don\u2019t relate yet they are grouped together. The first event’s description becomes the incident\u2019s description and someone is paged for the Network Link 5% Packet Loss item and the on-call responder dismissed that incident b\/c their quick scan of the incident in the mobile app doesn\u2019t prompt closer investigation or an urgent response. All the while, the critical business impacting MySQL Process Failure alert is unnoticed as it\u2019s grouped in with the Network Link 5% Packet Loss incident. See why the concern? Not a fun discussion with the boss…<\/p>\n\n\n\n
PagerDuty’s Intelligent Alert Grouping (IAG) \u2018learns\u2019 based upon historical TBAG grouping and responders manually merging alerts into incidents. If IAG makes sense in your future (and it will unless responders are dedicated to doing this manual correlation and merging, it can be challenging to do this within the PD Alert UI), you won’t want to influence what IAG may do with bogus groups that could happen with broad based \u201cMonitoring Services\u201d.<\/p>\n\n\n\n
Net net here is you probably don’t want to use alert grouping on big, broad based “Monitoring Services” for fear that things are grouped incorrectly and something uber important is missed. <\/p>\n\n\n\n
The Journey along the \u201cSignal to Insight to Action\u201d Path Leads to a Peaceful On-Call Experience!<\/em><\/strong><\/p>\n\n\n\n
![\"\"](\"https:\/\/cdn-images-1.medium.com\/max\/714\/1*OrfT4OOCuXzxuraUOKRFUg.jpeg\")
You can get there…<\/figcaption><\/figure><\/div>\n\n\n\n All PagerDuty customers are entitled to use Global Event Routing and certain Global Event Rules to process and route incoming events to the appropriate service. If you’re following the default Nagios – PagerDuty integration guide and directly integrating with the service, you’re
bypas<\/g>sing this very powerful feature.<\/p>\n\n\n\n Building upon this basic Global Event Routing capability is the broader Event Intelligence offering and its own associated Global Event Rules providing a growing toolbox of capabilities to deal with the operational realities of your environment. <\/p>\n\n\n\n
When deployed properly you’ll efficiently move from signal to insight to action by ensuring the right events land on the right services at the right time so the right responder\/team have the right context to take the right action.<\/em><\/strong> Whew, that’s a mouthful – but that’s the real goal here right? If you could avoid waking up Fred, Sally and Shika at 3 AM with non-actionable, low urgency events, WHY WOULDN’T YOU WANT TO DO THAT?<\/p>\n\n\n\n
Any of this sound familiar<\/strong>
?<\/strong><\/g><\/p>\n\n\n\n - Alert fatigue from too much noise in your monitoring tools? No problem, there\u2019s a rule for dealing with that!<\/em><\/li>
- False positive alerts waking people up at 3 AM due to reoccurring maintenance windows? No problem, there\u2019s a rule for dealing with that!<\/em><\/li>
- Crappy alert metadata leading to missed issues or long MTTA\/R because on-call responders don\u2019t grok what the alert is trying to tell them or don\u2019t know what to do next? No problem, there\u2019s a rule for dealing with that!<\/em><\/li><\/ul>\n\n\n\n
Don\u2019t Let \u201cBusiness As Usual\u201d or “We’ve Always Done it This Way” Hold You Back!<\/em><\/strong><\/p>\n\n\n\n
![\"\"](\"https:\/\/powerupyourmarketing.com\/wp-content\/uploads\/2018\/05\/Negative_Patterns.jpg\")
Insanity?<\/figcaption><\/figure><\/div>\n\n\n\n Imagine a situation where Nagios is deployed and monitoring ALL of your infrastructure – dozens, hundreds maybe thousands of nodes, services, interfaces, URLs, etc. This would take considerable time and effort to move away from! (Worse, you probably have at least a dozen tools all set up similarly with PagerDuty!)<\/strong><\/p>\n\n\n\n
Imagine the sheer amount of manual work to move from your \u201cBusiness as Usual\u201d configuration of Nagios and PagerDuty “Monitoring Services” to something better – maybe you’re nervously thinking how you might unpack your “Nagios Service” – it may go something like this:<\/p>\n\n\n\n
- Discover and map out exactly what\u2019s being monitored by Nagios – “I think Nagios Ned can help me with that\u2026”<\/strong><\/em><\/li>
- Discover server, application, \u2018thing\u2019 owners – “ugh, I have to talk with that group\/person…”<\/strong><\/em><\/li>
- Discover context of what that ‘thing’ does, what it supports, what is impacted when problems found – “uh oh, I\u2019m feeling really uncomfortable\u2026”<\/strong><\/em><\/li>
- Discover what the appropriate operational response needs to be for all event classes\/types and who\u2019s responsible – “more meetings\u2026fml…”<\/strong><\/em><\/li>
- Translate all of the above to appropriate PagerDuty configurations following best practices – “a whole lot of point+click coming my way\u2026”<\/strong><\/em><\/li>
- \u2026<\/em><\/li><\/ul>\n\n\n\n
There is a much better way and I have some \u2018magic pixie dust\u2019 that can help you optimize this!<\/em><\/p>\n\n\n\n
Where do we go from here?<\/em><\/strong><\/p>\n\n\n\n
The next few posts I’ve got in mind build out something like this:<\/p>\n\n\n\n
- Growing up from the Nagios – PagerDuty defaults – Crawling away from the default “Monitoring Service”<\/li>
- Introducing the Global Event Routing API – Walking in with your eyes wide open<\/li>
- Extending Nagios with Custom Attributes – Running with PagerDuty like a champ<\/li>
- Applying Event Intelligence to improve your Nagios + PagerDuty experience for on-call responders<\/li>
- Magic Pixie Dust – How PagerDuty can help you ADAPT to a better way of doing things in ops and on-call when using Nagios<\/li><\/ul>\n","protected":false},"excerpt":{"rendered":"
In the prior blog post, I walked through how following the PagerDuty – Nagios XI integration guides leads us to the creation of a \u201cMonitoring Service\u201d. At the end of that post, I mentioned I\u2019d talk about some of the reasons why running in this default configuration isn\u2019t best practice and how this impacts an […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1079,1082,1081,1084,1077,1078,1080],"tags":[931,929,1088,1087,1073,1072],"class_list":{"0":"post-7912","1":"post","2":"type-post","3":"status-publish","4":"format-standard","6":"category-event-intelligence","7":"category-event-routing","8":"category-event-rules","9":"category-integrations","10":"category-pagerduty","11":"category-pagerduty-best-practices","12":"category-service-design","13":"tag-best-practices","14":"tag-events","15":"tag-incident-response","16":"tag-nagios","17":"tag-nagios-xi","18":"tag-pagerduty"},"_links":{"self":[{"href":"https:\/\/dougmcclure.net\/blog\/wp-json\/wp\/v2\/posts\/7912","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dougmcclure.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dougmcclure.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dougmcclure.net\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dougmcclure.net\/blog\/wp-json\/wp\/v2\/comments?post=7912"}],"version-history":[{"count":5,"href":"https:\/\/dougmcclure.net\/blog\/wp-json\/wp\/v2\/posts\/7912\/revisions"}],"predecessor-version":[{"id":7936,"href":"https:\/\/dougmcclure.net\/blog\/wp-json\/wp\/v2\/posts\/7912\/revisions\/7936"}],"wp:attachment":[{"href":"https:\/\/dougmcclure.net\/blog\/wp-json\/wp\/v2\/media?parent=7912"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dougmcclure.net\/blog\/wp-json\/wp\/v2\/categories?post=7912"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dougmcclure.net\/blog\/wp-json\/wp\/v2\/tags?post=7912"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
- Discover server, application, \u2018thing\u2019 owners – “ugh, I have to talk with that group\/person…”<\/strong><\/em><\/li>
- False positive alerts waking people up at 3 AM due to reoccurring maintenance windows? No problem, there\u2019s a rule for dealing with that!<\/em><\/li>
- If the MTTA\/R is increasing for the \u201cNagios Service\u201d, what is the root cause? Is it due to a single server or systemic problem across all the things, certain teams?<\/em><\/li>