Duplicate OpsMgr Alert Notifications Caused by Service Manager’s OpsMgr Alert Connector And Ways Around It.
Few days ago, I posted an article: OpsMgr Alerts Push Notification to Android Devices. In my lab, I have created a subscription to notify me all new critical alerts using this channel:
In the first few days, I wasn’t paying too much attention when every time I get spammed by these push notifications on my phone, 2 days ago, I realised every new critical alerts gets pushed to my phone twice and two notifications are 1-2 minutes apart:
After some troubleshooting, I found the Service Manager’s OpsMgr Alert Connector is causing this issue. Well, I wouldn’t say it’s a fault or bug, it’s just the way that OpsMgr alert notifications handles alert.
In my lab, I have configured an OpsMgr Alert Connector in Service Manager. As a result, Service Manager updates the OpsMgr alert with a Ticket ID after the alert is generated:
And in alert history tab, you can see Service Manager updated the alert soon after it was initially raised:
Looks like this issue has been identified for a long time. some one has already posted a thread in TechNet forum back in 2009: http://social.technet.microsoft.com/Forums/en-US/operationsmanagergeneral/thread/4d2f7f8f-c616-483e-a5af-8c77f5d20953/
OpsMgr alerts get processed by notification subscriptions every time they are updated. i.e. If I take a critical alert with resolution state of “New” and set the resolution state to “New” again in the OpsMgr console, it will get processed again by subscriptions:
Therefore I wouldn’t blame service manager for this issue.
I can think of 3 possible solutions:
1. Put a 3 minutes delay in the OpsMgr alert subscription since Service Manager alert connector is configured to check OpsMgr every 60 seconds.
2. add another criteria to the alert subscription: with a “%” ticket ID – which means ticket ID must match a wildcard (%).
3. Modify the criteria to notify any new critical alert WITHOUT a ticket ID.
Each option has it’s pros an cons.
Option 1 would wait Service Manager 3 minutes so it can process the alert first. However, if the alert is generated by a monitor and the monitor has a recovery task, the alert is probably already closed after 3 minutes and you will not get notified. this may not be the desired result.
Option 2 would only pick up alerts that have already been processed by Service Manager. This adds a unnecessary dependency. If for some reasons Service Manager is down, Ticket ID field in the OpsMgr alert will not get updated, therefore alert subscription will not pick up the alert.
To me, Option 3 makes most of the sense. The subscription should only pick up the “brand new” new critical alerts, before Service Manager updates the Ticket ID. – But, it’s not that easy to configure. Why? because I cannot specify conditions like “and with a NULL ticket ID” via the OpsMgr Console. As you can see below, I can only specify a wildcard match to ticket ID:
Luckily I can do this outside of the console, by directly modifying the OpsMgr “Notification Internal Library” MP. Kevin Holman has an excellent blog post on this topic: Creating granular alert notifications – rule by rule, monitor by monitor.
So I’ve decided that I’m going to go for option 3. I’ll now go through the steps I took to modify the existing alert subscription.
1. Export AND BACKUP the “Notification Internal Library” MP from the OpsMgr console (Microsoft.SystemCenter.Notifications.Internal.xml)
2. Open Microsoft.SystemCenter.Notifications.Internal.xml with a text editor (i.e. Notepad++).
3. Towards the end of the file, in the <LanguagePack> section, find the <DisplayString> associated to the subscription. Make sure the Name in <Name> tag matches the subscription name in the OpsMgr console, and ElementID starts with “subscription”
4. Change the name in <Name> tag, add “-Do Not Change In UI” at the end (As shown above). this is to remind anyone not to change this subscription in the Operations console after it’s imported back.
5. Find the Rule with the same ID as the ElementID for the <DisplayString> taken from step 3. The rule should be under <Monitoring><Rules> tags.
6. Modify the DataSource module of this rule, Add below expression in <Expressions><And> tag:
7. Save the changes and import the MP back to OpsMgr management group.
8. Wait few minutes for the new configuration to become active. Once it’s active, you will no longer see the criteria in operations console:
Now, when I test this notification by creating a critical alert (using a test alert generating rule I’ve configured previously), I only get 1 notification pushed out to my phone (and it was pushed out before Service Manager updated it):
MP Schema Reference:
ExpressionType (GroupPopulationSchema): http://msdn.microsoft.com/en-us/library/ff472337.aspx
Note: I used UnaryExpression, which is only listed in GroupPopulation Schema ExpressionType. but it worked in this scenario.
Related Blog Articles:
Using Expressions and Wildcards to create groups, author rules and monitors, create console views and notification subscriptions, and in the Command Shell: http://blogs.technet.com/b/jonathanalmquist/archive/2010/10/13/regular-expression-syntax-in-scom-for-filtering-groups-monitor-elements-operational-views-notification-subscriptions-etc.aspx
Creating granular alert notifications – rule by rule, monitor by monitor: http://blogs.technet.com/b/kevinholman/archive/2008/10/12/creating-granular-alert-notifications-rule-by-rule-monitor-by-monitor.aspx