Duplicate OpsMgr Alert Notifications Caused by Service Manager’s OpsMgr Alert Connector And Ways Around It.

Background

Few days ago, I posted an article: OpsMgr Alerts Push Notification to Android Devices. In my lab, I have created a subscription to notify me all new critical alerts using this channel:

image

In the first few days, I wasn’t paying too much attention when every time I get spammed by these push notifications on my phone, 2 days ago, I realised every new critical alerts gets pushed to my phone twice and two notifications are 1-2 minutes apart:

image

Cause

After some troubleshooting, I found the Service Manager’s OpsMgr Alert Connector is causing this issue. Well, I wouldn’t say it’s a fault or bug, it’s just the way that OpsMgr alert notifications handles alert.

In my lab, I have configured an OpsMgr Alert Connector in Service Manager. As a result, Service Manager updates the OpsMgr alert with a Ticket ID after the alert is generated:

image

And in alert history tab, you can see Service Manager updated the alert soon after it was initially raised:

image

Looks like this issue has been identified for a long time. some one has already posted a thread in TechNet forum back in 2009: http://social.technet.microsoft.com/Forums/en-US/operationsmanagergeneral/thread/4d2f7f8f-c616-483e-a5af-8c77f5d20953/

OpsMgr alerts get processed by notification subscriptions every time they are updated. i.e. If I take a critical alert with resolution state of “New” and set the resolution state to “New” again in the OpsMgr console, it will get processed again by subscriptions:

image

Therefore I wouldn’t blame service manager for this issue.

Possible Solutions

I can think of 3 possible solutions:

1. Put a 3 minutes delay in the OpsMgr alert subscription since Service Manager alert connector is configured to check OpsMgr every 60 seconds.

image

2. add another criteria to the alert subscription: with a “%” ticket ID – which means ticket ID must match a wildcard (%).

image

3. Modify the criteria to notify any new critical alert WITHOUT a ticket ID.

Each option has it’s pros an cons.

Option 1 would wait Service Manager 3 minutes so it can process the alert first. However, if the alert is generated by a monitor and the monitor has a recovery task, the alert is probably already closed after 3 minutes and you will not get notified. this may not be the desired result.

Option 2 would only pick up alerts that have already been processed by Service Manager. This adds a unnecessary dependency. If for some reasons Service Manager is down, Ticket ID field in the OpsMgr alert will not get updated, therefore alert subscription will not pick up the alert.

To me, Option 3 makes most of the sense. The subscription should only pick up the “brand new” new critical alerts, before Service Manager updates the Ticket ID. – But, it’s not that easy to configure. Why? because I cannot specify conditions like “and with a NULL ticket ID” via the OpsMgr Console. As you can see below, I can only specify a wildcard match to ticket ID:

image

Luckily I can do this outside of the console, by directly modifying the OpsMgr “Notification Internal Library” MP. Kevin Holman has an excellent blog post on this topic: Creating granular alert notifications – rule by rule, monitor by monitor.

So I’ve decided that I’m going to go for option 3. I’ll now go through the steps I took to modify the existing alert subscription.

1. Export AND BACKUP the “Notification Internal Library” MP from the OpsMgr console (Microsoft.SystemCenter.Notifications.Internal.xml)

2. Open Microsoft.SystemCenter.Notifications.Internal.xml with a text editor (i.e. Notepad++).

3. Towards the end of the file, in the <LanguagePack> section, find the <DisplayString> associated to the subscription. Make sure the Name in <Name> tag matches the subscription name in the OpsMgr console, and ElementID starts with “subscription”

image

4. Change the name in <Name> tag, add “-Do Not Change In UI” at the end (As shown above). this is to remind anyone not to change this subscription in the Operations console after it’s imported back.

5. Find the Rule with the same ID as the ElementID for the <DisplayString> taken from step 3. The rule should be under <Monitoring><Rules> tags.

image

6. Modify the DataSource module of this rule, Add below expression in <Expressions><And> tag:

<Expression>
<UnaryExpression>
<ValueExpression>
<Property>TicketId</Property>
</ValueExpression>
<Operator>IsNull</Operator>
</UnaryExpression>
</Expression>

Before:

image

After:

image

7. Save the changes and import the MP back to OpsMgr management group.

8. Wait few minutes for the new configuration to become active. Once it’s active, you will no longer see the criteria in operations console:

image

Now, when I test this notification by creating a critical alert (using a test alert generating rule I’ve configured previously), I only get 1 notification pushed out to my phone (and it was pushed out before Service Manager updated it):

image

image

More Information

MP Schema Reference:

ExpressionType: http://msdn.microsoft.com/en-us/library/ee692979.aspx

ExpressionType (GroupPopulationSchema): http://msdn.microsoft.com/en-us/library/ff472337.aspx

Note: I used UnaryExpression, which is only listed in GroupPopulation Schema ExpressionType. but it worked in this scenario.

Related Blog Articles:

Using Expressions and Wildcards to create groups, author rules and monitors, create console views and notification subscriptions, and in the Command Shell: http://blogs.technet.com/b/jonathanalmquist/archive/2010/10/13/regular-expression-syntax-in-scom-for-filtering-groups-monitor-elements-operational-views-notification-subscriptions-etc.aspx

Creating granular alert notifications – rule by rule, monitor by monitor: http://blogs.technet.com/b/kevinholman/archive/2008/10/12/creating-granular-alert-notifications-rule-by-rule-monitor-by-monitor.aspx

2 comments

  1. Hey Tao,

    Did you try “last modified by a specific user”” I believe you can just check that off in the criteria and specify “System” as user.

    1. Hi Rob,

      Thanks for the advice. I didn’t think about it. It should work. My only concern is for those rule generated alerts that increase repeat count, you may get notified when the count increase. – but it might be something you want anyway.

      Thanks
      Tao

Leave a Reply

%d bloggers like this: