Using PowerShell Remote Sessions in a large SCOM environment

Written by Tao Yang

I have previously blogged using SCOM PowerShell snap-in and SDK client in a PowerShell remote session to avoid maintain a consistent SDK connection to RMS server.

I just found out there might be a potential issue when using this technique to run scripts against a reasonably large SCOM management group.

By default, the maximum amount of memory a PowerShell remote session can use is limited to 150MB. You can check this setting on your computer by using

winrm get winrm/config

image

Depending on how much data the remote session has to process, you might running into problems.

For example, compare these 2 scripts:

Sample #1:

$me = [System.Security.Principal.WindowsIdentity]::GetCurrent().Name

$RunAsCred = get-Credential ""

$RMS = ""

$AlertID = "8a5e738c-53d5-4a04-91af-cb8bc2d2e5d3"

$NewSession = new-pssession -ComputerName $env:COMPUTERNAME -Authentication Credssp -Credential (Get-Credential $me)

$alert = invoke-command  -session $NewSession -ScriptBlock {

param($RMS,$AlertID,$RunAsCred)

Add-PSSnapin Microsoft.EnterpriseManagement.OperationsManager.Client

New-PSDrive -Name:Monitoring -PSProvider:OperationsManagerMonitoring -Root:\

Set-Location "OperationsManagerMonitoring::"

new-managementGroupConnection -ConnectionString:$RMS -credential $RunAsCred | Out-Null

Set-Location $RMS

$Alert = Get-Alert | Where-Object {$_.Id -imatch $AlertID}

$Alert

} -ArgumentList $RMS, $AlertID, $RunAsCred

Remove-PSSession $NewSession

$alert

Sample #2:

$me = [System.Security.Principal.WindowsIdentity]::GetCurrent().Name

$RunAsCred = get-Credential ""

$RMS =  ""

$AlertID = "8a5e738c-53d5-4a04-91af-cb8bc2d2e5d3"

$NewSession = new-pssession -ComputerName $env:COMPUTERNAME -Authentication Credssp -Credential (Get-Credential $me)

$alert = invoke-command  -session $NewSession -ScriptBlock {

param($RMS,$AlertID,$RunAsCred)

Add-PSSnapin Microsoft.EnterpriseManagement.OperationsManager.Client

New-PSDrive -Name:Monitoring -PSProvider:OperationsManagerMonitoring -Root:\

Set-Location "OperationsManagerMonitoring::"

new-managementGroupConnection -ConnectionString:$RMS -credential $RunAsCred | Out-Null

Set-Location $RMS

$Alert = Get-Alert -Id $AlertID

$Alert

} -ArgumentList $RMS, $AlertID, $RunAsCred

Remove-PSSession $NewSession

$alert

The difference between these 2 scripts is, when using Get-Alert cmdlet,  #1 uses client side filtering and #2 uses server side filtering:

#1: $Alert = Get-Alert | Where-Object {$_.Id -imatch $AlertID}

#2: $Alert = Get-Alert -Id $AlertID

The management group my scripts connect to has just over 22,000 alerts still in the operational database. So when script #1 runs, because it uses client side filtering, it retrieves all 22,000+ alerts to the local remote session, Then filter out all the rest and retrieves ONE alert with that particular Alert ID. On the other hand, script #2 uses server side filtering, only ONE alert is returned from SCOM RMS server.

Here’s what happened when I tried to run both scripts:

#1:

image

#2:

image

As you can see, script #2 worked as expected but script #1 throws an exception:

Processing data for a remote command failed with the following error message: The WSMan provider host process did not return a proper response.  A provider in the host process may have behaved improperly. For more information, see the about_Remote_Troubleshooting Help topic.

    + CategoryInfo          : OperationStopped: (System.Manageme…pressionSyncJob:PSInvokeExpressionSyncJob) [], PSRemotingTransportException

    + FullyQualifiedErrorId : JobFailure

So now we know that we should use server side filtering because it is more efficient and there are less data returned.  However, not all SCOM cmdlets supports server side filtering. this statement is also true when talking about PowerShell cmdlets in general. not all cmdlets supports server side filtering. Sometimes, I have to use client side filering. For example, when working with Get-agent cmdlet, I have to use client side filtering.

Below script is very similar to script #1, I’ve changed a little bit to retrieve information of a particular SCOM agent:

$me = [System.Security.Principal.WindowsIdentity]::GetCurrent().Name

$RunAsCred = get-Credential ""

$RMS = ""

$AgentName = ""

$NewSession = new-pssession -ComputerName $env:COMPUTERNAME -Authentication Credssp -Credential (Get-Credential $me)

$agent = invoke-command  -session $NewSession -ScriptBlock {

param($RMS,$AgentName,$RunAsCred)

Add-PSSnapin Microsoft.EnterpriseManagement.OperationsManager.Client

New-PSDrive -Name:Monitoring -PSProvider:OperationsManagerMonitoring -Root:\

Set-Location "OperationsManagerMonitoring::"

new-managementGroupConnection -ConnectionString:$RMS -credential $RunAsCred | Out-Null

Set-Location $RMS

$Agent = Get-Agent | Where-Object {$_.PrincipalName -imatch $AgentName}

$Agent

} -ArgumentList $RMS, $AgentName, $RunAsCred

Remove-PSSession $NewSession

$agent

The management group I have connected to has around 5000 agents at the moment, so it’s a reasonable size. Below is what happened:

image

PowerShell threw an exception:

Processing data for a remote command failed with the following error message: <f:WSManFault xmlns:f=”http://schemas.microsoft.com/wbem/wsman/1/wsmanfault” Code=”3762507597″ Machine=”computername”><f:Message><f:ProviderFault provider=”microsoft.powershell” path=”C:\Windows\system32\pwrshplugin.dll”></f:ProviderFault></f:Message></f:WSManFault> For more information, see the about_Remote_Troubleshooting Help topic.

    + CategoryInfo          : OperationStopped: (System.Manageme…pressionSyncJob:PSInvokeExpressionSyncJob) [], PSRemotingTransportException

    + FullyQualifiedErrorId : JobFailure

To make the script run, I had to increase the maximum allowed memory for PowerShell Remote session. In this case, I increased from 150MB to 512MB. I used this command to increase it:

winrm set winrm/config/winrs <code>@</code>{MaxMemoryPerShellMB=<code>&quot;512</code>&quot;`}

After the increase, my get-agent script ran successfully:

Conclusion:

When writing a script, we normally test it against a development or test environment. generally speaking, these environments are much much small than production environments. scripts may run perfectly in dev / test environments but because of the size of environments, we may run into situations like this. Depending on the size of the production environments, we will have to adjust the “MaxMemoryPerShellMB” setting for PowerShell remote sessions accordingly.

3 comments on “Using PowerShell Remote Sessions in a large SCOM environment

  1. Pingback: My First Impression on PowerShell Web Access | Tao Yang's System Management Blog

  2. I want to be able to issue the $agent command from scripts, how can I leave the $NewSession running and send the $agent command to it from other powershell scripts?

    • Gavin,
      From my sample script in the blog post, the $agent object has been transferred from the remote session to local session before displaying on the screen.
      If you want to leave the $NewSession running, all you have to do is remove the “Remove-PSSession $NewSession” line. so you can enter the $NewSession later.

Leave a Reply