Tag Archives: MimboloveSCOM Migration

EventID 28001 and 29112 on SCOM 2012 Management Server after Operational Database Move

Written by Tao Yang

Recently I’ve moved databases for my lab SCOM 2012 management group to a new SQL 2012 server as part of of my RTM to SP1 migration.

I followed the How to Move the Operational Database guide from TechNet. After the migration, I have noticed that on one of my management servers, I kept getting a warning event 28001 and an error event 29112 every couple of minutes in the OperationsManager event log.

Event 28001:

image

The Root connector received an exception from the Config Service on StateSyncRequest:

System.Runtime.Remoting.RemotingException: Failed to connect to an IPC Port: The system cannot find the file specified.

Server stack trace:
at System.Runtime.Remoting.Channels.Ipc.IpcPort.Connect(String portName, Boolean secure, TokenImpersonationLevel impersonationLevel, Int32 timeout)
at System.Runtime.Remoting.Channels.Ipc.ConnectionCache.GetConnection(String portName, Boolean secure, TokenImpersonationLevel level, Int32 timeout)
at System.Runtime.Remoting.Channels.Ipc.IpcClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)
at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Microsoft.EnterpriseManagement.Mom.Internal.IConfigService.OnStateSyncRequest(Guid source, UInt64 messageIdentifier, String cookie)

Event 29112:

image

OpsMgr Management Configuration Service failed to execute bootstrap work item ‘ConfigurationStoreInitializeWorkItem’ due to the following exception

System.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 – Could not open a connection to SQL Server)
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
at System.Data.SqlClient.TdsParser.Connect(ServerInfo serverInfo, SqlInternalConnectionTds connHandler, Boolean ignoreSniOpenTimeout, Int64 timerExpire, Boolean encrypt, Boolean trustServerCert, Boolean integratedSecurity)
at System.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, SqlConnection owningObject)
at System.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, Boolean redirectedUserInstance, SqlConnection owningObject, SqlConnectionString connectionOptions, TimeoutTimer timeout)
at System.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(SqlConnection owningObject, TimeoutTimer timeout, SqlConnectionString connectionOptions, String newPassword, Boolean redirectedUserInstance)
at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, Object providerInfo, String newPassword, SqlConnection owningObject, Boolean redirectedUserInstance)
at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection)
at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnection owningConnection, DbConnectionPool pool, DbConnectionOptions options)
at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject)
at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject)
at System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection owningObject)
at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)
at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)
at System.Data.SqlClient.SqlConnection.Open()
at Microsoft.EnterpriseManagement.ManagementConfiguration.DataAccessLayer.ConnectionManagementOperation.Execute()
at Microsoft.EnterpriseManagement.ManagementConfiguration.DataAccessLayer.DataAccessOperation.ExecuteSynchronously(Int32 timeoutSeconds, WaitHandle stopWaitHandle)
at Microsoft.EnterpriseManagement.ManagementConfiguration.SqlConfigurationStore.ConfigurationStore.ExecuteOperationSynchronously(IDataAccessConnectedOperation operation, String operationName)
at Microsoft.EnterpriseManagement.ManagementConfiguration.SqlConfigurationStore.ConfigurationStore.Initialize()
at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.ConfigurationStoreInitializeWorkItem.ExecuteWorkItem()

This particular management server did not get rebuilt to Windows 2012, all other management servers I have in the MG has been completely rebuilt so they did not have this issue.

In the step 7 of the guide from TechNet, it mentioned updating the <Category Name=”Cmdb”> tag in the %ProgramFiles%\System Center 2012\Operations Manager\Server\ConfigService.config file.

image

However, the old DB server name also exists in the <Category Name=”ConfigStore”> tag in the same file. this was not mentioned in the guide:

image

After I updated <Category Name=”ConfigStore”> section and restarted all the SCOM services on the management server, the error went away.

Using SCOM to Automatically Stop MOM Agent Service

Written by Tao Yang

I’m currently working on a MOM 2005-to-SCOM 2007 migration project for a client. after months of work, we are finally ready to stop MOM service.

We created a new GPO to set MOM service (MOM agent) to “Disabled” and linked to the top of the domain. we also wanted to make sure all MOM service are actually STOPPED on domain member servers AS WELL AS standalone SCOM agents. Traditionally, I’d create a package in SMS/SCCM with a script that firstly detect if SCOM agent service (HealthService) is running, and secondly, if so, stop MOM service. However, this particular client I’m currently working for does not have SCCM infrastructure for their servers. I didn’t want to run a script to go out stop the MOM service on all servers because there are firewalls located in different segments of the network and running a script does not ENFORCE this setting…

To achieve my objective, I have created a basic service unit monitor in SCOM to detect the status of MOM service and created a recovery task for this monitor. The monitor generates an alert if MOM service is running and Recovery task will be executed to stop MOM service!

Below is what I’ve done:

1. Create a basic service monitor targeting Windows Operating System:

This monitor is targeting MOM service

If the service is NOT running, the health state is considered as healthy, otherwise, it is in warning state

We only subscribe to critical and warning alerts, I didn’t want get too much attention so I configured the alert severity to only Information:

Lastly, I created a diagnostic and a recovery task associated to this monitor.

The Diagnostic task is simply checking the state of MOM service:

And the recovery task stops MOM service:

So now the monitor is created. note from the first screen shot, I did not enable this monitor. I wanted to test it first! So I created an override to only enable it to a particular Windows 2003 server.

After few minutes, I could see the health state change in Health Explorer:

1 minute after the health state changed from Healthy to Warning, it changed back to Healthy – because the recovery task kicked in and stopped MOM service. Below are the results from Diagnostic and Recovery Tasks:

Diagnostic Task:

Recovery Task:

And now, the alert is automatically closed:

I logged on to the target server, checked MOM service, it is stopped:

Now, I can go ahead and enable this monitor!