EventID 28001 and 29112 on SCOM 2012 Management Server after Operational Database Move

Recently I’ve moved databases for my lab SCOM 2012 management group to a new SQL 2012 server as part of of my RTM to SP1 migration.

I followed the How to Move the Operational Database guide from TechNet. After the migration, I have noticed that on one of my management servers, I kept getting a warning event 28001 and an error event 29112 every couple of minutes in the OperationsManager event log.

Event 28001:

image

The Root connector received an exception from the Config Service on StateSyncRequest:

System.Runtime.Remoting.RemotingException: Failed to connect to an IPC Port: The system cannot find the file specified.

Server stack trace:
at System.Runtime.Remoting.Channels.Ipc.IpcPort.Connect(String portName, Boolean secure, TokenImpersonationLevel impersonationLevel, Int32 timeout)
at System.Runtime.Remoting.Channels.Ipc.ConnectionCache.GetConnection(String portName, Boolean secure, TokenImpersonationLevel level, Int32 timeout)
at System.Runtime.Remoting.Channels.Ipc.IpcClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)
at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Microsoft.EnterpriseManagement.Mom.Internal.IConfigService.OnStateSyncRequest(Guid source, UInt64 messageIdentifier, String cookie)

Event 29112:

image

OpsMgr Management Configuration Service failed to execute bootstrap work item ‘ConfigurationStoreInitializeWorkItem’ due to the following exception

System.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 – Could not open a connection to SQL Server)
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
at System.Data.SqlClient.TdsParser.Connect(ServerInfo serverInfo, SqlInternalConnectionTds connHandler, Boolean ignoreSniOpenTimeout, Int64 timerExpire, Boolean encrypt, Boolean trustServerCert, Boolean integratedSecurity)
at System.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, SqlConnection owningObject)
at System.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, Boolean redirectedUserInstance, SqlConnection owningObject, SqlConnectionString connectionOptions, TimeoutTimer timeout)
at System.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(SqlConnection owningObject, TimeoutTimer timeout, SqlConnectionString connectionOptions, String newPassword, Boolean redirectedUserInstance)
at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, Object providerInfo, String newPassword, SqlConnection owningObject, Boolean redirectedUserInstance)
at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection)
at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnection owningConnection, DbConnectionPool pool, DbConnectionOptions options)
at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject)
at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject)
at System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection owningObject)
at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)
at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)
at System.Data.SqlClient.SqlConnection.Open()
at Microsoft.EnterpriseManagement.ManagementConfiguration.DataAccessLayer.ConnectionManagementOperation.Execute()
at Microsoft.EnterpriseManagement.ManagementConfiguration.DataAccessLayer.DataAccessOperation.ExecuteSynchronously(Int32 timeoutSeconds, WaitHandle stopWaitHandle)
at Microsoft.EnterpriseManagement.ManagementConfiguration.SqlConfigurationStore.ConfigurationStore.ExecuteOperationSynchronously(IDataAccessConnectedOperation operation, String operationName)
at Microsoft.EnterpriseManagement.ManagementConfiguration.SqlConfigurationStore.ConfigurationStore.Initialize()
at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.ConfigurationStoreInitializeWorkItem.ExecuteWorkItem()

This particular management server did not get rebuilt to Windows 2012, all other management servers I have in the MG has been completely rebuilt so they did not have this issue.

In the step 7 of the guide from TechNet, it mentioned updating the <Category Name=”Cmdb”> tag in the %ProgramFiles%\System Center 2012\Operations Manager\Server\ConfigService.config file.

image

However, the old DB server name also exists in the <Category Name=”ConfigStore”> tag in the same file. this was not mentioned in the guide:

image

After I updated <Category Name=”ConfigStore”> section and restarted all the SCOM services on the management server, the error went away.

5 comments

  1. i checked the config in my all MS servers and its identical. Still have the error message like below…

    OpsMgr Management Configuration Service failed to execute ‘LocalHealthServiceDirtyNotification’ engine work item due to the following exception

    System.Runtime.Remoting.RemotingException: Failed to connect to an IPC Port: Access is denied.

    Server stack trace:
    at System.Runtime.Remoting.Channels.Ipc.IpcPort.Connect(String portName, Boolean secure, TokenImpersonationLevel impersonationLevel, Int32 timeout)
    at System.Runtime.Remoting.Channels.Ipc.ConnectionCache.GetConnection(String portName, Boolean secure, TokenImpersonationLevel level, Int32 timeout)
    at System.Runtime.Remoting.Channels.Ipc.IpcClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)
    at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)

    Exception rethrown at [0]:
    at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
    at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
    at Microsoft.EnterpriseManagement.Mom.Internal.IRootConnectorForConfigService.OnDirtyStateNotification(Guid target, TimeSpan messageExpiration, UInt64 messageIdentifier)
    at Microsoft.EnterpriseManagement.ManagementConfiguration.Communication.AgentCommunicationProvider.NotifyDirtyStateUnwrapped(Dictionary2 dirtyAgentDictionary)
    at Microsoft.EnterpriseManagement.ManagementConfiguration.Communication.AgentCommunicationProvider.NotifyDirtyState(Dictionary
    2 dirtyAgentDictionary)
    at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.TracingAgentCommunicationProvider.NotifyDirtyState(Dictionary`2 dirtyAgentDictionary)
    at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.LocalDirtyNotificationWorkItem.c__DisplayClass1.b__0()
    at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.HealthServiceLoggingCommunication.CallHealthService(HealthServiceCall healthServiceCall)
    at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.LocalDirtyNotificationWorkItem.ExecuteWorkItem()
    at Microsoft.EnterpriseManagement.ManagementConfiguration.Interop.ConfigServiceEngineWorkItem.Execute()

  2. Nice! This solution also worked for me. Be aware as I failed to do: if you are using a named instance of SQL, you should put the entire name, ie: “scdb01\scom”. I forgot to put the instance and that’s why I was having the issue. Also, thanks to this post, I changed the name on both sections of the config file.

  3. This was my problem, did a default install and didn’t create an instance on my SQL server. But when I installed I specified an instance thinking wow the install will take care of that. well it didn’t and was trying to connect to an instance that didnt exist. I went into the file and removed the \instance name that I had and just left the server name and works like a boss now !

Leave a Reply

%d bloggers like this: