We’ve encountered a strange issue, after what looked like connectivity connection error to the BizTalk databases which caused some of the host instances to restart, some of the 64 bit hosts did not restart and trying to start manually them failed instantly. Restarting a 64 bit host that appeared to be running state resulted in it failing as well. After a server reboot, all of the 64 bit hosts failed to start but the 32 bit ones were fine.
The issue was that the C:\Program Files (x86)\Microsoft BizTalk Server 2016\Bins64 was missing from the System Variable Path. Adding it back in resolved the issue
What we checked / tried
- Checked that no patches had been applied (not since March) and nothing new had been installed. The only odd things was that BizTalk was showing has having been installed on that day in Add / Remove programs.
- Tried creating a new 64 bit host and host instance (same error on start). Creating a 32 bit host & host instance works. Deleting a 64 bit host instance, changing the host to 32 bit and recreating the host instance also works.
- Checked that BTSNTSvc64.exe.config & BTSNTSvc.exe.config had not been changed or corrupted
- Check that both the 64 and 32 machine config had not changed.
- Rebooted the server
- Restarted MSDTC on the SQL Server.
- DTCP Ping showed no connection issues
The Escalation Microsoft Engineer used the following tools to attempt to diagnose the issue
In Process Monitor we could see that it was loading the 32 bit DLLs from the folder C:\Program Files (x86)\Microsoft BizTalk Server 2016\Bins32 but not the 64 bit ones.
Text from the above error
Starting instance of host * on server failed.
For help, click: http://go.microsoft.com/fwlink/?LinkId=47400&ProdName=Microsoft+BizTalk+Server+2016&ProdVer=3.12.774.0&EvtSrc=Microsoft.BizTalk.Administration.SnapIn.Properties.Errors&EvtID=HostInstance_UnmapServerHostInstanceFailed
Failed to start the BizTalk Host instance. Check the event log on the server “*” for more details.
Internal error: “The service did not respond to the start or control request in a timely fashion.” (WinMgmt)
at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo)
at System.Management.ManagementObject.InvokeMethod(String methodName, ManagementBaseObject inParameters, InvokeMethodOptions options)
at Microsoft.BizTalk.SnapIn.Framework.WmiProvider.Invoke(ManagementObject instance, String command)
at Microsoft.BizTalk.Administration.SnapIn.Nodes.HostInstanceNode.OnStart(Object sender, EventArgs e)