BizTalk 2013 R2 known bugs, issues & quirks

Below a list of know bugs, issues, & other quirks* of BizTalk 2013 R2 (mainly CU3) and Visual Studio 2013 (update 4) , but I will update with fixes as the CUs come out or when I know about a new issue, so if you have another one I will add it (and link to it if you have a blog about it) if you let me know.  This in addition to the ones listed by Microsoft on Known Issues in Installation, Configuration, and Deployment

I’ve now posted the below article on a TechNet wiki

* My definitions for classifying these are
Bug: Causes negative impact and no known work around or requires a lot of effort to work around.
Issue: Causes negative impact but has an easy or quick work around.
Missing: A feature that BizTalk doesn’t have that would make our life easier.
Quirk:  Something strange or unusual but does not cause a negative impact so far as known.

.Net 4.5 and 4.6, 4.6.1 end of life

Microsoft have announced that .Net 4.5, 4.6 and 4.6.1 are going to be end of life by April 26, 2022.

So as per my blog BizTalk supported Microsoft .Net Frameworks that is going to mean that BizTalk 2013 which can only run on .Net 4.5 that will no longer be getting patches. So if you haven’t been thinking about upgrading BizTalk, now is probably time to start thinking about that.

BizTalk 2013 R2 can run on .Net is 4.6, so make sure you have installed .Net 4.6.2

If you are running BizTalk 2016 with .Net 4.6 you will want to make sure you are at 4.6.2 or either go to .Net 4.7.2 (with at least CU2), or .Net 4.8 (with CU7 and above).

BizTalk 2020 will be running on 4.7.2, but I’m sure Microsoft will support .Net 4.8 for that soon as well.

As a side note, Microsoft is also retiring any contents signed with SHA-1 as per .NET Framework retiring SHA-1 content and UPDATE: SHA-1 signed content to be retired

The Messaging Engine failed to retrieve the configuration from the database. Details:”80131014″.

After changing a SAS key on the Azure Service Bus and updating the SAS credentials in the web.config for all the receive locations, I encountered some errors to do with one of the RelayEndpoint receive locations.

The Messaging Engine failed to retrieve the configuration from the database. Details:”80131014″.

And

WebHost failed to process a request.
Sender Information: System.ServiceModel.ServiceHostingEnvironment+HostingManager/52231697
Exception: System.ServiceModel.ServiceActivationException: The service ‘/Company.BT.DocMan.SharePoint/sync/Document/v1/Document.svc’ cannot be activated due to an exception during compilation. The exception message is: Receive location for address “/Company.BT.DocMan.SharePoint/sync/Document/v1/Document.svc” not found. (The BizTalk receive location may be disabled.). —> Microsoft.BizTalk.Adapter.Wcf.AdapterException: Receive location for address “/Company.BT.DocMan.SharePoint/sync/Document/v1/Document.svc” not found. (The BizTalk receive location may be disabled.)
at Microsoft.BizTalk.Adapter.Wcf.Runtime.ReceiveLocationManager2.GetEndpointContext(Uri uri) at Microsoft.BizTalk.Adapter.Wcf.Runtime.WebServiceHostFactory3.CreateServiceHost(String constructorString, Uri[] baseAddresses)
at System.ServiceModel.ServiceHostingEnvironment.HostingManager.CreateService(String normalizedVirtualPath, EventTraceActivity eventTraceActivity)
at System.ServiceModel.ServiceHostingEnvironment.HostingManager.ActivateService(ServiceActivationInfo serviceActivationInfo, EventTraceActivity eventTraceActivity)
at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath, EventTraceActivity eventTraceActivity)
— End of inner exception stack trace —
at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath, EventTraceActivity eventTraceActivity)
at System.ServiceModel.ServiceHostingEnvironment.EnsureServiceAvailableFast(String relativeVirtualPath, EventTraceActivity eventTraceActivity)
Process Name: w3wp
Process ID: 2496

Browsing to the receive location URL also resulted in the error that the receive location may be disabled.

The solution, recycle the App Pool that the web service was under in IIS.

BizTalk Server 2020 CU2 is available

As per the announcement by Microsoft BizTalk Server 2020 CU2 is available for download.

For the adapters, SFTP gets 4 fixes, the SB-Messaging adapters 3 fixes, the SAP adapter two fixes and one fix each for the WCF-SQL, Logic App adapters.

Also included is a fix to address configuring log shipping, and also a fix that prevented host instances and admin console not starting if the default web proxy is disabled, a fix for a receive location polling outside the service window and an Arithmetic overflow with convoy messages. Import/export of send handler will now include the configured send handler. And a fix for a null reference error when searching in Archived DTA DB.

Also three fixes for development either not surfacing errors or becoming non-responsive. As per the announcement you need to update your extension if you haven’t already done so.

As a quick reminder, for BizTalk Developer Tools please also update/install version 3.13.2.0 of the BizTalk Server Visual Studio extension in addition to installing the CU package. The extension can be installed from https://marketplace.visualstudio.com/items?itemName=ms-biztalk.BizTalk or from within Visual Studio – Manage Extensions. Please note that we have not updated the extension since CU1 so you can skip this step if you have already installed it previously.”

Microsoft BizTalk Server 2016 Feature Update 3 with CU8

Despite earlier indicating that they weren’t going to release updated Feature Updates with new CUs, Microsoft have actually released Microsoft BizTalk Server 2016 Feature Update 3 with CU8. So you can still get all those CU8 fixes if you have feature packs installed, as you should not install a CU only, if you have a feature pack installed, see BizTalk 2016 Feature Pack vs BizTalk 2016 CUs

Cumulative Update 8 for Microsoft BizTalk Server 2016

Note: Microsoft have release CU 8 for BizTalk 2016 Feature Pack CU 8 can be found here, you should not install a CU only, if you have a feature pack installed, see BizTalk 2016 Feature Pack vs BizTalk 2016 CUs

This CU contains fixes for following fixes for Adapters

• WCF-SQL, 2 fixes
  • The element orders in schemas generated and used in runtime by WCF-SQL adapter are random when large data exists in database
  • WCF-SQL adapter fails reading encrypted varchar(max) column from SQL
• SFTP, 3 fixes & 1 improvement
  • Using macro “%datetime.tz%” within TargetFileName in SFTP send port raises “NotSupportedException” error
  • SFTP receive location stops receiving files silently after running into file permission issues
  • SFTP receive location intermittently stops polling intermittently
  • Improvement: SFTP adapter improvements
    • WinSCP is upgraded to 5.17.8 or later versions.
    • The new “Error Threshold” property disables the receive location on error. The receive location is disabled when the error threshold is exceeded. If the error threshold is set to 0, then the receive location is not disabled.
    • The exceptions that occur in the SFTP receive location is written to the event log. This helps you troubleshoot any issues that affect the receive location.
• Dynamic port
  • Import/export send handler binding information for dynamic send ports
• SB-Messaging
  • When sending message to SB-Messaging send port, the binding property “DefaultScheduledEnqueueTimeUtc” causes an error
• Proxy settings, 2 fixes
  • Predefined WCF Bindings not taking in account a default Web proxy configuration in BizTalk Server
  • BizTalk Server runtime doesn’t use the authentication credentials configured in system web proxy settings

There the two remaining fixes are for BizTalk Server Setup and Configuration “Cannot insert duplicate key row in object ‘dbo.bts_LogShippingHistory’ with unique index” error while configuring log shipping destination

And BizTalk Server Accelerators Digest is not injected in the ACK in FileAct SnF PUSH mode when signature and non-repudiation are requested

MessageBox_DeadProcesses_Cleanup_BizTalkMsgBoxDb failing

I had a test environment where the SQL Agent job MessageBox_DeadProcesses_Cleanup_BizTalkMsgBoxDb had been failing for a while. This job runs exec bts_CleanupDeadProcesses in the message box DB. It was failing with the error

Executed as user: Domain\Account. Could not find stored procedure ‘dbo.int_ProcessCleanup_OLDHOSTNAME’. [SQLSTATE 42000] (Error 2812). The step failed.

Now the clue in this was that the stored procedure was trying to execute another stored procedure with a name of an a host that had been decommissioned, along with several others, after there had been some issues with them and replacement host and host instances had been created.

In that stored procedure it had this

DECLARE btsProcessCurse CURSOR FAST_FORWARD FOR
SELECT s.uidProcessID, s.nvcApplicationName FROM dbo.ProcessHeartbeats AS s WITH (ROWLOCK READPAST)
WHERE (s.dtNextHeartbeatTime < @dtCurrentTime) OPTION(KEEPFIXED PLAN)

So to see what that was getting I ran the below

declare @dtCurrentTime datetime

set @dtCurrentTime = GetUTCDate()

select *
FROM dbo.ProcessHeartbeats WITH (ROWLOCK READPAST)
WHERE (dtNextHeartbeatTime < @dtCurrentTime) OPTION(KEEPFIXED PLAN)

This gave a list of very old heartbeats for all the old host names. To resolve the issue I changed the select * into a delete, and ran that, and the SQL Agent Job is now running correctly.

BizTalk Server 2016 CU8 & future of Feature Packs

There is an an post asking for Feedback on the new Microsoft forums techcommunity that is talking about a CU8 for BizTalk 2016. However it indicates that they are considering releasing only for the base RTM and not for those with Feature Packs, and those that have feature packs are expected to upgrade to BizTalk 2020

Feature Updates to BizTalk Server 2016 provided early production ready access to BizTalk Server 2020 features. All those features are rolled up and now available with BizTalk Server 2020 release. We believe that Feature Updates to 2016 have served its purpose by providing early access to capabilities under work for 2020. 2020 release has further enhanced those features due to the freedom to make larger code changes in new release. BizTalk Server 2020 release has been available since the beginning of year and we released CU1 recently.

Feature Updates were available for select BizTalk Server customers strictly on an opt-in basis with an understanding that these customers will move to 2020 release soon after its availability. BizTalk Server Enterprise customers with Enterprise Agreement are eligible for free upgrade to BizTalk Server 2020. BizTalk Server 2020 supports in-place and seamless upgrade from BizTalk Server 2016.

We are in the planning process for CU8 release of BizTalk Server 2016 and were hoping, keeping above in mind, to continue releasing the CUs for the base 2016 (aka RTM) versions that caters to all SKUs and customers uniformly. Those who were on accelerated cadence by opting into Feature Packs, will move to BizTalk Server 2020 for their support requirements

We want to gather community response before finalizing our decision. Let us please use the discussion space under this post with constructive comments to help us arrive at a decision that helps us optimize engineering resources without compromising the customer needs.

https://techcommunity.microsoft.com/t5/biztalk-server/hotfixes-and-cus-for-biztalk-server-2016-feature-update/m-p/1569583/highlight/false#M54

Do you have an opinion about this? If so, now is your chance to provide feedback.

BizTalk 2020 CU1 released

BizTalk 2020 CU1 was released on July 28th 2020 as announced here

As per previous version of BizTalk it has a number of SAP/IDOC fixes.

Fixes

• 4 SAP/IDOX
• 2 fixes related to the WCF-SQL adapter (order of elements and FILESTREAM operations)
• 1 for the Oracle DB adapter to handle Database artifacts with nested types in Oracle 18c and later versions
• Fix for predefined WCF Bindings not taking in account a default Web proxy configuration
• 2 JSON Encoder fixes (1 one which was fixed for BizTalk 2016 in CU7)
• Fix for XLTS errors not being reported in XSLT stylesheet
• Fix for EDI receive location with named SQL instance
• Fix for ESB Toolkit to use gMSA.
• Fix to remove dependency on VS 2019 version 16.3.9
• FIX: Mapper toolbox not working for WCF Workflow Service Application after upgrade to BizTalk 2020

Improvements

• FTP improvement (macros for Before Put and After Put)
• Dynamic send ports able to use Officer 365 Outlook adapter types.

Visual Studio Extension

In addition to installing CU package, also update/install new version of BizTalk Server Visual Studio extension (3.13.2.0). The extension can be installed from https://marketplace.visualstudio.com/items?itemName=ms-biztalk.BizTalk or within Visual Studio – Manage Extensions.

As per other CUs, you can find them on the page Service Pack and cumulative update list for BizTalk Server

Max worker threads for SFTP, MQSC or DB and in BizTalk

A about March or earlier, the latest rules for BizTalk Health Monitor started to show the following important warning.

BizTalk Health Monitor Important warning – The host instances of ‘ReceivingSFTP’ need more worker threads per cpu to run correctly SFTP Receive Locations. Increase so the “Maximum Worker Threads” property to 500 for these host instances and be sure they are dedicated for this SFTP Receive Locations

I had already been advising clients to run SFTP receive locations on a separate host instance due to issues where it would silently stop receiving requiring a host instance restart to resolve, possibly related to FIX: SFTP adapter receive locations stop working intermittently during higher load in BizTalk Server

BHM rule 637 – Suggest to Increase MaxWorkerThreads for SFTP, MQSC or DB2 Receive Locations

The rule itself also talks about MQSC and DB, but as I didn’t have any receive locations for those it was only warning about SFTP.

I’ve seen that warning even on a BizTalk 2010 environment that has a Custom SFTP adapter installed, possibly because it was also called simply SFTP. However it does not warn about third party SFTP adapters such as the nSoftware one which has a different name.

BizTalk Host Instance Settings – Maximum worker threads.

However, even after changing the Max worker threads to 500 and restarting the Host Instance and re-running BHM, the warning did not go away. Even moving the receive locations running on the nSoftware onto another host instance, and un-configuring the adapter on that host instance didn’t make any difference.

The BizTalk adapter’s default configuration is to poll every 5 seconds. This is rather excessive and should be reduced.

Recommendations for SFTP Receive Locations

1. Have a dedicated host for receive locations using SFTP
2. Increase the Maximum Worker threads setting to 500 on that host instance
3. Check how frequently you poll (the default is 5 seconds)
4. Put a schedule on to only poll during the periods you need.
5. Disable message body tracking if it is not needed.

No BizTalk Server 2020 Dev Azure VM

Back in January I wrote that the BizTalk 2020 Azure VMs were available in the Azure Marketplace, but noted that the Dev one was not yet available.

Available Azure VM

I asked Microsoft when this was going to be available, and I finally got a response.

We will welcome BizTalk community to own and publish such an image. I cannot think of anything specific here that requires it to be done by Microsoft.

So, who is up for the challenge of setting up and publishing to the Azure Market Place a BizTalk Server 2020 VM in Azure?

Update: Sandro Pereira is planning to do this