How to fix Error 5838 on Windows Server 2025 / Windows 11

by

Troubleshooting Guide: SQL Server Error 5838 — Invalid Service Broker Connection Authentication Value

Why This Error Happens

This error occurs when the Service Broker endpoint receives a connection authentication value that does not match any of the supported authentication mechanisms (NTLM, Kerberos, or Certificate-based authentication). It is typically caused by a mismatch between the AUTHENTICATION clause defined on the Service Broker endpoint and the actual security context being used by the connecting instance, often introduced after a certificate rotation, domain trust change, or manual endpoint reconfiguration. Corrupted or expired certificates bound to the endpoint, as well as inconsistent Windows authentication protocols between Windows Server 2025 and Windows 11 clients, can also trigger this value mismatch.

Step-by-Step Resolution Guide

Step 1: Verify the Service Broker Endpoint Configuration

  • Open SQL Server Management Studio (SSMS) and connect to the affected instance.
  • Run SELECT * FROM sys.service_broker_endpoints; to inspect the current AUTHENTICATION and ENCRYPTION settings.
  • Confirm whether the endpoint is set to WINDOWS NEGOTIATE, WINDOWS KERBEROS, WINDOWS NTLM, or CERTIFICATE, and check if this matches the expected setting on the partner instance.

Step 2: Realign Authentication Settings Between Instances

  • On both the initiator and target instances, run the following to reset authentication to a consistent value:
  ALTER ENDPOINT [ServiceBrokerEndpoint]
  FOR SERVICE_BROKER (AUTHENTICATION = WINDOWS NEGOTIATE, ENCRYPTION = REQUIRED ALGORITHM AES);
  • Ensure both servers are using the same domain trust relationship if using Windows-based authentication.
  • If using certificates, verify they have not expired via SELECT name, expiry_date FROM sys.certificates;.

Step 3: Re-issue or Re-bind Certificates (If Using Certificate Authentication)

  • Navigate to the certificate store via certlm.msc on Windows Server 2025 to confirm the certificate used by Service Broker is present and valid.
  • Drop and recreate the certificate binding using:
  CREATE CERTIFICATE ServiceBrokerCert
  WITH SUBJECT = 'Service Broker Certificate';
  • Re-bind the new certificate to the endpoint and restart the SQL Server service via services.msc or net stop MSSQLSERVER && net start MSSQLSERVER.

Step 4: Restart the Service Broker and Clear Cached Sessions

  • Run ALTER DATABASE [YourDB] SET NEW_BROKER; only if conversations are non-critical, as this resets the Broker GUID.
  • Alternatively, use ALTER DATABASE [YourDB] SET ENABLE_BROKER; followed by a service restart to clear stale authentication tokens.
  • Check the SQL Server error log (ERRORLOG in C:\Program Files\Microsoft SQL Server\MSSQL17.MSSQLSERVER\MSSQL\Log\) for confirmation that Error 5838 no longer appears.

Alternative or Verification Steps

  • Verify the fix by querying SELECT * FROM sys.conversation_endpoints; to confirm conversations are establishing successfully without authentication failures.
  • If the issue persists, temporarily set the endpoint AUTHENTICATION = WINDOWS without specifying NTLM/Kerberos explicitly, allowing SQL Server to negotiate automatically between Windows Server 2025 and Windows 11 clients.