How to fix Error 5840 on Windows Server 2025 / Windows 11

by

Troubleshooting Guide: Service Broker Error 5840 – Invalid Message Forward Mode

Why This Error Happens

Error 5840 occurs when the Service Broker’s message forwarding configuration contains an invalid or unsupported value, typically caused by corrupted route metadata or an improperly configured FORWARD_MESSAGES setting at the endpoint level. This can also result from a failed SQL Server upgrade process where legacy broker configurations from an older instance did not migrate cleanly into SQL Server 2025. In some cases, manual edits to system catalogs (such as sys.routes or sys.service_broker_endpoints) introduce inconsistent state that the Broker engine cannot interpret correctly.

Step-by-Step Resolution Guide

Step 1: Verify Current Broker Endpoint Configuration

  • Open SQL Server Management Studio (SSMS) and connect to your Windows Server 2025 instance.
  • Run the following query to inspect current forwarding settings:
  SELECT name, is_message_forwarding_enabled, message_forwarding_size 
  FROM sys.service_broker_endpoints;
  • Confirm whether is_message_forwarding_enabled holds an unexpected or NULL value.

Step 2: Reset the Message Forwarding Mode

  • Disable and reconfigure the forwarding option explicitly rather than leaving it in a default/ambiguous state.
  • Execute the following T-SQL command:
  ALTER ENDPOINT [ServiceBrokerEndpoint]
  FOR SERVICE_BROKER (MESSAGE_FORWARDING = DISABLED);
  • If forwarding is required for your environment, re-enable it with explicit sizing:
  ALTER ENDPOINT [ServiceBrokerEndpoint]
  FOR SERVICE_BROKER (MESSAGE_FORWARDING = ENABLED, MESSAGE_FORWARD_SIZE = 10);

Step 3: Rebuild Corrupted Route Entries

  • Navigate to the affected database and drop any routes referencing invalid forwarding states.
  DROP ROUTE [BrokenRouteName];
  • Recreate the route with correct parameters pointing to the valid broker instance:
  CREATE ROUTE [FixedRoute]
  WITH SERVICE_NAME = '//YourApp/Service',
  ADDRESS = 'TCP://ServerName:4022';

Step 4: Restart the SQL Server Service

  • On Windows Server 2025, open Services.msc.
  • Locate SQL Server (MSSQLSERVER) or your named instance.
  • Right-click and select Restart to clear any cached broker session states tied to the invalid forward mode.

Alternative or Verification Steps

  • To verify the fix, re-run the sys.service_broker_endpoints query from Step 1 and confirm is_message_forwarding_enabled now returns a valid boolean (0 or 1) rather than an inconsistent value; also check the SQL Server Error Log for the absence of further Error 5840 entries after restart.
  • If the error persists, consider dropping and fully recreating the Service Broker endpoint from scratch using CREATE ENDPOINT, as this bypasses any residual corrupted configuration that ALTER commands cannot overwrite.