How to fix Error 11204 on Windows Server 2025 / Windows 11

by

Troubleshooting Guide: SQL Server Error 11204 — Broker Instance Exceeds Maximum Size

Why This Error Happens

This error occurs when a Service Broker message contains a “FROM” broker instance identifier (a GUID-based conversation handle string) that exceeds the internal size limit enforced by SQL Server, typically due to message corruption in transit or a malformed dialog initiated by a non-standard or third-party client. It can also result from network-layer packet corruption, mismatched Service Broker endpoint certificates causing garbled decryption, or a bug in custom application code that manually constructs Service Broker message headers. In rare cases, outdated SQL Server 2025 cumulative updates with known Service Broker parsing defects can trigger this behavior even with valid messages.

Step-by-Step Resolution Guide

Step 1: Verify Service Broker Configuration and Certificates

  • Run SELECT name, is_broker_enabled FROM sys.databases WHERE name = 'YourDatabaseName'; to confirm Service Broker is properly enabled.
  • Check certificate validity using SELECT * FROM sys.certificates; on both the initiator and target databases to rule out encryption/decryption mismatches corrupting message headers.
  • Ensure certificate expiration dates and subject names match expected configurations on both ends of the conversation.

Step 2: Inspect and Clean Up Broken Conversations

  • Query SELECT * FROM sys.conversation_endpoints WHERE state_desc <> 'CONVERSING'; to identify stuck or malformed dialogs.
  • End problematic conversations manually using END CONVERSATION 'conversation_handle' WITH CLEANUP; to purge corrupted message chains from the queue.
  • Check sys.transmission_queue for messages with transmission_status showing repeated Error 11204 entries and remove them if they are stale.

Step 3: Apply Latest SQL Server 2025 Cumulative Update

  • Navigate to the Microsoft Update Catalog and download the latest Cumulative Update (CU) for SQL Server 2025, as several Service Broker parsing fixes are frequently rolled into CUs.
  • Install using the setup executable with /ACTION=Patch /IACCEPTSQLSERVERLICENSETERMS flags from an elevated PowerShell session on Windows Server 2025.
  • Restart the SQL Server service via services.msc or Restart-Service -Name 'MSSQLSERVER' after patching completes.

Step 4: Rebuild the Affected Queue and Service

  • Drop and recreate the affected Service Broker QUEUE and SERVICE objects using DROP SERVICE [YourServiceName]; followed by CREATE SERVICE with the original contract definitions.
  • Re-link the SERVICE to its QUEUE and verify the ROUTE and remote service binding definitions are intact using SELECT * FROM sys.routes; and SELECT * FROM sys.remote_service_bindings;.
  • Restart any application components that initiate dialogs to ensure they reconnect to the rebuilt service cleanly.

Alternative or Verification Steps

  • To verify the fix, send a test message through the Service Broker dialog and monitor sys.transmission_queue to confirm it clears without generating new 11204 errors in the SQL Server Error Log (EXEC xp_readerrorlog 0, 1, N'11204';).
  • If the error persists, consider enabling Service Broker trace flags (such as 1222 combined with Extended Events for broker_message_classify) to capture the raw malformed message for deeper packet-level analysis.