How to fix Error 11235 on Windows Server 2025 / Windows 11

by

Troubleshooting Guide: SQL Server Error 11235 — Malformed Service Broker Message

Why This Error Happens

Error 11235 occurs when the Service Broker endpoint on SQL Server 2025 receives a TDS-encapsulated message with a binary message class it does not recognize, which typically means the data arriving on the dialog is not legitimate Service Broker traffic. This is most commonly caused by another non-SQL application (such as a monitoring agent, load balancer health check, or an unrelated TCP service) connecting to the same port configured for the Service Broker endpoint. It can also result from network-level corruption, a misconfigured firewall/NAT rule that redirects unrelated traffic to the Broker port, or a mismatch in endpoint configuration between two SQL Server instances attempting to communicate.

Step-by-Step Resolution Guide

Step 1: Identify the Port and Confirm What Is Actually Connecting

  • Run the following to confirm the Service Broker endpoint port: SELECT name, port, protocol_desc, state_desc FROM sys.tcp_endpoints WHERE type_desc = 'SERVICE_BROKER';
  • Use netstat -ano | findstr <port> on Windows Server 2025 / Windows 11 to see which process IDs are bound to or connecting on that port.
  • Cross-reference PIDs in Task Manager or Get-Process -Id <PID> in PowerShell to confirm whether a non-SQL process (e.g., a health-check script, antivirus proxy, or third-party agent) is hitting the port.

Step 2: Isolate the Service Broker Endpoint from Conflicting Traffic

  • If another application is sharing or scanning the port, reassign the Service Broker endpoint to a dedicated, non-standard port: ALTER ENDPOINT [ServiceBrokerEndpoint] AS TCP (LISTENER_PORT = 4023);
  • Update Windows Firewall rules to restrict that port strictly to SQL Server traffic: New-NetFirewallRule -DisplayName "SQL Broker Dedicated" -Direction Inbound -Protocol TCP -LocalPort 4023 -Program "C:\Program Files\Microsoft SQL Server\MSSQL17.MSSQLSERVER\MSSQL\Binn\sqlservr.exe" -Action Allow
  • Remove or reconfigure any load balancer, NAT rule, or third-party agent that was pointed at the original Broker port.

Step 3: Validate Endpoint and Certificate Configuration

  • Confirm the endpoint state is STARTED and authentication matches on both ends of the dialog: SELECT state_desc, is_encryption_enabled FROM sys.tcp_endpoints;
  • Check sys.transmission_queue for stuck or malformed messages: SELECT * FROM sys.transmission_queue;
  • If certificates are used for Broker security, verify they have not expired: SELECT name, expiry_date FROM sys.certificates;

Step 4: Capture Network Traffic to Confirm Root Cause

  • Use Network Monitor or Wireshark on Windows Server 2025 to capture traffic on the Broker port during the error window.
  • Filter by tcp.port == <port> and inspect payloads for non-TDS formatted data confirming an external application is the source.
  • Document the source IP/MAC of the offending connection for firewall blocking.

Step 5: Restart the Broker Session and Monitor Logs

  • Run ALTER DATABASE [YourDB] SET NEW_BROKER; only if conversations are corrupted beyond recovery (this is destructive — back up first).
  • Monitor the SQL Server Error Log and Windows Event Viewer (Application log) for recurrence of Event ID 11235 after applying the port/firewall changes.

Alternative or Verification Steps

  • After applying the port change and firewall rule, re-run a test conversation using BEGIN DIALOG CONVERSATION between two test queues and confirm no 11235 errors appear in the SQL Server Error Log for at least 24 hours.
  • If the error persists, as a fallback, disable the Service Broker endpoint temporarily (ALTER ENDPOINT [ServiceBrokerEndpoint] STATE = DISABLED;) and use SQL Profiler/Extended Events to trace exactly which client application is initiating the malformed connection before re-enabling it.