How to fix Error 11255 on Windows Server 2025 / Windows 11

by

Troubleshooting Guide: SQL Server Error 11255 – “A Corrupted Message Has Been Received. The Encryption Flag Is Set”

Why This Error Happens

This error typically occurs when there is a mismatch or corruption in the TDS (Tabular Data Stream) packet during the SSL/TLS handshake between the client and Microsoft SQL Server 2025. It is commonly caused by a mismatched or expired SSL certificate, a corrupted network packet due to an unstable connection, or an outdated TLS protocol configuration on either the client or the server. Additionally, third-party network security software or proxies intercepting encrypted traffic can corrupt the handshake, triggering this specific error.

Step-by-Step Resolution Guide

Step 1: Verify and Reconfigure SSL/TLS Certificate Bindings

  • Open SQL Server Configuration Manager on Windows Server 2025.
  • Navigate to SQL Server Network Configuration > Protocols for [Your Instance Name].
  • Right-click and select Properties, then go to the Certificate tab.
  • Ensure a valid, non-expired certificate is selected (do not rely on a self-signed certificate for production unless explicitly configured for it).
  • If the certificate is invalid, reissue one via certlm.msc and rebind it to the SQL Server instance.

Step 2: Update TLS Protocol Settings in the Windows Registry

  • Open the Registry Editor (regedit) on the server.
  • Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols.
  • Ensure TLS 1.2 is enabled explicitly, as Windows Server 2025 defaults to TLS 1.3, which some older SQL client drivers do not negotiate correctly.
  • Set the DisabledByDefault key to 0 under the TLS 1.2 Client and Server subkeys.
  • Restart the SQL Server Service via services.msc to apply the registry changes.

Step 3: Check Network Packet Integrity and Firewall Interception

  • Disable any third-party antivirus or deep packet inspection (DPI) firewall rules temporarily to rule out packet corruption.
  • Run netsh winhttp show proxy to check if a corporate proxy is intercepting the SSL handshake.
  • If using a Virtual Network Appliance (common in Azure or VMWare environments running Windows Server 2025), bypass it temporarily to isolate the fault.

Step 4: Update SQL Server Native Client and ODBC Drivers

  • Ensure the client machine is using the latest MSODBCSQL18 or SqlClient driver, as older drivers (pre-2022) often fail to handle the newer Encryption flags required by SQL Server 2025.
  • Download the latest driver from the Microsoft Download Center and reinstall it silently using:
  • msiexec /i msodbcsql.msi IACCEPTMSODBCSQLLICENSETERMS=YES ADDLOCAL=ALL /quiet

Alternative or Verification Steps

  • Verification: Use sqlcmd -S [ServerName] -E -N to force an encrypted connection test; if the connection succeeds without the 11255 error, the certificate/protocol fix was successful.
  • Alternative Fix: If the error persists, navigate to the connection string on the client application and explicitly set Encrypt=False (only for internal, trusted networks) to bypass the corrupted handshake while you continue investigating the certificate chain.