How to fix Error 9699 on Windows Server 2025 / Windows 11

by

Troubleshooting Guide: SQL Server Error 9699 – Memory Allocation Failure for Service Broker Tasks

Why This Error Happens

Error 9699 occurs when SQL Server dynamically detects additional CPUs (often via hot-add CPU operations or VM resource adjustments) and attempts to spin up corresponding Service Broker worker tasks, but the memory allocator cannot reserve sufficient contiguous memory for these new task structures. This is typically caused by memory pressure at the OS level, restrictive “max server memory” configurations that don’t account for the Service Broker overhead, or fragmentation within SQL Server’s memory clerks. In virtualized or cloud environments running Windows Server 2025, this is exacerbated when dynamic memory/CPU hot-add features are enabled without corresponding adjustments to SQL Server’s memory ceiling.

Step-by-Step Resolution Guide

Step 1: Verify Current Memory Configuration

  • Open SQL Server Management Studio (SSMS) and run EXEC sp_configure 'max server memory'; to check the current ceiling.
  • Compare this value against total physical RAM on the host; ensure at least 10-15% is reserved for the OS and SQL Server’s non-buffer-pool needs (including Service Broker).
  • Check Windows Task Manager (Performance tab) or Get-Counter "\Memory\Available MBytes" in PowerShell to confirm available system memory at the time of the error.

Step 2: Adjust Max Server Memory to Accommodate CPU Scaling

  • If your environment uses dynamic CPU hot-add (common in Hyper-V or Azure VMs), lower max server memory to leave more headroom, since each added CPU requires additional worker thread and Service Broker task memory.
  • Run the following to reconfigure:
  EXEC sp_configure 'show advanced options', 1;
  RECONFIGURE;
  EXEC sp_configure 'max server memory', <new_value_in_MB>;
  RECONFIGURE;
  • As a rule of thumb, reserve an additional 1-2 GB per expected hot-added CPU core beyond your baseline calculation.

Step 3: Disable or Audit Hot-Add CPU/Memory Settings

  • If CPUs are being added dynamically without a planned restart, verify whether Hyper-V Dynamic Memory or Processor Compatibility settings are triggering unplanned scale events.
  • In Hyper-V Manager, check the VM settings under Processor and Memory tabs; disable “Enable Dynamic Memory” if SQL Server is not tuned to handle runtime scaling.
  • For physical hosts, check BIOS/UEFI settings for CPU hot-plug support and disable if not explicitly required.

Step 4: Clear Memory Fragmentation via Service Restart

  • If the error persists under adequate memory, a restart of the SQL Server service will reinitialize memory clerks cleanly:
  Restart-Service -Name "MSSQLSERVER" -Force
  • Schedule this during a maintenance window, as this clears any fragmented allocations tied to the Service Broker’s internal task pool.

Step 5: Review SQL Server Error Log and DMVs for Memory Clerks

  • Query sys.dm_os_memory_clerks to identify if the MEMORYCLERK_SQLSERVICEBROKER clerk is consuming disproportionate memory:
  SELECT * FROM sys.dm_os_memory_clerks 
  WHERE type = 'MEMORYCLERK_SQLSERVICEBROKER';
  • Cross-reference findings with the SQL Server Error Log (via xp_readerrorlog) for repeated 9699 entries correlating with CPU count changes.

Alternative or Verification Steps

  • To verify the fix, monitor the SQL Server Error Log after the next CPU scaling event and confirm Error 9699 no longer appears; also check sys.dm_os_sys_info to validate that cpu_count matches the expected value without triggering allocation failures.
  • As an alternative, if hot-add CPU is a business requirement, consider pre-allocating a fixed, higher CPU count at VM/container creation time rather than relying on runtime scaling, which avoids triggering Service Broker’s dynamic task allocation entirely.