IL-0136 Microsoft Azure · 2026-09-29azureai-infra
Azure OpenAI Service and related Azure AI services look up each calling account's resource metadata and configuration in a regional backend service, which reads from a caching layer and a database layer. Internal services that call this backend keep their own caches of this information to reduce load on it.
A planned deployment of one of those internal services in Sweden Central cleared its caches at peak traffic, which produced a large number of cache misses. Two defects in how the service read from its caches made most lookups bypass the cache and call the backend directly. By about 10:00 UTC the backend received roughly 20 times its normal traffic. Its own requests to the caching layer drove that layer to full utilisation, latency rose, connections stayed open longer, client retries added requests, and the network connection capacity on the infrastructure node instances ran out, so no new outbound connections to the caching or database layers could be made.
The surge started the incident, but its length came mainly from the backend's health checks, which depended on connecting to the database layer. Instances that could not open new outbound connections failed the check and were restarted by design, which shrank the pool of healthy instances further. Scaling and resource changes did not help. At 15:40 UTC the health check was reconfigured to no longer depend on remote connectivity, the restart loop stopped, and availability recovered by 15:58 UTC. At 16:30 UTC the source of the extra traffic was traced to the internal service deployment.
From 10:03 UTC on 29 September 2026, requests to Azure OpenAI Service, Foundry Agent Service, Foundry Models and Azure AI services in Sweden Central failed intermittently, took longer, or returned HTTP 500 and 503 errors. At 10:17 UTC service monitoring raised an alert on elevated failure rates for Azure OpenAI Service requests in the region.
At 11:30 UTC engineers found that the service instances on the infrastructure nodes were timing out when they tried to connect to the backend service's caching and database layers. By 13:55 UTC the database layer had been confirmed healthy.
The backend service's instances were failing their automated health checks, because each check required a new connection to the database layer, and were being restarted automatically. As more instances entered the restart loop, fewer healthy instances were left, and retries from dependent services added more load to them.
The existing network and diagnostic metrics did not show that outbound connections were exhausted, and correlating the failures with that cause took a long time. Scaling the number of instances to reduce outbound connections, adding compute and memory, raising resource limits and restarting instances did not resolve the incident.
Symptom ? High error rateHigh latencyIntermittent connection failuresIntermittent timeoutsProbe failuresOutbound connection exhaustion
Root cause ? Faulty deployment
Service monitoring detected elevated failure rates for Azure OpenAI Service requests in the region at 10:17 UTC and alerted the engineering teams; engineers were engaged at 10:28 UTC.
A health check that depends on a remote dependency turns a dependency problem into a restart loop: instances that cannot reach the database are killed, the remaining ones take more load and retries, and adding instances or resources does not help until the check itself is changed.
Clearing a cache during peak traffic can multiply load on the backend behind it, especially when cache-read defects let lookups bypass the cache. Connection exhaustion was hard to see because the existing network and diagnostic metrics did not show it.
–