Regulating Concurrency vs. Regulating Rate, for CPU Overload Protection of Web Servers

We compare two techniques of preventing server overload by controlling dispatch of mostly short transactional requests at the edge of a system of servers: (1) a mechanism that imposes a limit on the number of active requests and (2) a mechanism that imposes a limit on the rate at which requests are dispatched. While either technique will work well for a workload with low variation, the story gets more complicated if the workload has extreme variation in service time. We suppose a mechanism/policy separation, in which the limits imposed by a dispatch mechanism are set periodically based on policy considerations and current traffic characteristics. Since the dispatch mechanisms follow simple parameterized rules, they can not make ideal decisions for each individual request even with optimal parameter setting. We study the sensitivity of these mechanisms to the variation among requests handled by each mechanism in steady load scenarios where limits are constant or change occasionally as a result of policy-driven automation. The evaluation criterion is the quality of the regulation of the CPU utilization. We find that when the limits are manually set to fixed values neither dispatch mechanism is clearly better than the other. However, when matching automation of the limits is added then the rate-based dispatch mechanism produces more accurate results. On the other hand, the occupancy-based dispatch mechanism produces results that underload less often.