Right now, the management server synchronously waits for functions to pin, which can cause huge backlogs of status updates. This is bad because -- when the function takes a long time to pin -- on the next pin round, it will think there was a huge influx of thousands of requests, and it will try to add some absurd number of replicas.
Two changes need to be made:
- Status updates should be eagerly pulled of the queue (i.e., dequeue everything that's there, not one per loop) because otherwise they get starved after long pin operations.
- Pins should be made asynchronous. In the interim, there should be a set of sequestered executors that are waiting for pin responses. Those executors should not be used for other pin operations. If the pin succeeds, put them into
function_locations. If it fails, put it back into the pool of available executors.
Right now, the management server synchronously waits for functions to pin, which can cause huge backlogs of status updates. This is bad because -- when the function takes a long time to pin -- on the next pin round, it will think there was a huge influx of thousands of requests, and it will try to add some absurd number of replicas.
Two changes need to be made:
function_locations. If it fails, put it back into the pool of available executors.