The use of remotely distributed computing resources as a single system offers great potential for compute-intensive applications. Increasingly, users have access to hundreds or thousands of machines at once and wish to utilize those resources concurrently. To provide a reasonable user experience, such systems must provide an effective, scalable scheduling system. Unfortunately, the great majority of job schedulers are centralized and many do not scale well to thousands or even hundreds of nodes. In this paper we study how distributed scheduling systems can be designed most effectively; we focus on the problem of selecting an optimal arrangement of schedulers, or a deployment, for hierarchically organized systems. We show that the optimal deployment is a complete spanning d-ary tree; this result conforms with results from the scheduling literature. More importantly, we present an approach for determining the optimal degree d for the tree. To test our approach, we use DIET, a middleware system that uses hierarchical schedulers. We develop detailed performance models for DIET and validate these models in a real-world environment. Finally, we demonstrate that our approach selects deployments that are near-optimal in practice.