Characterizing, Constructing and Managing Resource Usage Profiles of System S Applications: Challenges and Experience

We describe the challenges of and our experience in characterizing, constructing and managing the usage profiles of System S applications. A System S application deployed at runtime is a directed graph with software processing elements (PEs) as vertices and data streams as edges connecting the PEs. The resource usage of each PE is a critical input to the runtime scheduler for proper resource allocation. We represent the resource usage of PEs in terms of resource functions (RFs) that are used by the System S scheduler, with one RF per resource per PE. The first challenge is that building good RFs that can accurately predict the resource usage of a PE can be difficult because the PEs perform arbitrary computations. A second set of challenges arises in managing the RFs and data so that we can apply them for PEs that are re-run and/or reused by the same or different applications or users. We report our experience in overcoming these challenges. Specifically, we present an empirical characterization of PE RFs from several real streaming applications running in a System S testbed. This justifies our simple, yet effective, models of resource usage that build on the data-flow nature of the underlying application. We show that simple piecewise linear models are generally effective in practice, even for complex PEs. To illustrate our methodology, we evaluate and analyze the performance of several real System S applications as a function of the quality of our resource profile models. To obtain these resource profiles, the system automatically learns the models from the raw metrics data collected from running PEs. We describe our approach to managing the metrics and RF models, which allows us to construct generalizable RFs and eliminates or reduces the learning time for new PEs by intelligently storing and reusing the metrics data.