Performance Modeling and Placement of Transforms for Stateful Mediations

In this paper we propose a new technique for placing large delivery plans for streaming systems on a network of machines to optimize efficiency measures such as latency. In the model we consider, there is a large network of machines and the different fixed end-points of the network act as publishers and subscribers of information. Information demanded by subscribers is a transformed view of the information published by the publishers. The transformed view is the outcome of an acyclic network of simple transformations operating on the publishers' information or some intermediate transformed view of it. We propose algorithms for the optimal placement of the acyclic transform network on the network of machines. As an example scenario to evaluate the efficacy of our algorithms we consider SQL queries on streaming relational tables. The transform network in this case is the SQL operator tree for the query. We first show how to model the performance of individual operators acting on distributed streams and then develop the optimal placement strategy for different optimization measures. We present our work on a distributed message-oriented middleware and a programming platform for large-scale publish-subscribe applications called SMILE. In our system, we use incremental implementation each of the relational operator for streaming data. We demonstrate that our technique performs significantly better than straightforward approaches like greedy, and random placement.

By: Vinayaka Pandit, Rob Strom, Gerry Buttner and Roman Ginis

Published in: RI08002 in 2008

LIMITED DISTRIBUTION NOTICE:

This Research Report is available. This report has been submitted for publication outside of IBM and will probably be copyrighted if accepted for publication. It has been issued as a Research Report for early dissemination of its contents. In view of the transfer of copyright to the outside publisher, its distribution outside of IBM prior to publication should be limited to peer communications and specific requests. After outside publication, requests should be filled only by reprints or legally obtained copies of the article (e.g., payment of royalties). I have read and understand this notice and am a member of the scientific community outside or inside of IBM seeking a single copy only.

RI08002.pdf

Questions about this service can be mailed to reports@us.ibm.com .