Challenges and Experience in Prototyping a Multi-Modal Stream Analytic and Monitoring Application on System S

In this paper, we describe the challenges of prototyping a reference application on System S, a distributed stream processing middle-ware under development at IBM Research. With a large number of stream PEs (Processing Elements) implementing various stream analytic algorithms, running on a large-scale, distributed cluster of nodes, and collaboratively digesting several multi-modal source streams with vastly divergent rates, prototyping a reference application on System S faces many challenges. Specifically, we focus on three critical challenges of, and our experience in, prototyping DAC (Disaster Assistance Claim monitoring), a reference application dealing with multi-modal stream analytic and monitoring. The three challenges are: (1) How do we generate correlated, multi-modal source streams for DAC? (2) How do we design and implement a comprehensive stream application, like DAC, from many divergent stream analytic PEs? (3) How do we deploy DAC in light of source streams with extremely different rates? We report our experience in addressing these challenges, including modeling a disaster claim processing center to generate correlated source streams, constructing the PE flow graph, utilizing programming supports from System S, adopting parallelism, and exploiting resource-adaptive computation.