A Checkpointing Strategy for Scalable Recovery on Distributed Parallel Systems

In this paper, we describe a new scheme for checkpointing parallel
applications on message-passing scalable distributed memory systems.
The novelty of our scheme is that a checkpointed application can be
restored, from its checkpointed state, in a reconfigured form. Thus, a
parallel application may be checkpointed while executing with $t_1$
tasks on $p_1$ processors, and then restarted from the checkpointed
state with $t_2$ tasks on $p_2$ processors. As a result, applications
can recover from partial failures in the underlying system. Also, the
reconfigurable checkpointed states can be migrated from one parallel
system to another even if they do not have the same number of
processors. We describe a new programming model for implementing a
reconfigurable checkpointing scheme for parallel programs. This new
model is derived from the DRMS programming model, developed in the
context of run-time reconfiguration of parallel applications. A key
component of our implementation is the distribution-independent
representation of application array data structures in persistent
storage. For further optimizing the performance of checkpoint/restart
operations, we provide parallel array section streaming operations for
such distributed arrays. We present performance data for the
reconfigurable checkpointing and restarting of parallel applications
and compare that with the performance of conventional forms of
checkpointing. Our results demonstrate the advantages of the new
scheme we describe.