Clock Synchronization on a Multicomputer

We describe hardware and software schemes for achieving precise clock synchronization on SP2 parallel system nodes. The SP2 multistage interconnection network has an unusual hardware feature, a set of distributed counters that the processor nodes may utilize for synchronizing their time--of--day clocks. We describe an algorithm for synchronizing the counters within less than 200 nanoseconds of each other in a network of up to 512 processor nodes. This is 4--5 orders of magnitude better than what can be achieved by existing software schemes. We also describe an experimental system software, called sptimed, for synchronizing the node clocks to the Internet time--of--day, utilizing the synchronous counters in the SP2 network. Sptimed synchronizes the node clocks typically within 5 microseconds of each other, which is up to 2--3 orders of magnitude better than what can be achieved by previous methods on the SP2 system. Synchronized clocks are useful in parallel and distributed environments, for example for performance measurement, tuning, tracing, debugging, gang scheduling of parallel processes, and timestamping of transactions, to name a few. We also measure the performance of a widely used time synchronization utility, the Network Time Protocol, using the synchronous counters of the SP2 interconnection network.