Keywords: master-worker, parallel computing, cluster computing, grid computing, volunteer computing, interpreted language.

In this paper we describe a general purpose master-worker parallel computation Python module called PyMW. Other tools for writing parallel programs, such as OpenMP, MPI or Globus, support a limited range of platforms. For example, an OpenMP program cannot fully utilize a computing Grid, and a Grid program contains more functionality than is required to run on a cluster. PyMW provides a unified interface to multiple computation environments including multicore processors, networked clusters and the Berkeley Open Infrastructure for Network Computing (BOINC) software platform. This allows researchers to easily write parallel master-worker style programs in Python that can run on a variety of platforms.

PyMW is intended to support rapid development, testing and deployment of large scale master-worker style computations, such as Monte Carlo simulations, parameter sweeps, or analysis of large data sets such as in SETI@home. PyMW allows researchers to quickly develop a parallel program on a multicore machine, test it on a cluster, and finally deploy it on a large scale Grid.

In this paper we discuss the organization of PyMW, and the functionality and abstractions necessary to support a generalized master-worker programming library. PyMW is designed to be nonblocking to allow normal user program flow, and to enable parallel computing with Python which does not support parallel execution. We describe the internal PyMW architecture which allows this functionality while enabling easy usage by programmers.

PyMW is designed to allow easy extension to additional parallel computing platforms through the use of an interface. We detail three interfaces already implemented for PyMW (multicore, MPI and BOINC) and describe the organization of an interface. Interfaces may be implemented with only a single function, while allowing expansion for interface specific capabilities such as redundant computation and worker reservation.

We demonstrate the effectiveness and scalability of PyMW by performing several master-worker style parallel computations on a multicore machine, a networked cluster and a BOINC project. The results of these tests demonstrate that PyMW adds little overhead to master-worker computations and scales well on these systems. These experiments demonstrate that the architecture of PyMW provides flexible and extensible master-worker style parallel computing ability on multiple platforms for the Python programming language.

purchase the full-text of this paper (price £20)

go to the previous paper
go to the next paper
return to the table of contents
return to the book description
purchase this book (price £72 +P&P)