PIFD (Parallel Items-Files Distribution) Project

PIFD is a parallel read/write engine to distribute data items from/to files located over multiple machines.
It provides a transparent and efficient way to incorporate data reading, distributing, and processing in the same MPI program.

The system was mainly developed for ParInt, to save/load large number of integration regions, however it can be easily integrated in any application that reads/writes large numbers of independent items. PIFD helps maximize the use of the underling storage topology, exploiting in particular the benefit of using local disks.

Given a collection of items distributed over a number of machines, PIFD:
       - reads each item exactly once (when the item is processed)
       - minimizes the communication and efficiently distributes all items among all participating nodes.
       - provides an easy and transparent way to use local hardisks
       - uses a configurable size buffer for interprocess communication
       - provides an array of options to read and distribute the data such that:
             - each node will process the same number of items, or (EQUAL)
             - allow faster nodes to process more data, or (COMPETE)
             - only nodes with direct access to data will process items. (LOCAL)

The system is written in C, and can be integrated with the user application trough a small set of read/write/pack/unpack functions.
The source archive contains a demo application that shows how the system is used.

Download the source code (version 1.1): pifd-1.1.tgz

Tests:
Test Set gRpas HTML
Comprehensive applet view
Non Uniform Distribution applet -

PIFD and ParInt



...back to my home page.