CFD General Notation System (CGNS)
Parallel I/O for the CGNS system

Thomas Hauser
thomas.hauser@usu.edu
Center for High Performance Computing
Utah State University

Outline

Why parallel I/O

I/O Needs on Parallel Computers

Distributed File Systems

Diagram of a distributed file system

Parallel File Systems

Diagram of a parallel file system

PVFS2 - Parallel Virtual File System

Diagram of a parallel virtual file system

Simplistic parallel I/O

I/O in parallel programs without using a parallel I/O interface

Not scalable for large applications

Single Process I/O

Diagram illustrating single process I/O

Post-Mortem Reassembly

Diagram illustrating need for reassembly

Parallel CGNS I/O

Diagram illustrating parallel I/O with CGNS

Parallel CGNS I/O API

MPI-IO File System Hints

Parallel I/O implementation

Examples

Example Reading

   if (cgp_open(comm, info, fname, MODE_READ, &cgfile)) cg_error_exit();
   if (cg_nbases(cgfile, &nbases)) cg_error_exit();
   cgbase = 1;
   if (cg_base_read(cgfile, cgbase, basename, &cdim, &pdim)) cg_error_exit();
   if (cg_goto(cgfile, cgbase, "end")) cg_error_exit();
   if (cg_units_read(&mu, &lu, &tu, &tempu, &au)) cg_error_exit();
   if (cg_simulation_type_read(cgfile, cgbase, &simType)) cg_error_exit();

   if (cg_nzones(cgfile, cgbase, &nzones)) cg_error_exit();
   nstart[0] = 1; nstart[1] = 1; nstart[2] = 1;
   nend[0] = SIDES; nend[1] = SIDES; nend[2] = SIDES;
   for(nz=1; nz <= nzones; nz++) {
     if (cg_zone_read(cgfile, cgbase, nz, zname, zsize)) cg_error_exit
     if (mpi_rank == nz-1) {
       if (cg_ncoords(cgfile, cgbase, nz, &ngrids)) cg_error_exit();
       if (cg_coord_read(cgfile, cgbase, nz, "CoordinateX", RealDouble,
                         nstart, nend, coord)) cg_error_exit();
     }
   }

Each Process writes one Zone - 1

   if (cgp_open(comm, info, fname, MODE_WRITE, &cgfile) ||
       cg_base_write(cgfile, "Base", 3, 3, &cgbase) ||
       cg_goto(cgfile, cgbase, "end") ||
       cg_simulation_type_write(cgfile, cgbase, NonTimeAccurate)) cg_error_exit();
   for(nz=0; nz < nzones; nz++) {
     if (cg_zone_write(cgfile, cgbase, name, size, Structured, &cgzone[nz][0]))
        cg_error_exit();
     if (cgp_coord_create(cgfile, cgbase, cgzone[nz][0], RealDouble, "CoordinateX",
                          &cgzone[nz][1])) cg_error_exit();
   }

Each Process writes one Zone - 2

   for(nz=0; nz < nzones; nz++) {
     if (mpi_rank == nz) {
       if (cgp_coord_write(cgfile, cgbase, cgzone[mpi_rank][0], cgzone[mpi_rank][1],
                           coord) ||
           cgp_coord_write(cgfile, cgbase, cgzone[mpi_rank][0], cgzone[mpi_rank][2],
                           coord) ||
           cgp_coord_write(cgfile, cgbase, cgzone[mpi_rank][0], cgzone[mpi_rank][3],
                           coord)) cg_error_exit();
     }
   }

Writing One Zone

   /* size is the total size of the zone */
   if (cg_zone_write(cgfile, cgbase, name, size, Structured, &cgzone[nz][0])) cg_error_exit();
   if (cgp_coord_create(cgfile, cgbase, cgzone[nz][0], RealDouble, "CoordinateX",
                        &cgzone[nz][1]) ||
       cgp_coord_create(cgfile, cgbase, cgzone[nz][0], RealDouble, "CoordinateY",
                        &cgzone[nz][2]) ||
       cgp_coord_create(cgfile, cgbase, cgzone[nz][0], RealDouble, "CoordinateZ",
                        &cgzone[nz][3])) cg_error_exit();

   if (cgp_coord_partial_write(cgfile, cgbase, cgzone, cgcoord, rmin, rmax, coord))
      cg_error_exit;

Write Performance

Total write time - NFS

Graph showing total write time with NFS vs number of processors

Total write time - PVFS2

Graph showing total write time with PVFS2 vs number of processors

Creation time - PVFS2

Graph showing creation time with PVFS vs number of processors

Write time - PVFS2

Graph showing write time with PVFS vs number of processors

Conclusion