Jun 23, 2008

Climate model on a mac project: #11 sysctl.conf and mpich

 

A key component for running the climate model I’m using is MPICH. MPICH is an implementation of the MPI (Message Passing Interface) library for supercomputing maintained at Argonne National Laboratory (disclosure: I am a former employee of the lab). The climate model uses MPICH to break down the world into smaller parts and distribute those parts to each CPU. However, these parts must also talk to each other because climate information must move from processor to processor so that each processor knows what’s going on around it.

MPICH comes installed in Leopard (I’m not sure if it’s installed with the developer installation or the standard client installation) and is meant for use with GCC only. It is also likely that it’s intended for XGrid as well. So, for this project, the pre-installed MPICH does not have the required fortran to run my simulations.

As it turns out, the default Leopard configuration is lacking a critical configuration for running MPICH on a single mac. It does not allow for shared memory. Using shared memory can speed up computation when you’re using multi-processor or multi-core machines because it helps limit how much ethernet bandwidth the processes will use. Ethernet bandwidth is a major bottleneck for processing speed, which is why many high performance clusters use optical networking such as Myranet.

To allow shared memory on Leopard, /etc/sysctl.conf must be created. The contents of the file include something like:

kern.sysv.shmmax=536870912 kern.sysv.shmmin=1
kern.sysv.shmmni=128
kern.sysv.shmseg=32
kern.sysv.shmall=131072

Unfortunately, I don’t know the optimal settings for these parameters, I just found these settings off the net. However, they do work.

And now, MPICH can be compiled and installed on the system. Here is the contents of my script to build MPICH:

make distclean
export CC=pgcc
export CXX=pgcpp
export FC=pgf90
export F90=pgf90
export F77=pgf77

export CFLAGS=’-Msignextend ‘
export FC=pgf77
export FFLAGS=
export CCFLAGS=

./configure -prefix=/opt/mpich/pgi/7.2/ –with-device=ch_shmem –enable-g –enable-sharedlib