At last, the new machine is now doing production work and I've run almost 24 hours so far. First, it's running a bit slower than anticipated, about 13.5 model years per day. I'm not sure what's causing the slower speeds, but it's still an acceptable speed considering my original estimate was only 7 model years per day.
Temperature remains a worry for me. The CPU temps seemed reasonable, but the RAM temps seemed rather high for long-term processing. So, I ordered some fans to cool the RAM boards.
Tomorrow, I'll take a look at the output and start planning a possible Cocoa front-end for running the model.
A key component for running the climate model I'm using is MPICH. MPICH is an implementation of the MPI (Message Passing Interface) library for supercomputing maintained at Argonne National Laboratory (disclosure: I am a former employee of the lab). The climate model uses MPICH to break down the world into smaller parts and distribute those parts to each CPU. However, these parts must also talk to each other because climate information must move from processor to processor so that each processor knows what's going on around it.
MPICH comes installed in Leopard (I'm not sure if it's installed with the developer installation or the standard client installation) and is meant for use with GCC only. It is also likely that it's intended for XGrid as well. So, for this project, the pre-installed MPICH does not have the required fortran to run my simulations.
As it turns out, the default Leopard configuration is lacking a critical configuration for running MPICH on a single mac. It does not allow for shared memory. Using shared memory can speed up computation when you're using multi-processor or multi-core machines because it helps limit how much ethernet bandwidth the processes will use. Ethernet bandwidth is a major bottleneck for processing speed, which is why many high performance clusters use optical networking such as Myranet.
To allow shared memory on Leopard, /etc/sysctl.conf must be created. The contents of the file include something like:
Unfortunately, I don't know the optimal settings for these parameters, I just found these settings off the net. However, they do work.
And now, MPICH can be compiled and installed on the system. Here is the contents of my script to build MPICH:
export CFLAGS='-Msignextend '
./configure -prefix=/opt/mpich/pgi/7.2/ –with-device=ch_shmem –enable-g –enable-sharedlib
Now that the model clearly works using the test license from Portland, it's time to rebuild the system. So far, I've reinstalled MacOS onto the system and installed the developer tools.
Next, I installed required version control software. At this point, subversion and cvs are already on the system so I only needed to install GIT.
The next step in the process is to migrate my subversion and git repositories to the new system and get the servers going.
The version control systems will handle a few things. First, one of the systems will be used to handle the /etc directory. I found that I end up many different versions of mpich and other libraries and not only to I want to prevent messing up this directory, but possibly use branches to determine which versions of these software versions.
Model configurations will also be under version control. Right now, subversion is used to maintain these files. However, git will probably take over this role since a distributed model (since I use more than one cluster) make more sense for configuration management.
Next up: Modifying the system for shared memory runs.
One misgiving about running a climate model on the mac is that I don't trust linux to handle the Mac's fans to keep the mac cool. Apparently, there is even more to it. The mac also will throttle memory bandwidth if the ram gets too hot (see technote TN2156). Whether or not a Linux install not targeted to a Mac would handle these issues, I don't know.
All those worries seem to be moot, however. I managed to get the model running within Leopard. The next few posts will concern some of the details of how this was accomplished.