Monthly Archives: January 2010

FOAM Output Variables

Since I get many questions on what's contained in FOAM output, here's a list of all the variables contained in the standard atmosphere, coupler, and ocean output files.

Atmosphere Output
ALB Albedo
ALBCLR Clear sky albedo
CLDHGH Vertically-integrated, random overlap, total cloud amount
CLDLOW Vertically-integrated, random overlap, low cloud amount
CLDMED Vertically-integrated, random overlap, mid-level cloud amount
CLDTOT Vertically-integraed, random overlap, total cloud amount
CLOUD Cloud fraction
CMFDQ Q tendency – moist convetsion
CMFDT T tendency-moist convection
CMFMC Moist convection mass flux
CNVCLD Random overlap total convective cloud amount
DC01 convective adjustment tendency
DTCOND T tendency – convective adjustment
DTH T horizontal diffusive heating
DTV T vertical diffusion
FLNS Net longwave flux at surface
FLNSC Clearsky net longwave flux at surface
FLNT Net longwave flux at top
FLNTC Clearsky net longwave flux at top
FLUT Top of Atmosphere Outgoing Longwave Flux
FLUTC Clearsky Top of Atmosphere Outgoing Longwave Flux
FSDS Flux Shortwave Downwelling at Surface
FSDSC Clearsky Flux Shortwave Downwelling at Surface
FSNS Net solar flux at surface
FSNSC Clearsky net solor flux at surface
FSNT Net solar flux at top
FSNTC Clearsky net solar flux at top
FSNTOA Net Solar Flux at the Top of the Atmosphere
FSNTOAC Clearsky Net Solar Flux at the Top of the Atmosphere
LHFLX Surface latent heat flux
LWCF Longwave Cloud Forcing
OMEGA Vertical pressure velocity
ORO ocean(0), land(1), sea ice(2)
PBLH Planetary Boundary Layer Height
PHIS surface geopotential
PRECC Convective precipitation rate
PRECL Large-scale (stable) precipitation rate
PRECSC Convective snow rate (water equivalent)
PRECSL Large-scale (stable) snow rate (water equivalent)
PS surface pressure
PSL Sea level pressure
Q specific humidity
QFLX Surface water flux
QPERT Perturbation specific humidity (eddies in PBL)
QRL Longwave heating rate
QRS Solar heating rate
RELHUM Relative humidity
SHFLX Surface sensible heat flux
SNOWH Water equivalent snow depth
SOLIN Solar insolation
SRFRAD Net radiative flux at surface
SWCF Shortwave Cloud Forcing
T temperature
TAUGWX East-west gravity wave drag surface stress
TAUGWY North-south gravity wave drag surface stress
TAUX X-component (eat-weat) at surface stress
TAUY Y-component (north-south) of surface stress
TMQ Water Vapor
TPERT Perturbation temperature (eddies in PBL)
TS1 Surface temperature (level 1)
TS2 Subsurface temperature (level 2)
TS3 Subsurface temperature (level 3)
TS4 Subsurface temperature (level 4)
U zonal wind component
USTAR Surface friction velocity
UTGW U tendency – gravity wave drag
V meridional wind component
VD01 vertical diffusion tenency of water vapor
VQ Meridional water transport
VT Meridional heat transport
VTGW V tendency – gravity wave drag
VVPUU Kenetic Energy
VZ Meridional transport
WET Soil moisture
Z3 Geopotential Height

Coupler

EVP moisture flux
FRAC ice fraction
ICET1 ice layer 1 temp
ICET2 ice layer 2 temp
INTERT interface temp
LHF latent heat flux
LWV longwave out
MELTP melt potential
OHEAT ocean heat forcing field
OPREC precip from atm
OQFLX ocean freshwater forcing field
ORNF runoff into the ocean
ORO land mask
RAD surface radiation
RNF land runoff
SHF sensible heat flux
SNDPTH snow depth
SNM snow melt
SNOWT snow temp
TAUX ocean taux foring field
TAUY ocean tauy forcing filed
THCK seaice thickness flag
TSSUB1 top soil layer temp
TSSUB2 soil temp layer 2
TSSUB3 soil temp layer 3
TSSUB4 bottom soil temp layer
VOLR river volume
WS soil moisture

Ocean

CONVEC Upper Ocean Convective adjustment frequencey
CONVEC2 Deep Ocean Convective adjustment frequencey
Currents Currents
HEATF Ocean heat forcing
P Normalized perturbation presure
S Salinity
SALTF Sfc salinity tendency due to freshwater forcing
Sconv Surface layer S convective adjustment
Sconvm1 Near surface layer S convective adjustment
Szz Surface layer S vertical mixing
Szzm1 Near surface layer S vertical mixing
T Temperature
TAUX X-component (east-west) of surface stress
TAUY Y-component (north-south) of surface stress
Tconv Surface layer T convective adjustment
Tconvm1 Near surface layer T convective adjustment
Tzz Surface layer T vertical mixing
Tzzm1 Near surface layer T vertical mixing
U Zonal current component
V Meridional current component
W Vertical velocity

Using SQLite and Python to Store Model Metadata

As I continue to run a range of climate models, I've learned from painful lessons that I need to record as much information about the model run as possible. When I first started this process, I simply kept files used to make the run (the geography and configuration files for the model) and the model output. At first, this seemed sufficient because, in the end, these were the data that were most important. As it turns out, however, that having a history of everything you did during the model run, such as adjustments to the settings or geography, is also important both historically to the run and possibly sorting out problems later.

My initial solution to this problem was to create a log file. Every time I ran the model, the important setting information was sent to a simple flat-file log. It turned out that this log was very important to debugging a model-run issue because it kept a record of how the model was initially run. I also started keeping information about the hardware in this log. Along with the model information, I began to store hardware temperature data from before and after the run in the log just in case I needed to debug hardware issues. However, these data turned out to be virtually useless in a flat log file. Other information I haven't been keeping that I wanted in the log was geography version control information. I use version control to track all my geography work, so I can both track how I change the geography and get an idea how much time I spend on it. However, the exact geography used in a run is important to know. However, even more info in a flat log file makes it even more difficult to review.

My new solution is to dump the flat file approach and go with SQLite. SQLite is a lightweight, public domain SQL file format that works well with a variety of languages. SQLite has become one of my preferred file formats over the years (nudging out XML for custom data structures). The Python scripting language seems a natural fit to work with SQLite as well.

So, how does this solution work? FIrst, I have a simple run script for the model using bash (for some reason, I could never get the model to run using Python). This script calls my python script before the model starts and after the model ends. It sends two pieces of information, a uuid and the model directory path. The python script assembles everything it needs on its own.

Why a uuid? Each time I run the model, I need to identify the run in the database with a unique id that can be used to link across a number of SQL tables. A uuid ensures that the id is unique. I've considered using a uuid for the overall simulation but I haven't implemented that.

To pull in settings data and temperature data, I've written parsers for each format. For the model I've been running, FOAM, I have parsers that read the atmos_params and run_params files in addition to parsing the temperature monitor software and subversion “svn info” command. The script then inserts these data into their own tables marked by the uuid. While most of these tables have fields for each value I pull out of files, the temperature data is stored in key->value type table since the number of temperature sensors is dependent on hardware and thus may change from machine to machine (and is also Mac only).

Here is the schema for the main table, “runs”:

CREATE TABLE runs (
uuid text,
starttime text,
endtime text,
runduration text,
std_out blob,
completed text,
comments text,
runcmd text,
yearsPerDay text
);

Some of these fields are not yet used. std_out and the rancmd is not yet implemented in the script . Right now, I'll do the comments field manually. My currently running simulation looks like this at the moment:

uuid = **deleted
starttime = 2010-01-16 23:33:45
endtime = 2010-01-17 10:29:57
runduration = 39372.0
std_out =
completed = NO
comments = manual shutdown because of memory problem
runcmd =
yearsPerDay = 14.2

For the geography source location, here's the results for the run above:

uuid = **deleted
url = file:///Volumes/**deleted
type = svn
date = 2009-08-28 09:42:01 -0500 (Fri, 28 Aug 2009)
author = tlmoore
rev = 191
branch =

the branch is empty in anticipation for moving from subversion to git.

For temperatures, I can now look at before and after values for specific sensors for a run:

uuid = *deleted
sensor = SMC CPU A HEAT SINK
temperature = 64.4

uuid = *deleted
sensor = SMC CPU A HEAT SINK
temperature = 98.6

One thing I'd change here is specifying pre- versus post-run measurements.

So far, I'm happy with most of this new solution. It just need refinements.