Commonly used general-purpose software is made available to NERSC users through the use of modules. In the same way, we are now making the most popular CMB-specific software available to NERSC CMB users through a dedicated module.

Unlike general purpose software, the cmb module is not supported by NERSC, but by CMB data analysts in the Computational Research Division at Berkeley Lab; if you have any problems with this module, or would like to see new software added to it, you should therefore contact us, and not NERSC support.

The latest release of the CMB module is installed at NERSC on franklin, jacquard, bassi, and davinci. See notes at the bottom of this page describing how to use this software on each machine. Also, this software has been installed on several machines outside of NERSC. These machines also have useage notes below.

Individual Software Packages

The CMB module is made up of a number of packages, each of which is also a module in its own right. These fall into 2 categories:

  1. Packages for which we support the installation, but not the software itself. We are "package maintainers" for this software, similar to package maintainers for various GNU/Linux distributions. We take the original source tarball, apply a patch which allows the software to build in our environment, and then build and install the software.


  2. Packages for which we support both the NERSC installation and the software itself.

    • The CMB Data Management Framework
    • General Compressed Pointing (GCP) Library
    • M3 Data Abstraction Library
    • The Microwave Anisotropy Dataset Computational Analysis Package (MADCAP)
    • MADmap (PCG Mapmaker)

All of the software has been compiled in 64-bit mode when possible, and with extensive optimization flags. Codes using OpenMP have been compiled with this feature enabled.

Using the CMB module

The precise location of the installed software modules varies on each system. In order to gain access to the modules, a few lines may need to be added to your shell resource file(s). See the machine-specific notes below to see how to initialize the modules environment on each system.

On many NERSC machines and installations elsewhere, there are multiple "Programming Environments" or compiler toolchains available. Before loading the desired cmb module version, you must decide which compiler environment you want to use. This selection is done with the "cmbenv" command: 

%> cmbenv
usage:  cmbenv [ pgi | gnu ]
After deciding which environment you will be using, run cmbenv with the appropriate selection: 
%> cmbenv pgi

After you have set up your environment, the modules should show up at the top of the list when you execute the module avail command: 

%> module avail
You should see a listing for both the "cmb" and "cmbpath" modules. You can disregard the cmbpath module unless you are trying to do specialized things with the modules. You can see which modules are currently loaded by typing: 
%> module list
Now load the cmb module and verify that it has in turn loaded the modules for all of the individual software packages. 
%> module load cmb
%> module list
More information can be found in the "help" page for each module listed above. For example, after using "module avail" to see what modules are available, you could then get more information about specific modules: 
%> module help healpix

All of the executables contained within these modules are now in your default PATH. You can run them just by typing the name of the program. If you want to verify exactly which executable is going to launch when you type the name of a program, use the "which" command: 

%> which anafast
Many of the tools in the CMB module provide libraries that you can use for building your own software. The necessary include and linking flags for these libraries are provided by the "cmbconfig" script. After loading the cmb or cmbpath modules, you will have access to this script. Running this script without any arguments displays useage information: 
%> cmbconfig


<************************************************************>
  Usage:  cmbconfig
          --comp (print compiler)
          --cflags (print compile options)
          --omp (print OpenMP flags)
          --incl (print include directories)
          --lflags (print linking directories)
          --libs (print libraries)
          --lang < C | CXX | F77 | F90 | MPIC | MPICXX | MPIF77 | MPIF90 >
          --fcaps (does F90 produce capitalized modules?)
          --fpref (print F90 prefix needed for "-D")
          -I (DEPRECATED:  same as --incl)
          -L (DEPRECATED:  same as --lflags --libs)
          [package1] [package2] [package3]...

          Where the following package names are accepted:
            cfitsio (print the flags for CFITSIO)
            xml2 (print the flags for XML2)
            healpix (print the flags for F90 HEALPix)
            chealpix (print the flags for C HEALPix)
            cxxhealpix (print the flags for C++ HEALPix)
            hpic (print the flags for HPIC)
            gcp (print the flags for GCP)
            fftw2 (print the flags for FFTW2)
            fftw2_mpi (print the flags for MPI FFTW2)
            fftw3a (print the flags for FFTW3-alpha)
            fftw3a_mpi (print the flags for MPI FFTW3-alpha)
            fftw3 (print the flags for FFTW3)
            camb (print the flags for CAMB)
            lenspix (print the flags for LensPix)
            s2hat (print the flags for S2hat)
            wmap (print the flags for WMAP)
            m3 (print the flags for M3)
            m3_mpi (print the flags for MPI M3)
            scalapack (print the flags for scalapack)
            madspec (print the flags for MADspec)
            lapack (print the flags for LaPACK)
            blas (print the flags for BLAS)
            sprng (print the flags for SPRNG)
<************************************************************>
For example, to compile a C++ program called "mycode" that requires the FFTW3, healpix, and GCP libraries, here is how you would use cmbconfig on bassi. Note that we make use of the "backtick" shell operator to insert the output of the cmbconfig script into our compile command: 
%> xlC_r -O3 `cmbconfig --incl fftw3 gcp healpix` -o mycode mycode.cpp \
`cmbconfig --lflags --libs fftw3 gcp healpix`
You can also use cmbconfig inside the Makefiles of your own software projects. In order to create a portable Makefile, you could use cmbconfig to obtain the compiler name as well. For example, a simple Makefile for the the "mycode" project might look like this: 
CXX = $(shell cmbconfig --lang CXX --comp)
CXXFLAGS = $(shell cmbconfig --lang CXX --cflags)
INCL = $(shell cmbconfig --incl fftw3 gcp healpix)
LINK = $(shell cmbconfig --lflags --libs fftw3 gcp healpix)


mycode : mycode.o
    $(CXX) -o mycode mycode.o $(LINK)


mycode.o : mycode.c
    $(CXX) $(CXXFLAGS) $(INCL) -o mycode.o -c mycode.c

If you use cmbconfig in your Makefile, then be sure to load the cmb module before running make.

IDL Notes

The HEALPix IDL tools have been installed systemwide, and the locations of these tools are added to the $IDL_PATH environment variable when the module is loaded. No other configuration is needed. You should NOT use the the “hidl” shell alias to start IDL. You should simply run “idl”, and all the HEALPix tools will be available.

Machine Specific Notes

On all supported machines, you must initialize the cmb modules in your shell resource file(s). This consists of adding a couple lines to the resource file which "sources" (inserts) a piece of shell code into your shell initialization process. There are 2 files that can be sourced depending on whether you are using a bourne-type shell or a C-type shell. If you don't know what your default shell is, just log into the machine and type: 

%> echo $SHELL

After determining your shell, follow the directions below for the machine of interest. NOTE: if you set up only one type of shell (e.g. bash) to access the cmb modules, remember that any batch scripts which use a different shell (like csh) may not have access to the cmb modules.

franklin.nersc.gov

On franklin, you should add the following lines to your shell resource files:
bash
(~/.bashrc.ext) 
. /project/projectdirs/cmb/modules/franklin/cmbenv.sh
(t)csh
(~/.cshrc.ext) 
source /project/projectdirs/cmb/modules/franklin/cmbenv.csh
The next time you log in, the "cmbenv" command will be available. You can specify either the pgi or gnu environments, and then load the cmb module. 
%> cmbenv pgi
%> module load cmb

hopper.nersc.gov

On hopper, you should add the following lines to your shell resource files:
bash
(~/.bashrc.ext) 
. /project/projectdirs/cmb/modules/hopper/cmbenv.sh
(t)csh
(~/.cshrc.ext) 
source /project/projectdirs/cmb/modules/hopper/cmbenv.csh
The next time you log in, the "cmbenv" command will be available. You can specify either the pgi or gnu environments, and then load the cmb module. 
%> cmbenv pgi
%> module load cmb

carver.nersc.gov

On carver, you should add the following lines to your shell resource files:
bash
(~/.bashrc.ext) 
. /project/projectdirs/cmb/modules/carver/cmbenv.sh
(t)csh
(~/.cshrc.ext) 
source /project/projectdirs/cmb/modules/carver/cmbenv.csh
The next time you log in, the "cmbenv" command will be available. You can specify either the pgi or gnu environments, and then load the cmb module. 
%> cmbenv gnu
%> module load cmb

Experiment-Specific Modules

The module system is also a convenient way to provide access to experiment-specific or proprietary software to a defined set of users. The first such module, specifically for Planck users, is already available. You can find documentation about this module here. Other users/experiments interested in this approach should contact us for more information.