How to run WRF-SFIRE on Janus

From CCM
Jump to: navigation, search
This page contains instructions for the Wildfire modeling project. See also How to use the Janus cluster for general information.

Contents

Basics

Your /home directory has a tiny quota not suitable for even building a couple of copies of the code. It is best to build your codes in /projects which has 250GB quota. Data intensive runs should run in /lustre/janus_scratch. It is best to commit all code mods the remote git repository (which you should do anyway) because /projects is not as redundant as /home. See Janus storage for details and current quotas.

Set up the environment for one of the compilers below (because this also sets path for git). I like to put in my .bashrc file the lines

alias intel="source /projects/jmandel/intel.rc"
alias gnu="source /projects/jmandel/gnu.rc"

Then I can switch between the GNU and Intel environments just by typing gnu or intel. This needs to be done in every login session.

Follow How to get WRF-Fire to get the software. Developers use

git clone ssh://yourusername@repo.openwfm.org/home/git/wrf-fire

GNU compilers

Set up the environment

source /projects/jmandel/gnu.rc

Here is a copy of gcc.rc file

Build the code

Parallel build

  • In configure, choose gfortran with gcc , and DM
    • SM = OpenMP is not recommended, slower. The only reason to compile with SM+DM may be when you would run out of memory by running 12 MPI processes per node. WRF-Chem does not support SM. If you do compile with SM+DM and want to use fewer threads than 12 per node, add to your mpirun command flag like -x OMP_NUM_THREADS=1 - you have to do this when running ideal.exe in any case, because it does not support multiple threads
  • After configuration, edit manually the file configure.wrf and add to CFLAGS_LOCAL the flag -DMPI2_SUPPORT
  • Add source /projects/jmandel/gnu.rc to your PBS script in case you change compilers between build and run
  • Submit by qsub -q janus-normal

Intel compilers

Set up the environment

source /projects/jmandel/intel.rc

Here is a copy of intel.rc file

Build the code

Parallel build

  • In configure, choose ifort with icc , and DM
    • SM = OpenMP is not recommended, slower. The only reason to compile with SM+DM may be when you would run out of memory by running 12 MPI processes per node. WRF-Chem does not support SM. If you do compile with SM+DM and want to use fewer threads than 12 per node, add to your mpirun command flag like -x OMP_NUM_THREADS=1 - you have to do this when running ideal.exe in any case, because it does not support multiple threads
  • You can add source /projects/jmandel/intel.rc to your PBS script in case you change compilers between build and run
  • Submit by qsub -q janus-normal

PGI compilers

A build with PGI compilers is not set up at the moment, please check later.

Debugging

Use Intel or PGI if you can, and run serial. GNU fortran debugging is terrible. See for example http://people.sc.fsu.edu/~jburkardt/f_src/gdb/gdb.html http://www.gnu.org/software/gdb http://gcc.gnu.org/wiki/GFortran http://gcc.gnu.org/wiki/GfortranFAQ

The GNU debugger is gdb. For Intel, use idbc for command line. idb with try to start X Window System gui. Bad idea over the network.

In any case, use configure -d to build.

If you must debug in MPI,

  • uncomment #define DEBUG_HOOK in module_fr_sfire_util.F and rebuild
  • Start the code using qsub as usual. It will keep spinning in infinite loop, waiting for you.
  • Find out what nodes it runs on using qstat -n, start a new terminal window for each process, ssh to the nodes where the process running, find out the process number by ps -ef, and attach to it using the debugger with -p and the process number.
  • You should find the debugger stopped in the system sleep routine. For each process, do it its debug session
up 2
set go=1
cont

See also

Personal tools
Namespaces

Variants
Actions
Navigation
Toolbox