Model-Forum for Levante

For an overview of the WRF modeling system, documentation and in-depth user support, please refer to the WRF Model Users' Page.¶

Building WRF¶

On Levante the spack package manager is used to install software under /sw/spack-levante.
We describe here how to use spack to search the software tree for compatible sets of external
libraries (e.g. MPI, netCDF, HDF5 etc.) required by WRF.

Intel compiler and Open MPI¶

The following script demonstrates how to build WRF on Levante using the Intel compiler and Open MPI.

#!/bin/bash
module purge
module load intel-oneapi-compilers/2022.0.1-gcc-11.2.0
module load openmpi/4.1.2-intel-2021.5.0
module load hdf5/1.12.1-openmpi-4.1.2-intel-2021.5.0
module load netcdf-c/4.8.1-openmpi-4.1.2-intel-2021.5.0
module load netcdf-fortran/4.5.3-openmpi-4.1.2-intel-2021.5.0
module load flex
spack load bison@3.8.2%intel@2021.5.0

export LC_ALL=en_US.utf8
export NETCDF=/sw/spack-levante/netcdf-fortran-4.5.3-k6xq5g
export NETCDF_C=/sw/spack-levante/netcdf-c-4.8.1-2k3cmu
export HDF5=/sw/spack-levante/hdf5-1.12.1-tvymb5
export PHDF5=$HDF5

export FLEX_LIB_DIR=/usr/lib64
export YACC='yacc -y -d'
#export JASPERINC=/usr/include
#export JASPERLIB=/usr/lib64

export WRFIO_NCD_LARGE_FILE_SUPPORT=1
export NETCDF4=1
unset NETCDF_classic
export WRF_EM_CORE=1
export WRF_NMM_CORE=0
#export WRF_CHEM=1
#export WRF_KPP=1

echo 'Cleaning WRF ...'
./clean -a

# patch rule for building nc4_test
sed >GNUmakefile <<'EOF' -e '3s/^/\t/'
include Makefile
nc4_test:
@cd tools ; /bin/rm -f nc4_test.{exe,nc,o} ; $(SCC) -o nc4_test.exe nc4_test.c -I$(NETCDF_C)/include -L$(NETCDF_C)/lib -lnetcdf ; cd ..
EOF

echo 'Configure WRF ...'
#./configure -D<<EOF
./configure<<EOF
15
1
EOF
rm -f GNUmakefile

# modify macros to use rpath while linking and rename libhdf5hl_fortran to libhdf5_hl_fortran
sed -i -e "s@-lnetcdff@-Wl,-rpath,${NETCDF}/lib -lnetcdff@" \
       -e "s@-lhdf5hl_fortran@-Wl,-rpath,${HDF5}/lib -lhdf5_hl_fortran@" configure.wrf

echo 'Compile WRF ...'
rm -f log_compile
export J=16
./compile em_real 2>&1 | tee -a log_compile

You can put the above script into your WRF root directory and execute it like this:

chmod u+x build_WRF_levante_openmpi_intel.sh
./build_WRF_levante_openmpi_intel.sh

Intel compiler and Intel MPI¶

The below is a script to build with Intel MPI instead of Open MPI. Please note the changed environment modules and library directories.

#!/bin/bash
module purge
module load intel-oneapi-compilers/2022.0.1-gcc-11.2.0
module load intel-oneapi-mpi/2021.5.0-gcc-11.2.0
module load hdf5/1.12.1-intel-oneapi-mpi-2021.5.0-intel-2021.5.0
module load netcdf-c/4.8.1-intel-oneapi-mpi-2021.5.0-intel-2021.5.0
module load netcdf-fortran/4.5.3-intel-oneapi-mpi-2021.5.0-intel-2021.5.0
module load flex
spack load bison@3.8.2%intel@2021.5.0

export LC_ALL=en_US.utf8
export NETCDF=/sw/spack-levante/netcdf-fortran-4.5.3-r5r3ev
export NETCDF_C=/sw/spack-levante/netcdf-c-4.8.1-7dq6g2
export HDF5=/sw/spack-levante/hdf5-1.12.1-jmeuy3
export PHDF5=$HDF5

export FLEX_LIB_DIR=/usr/lib64
export YACC='yacc -y -d'
#export JASPERINC=/usr/include
#export JASPERLIB=/usr/lib64

export WRFIO_NCD_LARGE_FILE_SUPPORT=1
export NETCDF4=1
unset NETCDF_classic
export WRF_EM_CORE=1
export WRF_NMM_CORE=0
#export WRF_CHEM=1
#export WRF_KPP=1

echo 'Cleaning WRF ...'
./clean -a

# patch rule for building nc4_test
sed >GNUmakefile <<'EOF' -e '3s/^/\t/'
include Makefile
nc4_test:
@cd tools ; /bin/rm -f nc4_test.{exe,nc,o} ; $(SCC) -o nc4_test.exe nc4_test.c -I$(NETCDF_C)/include -L$(NETCDF_C)/lib -lnetcdf ; cd ..
EOF

echo 'Configure WRF ...'
./configure<<EOF
15
1
EOF
rm -f GNUmakefile

# modify macros to use rpath while linking and rename libhdf5hl_fortran to libhdf5_hl_fortran
sed -i -e "s@-lnetcdff@-Wl,-rpath,${NETCDF}/lib -lnetcdff@" \
       -e "s@-lhdf5hl_fortran@-Wl,-rpath,${HDF5}/lib -lhdf5_hl_fortran@" configure.wrf

echo 'Compile WRF ...'
rm -f log_compile
export J=16
./compile em_real 2>&1 | tee -a log_compile

Other compilers¶

Building the model with GCC is also possible. You can use spack to find appropriate compatible libraries.

Running WRF¶

Please note, no test of run-time performance or appropriateness of the above settings has been made by DKRZ staff so far.
We received some user reports that the stack size needs to be adjusted for a successful run. You can start with the unlimited size stack:

ulimit -a unlimited

and then reduce it to an appropriate value. Unfortunately, there are no simple methods to estimate the stack size required for program execution.

For more information, please see Example Batch Scripts and recommended MPI runtime settings.

Requesting help from DKRZ¶

Please, try to write up the steps needed to reproduce the problem you are encountering. In most cases we would ask you for the information below for further analysis:

1. Source code
2. Compiler and MPI combination used
3. Run script
4. Access to input data needed to make a test run