Compiling WPS for multicore

Compiling NMM/ARW code for EMS system, upgrading WRF cores on your own, changing EMS scripts to suit your particular needs, and other modifications to original EMS distribution goes in this forum. These are officially unsupported actions.
meteoadriatic
Posts: 1512
Joined: Wed Aug 19, 2009 10:05 am

Compiling WPS for multicore

Post by meteoadriatic » Fri Mar 29, 2013 5:58 pm

Hello,

original metgrid binary from EMS distribution works on all CPU cores. However, the one I make with Intel compiler does not, it uses only one thread.

Does anybody know a trick to make WPS (metgrid) with multicore support?

Thanks

smartie
Posts: 94
Joined: Sat May 21, 2011 7:34 am

Re: Compiling WPS for multicore

Post by smartie » Fri Mar 29, 2013 8:22 pm

Hi,

are you sure you're selecting the correct option? Here's my configure:

Code: Select all

#
#   Settings for Linux x86_64, Intel compiler    (dmpar) 
#
#
COMPRESSION_LIBS    = -L/contrib/jasper/lib -L/opt/freeware/lib -ljasper -lpng -lz
COMPRESSION_INC     = -I/contrib/libpng/include -I/contrib/zlib/include -I/contrib/jasper/include
COMPRESSION_LIBS    = -L/home/david/wrf_misc/Build_WRF_and_WPS/COMPILER_ifort/jasper-install/lib -ljasper \
                      -L/home/david/wrf_misc/Build_WRF_and_WPS/COMPILER_ifort/libpng-install/lib -lpng    \
                      -L/home/david/wrf_misc/Build_WRF_and_WPS/COMPILER_ifort/zlib-install/lib   -lz 
COMPRESSION_INC     = -I/home/david/wrf_misc/Build_WRF_and_WPS/COMPILER_ifort/jasper-install/include      \
                      -I/home/david/wrf_misc/Build_WRF_and_WPS/COMPILER_ifort/libpng-install/include      \
                      -I/home/david/wrf_misc/Build_WRF_and_WPS/COMPILER_ifort/zlib-install/include   
FDEFS               = -DUSE_JPEG2000 -DUSE_PNG
SFC                 = ifort
SCC                 = gcc 
SFC		= ifort 
SCC		= icc 
DM_FC               = mpif90 -f90=ifort
DM_CC               = mpicc -cc=icc
FC                  = $(DM_FC)
CC                  = $(DM_CC) 
LD                  = $(FC)
FFLAGS              = -FR -convert big_endian
F77FLAGS            = -FI -convert big_endian
FCSUFFIX            = 
FNGFLAGS            = $(FFLAGS)
LDFLAGS             = 
CFLAGS              = -w
CPP                 = /lib/cpp -C -P -traditional
CPPFLAGS            = -D_UNDERSCORE -DBYTESWAP -DLINUX -DIO_NETCDF -DIO_BINARY -DIO_GRIB1 -DBIT32 -D_MPI
ARFLAGS             =
CC_TOOLS            = 
As we've been building a lot of slightly different versions I've taken to using this (3.4 only)-
http://www.mmm.ucar.edu/wrf/users/Build ... PS.tar.bz2
It builds everything more or less 'out-of-the-box'
(including multi-core metgrid)

Best,
David

meteoadriatic
Posts: 1512
Joined: Wed Aug 19, 2009 10:05 am

Re: Compiling WPS for multicore

Post by meteoadriatic » Fri Mar 29, 2013 8:46 pm

Thanks David,

is that in code tags your whole configure.wps file or just architecture specific part of it?

smartie
Posts: 94
Joined: Sat May 21, 2011 7:34 am

Re: Compiling WPS for multicore

Post by smartie » Fri Mar 29, 2013 9:01 pm

Here's the whole lot-

Code: Select all

# configure.wps
#
# This file was automatically generated by the configure script in the
# top level directory. You may make changes to the settings in this
# file but be aware they will be overwritten each time you run configure.
# Ordinarily, it is necessary to run configure once, when the code is
# first installed.
#
# To permanently change options, change the settings for your platform
# in the file arch/configure.defaults, the preamble, and the postamble -
# then rerun configure.
#

.SUFFIXES: .F .f .c .o

SHELL           	=       /bin/sh

NCARG_LIBS		=	-L$(NCARG_ROOT)/lib -lncarg -lncarg_gks -lncarg_c \
				-L/usr/X11R6/lib -lX11

NCARG_LIBS2		=	# May be overridden by architecture specific value below

FDEFS			=	-DUSE_JPEG2000 -DUSE_PNG

# Listing of options that are usually independent of machine type.
# When necessary, these are over-ridden by each architecture.

ARFLAGS			=	

PERL			=	perl

RANLIB			=	echo

WRF_DIR			=	../WRFV3

WRF_INCLUDE     =       -I$(WRF_DIR)/external/io_netcdf \
                        -I$(WRF_DIR)/external/io_grib_share \
                        -I$(WRF_DIR)/external/io_grib1 \
                        -I$(WRF_DIR)/external/io_int \
                        -I$(WRF_DIR)/inc

WRF_LIB         =       -L$(WRF_DIR)/external/io_grib1 -lio_grib1 \
                        -L$(WRF_DIR)/external/io_grib_share -lio_grib_share \
                        -L$(WRF_DIR)/external/io_int -lwrfio_int \
                        -L$(WRF_DIR)/external/io_netcdf -lwrfio_nf \
                        -L$(NETCDF)/lib -lnetcdff -lnetcdf

#### Architecture specific settings ####

COMPRESSION_LIBS	=  # intentionally left blank, fill in COMPRESSION_LIBS below

COMPRESSION_INC		=  # intentionally left blank, fill in COMPRESSION_INC below

#
#   Settings for Linux x86_64, Intel compiler    (dmpar) 
#
#
COMPRESSION_LIBS    = -L/contrib/jasper/lib -L/opt/freeware/lib -ljasper -lpng -lz
COMPRESSION_INC     = -I/contrib/libpng/include -I/contrib/zlib/include -I/contrib/jasper/include
COMPRESSION_LIBS    = -L/home/david/wrf_misc/Build_WRF_and_WPS/COMPILER_ifort/jasper-install/lib -ljasper \
                      -L/home/david/wrf_misc/Build_WRF_and_WPS/COMPILER_ifort/libpng-install/lib -lpng    \
                      -L/home/david/wrf_misc/Build_WRF_and_WPS/COMPILER_ifort/zlib-install/lib   -lz 
COMPRESSION_INC     = -I/home/david/wrf_misc/Build_WRF_and_WPS/COMPILER_ifort/jasper-install/include      \
                      -I/home/david/wrf_misc/Build_WRF_and_WPS/COMPILER_ifort/libpng-install/include      \
                      -I/home/david/wrf_misc/Build_WRF_and_WPS/COMPILER_ifort/zlib-install/include   
FDEFS               = -DUSE_JPEG2000 -DUSE_PNG
SFC                 = ifort
SCC                 = gcc 
SFC		= ifort 
SCC		= icc 
DM_FC               = mpif90 -f90=ifort
DM_CC               = mpicc -cc=icc
FC                  = $(DM_FC)
CC                  = $(DM_CC) 
LD                  = $(FC)
FFLAGS              = -FR -convert big_endian
F77FLAGS            = -FI -convert big_endian
FCSUFFIX            = 
FNGFLAGS            = $(FFLAGS)
LDFLAGS             = 
CFLAGS              = -w
CPP                 = /lib/cpp -C -P -traditional
CPPFLAGS            = -D_UNDERSCORE -DBYTESWAP -DLINUX -DIO_NETCDF -DIO_BINARY -DIO_GRIB1 -DBIT32 -D_MPI
ARFLAGS             =
CC_TOOLS            = 

########################################################################################################################
#
#	Macros, these should be generic for all machines

LN		=	ln -sf
MAKE		=	make -i -r
RM		= 	/bin/rm -f
CP		= 	/bin/cp
AR		=	ar ru

.IGNORE:
.SUFFIXES: .c .f .F .o

#	There is probably no reason to modify these rules

.c.o:
	$(RM) $@
	$(CC) $(CPPFLAGS) $(CFLAGS) -c $<	

.f.o:
	$(RM) $@ $*.mod
	$(FC) $(F77FLAGS) -c $< $(WRF_INCLUDE)

.F.o:
	$(RM) $@ $*.mod
	$(CPP) $(CPPFLAGS) $(FDEFS) $(WRF_INCLUDE) $< > $*.f90
	$(FC) $(FFLAGS) -c $*.f90 $(WRF_INCLUDE)
#	$(RM) $*.f90
David

meteoadriatic
Posts: 1512
Joined: Wed Aug 19, 2009 10:05 am

Re: Compiling WPS for multicore

Post by meteoadriatic » Fri Mar 29, 2013 9:18 pm

Yes sure. It is not problem in configure file. It must be something else a fault. I didn't tried this whole building system yet...

Gippox
Posts: 59
Joined: Sat Mar 02, 2013 1:16 am

Re: Compiling WPS for multicore

Post by Gippox » Tue Nov 12, 2013 10:11 am

Hello Meteoadriatic i'm trying to compile wps 3.5.1 with the intel compiler and mpich3 on a new machine with Intel64 architecture i7 4770k
On wrfems 3.4.1.13.43 i managed to compile the solver wrf 3.5.1 and this works well, however for wps no way to make it go, metgrid crashes continuously... :cry:
This is the log of the error (prep_metgrid.log)

Code: Select all

 
forrtl: severe (17): syntax error in NAMELIST input, unit 10, file /home/andrea/wrfems/runs/arw/namelist.wps, line 17, position 18
Image              PC                Routine            Line        Source             
metgrid            0000000000652852  Unknown               Unknown  Unknown
metgrid            000000000040CB26  Unknown               Unknown  Unknown
metgrid            000000000041177D  Unknown               Unknown  Unknown
metgrid            000000000040A276  Unknown               Unknown  Unknown
libc.so.6          0000003E91621B45  Unknown               Unknown  Unknown
metgrid            000000000040A169  Unknown               Unknown  Unknown
forrtl: severe (17): syntax error in NAMELIST input, unit 10, file /home/andrea/wrfems/runs/arw/namelist.wps, line 17, position 18
Image              PC                Routine            Line        Source             
metgrid            0000000000652852  Unknown               Unknown  Unknown
metgrid            000000000040CB26  Unknown               Unknown  Unknown
metgrid            000000000041177D  Unknown               Unknown  Unknown
metgrid            000000000040A276  Unknown               Unknown  Unknown
libc.so.6          0000003E91621B45  Unknown               Unknown  Unknown
metgrid            000000000040A169  Unknown               Unknown  Unknown
forrtl: severe (17): syntax error in NAMELIST input, unit 10, file /home/andrea/wrfems/runs/arw/namelist.wps, line 17, position 18
Image              PC                Routine            Line        Source             
metgrid            0000000000652852  Unknown               Unknown  Unknown
metgrid            000000000040CB26  Unknown               Unknown  Unknown
metgrid            000000000041177D  Unknown               Unknown  Unknown
metgrid            000000000040A276  Unknown               Unknown  Unknown
libc.so.6          0000003E91621B45  Unknown               Unknown  Unknown
metgrid            000000000040A169  Unknown               Unknown  Unknown
I have modified these scripts to force the use of mpich3
.cshrc

Code: Select all

  

#!/bin/tcsh
#-------------------------------------------------------
#
# cshrc file for Common usage
#
# Please read all the comments below and when requested,
# make changes to reflect your system set-up.
#
# Log
# R.Rozumalski   January 2009 Added for WRF EMS support
#-------------------------------------------------------
#
# Note that the man page search path is set in the /etc/man.config
#

setenv EDITOR /usr/bin/nano
#
# Set the default file protection to be mode 0644 (-rw-r--r--)
#
umask 002

#
# Set the path
#
set path=( . $HOME/bin /usr/bin /usr/local/bin /usr/X11R6/bin /usr/bin/X11 \
             /bin /usr/sbin /sbin /etc )


source /opt/intel/bin/compilervars.csh intel64
setenv LD_LIBRARY_PATH /home/andrea/wrfems/util/NETCDF/lib:$LIBRARY_PATH
setenv LD_LIBRARY_PATH /opt/intel/bin:$LD_LIBRARY_PATH
set path=(/opt/mpich2/bin:$path)

# Source EMS.cshrc
#
if (-f /home/andrea/wrfems/etc/EMS.cshrc) source /home/andrea/wrfems/etc/EMS.cshrc

#  Uncomment and modify if run directory something other than default wrfems/runs.
#  
#  setenv EMS_RUN <your run directory>

#
# Various definitions
#
set filec
set cdpath = ( ~ )
set notify
set history = 200 
set savehist = 200
limit coredumpsize 0
unset limits
limit stacksize unlimited


# Set a user prompt
#
set host=`hostname | cut -d"." -f1`
set prompt="${USER}@${host}-> "

# Set a few aliases
#
alias dir       'ls -F'
alias la        'ls -a'
alias ll        'ls -lt'
alias lla       'ls -la'
alias ls        'ls -CF'
alias c         clear
alias sl        ls
alias h         history
alias hh        'history -h'
alias .         'echo $cwd'
alias ..        'set dot=$cwd;cd ..'
alias cd        'set old=$cwd;chdir \!*;pwd'
alias lwd       'set lwd=$old;set old=$cwd;cd $lwd; unset lwd '
ems_util/ems_mpi.pm

Code: Select all

#  Return the mpich2 portion of the command to run
#
my $mpiexec = $system{mpich2}{hydra} ? "/opt/mpich3/bin/mpiexec $hostfile" : "/opt/mpich /bin/mpiexec.hydra"; 
$mpiexec = "$mpiexec -n $system{ncpus} $iface";
ems_util/ems_mpich.pm

Code: Select all

#  Return the mpich2 portion of the command to run 
#
my $mpiexec = $process{mpich2}{hydra} ? "/opt/mpich3/bin/mpiexec $hostfile" : "/opt/mpich3/bin/mpiexec.hydra";
$mpiexec = "$mpiexec -n $process{ncpus} $iface";
/home/andrea/wrfems/strc/ems_prep/Pinterp.pm

Code: Select all

my $mpiexec = $hydra ? "$ENV{EMS_MPI}/bin/mpiexec" : "/opt/mpich3/bin/mpiexec.";
my $exe     = $hydra ? "$mpiexec -exitinfo -n $ncpus $metgrid" : $maxtime ? "$mpiexec -exitinfo -n $ncpus -maxtime $maxtime $metgrid" : "$mpiexec -exitinfo -n $ncpus $metgrid";
On a single-processor IA32 old machine and wrfems 3.2 i was able to work everything.
Did I forget something?

Thank you very much.

Gippox

meteoadriatic
Posts: 1512
Joined: Wed Aug 19, 2009 10:05 am

Re: Compiling WPS for multicore

Post by meteoadriatic » Tue Nov 12, 2013 3:43 pm

If I remember correctly, metgrid 3.5 built with Intel and put in EMS environment will crash if some of these (or both) are not satisfied:

a) original ungrib is not replaced with 3.5 version
b) mpiexec is not replaced with Intel version

Try that first and report if you're still unsuccessful.

P.S. Probably unrelated to your issue, but still as suggestion... I found much more trouble upgrading EMS v3.4 then older versions. I had lot of headache with this and simply gave up of this adventure. For example, you WILL NOT have precipitation outputted in grib files from EMSUPP if you put your own WRF binaries. To fix that you will have to either patch EMS v3.5 with wrfems (3.4!) patches (almost impossible if you aren't VERY familiar with WRF source code, or, build your own UPP. But, if you build your own UPP, you won't be able to start it easy with ems_post unless you make massive changes in strc scripts. You will have to modify some other things if I remember correctly.

So, if you don't require specific v3.4 EMS functions, better use EMS v3.1 for custom upgrades, because it is very easy to modify and works almost out of the box. You just need to build UPP and start it on your own (I can give you script for that, ems_post from v3.1 can't run UPP). Or, just use wrfpost if you don't need any UPP advantage.

Gippox
Posts: 59
Joined: Sat Mar 02, 2013 1:16 am

Re: Compiling WPS for multicore

Post by Gippox » Tue Nov 12, 2013 7:01 pm

Hi Meteo thanks for the suggestions ! :)
I managed to run WPS 3.5.1, i commented out this line in / wrfems / STRC / ems_prep / Pinterp:

# " process_only_bdy = $EMSprep{MASTERNL}{METGRID}{process_only_bdy}[0]\n",

For the moment (i still have to go to the post processing and displaying with Grads) also Ungrib from WPS 3.5.1 works.
Only to WPS, i used a version of MPICH2 to compile mpiexec.gf (instead mpich3 for WRF).
In this way i can use mpich2 compiled for intel. At the moment mpiexec.hydra does not work with WPS.

I having a new PC and i'm reinstalling everything :mrgreen:
How could i install wrfems 3.2, with the same installer of wrfems 3.4.1 ? This old version is still present on the server?

Thanks

Gippox
Posts: 59
Joined: Sat Mar 02, 2013 1:16 am

Re: Compiling WPS for multicore

Post by Gippox » Tue Nov 12, 2013 7:30 pm

How do you say, i have just seen that precipitation and other fields are not plotted with UPP and wrems 3.4.1 ... :(
I think i will do as you said and will install wrfems 3.1 or 3.2 (which is better in your opinion?) which are actually easier to hack.
If you explain to me how to install UPP on wrfems 3.2 i would be very grateful ! :D
Last edited by Gippox on Tue Nov 12, 2013 7:39 pm, edited 1 time in total.

meteoadriatic
Posts: 1512
Joined: Wed Aug 19, 2009 10:05 am

Re: Compiling WPS for multicore

Post by meteoadriatic » Tue Nov 12, 2013 7:35 pm

Gippox wrote:How could i install wrfems 3.2, with the same installer of wrfems 3.4.1 ? This old version is still present on the server?
I have old installations on my computers and in backups, I just copy that to every new computer I want. No installation routine... then edit what is different and it is done.

Yes, you can't have precipitation (and some other fields) because something is coded differently in wrfout files for EMS UPP version... not compatible with original WRF code (neither with 3.4.1 if you compile it without EMS patches).

OK, for UPP, give me some time, I'll open new thread for that. You can use your own UPP in EMS 3.4 too, if 3.4 works OK for you.

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests