Fatal Error in PMPI_Wait?

[fsugrizz]

I've been getting the following error over the pass 3 days in a model run that we've been using daily with no issues for a couple of months now.

Code: Select all

taskid: 9 hostname:node1
Fatal error in PMPI_Wait: Unknown error class, error stack:
PMPI_Wait(203)..................: MPI_Wait(request)=0x4e52440, status=0x7ffff288ffc0) failed
MPIR_Wait_impl(100)
MPIDU_Complete_posted_with_error(1149):Process failed

The cluster still passes MPI_check and I can provide the rest of the logs but I'm just curious if anyone has an idea. I did notice that NASA Sports database changed the ftp location the other day so my gribinfo file was incorrect but I'm still at a loss now.

[Hachi-ait]

Hello,
Did you solve this problem?
Could you please share for me?
I encountered for the same, I run successfully the cluster for several times, using the command:

Code: Select all

mpiexe --machinefile machinefile.txt ./wrf.exe

But now I ran for a larger nesting domain, after 20mins it got the error:

Code: Select all

WRF TILE   1 IS      1 IE    154 JS     41 JE     80
WRF NUMBER OF TILES =   1
Fatal error in PMPI_Wait: Unknown error class, error stack:
PMPI_Wait(216)........................: MPI_Wait(request=0x87a719c, status=0x7ffe675bdca0) failed
MPIR_Wait(112)........................:
MPIDU_Complete_posted_with_error(1137): Process failed

Thanks