Write failed: broken pipe on cluster

All issues/questions about EMS v3.4 package, please ask here.
Post Reply
metzag
Posts: 3
Joined: Thu Dec 12, 2013 11:40 am

Write failed: broken pipe on cluster

Post by metzag » Thu Apr 24, 2014 11:43 am

We have wrf running on a two machine cluster (made by following instructions on this thread viewtopic.php?p=1258 ). Most of the time it works great but sometimes the run fails. Thinking it's because of ssh closing idle connections, I set the sshd_config file to have ClientAliveInterval 120
ClientAliveCountMax 3
but the problem still shows up. Any idea what might be causing it?

This is the log:

starting wrf task 0 of 16
starting wrf task 9 of 16
starting wrf task starting wrf task 11 of 16
starting wrf task 1 of 16
starting wrf task 12 of 16
starting wrf task 2 of 16
starting wrf task 3 of 16
starting wrf task 8 of 16
starting wrf task 4 of 16
starting wrf task 5 of 16
6 of 16
starting wrf task 7 of 16
starting wrf task 10 of 16
starting wrf task 13 of 16
starting wrf task 15 of 16
starting wrf task 14 of 16
error_dup: cannot open rsl.out.nnnn: Permission denied
...sending output to standard output and continuing.
error_dup: cannot open rsl.out.nnnn: Permission denied
...sending output to standard output and continuing.
error_dup: cannot open rsl.out.nnnn: Permission denied
...sending output to standard output and continuing.
error_dup: cannot open rsl.out.nnnn: Permission denied
...sending output to standard output and continuing.
error_dup: cannot open rsl.out.nnnn: Permission denied
...sending output to standard output and continuing.
error_dup: cannot open rsl.out.nnnn: Permission denied
...sending output to standard output and continuing.
error_dup: cannot open rsl.out.nnnn: Permission denied
...sending output to standard output and continuing.
error_dup: cannot open rsl.out.nnnn: Permission denied
...sending output to standard output and continuing.
Quilting with 1 groups of 0 I/O tasks.
Quilting with 1 groups of 0 I/O tasks.
Quilting with 1 groups of 0 I/O tasks.
Quilting with 1 groups of 0 I/O tasks.
Quilting with 1 groups of 0 I/O tasks.
Quilting with 1 groups of 0 I/O tasks.
Quilting with 1 groups of 0 I/O tasks.
Quilting with 1 groups of 0 I/O tasks.
Quilting with 1 groups of 0 I/O tasks.
Quilting with 1 groups of 0 I/O tasks.
Quilting with 1 groups of 0 I/O tasks.
Quilting with 1 groups of 0 I/O tasks.
Quilting with 1 groups of 0 I/O tasks.
Quilting with 1 groups of 0 I/O tasks.
Quilting with 1 groups of 0 I/O tasks.
Quilting with 1 groups of 0 I/O tasks.
Error reading namelist &logging from namelist.input. Using default logging config.
Error reading namelist &logging from namelist.input. Using default logging config.
starting wrf task 14 of 16
Error reading namelist &logging from namelist.input. Using default logging config.
Error reading namelist &logging from namelist.input. Using default logging config.
Error reading namelist &logging from namelist.input. Using default logging config.
Error reading namelist &logging from namelist.input. Using default logging config.
Error reading namelist &logging from namelist.input. Using default logging config.
starting wrf task 12 of 16
Error reading namelist &logging from namelist.input. Using default logging config.
starting wrf task 15 of 16
starting wrf task 10 of 16
Error reading namelist &logging from namelist.input. Using default logging config.
Error reading namelist &logging from namelist.input. Using default logging config.
Error reading namelist &logging from namelist.input. Using default logging config.
Error reading namelist &logging from namelist.input. Using default logging config.
starting wrf task Error reading namelist &logging from namelist.input. Using default logging config.
Error reading namelist &logging from namelist.input. Using default logging config.
Error reading namelist &logging from namelist.input. Using default logging config.
starting wrf task 8 of 16
starting wrf task 9 of 16
Error reading namelist &logging from namelist.input. Using default logging config.
13 of 16
error_dup: cannot open rsl.out.nnnn: Permission denied
...sending output to standard output and continuing.
starting wrf task 11 of 16
error_dup: cannot open rsl.out.nnnn: Permission denied
...sending output to standard output and continuing.
Error reading namelist &logging from namelist.input. Using default logging config.
Error reading namelist &logging from namelist.input. Using default logging config.
error_dup: cannot open rsl.out.nnnn: Permission denied
...sending output to standard output and continuing.
Error reading namelist &logging from namelist.input. Using default logging config.
Error reading namelist &logging from namelist.input. Using default logging config.
error_dup: cannot open rsl.out.nnnn: Permission denied
...sending output to standard output and continuing.
Error reading namelist &logging from namelist.input. Using default logging config.
Error reading namelist &logging from namelist.input. Using default logging config.
error_dup: cannot open rsl.out.nnnn: Permission denied
...sending output to standard output and continuing.
Error reading namelist &logging from namelist.input. Using default logging config.
Error reading namelist &logging from namelist.input. Using default logging config.
error_dup: cannot open rsl.out.nnnn: Permission denied
...sending output to standard output and continuing.
Error reading namelist &logging from namelist.input. Using default logging config.
Error reading namelist &logging from namelist.input. Using default logging config.
error_dup: cannot open rsl.out.nnnn: Permission denied
...sending output to standard output and continuing.
Error reading namelist &logging from namelist.input. Using default logging config.
Error reading namelist &logging from namelist.input. Using default logging config.
error_dup: cannot open rsl.out.nnnn: Permission denied
...sending output to standard output and continuing.
Error reading namelist &logging from namelist.input. Using default logging config.
Error reading namelist &logging from namelist.input. Using default logging config.
Error reading namelist &logging from namelist.input. Using default logging config.
Error reading namelist &logging from namelist.input. Using default logging config.
Ntasks in X 4 , ntasks in Y 4
Ntasks in X 4 , ntasks in Y 4
Ntasks in X 4 , ntasks in Y 4
Ntasks in X 4 , ntasks in Y 4
Ntasks in X 4 , ntasks in Y 4
Ntasks in X 4 , ntasks in Y 4
Ntasks in X 4 , ntasks in Y 4
Ntasks in X 4 , ntasks in Y 4
Ntasks in X 4 , ntasks in Y 4
Ntasks in X 4 , ntasks in Y 4
Ntasks in X 4 , ntasks in Y 4
Ntasks in X 4 , ntasks in Y 4
Ntasks in X 4 , ntasks in Y 4
Ntasks in X 4 , ntasks in Y 4
Ntasks in X 4 , ntasks in Y 4
Ntasks in X 4 , ntasks in Y 4
--- NOTE: sst_update is 0, setting io_form_auxinput4 = 0 and auxinput4_interval = 0 for all domains
--- NOTE: sst_update is 0, setting io_form_auxinput4 = 0 and auxinput4_interval = 0 for all domains
--- NOTE: sst_update is 0, setting io_form_auxinput4 = 0 and auxinput4_interval = 0 for all domains
--- NOTE: grid_fdda is 0 for domain 1, setting gfdda interval and ending time to 0 for that domain.
--- NOTE: sst_update is 0, setting io_form_auxinput4 = 0 and auxinput4_interval = 0 for all domains
--- NOTE: grid_fdda is 0 for domain 1, setting gfdda interval and ending time to 0 for that domain.
--- NOTE: sst_update is 0, setting io_form_auxinput4 = 0 and auxinput4_interval = 0 for all domains
--- NOTE: grid_fdda is 0 for domain 1, setting gfdda interval and ending time to 0 for that domain.
--- NOTE: sst_update is 0, setting io_form_auxinput4 = 0 and auxinput4_interval = 0 for all domains
--- NOTE: grid_fdda is 0 for domain 1, setting gfdda interval and ending time to 0 for that domain.
--- NOTE: both grid_sfdda and pxlsm_soil_nudge are 0 for domain 1, setting sgfdda interval and ending time to 0 for that domain.
--- NOTE: obs_nudge_opt is 0 for domain 1, setting obs nudging interval and ending time to 0 for that domain.
--- NOTE: sst_update is 0, setting io_form_auxinput4 = 0 and auxinput4_interval = 0 for all domains
--- NOTE: grid_fdda is 0 for domain 1, setting gfdda interval and ending time to 0 for that domain.
--- NOTE: both grid_sfdda and pxlsm_soil_nudge are 0 for domain 1, setting sgfdda interval and ending time to 0 for that domain.
--- NOTE: obs_nudge_opt is 0 for domain 1, setting obs nudging interval and ending time to 0 for that domain.
--- NOTE: grid_fdda is 0 for domain 1, setting gfdda interval and ending time to 0 for that domain.
--- NOTE: both grid_sfdda and pxlsm_soil_nudge are 0 for domain 1, setting sgfdda interval and ending time to 0 for that domain.
--- NOTE: obs_nudge_opt is 0 for domain 1, setting obs nudging interval and ending time to 0 for that domain.
--- NOTE: sst_update is 0, setting io_form_auxinput4 = 0 and auxinput4_interval = 0 for all domains
--- NOTE: grid_fdda is 0 for domain 1, setting gfdda interval and ending time to 0 for that domain.
--- NOTE: both grid_sfdda and pxlsm_soil_nudge are 0 for domain 1, setting sgfdda interval and ending time to 0 for that domain.
--- NOTE: obs_nudge_opt is 0 for domain 1, setting obs nudging interval and ending time to 0 for that domain.
--- NOTE: num_soil_layers has been set to 4 --- NOTE: sst_update is 0, setting io_form_auxinput4 = 0 and auxinput4_interval = 0 for all domains
--- NOTE: grid_fdda is 0 for domain 1, setting gfdda interval and ending time to 0 for that domain.
--- NOTE: both grid_sfdda and pxlsm_soil_nudge are 0 for domain 1, setting sgfdda interval and ending time to 0 for that domain.
--- NOTE: obs_nudge_opt is 0 for domain 1, setting obs nudging interval and ending time to 0 for that domain.
--- NOTE: sst_update is 0, setting io_form_auxinput4 = 0 and auxinput4_interval = 0 for all domains
--- NOTE: grid_fdda is 0 for domain 1, setting gfdda interval and ending time to 0 for that domain.
--- NOTE: both grid_sfdda and pxlsm_soil_nudge are 0 for domain 1, setting sgfdda interval and ending time to 0 for that domain.
--- NOTE: obs_nudge_opt is 0 for domain 1, setting obs nudging interval and ending time to 0 for that domain.
--- NOTE: grid_fdda is 0 for domain 1, setting gfdda interval and ending time to 0 for that domain.
--- NOTE: both grid_sfdda and pxlsm_soil_nudge are 0 for domain 1, setting sgfdda interval and ending time to 0 for that domain.
--- NOTE: obs_nudge_opt is 0 for domain 1, setting obs nudging interval and ending time to 0 for that domain.
--- NOTE: num_soil_layers has been set to 4
WRF V3.4.1 MODEL
--- NOTE: both grid_sfdda and pxlsm_soil_nudge are 0 for domain 1, setting sgfdda interval and ending time to 0 for that domain.
--- NOTE: obs_nudge_opt is 0 for domain 1, setting obs nudging interval and ending time to 0 for that domain.
--- NOTE: num_soil_layers has been set to 4
WRF V3.4.1 MODEL
--- NOTE: sst_update is 0, setting io_form_auxinput4 = 0 and auxinput4_interval = 0 for all domains
--- NOTE: grid_fdda is 0 for domain 1, setting gfdda interval and ending time to 0 for that domain.
--- NOTE: both grid_sfdda and pxlsm_soil_nudge are 0 for domain 1, setting sgfdda interval and ending time to 0 for that domain.
--- NOTE: obs_nudge_opt is 0 for domain 1, setting obs nudging interval and ending time to 0 for that domain.
--- NOTE: num_soil_layers has been set to 4
WRF V3.4.1 MODEL
--- NOTE: both grid_sfdda and pxlsm_soil_nudge are 0 for domain 1, setting sgfdda interval and ending time to 0 for that domain.
--- NOTE: obs_nudge_opt is 0 for domain 1, setting obs nudging interval and ending time to 0 for that domain.
--- NOTE: num_soil_layers has been set to 4
WRF V3.4.1 MODEL
--- NOTE: both grid_sfdda and pxlsm_soil_nudge are 0 for domain 1, setting sgfdda interval and ending time to 0 for that domain.
--- NOTE: obs_nudge_opt is 0 for domain 1, setting obs nudging interval and ending time to 0 for that domain.
--- NOTE: num_soil_layers has been set to 4
WRF V3.4.1 MODEL
--- NOTE: sst_update is 0, setting io_form_auxinput4 = 0 and auxinput4_interval = 0 for all domains
--- NOTE: grid_fdda is 0 for domain 1, setting gfdda interval and ending time to 0 for that domain.
--- NOTE: both grid_sfdda and pxlsm_soil_nudge are 0 for domain 1, setting sgfdda interval and ending time to 0 for that domain.
--- NOTE: obs_nudge_opt is 0 for domain 1, setting obs nudging interval and ending time to 0 for that domain.
--- NOTE: num_soil_layers has been set to 4
WRF V3.4.1 MODEL
--- NOTE: sst_update is 0, setting io_form_auxinput4 = 0 and auxinput4_interval = 0 for all domains
--- NOTE: grid_fdda is 0 for domain 1, setting gfdda interval and ending time to 0 for that domain.
--- NOTE: both grid_sfdda and pxlsm_soil_nudge are 0 for domain 1, setting sgfdda interval and ending time to 0 for that domain.
--- NOTE: obs_nudge_opt is 0 for domain 1, setting obs nudging interval and ending time to 0 for that domain.
--- NOTE: num_soil_layers has been set to 4
WRF V3.4.1 MODEL
--- NOTE: num_soil_layers has been set to 4
WRF V3.4.1 MODEL
--- NOTE: num_soil_layers has been set to 4
WRF V3.4.1 MODEL
--- NOTE: sst_update is 0, setting io_form_auxinput4 = 0 and auxinput4_interval = 0 for all domains
--- NOTE: grid_fdda is 0 for domain 1, setting gfdda interval and ending time to 0 for that domain.
--- NOTE: both grid_sfdda and pxlsm_soil_nudge are 0 for domain 1, setting sgfdda interval and ending time to 0 for that domain.
--- NOTE: obs_nudge_opt is 0 for domain 1, setting obs nudging interval and ending time to 0 for that domain.
--- NOTE: num_soil_layers has been set to 4
WRF V3.4.1 MODEL
--- NOTE: num_soil_layers has been set to 4
WRF V3.4.1 MODEL

WRF V3.4.1 MODEL
--- NOTE: num_soil_layers has been set to 4
WRF V3.4.1 MODEL
--- NOTE: num_soil_layers has been set to 4
WRF V3.4.1 MODEL
--- NOTE: sst_update is 0, setting io_form_auxinput4 = 0 and auxinput4_interval = 0 for all domains
*************************************
Parent domain *************************************
*************************************
Parent domain
ids,ide,jds,jde 1 114 1 94
*************************************
Parent domain
*************************************
*************************************
Parent domain
*************************************
Parent domain
ids,ide,jds,jde 1 114 1 94
ims,ime,jms,jme 79 119 41 77
ips,ipe,jps,jpe 86 114 48 70 *************************************
Parent domain
ids,ide,jds,jde 1 114 1 94 *************************************
Parent domain
ids,ide,jds,jde 1 114 1 94
ims,ime,jms,jme 23 64 64 99
ids,ide,jds,jde 1 114 1 94
ims,ime,jms,jme 51 92 64 99
ips,ipe,jps,jpe 58 85 71 94
*************************************
DYNAMICS OPTION: Eulerian Mass Coordinate
*************************************
Parent domain
ids,ide,jds,jde 1 114 1 94
ims,ime,jms,jme 79 119 64 99
ips,ipe,jps,jpe 86 114 71 94
*************************************
DYNAMICS OPTION: Eulerian Mass Coordinate
*************************************
Parent domain
ids,ide,jds,jde 1 114 1 94
ims,ime,jms,jme 23 64 41 77
ips,ipe,jps,jpe 30 57 48 70
*************************************
*************************************
Parent domain
ids,ide,jds,jde 1 114 1 94
ims,ime,jms,jme 51 92 41 77
ips,ipe,jps,jpe 58 85 48 70
*************************************
Parent domain
ids,ide,jds,jde 1 114 1 94
ims,ime,jms,jme 79 119 41 77
ips,ipe,jps,jpe 86 114 48 70
*************************************
DYNAMICS OPTION: Eulerian Mass Coordinate
*************************************
Parent domain
ids,ide,jds,jde 1 114 1 94
ims,ime,jms,jme -4 36 64 99
ips,ipe,jps,jpe 1 29 71 94
*************************************
DYNAMICS OPTION: Eulerian Mass Coordinate
*************************************
Parent domain
ids,ide,jds,jde 1 114 1 94
ims,ime,jms,jme 23 64 64 99
ips,ipe,jps,jpe 30 57 71 94
*************************************
DYNAMICS OPTION: Eulerian Mass Coordinate
ims,ime,jms,jme 51 92 64 99
ips,ipe,jps,jpe 58 85 71 94
*************************************
DYNAMICS OPTION: Eulerian Mass Coordinate
ids,ide,jds,jde 1 114 1 94
ims,ime,jms,jme 79 119 64 99
ips,ipe,jps,jpe 86 114 71 94
*************************************
DYNAMICS OPTION: Eulerian Mass Coordinate
Parent domain
ids,ide,jds,jde 1 114 1 94
ims,ime,jms,jme 23 64 41 77
ips,ipe,jps,jpe 30 57 48 70
*************************************
DYNAMICS OPTION: Eulerian Mass Coordinate
ids,ide,jds,jde 1 114 1 94
ims,ime,jms,jme 51 92 41 77
ips,ipe,jps,jpe 58 85 48 70
*************************************
DYNAMICS OPTION: Eulerian Mass Coordinate

*************************************
DYNAMICS OPTION: Eulerian Mass Coordinate

ims,ime,jms,jme -4 36 64 99
ips,ipe,jps,jpe 1 29 71 94
*************************************
DYNAMICS OPTION: Eulerian Mass Coordinate

ips,ipe,jps,jpe 30 57 71 94
*************************************
DYNAMICS OPTION: Eulerian Mass Coordinate
DYNAMICS OPTION: Eulerian Mass Coordinate
DYNAMICS OPTION: Eulerian Mass Coordinate
--- NOTE: sst_update is 0, setting io_form_auxinput4 = 0 and auxinput4_interval = 0 for all domains
--- NOTE: grid_fdda is 0 for domain 1, setting gfdda interval and ending time to 0 for that domain.
--- NOTE: grid_fdda is 0 for domain 1, setting gfdda interval and ending time to 0 for that domain.
--- NOTE: both grid_sfdda and pxlsm_soil_nudge are 0 for domain 1, setting sgfdda interval and ending time to 0 for that domain.
--- NOTE: both grid_sfdda and pxlsm_soil_nudge are 0 for domain 1, setting sgfdda interval and ending time to 0 for that domain.
--- NOTE: obs_nudge_opt is 0 for domain 1, setting obs nudging interval and ending time to 0 for that domain.
--- NOTE: obs_nudge_opt is 0 for domain 1, setting obs nudging interval and ending time to 0 for that domain.
--- NOTE: num_soil_layers has been set to 4
--- NOTE: num_soil_layers has been set to 4
WRF V3.4.1 MODEL
WRF V3.4.1 MODEL
*************************************
*************************************
Parent domain
Parent domain
ids,ide,jds,jde 1 114 1 94
ids,ide,jds,jde 1 114 1 94
ims,ime,jms,jme -4 36 41 77
ims,ime,jms,jme -4 36 41 77
ips,ipe,jps,jpe 1 29 48 70
ips,ipe,jps,jpe 1 29 48 70
*************************************
*************************************
DYNAMICS OPTION: Eulerian Mass Coordinate
DYNAMICS OPTION: Eulerian Mass Coordinate
alloc_space_field: domain 1 , 47854904 bytes allocated
alloc_space_field: domain 1 , 47854904 bytes allocated
alloc_space_field: domain 1 , 49133360 bytes allocated
alloc_space_field: domain 1 , 49133360 bytes allocated
alloc_space_field: domain 1 , 47986152 bytes allocated
alloc_space_field: domain 1 , 47986152 bytes allocated
alloc_space_field: domain 1 , 48926424 bytes allocated
alloc_space_field: domain 1 , 48926424 bytes allocated
alloc_space_field: domain 1 , 50107104 bytes allocated
alloc_space_field: domain 1 , 50107104 bytes allocated
alloc_space_field: domain 1 , 50107104 bytes allocated
alloc_space_field: domain 1 , 50107104 bytes allocated
alloc_space_field: domain 1 , 48926424 bytes allocated
alloc_space_field: domain 1 , 48926424 bytes allocated
alloc_space_field: domain 1 , 49007424 bytes allocated
alloc_space_field: domain 1 , 49007424 bytes allocated
med_initialdata_input: calling input_input
med_initialdata_input: calling input_input
med_initialdata_input: calling input_input
med_initialdata_input: calling input_input
med_initialdata_input: calling input_input
med_initialdata_input: calling input_input
med_initialdata_input: calling input_input
med_initialdata_input: calling input_input
med_initialdata_input: calling input_input
med_initialdata_input: calling input_input
med_initialdata_input: calling input_input
med_initialdata_input: calling input_input
med_initialdata_input: calling input_input
med_initialdata_input: calling input_input
med_initialdata_input: calling input_input
med_initialdata_input: calling input_input
INPUT LandUse = "USGS"
INPUT LandUse = "USGS"
INPUT LandUse = "USGS"
INPUT LandUse = "USGS"
INPUT LandUse = "USGS"
INPUT LandUse = "USGS"
INPUT LandUse = "USGS"
INPUT LandUse = "USGS"
INPUT LandUse = "USGS"
INPUT LandUse = "USGS"
INPUT LandUse = "USGS"
INPUT LandUse = "USGS"
INPUT LandUse = "USGS"
INPUT LandUse = "USGS"
INPUT LandUse = "USGS"
INPUT LandUse = "USGS"
LANDUSE TYPE = "USGS" FOUND 33 CATEGORIES 2 SEASONS WATER CATEGORY = 16 SNOW CATEGORY = 24
LANDUSE TYPE = "USGS" FOUND 33 CATEGORIES 2 SEASONS WATER CATEGORY = 16 SNOW CATEGORY = 24
LANDUSE TYPE = "USGS" FOUND 33 CATEGORIES 2 SEASONS WATER CATEGORY = 16 SNOW CATEGORY = 24
LANDUSE TYPE = "USGS" FOUND 33 CATEGORIES 2 SEASONS WATER CATEGORY = 16 SNOW CATEGORY = 24
LANDUSE TYPE = "USGS" FOUND 33 CATEGORIES 2 SEASONS WATER CATEGORY = 16 SNOW CATEGORY = 24
LANDUSE TYPE = "USGS" FOUND 33 CATEGORIES 2 SEASONS WATER CATEGORY = 16 SNOW CATEGORY = 24
LANDUSE TYPE = "USGS" FOUND 33 CATEGORIES 2 SEASONS WATER CATEGORY = 16 SNOW CATEGORY = 24
LANDUSE TYPE = "USGS" FOUND 33 CATEGORIES 2 SEASONS WATER CATEGORY = 16 SNOW CATEGORY = 24
LANDUSE TYPE = "USGS" FOUND 33 CATEGORIES 2 SEASONS WATER CATEGORY = 16 SNOW CATEGORY = 24
LANDUSE TYPE = "USGS" FOUND 33 CATEGORIES 2 SEASONS WATER CATEGORY = 16 SNOW CATEGORY = 24
LANDUSE TYPE = "USGS" FOUND 33 CATEGORIES 2 SEASONS WATER CATEGORY = 16 SNOW CATEGORY = 24
LANDUSE TYPE = "USGS" FOUND 33 CATEGORIES 2 SEASONS WATER CATEGORY = 16 SNOW CATEGORY = 24
LANDUSE TYPE = "USGS" FOUND 33 CATEGORIES 2 SEASONS WATER CATEGORY = 16 SNOW CATEGORY = 24
LANDUSE TYPE = "USGS" FOUND 33 CATEGORIES 2 SEASONS WATER CATEGORY = 16 SNOW CATEGORY = 24
LANDUSE TYPE = "USGS" FOUND 33 CATEGORIES 2 SEASONS WATER CATEGORY = 16 SNOW CATEGORY = 24
LANDUSE TYPE = "USGS" FOUND 33 CATEGORIES 2 SEASONS WATER CATEGORY = 16 SNOW CATEGORY = 24
INITIALIZE THREE Noah LSM RELATED TABLES
INITIALIZE THREE Noah LSM RELATED TABLES
INITIALIZE THREE Noah LSM RELATED TABLES
INITIALIZE THREE Noah LSM RELATED TABLES
INITIALIZE THREE Noah LSM RELATED TABLES
INITIALIZE THREE Noah LSM RELATED TABLES
INITIALIZE THREE Noah LSM RELATED TABLES
INITIALIZE THREE Noah LSM RELATED TABLES
INITIALIZE THREE Noah LSM RELATED TABLES
INITIALIZE THREE Noah LSM RELATED TABLES
INITIALIZE THREE Noah LSM RELATED TABLES
INITIALIZE THREE Noah LSM RELATED TABLES
INITIALIZE THREE Noah LSM RELATED TABLES
INITIALIZE THREE Noah LSM RELATED TABLES
INITIALIZE THREE Noah LSM RELATED TABLES
INITIALIZE THREE Noah LSM RELATED TABLES
WRF TILE 1 IS 1 IE 29 JS 48 JE 70
WRF NUMBER OF TILES = 1
WRF TILE 1 IS 1 IE 29 JS 48 JE 70
WRF NUMBER OF TILES = 1
WRF TILE 1 IS 86 IE 114 JS 48 JE 70
WRF NUMBER OF TILES = 1
WRF TILE 1 IS 1 IE 29 JS 71 JE 94
WRF NUMBER OF TILES = 1
WRF TILE 1 IS 30 IE 57 JS 71 JE 94
WRF NUMBER OF TILES = 1
WRF TILE 1 IS 30 IE 57 JS 48 JE 70
WRF NUMBER OF TILES = 1
WRF TILE 1 IS 58 IE 85 JS 48 JE 70
WRF NUMBER OF TILES = 1
WRF TILE 1 IS 86 IE 114 JS 48 JE 70
WRF NUMBER OF TILES = 1
WRF TILE 1 IS 1 IE 29 JS 71 JE 94
WRF NUMBER OF TILES = 1
WRF TILE 1 IS 30 IE 57 JS 71 JE 94
WRF NUMBER OF TILES = 1
WRF TILE 1 IS 58 IE 85 JS 71 JE 94
WRF NUMBER OF TILES = 1
WRF TILE 1 IS 30 IE 57 JS 48 JE 70
WRF NUMBER OF TILES = 1
WRF TILE 1 IS 58 IE 85 JS 48 JE 70
WRF NUMBER OF TILES = 1
WRF TILE 1 IS 58 IE 85 JS 71 JE 94
WRF NUMBER OF TILES = 1
WRF TILE 1 IS 86 IE 114 JS 71 JE 94
WRF TILE 1 IS 86 IE 114 JS 71 JE 94
WRF NUMBER OF TILES = 1
WRF NUMBER OF TILES = 1

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= EXIT CODE: 1
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
Write failed: Broken pipe
[mpiexec@linux-dl08] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting
[mpiexec@linux-dl08] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion
[mpiexec@linux-dl08] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:217): launcher returned error waiting for completion
[mpiexec@linux-dl08] main (./ui/mpich/mpiexec.c:331): process manager error waiting for completion

metzag
Posts: 3
Joined: Thu Dec 12, 2013 11:40 am

Re: Write failed: broken pipe on cluster

Post by metzag » Fri Apr 25, 2014 8:52 am

Nevermind, it turns out one of the machines randomly restarts itself.

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests