32bit wgrib cant read >2GB files (was: Number of points...)

Forum dedicated to older versions of EMS package (WRFEMS v3.2, v3.1 or older). Support is user-to-user based, so please help others if you can.
nebojsa
Posts: 133
Joined: Sun Mar 21, 2010 3:40 pm

32bit wgrib cant read >2GB files (was: Number of points...)

Post by nebojsa » Sat Feb 19, 2011 1:55 pm

Is there any maximum number of points per domain?

Best regards!

Nebojsa

meteoadriatic
Posts: 1587
Joined: Wed Aug 19, 2009 10:05 am

Re: Number of points per domain

Post by meteoadriatic » Sat Feb 19, 2011 7:25 pm

I think there is no such limit, but, your maximum number of points will depend on RAM amount. Bigger domains are - more RAM you need to run model.

nebojsa
Posts: 133
Joined: Sun Mar 21, 2010 3:40 pm

Re: Number of points per domain

Post by nebojsa » Sun Feb 20, 2011 8:09 am

Right.
But when I asked this question I had something else on my mind.
I've run north half of Europe on 24km resolution, 180/3hrs, and it worked ok. Then I've tried to run whole Europe but it didn't work properly. During ./ems_post it stopped to work with last message:
"Combining individual grib files into a single monolithic file - 201102170000_nmm_wrfout_d01.grb
could not open file: 201102170000_nmm_wrfout_d01.grb
Unrecognized date/time format (0) in ems_utils::compute_date"
I've created the same domain several times but the result was always the same.
So, I'm tryning to figure out what's going on.

meteoadriatic
Posts: 1587
Joined: Wed Aug 19, 2009 10:05 am

Re: Number of points per domain

Post by meteoadriatic » Sun Feb 20, 2011 8:55 am

nebojsa wrote:Right.
But when I asked this question I had something else on my mind.
I've run north half of Europe on 24km resolution, 180/3hrs, and it worked ok. Then I've tried to run whole Europe but it didn't work properly. During ./ems_post it stopped to work with last message:

Code: Select all

Combining individual grib files into a single monolithic file - 201102170000_nmm_wrfout_d01.grb
could not open file: 201102170000_nmm_wrfout_d01.grb
Unrecognized date/time format (0) in ems_utils::compute_date
The first logical move will be to check if that file - 201102170000_nmm_wrfout_d01.grb exist or not. Not knowing that it is very hard to guess what could be a problem.

nebojsa
Posts: 133
Joined: Sun Mar 21, 2010 3:40 pm

Re: Number of points per domain

Post by nebojsa » Sun Feb 20, 2011 11:14 am

Yep! It exists!
During the run it creates grads/<actual-date>_nmm_wrfout_d01.grb (2.5 GB) and that's it. No *.ctl file.
After a while I delete this <actual-date>_nmm_wrfout_d01.grb and start manualy ./ems_post --domain 1 and it creates 62 files. Then stops with this messagge:
"Combining individual grib files into a single monolithic file - <actual-date>_nmm_wrfout_d01.grb
could not open file: 201102170000_nmm_wrfout_d01.grb
Unrecognized date/time format (0) in ems_utils::compute_date"

I have another domain which works properly so this is confuzing for me. First I wanted to remove possibility of exceeding the max num og points per domain (stupid, but not impossible).

meteoadriatic
Posts: 1587
Joined: Wed Aug 19, 2009 10:05 am

Re: Number of points per domain

Post by meteoadriatic » Sun Feb 20, 2011 11:24 am

Check that there is enough free space on wrfems partition. Your grib file might not be complete if you run out of free space. Look at this command during postprocessing, especially just before you get error:

Code: Select all

watch -n 1 df -h
Finally, try to create .ctl and .idx files manually to see what's going on:

Code: Select all

cd $EMS/runs/yourdomain/emsprd/grads
grib2ctl.pl -verf  201102170000_nmm_wrfout_d01.grb >201102170000_nmm_wrfout_d01.grb.ctl
gribmap -i 201102170000_nmm_wrfout_d01.grb.ctl
Hope then we will have more info to resolve this.

Edit: I see now file is over 2GB. This could be a problem but I'm not sure yet.

nebojsa
Posts: 133
Joined: Sun Mar 21, 2010 3:40 pm

Re: Number of points per domain

Post by nebojsa » Sun Feb 20, 2011 11:46 am

Checked in other run which works proprely, main domain grb is 0.5GB and nested is 0.8GB.
Il create new "problematic" run but with less points.
But, first, I'll follow yout advices and instructions.
Thanks!

nebojsa
Posts: 133
Joined: Sun Mar 21, 2010 3:40 pm

Re: Number of points per domain

Post by nebojsa » Sun Feb 20, 2011 11:50 am

OK, here's the output in ./ems_post --domain 1
61 of 61 : Writing fields from 2011-02-27_12:00:00 to GRIB 1 - 1102200000_nmm_d01.grb1f800000
Converting GRIB 1 to GRIB 2 - 1102200000_nmm_d01.grb2f800000


Processing WRF netCDF output to GRIB format completed at Sun Feb 20 11:41:36 2011 UTC


III. Creating GrADS control and index files from domain 01 WRF GRIB data


* Combining individual grib files into a single monolithic file - 201102200000_nmm_wrfout_d01.grb
could not open file: 201102200000_nmm_wrfout_d01.grb
Unrecognized date/time format (0) in ems_utils::compute_date

nebojsa
Posts: 133
Joined: Sun Mar 21, 2010 3:40 pm

Re: Number of points per domain

Post by nebojsa » Sun Feb 20, 2011 11:56 am

Some more information:
nebojsa@linux-06g8-> grib2ctl.pl -verf 201102200000_nmm_wrfout_d01.grb >201102200000_nmm_wrfout_d01.grb.ctl
could not open file: 201102200000_nmm_wrfout_d01.grb


Big problem:
either 201102200000_nmm_wrfout_d01.grb is missing or not a grib file
wgrib is not on your path or can not write to /tmp/g11065.tmp (full disk or permissions)
Whole computer is on one partition, so, that couldn't be the problem. Perhaps the *grb file is to big.

nebojsa
Posts: 133
Joined: Sun Mar 21, 2010 3:40 pm

Re: Number of points per domain

Post by nebojsa » Sun Feb 20, 2011 11:58 am

And another information:
nebojsa@linux-06g8-> gribmap -i 201102200000_nmm_wrfout_d01.grb.ctl
Open Error: missing XDEF record
Open Error: missing YDEF record
Open Error: missing ZDEF record
Open Error: missing TDEF record
Open Error: missing UNDEF record
Open Error: missing DSET record
Open Error: missing VARS record
The data file was not opened.
nebojsa@linux-06g8->

Post Reply