problem with large outputed grb2 files for GRADS

All issues/questions about EMS v3.4 package, please ask here.
Post Reply
emsiwx
Posts: 80
Joined: Sun Aug 12, 2012 11:07 am

problem with large outputed grb2 files for GRADS

Post by emsiwx » Sat Nov 01, 2014 1:43 pm

I would like to ask you for help.

I am playing with WRF EMS and currently trying the Europe domain at 15km resolution.

If I do 72 hours forecast, the GRADS outputed grb2 file is about 1,4 GB and ctl and idx files are created correctly.

Then I increase to 96+ hours forecast the grb2 goes more than 2GB and in post process to get ctl and idx files I get error
grb2 is either not valid, or missing or you have not persmission.

I also tried manually with g2ctl, with the same result.

Any ideas please?

surgeon
Posts: 45
Joined: Wed Dec 08, 2010 1:34 pm
Location: Poland
Contact:

Re: problem with large outputed grb2 files for GRADS

Post by surgeon » Mon Nov 03, 2014 4:12 pm

gribmap -help

This is gribmap for GrADS Version 2.0.2.oga.1
creates the index file for using GRIB data in grads

Command line options are as follows:

-0 ignores the forecast time in the mapping; only uses the base time
-b set the valid time for averages to be the beginning of the period rather than the end (default)

-big required if the GRIB1 or GRIB2 files are > 2 Gb

-e ignores junk bytes (non GRIB msg) at end of file (e.g., ECMWF GRIB files)
-E ignores junk bytes in middle and/or end of GRIB file
-fNNNN forces a match for forecast time in hours = NNNNN (e.g., f24 for t=24 h)
-hNNNN where NNNN is the # of header bytes to look for first GRIB messages (default is 100)
-help prints out this help
-i fname provides the name of data descriptor file to map
-m uses the initial time from the descriptor file instead of the base time in the grib header
-min0 ignores minutes code
-N does NOT write out the map
-q quiet mode gives no listing except for errors
-sNNNN where NNNN is the maximum # of bytes to skip between GRIB messages (default is 1000)
-t0 forces a match if base time in the GRIB header equals the initial time in the descriptor file
-v verbose option shows details of the mapping

emsiwx
Posts: 80
Joined: Sun Aug 12, 2012 11:07 am

Re: problem with large outputed grb2 files for GRADS

Post by emsiwx » Mon Nov 03, 2014 6:20 pm

Thank you surgeon,

I will try that manually...Is it possible to change ems_autorun procedure do use -big option as default?

EmsiWx

meteoadriatic
Posts: 1565
Joined: Wed Aug 19, 2009 10:05 am

Re: problem with large outputed grb2 files for GRADS

Post by meteoadriatic » Mon Nov 03, 2014 7:39 pm

You should edit perl file that does this thing. It should be somewhere inside wrfems/strc directory.

I have no idea in EMS v3.4 where it is exactly and what to edit, I gave up from >v3.1 EMS system... maybe somebody else can help you if you can't find that yourself.

emsiwx
Posts: 80
Joined: Sun Aug 12, 2012 11:07 am

Re: problem with large outputed grb2 files for GRADS

Post by emsiwx » Mon Nov 03, 2014 8:19 pm

Thank you meteoadriatic, will try to go through the perl scripts to find where and how it points to gribmap.

But first I will try manually if it is only the question of -big flag. I think it is, because grb2 and ctl files are created, except the idx file.

EmsiWx

meteoadriatic
Posts: 1565
Joined: Wed Aug 19, 2009 10:05 am

Re: problem with large outputed grb2 files for GRADS

Post by meteoadriatic » Mon Nov 03, 2014 8:24 pm

I have ddifferent experience. In my case, when I create ctl and idx from >2GB grib, without -big flag, then all looks good, until I actually want to display in grads data that falls outside of 2GB limit inside file, then I get garbage instead of data. So I think your problem is possibly somewhat else.

emsiwx
Posts: 80
Joined: Sun Aug 12, 2012 11:07 am

Re: problem with large outputed grb2 files for GRADS

Post by emsiwx » Mon Nov 03, 2014 8:56 pm

It seems that the script (Ograds.pm) uses -big by default:

my $exe = &EMS_utils::popit($gribmap);
&EMS_style::emsprint(1,11,144,1,0,"Creating GrADS index file for $fcst - $gradsfil.idx");
my $rc = &EMS_system::emsExec("$gribmap -big -q -E -i $gradsfil.ctl > /dev/null 2>&1");
if (&EMS_system::checkFile("$gradsfil.idx")) {
&EMS_style::emsprint(0,1,24,0,1," - Failed ($exe code $rc) No GrADS Love for You!");
@grads=(); return \@grads;
} else {
&EMS_style::emsprint(0,1,24,0,1," - More GrADS Love for You!");

The other thing could be the part where the perl script checks if the grb>2 GB on 32bit systems:


# Test the grib file to determine whether it exceeds the 2gb limit on
# 32-bit systems.
#
return \@grads if &EMS_files::test2Gb($monolith);

But it should not be my problem, because I am running 64bit Ubuntu.

This drives me crazy :?

EmsiWX

meteoadriatic
Posts: 1565
Joined: Wed Aug 19, 2009 10:05 am

Re: problem with large outputed grb2 files for GRADS

Post by meteoadriatic » Mon Nov 03, 2014 10:00 pm

As I suspected unfortunatelly.

emsiwx
Posts: 80
Joined: Sun Aug 12, 2012 11:07 am

Re: problem with large outputed grb2 files for GRADS

Post by emsiwx » Tue Nov 04, 2014 6:23 am

Hello,

so I tried the 126 hours run over EU 15km and got 2,7gb grb2...I got this in WRFEMS:

Image

I have found out that even ctl file is not created correctly. idx file is not created at all.
I also tried to make those files manually but without succes :(

Then I copied the grb2 file to my WIN XP and manually created ctl file using g2ctl.pl -verf, and idx file using gribmap -big -i -e.
Both files were created succesfully and maps are generated correctly under Opengrads in WIN XP.

Any ideas guys what's wrong in my Ubuntu?

Thank you for your answers.

EmsiWx

Post Reply

Who is online

Users browsing this forum: Baidu [Spider] and 1 guest