Model running times not consistent, why?

Forum dedicated to older versions of EMS package (WRFEMS v3.2, v3.1 or older). Support is user-to-user based, so please help others if you can.
Post Reply
meteoadriatic
Posts: 1512
Joined: Wed Aug 19, 2009 10:05 am

Model running times not consistent, why?

Post by meteoadriatic » Sat May 15, 2010 6:58 pm

Do you experience different model running times from run to run? I don't use adaptive time step and so I would expect that they shoud be consistent? Of course, computer is not used for anything else than WRF. But.. running times vary significantly, as much as 20 or 30 percent between different runs.

How's that?

meteoadriatic
Posts: 1512
Joined: Wed Aug 19, 2009 10:05 am

Re: Model running times

Post by meteoadriatic » Sun Jul 11, 2010 12:00 pm

In fact I found some regularity here. There is difference between my 2 NMM runs in every day. I run NMM, the same domain of course, 2 times a day, once with 00z and second time with 12z GFS init data.

I found that every time run started with 12z data is much slower than run started with 00z data :shock:

It takes about 3,5h to finish 00z run, and about 4,5h to finish 12z run ?! :?

Of course, computer is completely idle otherwise, there is nothing else running on the system that could slow down 12z run. The only difference between these two runs is starting time in crontab file. Nothing else is different. How is that possible? Anybody has an idea?

I did some changes to scripts that allow for logging all runs in separate log files. This way I can track down what's going on. These are running times for 00z run:

Code: Select all

wrf@wrf-> cat *00z* | grep 'Model run time'
            >  Model run time was 3 hours 39 minutes 28 seconds.
            >  Model run time was 3 hours 27 minutes 48 seconds.
            >  Model run time was 3 hours 30 minutes 55 seconds.
            >  Model run time was 3 hours 27 minutes 2 seconds.
            >  Model run time was 3 hours 23 minutes 28 seconds.
            >  Model run time was 3 hours 40 minutes 2 seconds.
            >  Model run time was 3 hours 30 minutes 16 seconds.
            >  Model run time was 3 hours 32 minutes 44 seconds.
            >  Model run time was 3 hours 23 minutes 15 seconds.
            >  Model run time was 3 hours 42 minutes 59 seconds.
            >  Model run time was 3 hours 27 minutes 20 seconds.
            >  Model run time was 3 hours 40 minutes 22 seconds.
            >  Model run time was 3 hours 21 minutes 23 seconds.
            >  Model run time was 3 hours 41 minutes 5 seconds.
            >  Model run time was 3 hours 54 minutes 56 seconds.
            >  Model run time was 3 hours 37 minutes 38 seconds.
            >  Model run time was 3 hours 41 minutes 11 seconds.
            >  Model run time was 3 hours 31 minutes 9 seconds.
            >  Model run time was 3 hours 34 minutes 44 seconds.
            >  Model run time was 3 hours 22 minutes 19 seconds.
            >  Model run time was 3 hours 39 minutes 36 seconds.
These are running times for 12z runs, I repeat again, the same domain!

Code: Select all

wrf@wrf-> cat *12z* | grep 'Model run time'
            >  Model run time was 4 hours 25 minutes 22 seconds.
            >  Model run time was 4 hours 11 minutes 13 seconds.
            >  Model run time was 4 hours 33 minutes 6 seconds.
            >  Model run time was 4 hours 38 minutes 28 seconds.
            >  Model run time was 4 hours 23 minutes 10 seconds.
            >  Model run time was 4 hours 29 minutes 43 seconds.
            >  Model run time was 4 hours 32 minutes 55 seconds.
            >  Model run time was 4 hours 34 minutes 28 seconds.
            >  Model run time was 4 hours 26 minutes 30 seconds.
            >  Model run time was 4 hours 22 minutes 28 seconds.
            >  Model run time was 4 hours 10 minutes 28 seconds.
            >  Model run time was 4 hours 22 minutes 55 seconds.
            >  Model run time was 4 hours 32 minutes 36 seconds.
            >  Model run time was 4 hours 35 minutes 23 seconds.
            >  Model run time was 4 hours 37 minutes 7 seconds.
            >  Model run time was 4 hours 24 minutes 24 seconds.
            >  Model run time was 4 hours 36 minutes 12 seconds.
            >  Model run time was 4 hours 22 minutes 56 seconds.
            >  Model run time was 4 hours 39 minutes 21 seconds.
            >  Model run time was 4 hours 36 minutes 34 seconds.
            >  Model run time was 4 hours 14 minutes 53 seconds.
            >  Model run time was 4 hours 17 minutes 51 seconds.
            >  Model run time was 4 hours 20 minutes 12 seconds.
            >  Model run time was 4 hours 30 minutes 31 seconds.
            >  Model run time was 4 hours 25 minutes 40 seconds.
            >  Model run time was 4 hours 27 minutes 53 seconds.
            >  Model run time was 4 hours 28 minutes 29 seconds.
            >  Model run time was 4 hours 28 minutes 53 seconds.
            >  Model run time was 4 hours 33 minutes 2 seconds.
            >  Model run time was 4 hours 19 minutes 30 seconds.
            >  Model run time was 4 hours 25 minutes 52 seconds.
            >  Model run time was 4 hours 25 minutes 8 seconds.
            >  Model run time was 4 hours 14 minutes 33 seconds.
:roll:

meteoadriatic
Posts: 1512
Joined: Wed Aug 19, 2009 10:05 am

Re: Model running times not consistent, why?

Post by meteoadriatic » Wed Nov 09, 2011 5:55 pm

Unfortunately, I still have this problem. Now when we have lot more people browsing this forum, maybe someone will have an idea what is going on here, that's why I bump this tread again.

Again, My 12z run is about 30% slower than 00z run. And always has been like that, I never solved it.

Everything is the same, I start the same domain through cron, the only difference in cron jobs is that they starts 12 hours apart from each other, on same hardware, nothing is running on machine except wrf, CPU usage is always 100% during both runs... timestep is fixed manually (in fact it is NMM run so it can't be adaptive even if I wish...).

So, why do I loose one hour of time in 12z run? Just throw some idea if you have any?

nebojsa
Posts: 133
Joined: Sun Mar 21, 2010 3:40 pm

Re: Model running times not consistent, why?

Post by nebojsa » Thu Nov 10, 2011 5:11 pm

Only significant difference I've noticed is during GFS data download depending on the source.

tornado
Posts: 2
Joined: Thu Jul 18, 2013 7:25 pm

Re: Model running times not consistent, why?

Post by tornado » Wed Apr 30, 2014 6:17 pm

This is quite an old topic, but I would like to ask if you could find out the reason for the problem.

Did you check the core usage percentages? Maybe unlikely, but can it be because of higher room temperatures during afternoon run? :)

meteoadriatic
Posts: 1512
Joined: Wed Aug 19, 2009 10:05 am

Re: Model running times not consistent, why?

Post by meteoadriatic » Wed Apr 30, 2014 7:27 pm

No, it was not temperature issue (this is first that was checked, room is deep under building, no windows, usually no people access... air-conditioned at steady 19°C). Problem was because of NUMA topology. Without optimized NUMA access to RAM, sometimes CPU 1 takes data from RAM BANKS 1 (faster) and sometimes from RAM BANKS 2 (much slower). This leads to different performance between runs. I can't explain why there was apparent rule of 00z run being faster than 12z. After I started using numad daemon, all NMM times consistently finish in not more than 5 minute difference.

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest