Dear forum,
I was wondering if I can get some help/guidance in the definition of HPC requirements (mainly in terms of tflops required, but also some other recommendations are welcome) for an operational WRF model. I have been trying to get information through several sources with limited success. I was wondering if somebody can help me and/or point in the right direction for this so I can find the right requirements. The idea is to run the following operational models with WRF:
 One 7km grid (approximately 1400km by 700km).
 One 2km grid (approximately 700km by 400km nested to the previous one)
 Two 0.5km grid (approximately 300km by 150km nested to the previous one)
Forecast for 57 days. Ideally these grids should be run operationally in less than 3 hours. Any help or guidance would be highly appreciated.
Thanks!
HPC requirements

 Posts: 1604
 Joined: Wed Aug 19, 2009 10:05 am
Re: HPC requirements
Hello
First of all you can not do such nesting ratio if you will use normal (synchroneous) nesting type. In such case, ratio must be odd number.
0.5km grid 300x150 km has 1800.000 grid points (domain size 600x300). With this grid distance time step should be around 3 seconds.
The similar runs I did were around 420x350 points but with grid size 300m, which is pretty much the same as your domain in terms of requirement. If I remember correctly, to calculate 1 hour of simulation, it took more than 2 hours of time which for 6 days would translate to almost 300 hours, or, around 1112 days of calculation.
For that task I had i78700K CPU that has 16.000 cpumark result:
https://www.cpubenchmark.net/cpu.php?cp ... Hz&id=3098
In order to cut down time to required 3 hours, you need around 100 times more cpumark power, which equals 1.600.000 cpumark. I don't know how much terraflops that is, but make the research from given data.
Now, this assumption is that your multicpu calculation will scale equally well as on 12thread single CPU (which is false so you will need more power than you think and how much I can't tell you. Probably something like at least 50% more than theoretical calculation driven from above figures gives).
Another thing. If you think about asynchroneous nesting in order to reach your required grid distances, you will have very bad ratio of time spent in I/O calculations vs actual wrf.exe integration time so my suggestion is to readjust domain grid distances to odd number ratios and use synchroneous nesting.
First of all you can not do such nesting ratio if you will use normal (synchroneous) nesting type. In such case, ratio must be odd number.
0.5km grid 300x150 km has 1800.000 grid points (domain size 600x300). With this grid distance time step should be around 3 seconds.
The similar runs I did were around 420x350 points but with grid size 300m, which is pretty much the same as your domain in terms of requirement. If I remember correctly, to calculate 1 hour of simulation, it took more than 2 hours of time which for 6 days would translate to almost 300 hours, or, around 1112 days of calculation.
For that task I had i78700K CPU that has 16.000 cpumark result:
https://www.cpubenchmark.net/cpu.php?cp ... Hz&id=3098
In order to cut down time to required 3 hours, you need around 100 times more cpumark power, which equals 1.600.000 cpumark. I don't know how much terraflops that is, but make the research from given data.
Now, this assumption is that your multicpu calculation will scale equally well as on 12thread single CPU (which is false so you will need more power than you think and how much I can't tell you. Probably something like at least 50% more than theoretical calculation driven from above figures gives).
Another thing. If you think about asynchroneous nesting in order to reach your required grid distances, you will have very bad ratio of time spent in I/O calculations vs actual wrf.exe integration time so my suggestion is to readjust domain grid distances to odd number ratios and use synchroneous nesting.
Re: HPC requirements
Hello meteoadriatic,
Thank you very much for this and for your recommendations. Yes, the nesting ratio will be reviewed, thanks for that. I was wondering, this values you are providing will be for running just one domain, equivalent to one of my 0.5km domains, ins't it?
Thanks again!
Best regards
Thank you very much for this and for your recommendations. Yes, the nesting ratio will be reviewed, thanks for that. I was wondering, this values you are providing will be for running just one domain, equivalent to one of my 0.5km domains, ins't it?
Thanks again!
Best regards

 Posts: 1604
 Joined: Wed Aug 19, 2009 10:05 am
Re: HPC requirements
Figures I provided actually had 2 coarser domains around 300m (1.5km and 7.5km) and their dimensions were again very similar to your requirements for coverage with coarse domains.
Moreover, with the figures you gave, and having in mind that resources rise exponentially with increasing resolution, your 0.5km nest will use very most of total computing resources and very minor resources will be spent on their parent/grandparent domain.
Moreover, with the figures you gave, and having in mind that resources rise exponentially with increasing resolution, your 0.5km nest will use very most of total computing resources and very minor resources will be spent on their parent/grandparent domain.