Help! network problem - is there a wizard in the house?

Forum dedicated to older versions of EMS package (WRFEMS v3.2, v3.1 or older). Support is user-to-user based, so please help others if you can.
pattim
Posts: 173
Joined: Sun Jun 24, 2012 8:42 pm
Location: Los Angeles, CA, USA

Help! network problem - is there a wizard in the house?

Post by pattim » Fri Jun 29, 2012 9:29 pm

Can anyone help me? I know we have a proxy, but the installer seems to be negotiating that OK... My install of ModelE also uses a similar method of downloading files online automatically and it has no trouble.

Code: Select all

~/00_GCMs/EMS_WRF> ./ems_install.pl --install 3.2.1.5.45.beta --debug
    System Date         : Fri Jun 29 21:23:08 2012 UTC
    Linux Release       : CODENAME  Asparagus
    Kernel              : 3.1.10-1.9-desktop
    CPU Type            : x64
    CPU Instructions    : x64
    Physical CPUs       : 4             
    Cores per CPU       : 6             
    Total Processors    : 24             
    Hyper-Threading     : On                
    Installed Binaries  : None Installed  
    Greetings clearly intelligent and devastatingly good-looking WRF EMS user!
    Welcome to the NWS SOO Science and Training Resource Center (SOO/STRC) Weather Research
    and Forecasting (WRF) Environmental Modeling System (EMS) installation and update routine.
    Phew! That's more than a mouthful.
  What I have so far:
    TARGET : x64
    HWTYPE : x64
    hwtype : x64
    binems : None Installed
    DEF X32: Undefined
    DEF X64: Undefined
    BINONLY: Yes
    *  Checking http connection to emsdata1.comet.ucar.edu   - Contacted
    *  Connecting to emsdata1.comet.ucar.edu to get requested information
        !  There was a problem retrieving information from emsdata1.comet.ucar.edu
           * <url> malformed
            * About to connect() to proxy *************************************** (#0)
            *   ... connected
            > GET http://emsdata1.comet.ucar.edu/cgi-bin/ems_info.pl?list HTTP/1.1
            > User-Agent: curl/7.22.0 (x86_64-unknown-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.0e zlib/1.2.5 c-ares/1.7.5 libidn/1.22
           libssh2/1.2.9
            > Host: emsdata1.comet.ucar.edu
            > Accept: */*
            > Proxy-Connection: Keep-Alive
            > 
            * The requested URL returned error: 407
            * Closing connection #0
       Your Installation is Incomplete - Now go forth and fix the problem!
    Nigel Says - "WRF EMS, It's taking mesoscale modeling to 11"

Code: Select all

~/00_GCMs/EMS_WRF> ./ems_install.pl --install 3.2.1.5.45.beta --debug --wget
    System Date         : Fri Jun 29 21:23:17 2012 UTC
    Linux Release       : CODENAME  Asparagus
    Kernel              : 3.1.10-1.9-desktop
    CPU Type            : x64
    CPU Instructions    : x64
    Physical CPUs       : 4             
    Cores per CPU       : 6             
    Total Processors    : 24             
    Hyper-Threading     : On                
    Installed Binaries  : None Installed  
    Greetings clearly intelligent and devastatingly good-looking WRF EMS user!
    Welcome to the NWS SOO Science and Training Resource Center (SOO/STRC) Weather Research
    and Forecasting (WRF) Environmental Modeling System (EMS) installation and update routine.
    Phew! That's more than a mouthful.
  What I have so far:
    TARGET : x64
    HWTYPE : x64patti@TYAN-III:~/ModelE/modelE/decks> ../exec/get_input_data -w E000PMS /home/patti/ModelE/cmrun
Will search for data files in: http://portal.nccs.nasa.gov/GISS_modelE/modelE_input_data
Will store the data files  in: /home/patti/ModelE/cmrun
Will use rundeck: E000PMS.R
<...>
Already present:     ISCCP.tautables
Downloading:     GHG.Mar2009.txt
--2012-06-29 14:34:50--  http://*********************/GISS_modelE/modelE_input_data/GHG.Mar2009.txt
Resolving ************************* (*************************)... ***.***.***.***
Connecting ************************* (*************************)|***.***.***.***|:****... connected.
Proxy request sent, awaiting response... 200 OK
Length: 7745 (7.6K) [text/plain]
Saving to: `GHG.Mar2009.txt'
100%[======================================================================================================>] 7,745       --.-K/s   in 0.08s   
2012-06-29 14:34:50 (96.0 KB/s) - `GHG.Mar2009.txt' saved [7745/7745]
Already present:     dH2O_by_CH4_monthly
Already present:     BC.Dry+Wet.depositions.ann
<...>
patti@T:~/ModelE/modelE/decks> 
    hwtype : x64
    binems : None Installed
    DEF X32: Undefined
    DEF X64: Undefined
    BINONLY: Yes
    *  Checking http connection to emsdata1.comet.ucar.edu   - Contacted
    *  Connecting to emsdata1.comet.ucar.edu to get requested information
        !  There was a problem retrieving information from emsdata1.comet.ucar.edu
       Your Installation is Incomplete - Now go forth and fix the problem!
    Nigel Says - "WRF EMS, It's taking mesoscale modeling to 11"
patti@TYAN-III:~/00_GCMs/EMS_WRF>
Just for reference, here's the same sort of thing that ModelE does to download the files it needs. So I know there is a good network connection on my end...

Code: Select all

patti:~/ModelE/modelE/decks> ../exec/get_input_data -w E000PMS /home/patti/ModelE/cmrun
Will search for data files in: http://portal.nccs.nasa.gov/GISS_modelE/modelE_input_data
Will store the data files  in: /home/patti/ModelE/cmrun
Will use rundeck: E000PMS.R
<...>
Already present:     ISCCP.tautables
Downloading:     GHG.Mar2009.txt
--2012-06-29 14:34:50--  http://*********************/GISS_modelE/modelE_input_data/GHG.Mar2009.txt
Resolving ************************* (*************************)... ***.***.***.***
Connecting ************************* (*************************)|***.***.***.***|:****... connected.
Proxy request sent, awaiting response... 200 OK
Length: 7745 (7.6K) [text/plain]
Saving to: `GHG.Mar2009.txt'
100%[====================================================>] 7,745       --.-K/s   in 0.08s   
2012-06-29 14:34:50 (96.0 KB/s) - `GHG.Mar2009.txt' saved [7745/7745]
Already present:     dH2O_by_CH4_monthly
Already present:     BC.Dry+Wet.depositions.ann
<...>
patti@T:~/ModelE/modelE/decks> 
Here's how ModelE gets its data...

Code: Select all

	    print "Downloading:     $filename\n";
	    `cd $file_dir &&  wget $DATA_PORTAL_URL/$filename`;
	    $rcode = $? >> 8;
...and here's how EMS does it -

Code: Select all

    my $cmd = ($util =~ /curl/) ? "$util -v -C - -s --connect-timeout 25 --max-time 7200 $rfile -o $lfile >& $dlog" :
              ($util =~ /wget/) ? "$util -a $dlog -v -t 25 -T 25 --read-timeout=7200 --connect-timeout=10 -O $lfile  $rfile" : 0;
...Does this tell anyone something they could help me debug the EMS file-getter with? :cry:

meteoadriatic
Posts: 1574
Joined: Wed Aug 19, 2009 10:05 am

Re: Help! network problem - is there a wizard in the house?

Post by meteoadriatic » Sat Jun 30, 2012 6:11 am

I think that the best person that can help you is your system/network administrator. It is impossible that anyone of us know how your network proxy is set up and what has to be done at that computer to make a pass through that proxy.

pattim
Posts: 173
Joined: Sun Jun 24, 2012 8:42 pm
Location: Los Angeles, CA, USA

Re: Help! network problem - is there a wizard in the house?

Post by pattim » Sun Jul 01, 2012 8:50 pm

meteoadriatic wrote:I think that the best person that can help you is your system/network administrator. It is impossible that anyone of us know how your network proxy is set up and what has to be done at that computer to make a pass through that proxy.
Well, actually our proxy is really transparent. The only time I've had trouble in the past is with SFTP, and also that ModelE script's wget works flawlessly. Trust me, given our network people, it would be easier to simplify the EMS script than try to bend the rules on the proxy (since it's already very transparent). Patricia

meteoadriatic
Posts: 1574
Joined: Wed Aug 19, 2009 10:05 am

Re: Help! network problem - is there a wizard in the house?

Post by meteoadriatic » Sun Jul 01, 2012 9:22 pm

Going throuhg proxy shouldn't have anything with EMS scripts.

I assisted some people who needed to setup model behind proxy and they needed to set linux environment variable to be able to pass proxy. That's why I suggested to contact your network administrator.

Something like this might help, try it:
http://linux.byexamples.com/archives/18 ... t-console/

pattim
Posts: 173
Joined: Sun Jun 24, 2012 8:42 pm
Location: Los Angeles, CA, USA

Re: Help! network problem - is there a wizard in the house?

Post by pattim » Sun Jul 01, 2012 11:26 pm

meteoadriatic wrote:Going throuhg proxy shouldn't have anything with EMS scripts.

I assisted some people who needed to setup model behind proxy and they needed to set linux environment variable to be able to pass proxy. That's why I suggested to contact your network administrator.

Something like this might help, try it:
http://linux.byexamples.com/archives/18 ... t-console/
That's what I've done, but I use the KDE proxy tool to do it. I believe the system proxy is needed for curl to work correctly, but come to think of it, I have not tried wget with the proxy configured incorrectly (I can try giving the proxy the wrong password and see if the ModelE wget still works). I know browsers like Firefox have their own proxy settings, and don't usually rely on the linux system proxy.

pattim
Posts: 173
Joined: Sun Jun 24, 2012 8:42 pm
Location: Los Angeles, CA, USA

Re: Help! network problem - is there a wizard in the house?

Post by pattim » Tue Jul 03, 2012 8:40 pm

OK, so still working this. A simple command line with wget works through our proxy (using the linux system proxy settings, the method the ModelE file-fetcher uses):

Code: Select all

patti:~/ModelE/modelE/exec> wget http://portal.nccs.nasa.gov/GISS_modelE/modelE_input_data/GHG.Mar2009.txt
--2012-07-03 13:35:36--  http://portal.nccs.nasa.gov/GISS_modelE/modelE_input_data/GHG.Mar2009.txt
Resolving ***********************.org (*************************************)... ***.***.***.***
Connecting to ***********************.org (*************************************)... |***.***.***.***|:*  ... connected.
Proxy request sent, awaiting response... 200 OK
Length: 7745 (7.6K) [text/plain]
Saving to: `GHG.Mar2009.txt'
100%[==========================================>] 7,745       --.-K/s   in 0.005s  
2012-07-03 13:35:36 (1.45 MB/s) - `GHG.Mar2009.txt' saved [7745/7745]
patti:~/ModelE/modelE/exec> 
Evidently this is not what ems_install.pl does?

meteoadriatic
Posts: 1574
Joined: Wed Aug 19, 2009 10:05 am

Re: Help! network problem - is there a wizard in the house?

Post by meteoadriatic » Wed Jul 04, 2012 6:16 am

You can download files manually, then execute ems_install.pl and give it local directory as source for files. This will be easiest way probably.

pattim
Posts: 173
Joined: Sun Jun 24, 2012 8:42 pm
Location: Los Angeles, CA, USA

Re: Help! network problem - is there a wizard in the house?

Post by pattim » Wed Jul 04, 2012 6:43 pm

meteoadriatic wrote:You can download files manually, then execute ems_install.pl and give it local directory as source for files. This will be easiest way probably.
Do you know where I could download the files? I believe after installation the perl scripts have proxy entries so I assume they would work better than the installer itself. I have an installation at home which is about 18 GB, and I can just tar that and copy it to my home folder at work.

meteoadriatic
Posts: 1574
Joined: Wed Aug 19, 2009 10:05 am

Re: Help! network problem - is there a wizard in the house?

Post by meteoadriatic » Wed Jul 04, 2012 6:52 pm

pattim wrote:Do you know where I could download the files?
Sure,
here: http://soostrc.comet.ucar.edu/wrfems/releases/

pattim
Posts: 173
Joined: Sun Jun 24, 2012 8:42 pm
Location: Los Angeles, CA, USA

Re: Help! network problem - is there a wizard in the house?

Post by pattim » Wed Jul 04, 2012 7:19 pm

meteoadriatic wrote:
pattim wrote:Do you know where I could download the files?
Sure,
here: http://soostrc.comet.ucar.edu/wrfems/releases/
Surprise, surprise!! I thought one could only use the .pl file (that's what Robert sent me). OK, you just saved me a week's time :D

EDIT: can I do a wget *.* from that url?

EDIT2: I answered my own question...
wget -r http://soostrc.comet.ucar.edu/wrfems/re ... 5.45.beta/
...seems to be downloading everything in the "beta" directory. Good, good, good!

Actually, this fixes everything! I know wget goes just fine through a proxy/firewall using the linux system proxy settings. Too bad the installer.pl doesn't do this... :cry:
Last edited by pattim on Wed Jul 04, 2012 7:31 pm, edited 1 time in total.

Post Reply