Zezo forum: Dowload "all.grib" file

Dowload "all.grib" file

Post by Ben | 2020-04-15 | 10:04:02

Hi
Just a note that for some time, the all.grib file has been very long to download.
Benoit

25 Comments

Post by zezo | 2020-04-15 | 13:08:15

Maybe some part of the internet between us is congested because of all people staying home.

Anyone else having problems?

Post by Ben | 2020-04-16 | 07:25:57

Hi
This morning, very fast !
Subject closed....
Benoit

Post by zezo | 2020-04-16 | 11:51:33

I also tried from cloud instances in EU and US and got 8-10MB/sec, which is the theoretical limit as the server sits on a 100Mbit link.

But the congestion can be intermittent and/or affect different ISPs.

There was some packet loss to your IP address yesterday, but hard to localize exactly - could be in your last mile provider, my ISP international links or somewhere in between.

Now it's better, and the round-trip time is also considerably better (which also suggests congestion):

yesterday:

ping -s 1500 xx.yy.92.27
PING xx.yy.92.27 (xx.yy.92.27): 1500 data bytes
1508 bytes from xx.yy.92.27: icmp_seq=0 ttl=58 time=212.698 ms
1508 bytes from xx.yy.92.27: icmp_seq=1 ttl=58 time=204.227 ms

vs.

ping -s 1500 xx.yy.92.27
PING xx.yy.92.27 (xx.yy.92.27): 1500 data bytes
1508 bytes from xx.yy.92.27: icmp_seq=0 ttl=58 time=45.454 ms
1508 bytes from xx.yy.92.27: icmp_seq=1 ttl=58 time=44.758 ms

Post by Shelisteam | 2020-05-22 | 11:31:11

all grib - has the diferent data of the wind compared to the game even by 2-3 kn.

GFS - forecast more than 3

What forecast is correct for the game?

Any advice?

Post by zezo | 2020-05-22 | 12:26:36

It should contain almost exactly the game data at points with integer lat/lon (subject to specific rounding difference), but within the square your software uses different interpolation and gives different result.

There is also a 2.0 knots minimum wind limit in the game. It never gets below that, even if the GRIB says 0. That's applied at the last stage of the calculation, so the direction is preserved.

Post by Shelisteam | 2020-05-22 | 19:21:24

ALL GRIB
Data from the daschboard TWS - 7,38
TWS GRIB with standard interpolation - 7,49
TWS GRIB with spatial only interpolation - 7,81
TWS GRIB with SOL interpolation - 7,19

GFS
Data from the daschboard TWS - 7,38
TWS GRIB with standard interpolation - 7,61
TWS GRIB with spatial only interpolation - 8,14
TWS GRIB with SOL interpolation - 7,57

This is on light winds on strong one, differences are highier

Post by zezo | 2020-05-22 | 19:54:38

SOL should be using temporal interpolation.

Without temporal interpolation the results will only match every 3 hours (round 3 hours on UTC)

What tool is this?

Post by Shelisteam | 2020-05-22 | 20:01:13

Qtvlm

Post by zezo | 2020-05-22 | 20:36:05

Vlm used 0.5� gribs, not sure how well qtvlm deals with 1� resolution.

Also found this https://forum-marinsvirtuels.forumactif.com/t849-la-meteo-chez-vlm. Not sure what to make of it ;-)

Post by zezo | 2020-05-22 | 21:36:29

It's also possible that the data in all.grib is incorrect. It used to be a part of the data conversion years ago when VR used V1 data but V2 was easier to use. There were some rounding errors between v1 and v2, so I was converting to get in sync, or maybe my toolchain at the time did not support v2.

So if someone spots a significant difference between NOAA GRIB or VR winds, at integer coordinates and at UTC time of 0,3,6,9,12,... , please drop me a line with the numbers.

Post by Shelisteam | 2020-05-23 | 12:15:46

38�N47'25", 001�E51'05" TWS game 7,65 (dashboard)
TWS from the all.grib - Standard interpolation 7,81
TWS Xygrib 7,17 - Standard interpolation
TWS VLM=GFS NOAA grib - 7,33 - Standard interpolation

Post by zezo | 2020-05-23 | 17:49:04

You have to check the integer points. Like 40N 0E (there are lines there)

Post by zezo | 2020-05-23 | 17:55:35

Right now - 10 minutes before the 18h GRIB

game - 2.9 @ 124
all.grib via zyGrib -> 3.0 @ 125

You should get the same number in qtvlm at 18 UTC, and probably the next 3 hours without interpolation.

P.S @18 UTC - game says 3.0 @118, so 7 degree difference to the grib

data for 21:Z:

game: 6.8@64
grib: 6.1@67

The difference comes from the way VR rounds the wind U/V components. Repeating, all.grib is not modified according to VR rounding rules, just converted from v2

Post by CoreTeam | 2020-05-23 | 12:27:59

@zezo when is the approx GTM time that allgrib is finalized?

Post by zezo | 2020-05-23 | 17:51:11

The file time is in UTC when you list the directory

all.grib 2020-May-23 17:10:11 21.5M application/octet-stream

job is stared at 5/11/17/23 + 10 minutes. It has 2-3 files to get and then generates the output data.

Post by CoreTeam | 2020-05-24 | 07:59:05

Thanks! Very clear.

Post by lukla49 | 2020-06-01 | 17:31:31

Hi, Thank you for this wonderful work.
I try to download the allgrib to Qtvlm, but the file seems not to be the last one. Ex : file 2020-Jun-01 17:10:10 should begin on 2020-Jun-01 12h UTC and Qtvlm detect a file wich begin at 6h UTC. I use the link http://zezo.org/grib/gribv1/.
Where is the mistake ?
Thanks

Post by zezo | 2020-06-01 | 21:35:44

You have a point.

The first file in that bundle has date/time of
dataDate = 20200601;
dataTime = 0;

Then two from 0600

and then the current run.

So the data actually starts at the start of the day. That's the internal format I use for the router, but probably redundant for external GRIB viewer/router. I'll take a look tomorrow.

Post by zezo | 2020-06-01 | 21:43:42

The actual code is

find /...../gribv1/ -type f -name "2*" -mmin -840 ...

So that bundles all grib files less than 16 hours old.

That's done because the file is recreated during the update, i.e. it contains what the router currently uses, a mix of old and new data

And then during the update the old files are 6 hours old.

If only one, consistent, 4 times/day file is expected then the logic could be changed.

Post by zezo | 2020-06-01 | 21:54:37

To put it another way, all.grib consists exactly of all the 2020*.grib files above it in the directory listing.

It's a property of grib format that if you just concatenate few files together, the result is also a valid grib file.

Post by lukla49 | 2020-06-02 | 06:35:30

Thank you for your answer, It's clear. I wasn't sure that the new data were the good one. So it's not a problem to get also old data before the theorical date. No way to go in the past !

Post by Shelisteam | 2020-06-02 | 08:19:52

Position 56�06'18"N 7�06'16"E
TWS from the game 5,3
TWS from the grib 4,7 - all.grib

Position 14�24'11"N 127�26'24"W
from game 9,44
From grib 8,7

Interpolation Standard.

Where is the mistake? Mayby i'm doing something wrong.

Post by CoreTeam | 2020-06-02 | 08:46:06

@zezo I have a question that I didn't found the answer here.
I know that VR interpolates the GRIB data in some maybe mysterious way. I'm trying to have it as similar as possible in qtVlm using your all.grib and the best match I found is with "Standard" GRIB interpolation.
But looking in google I saw that the best for Virtual Regatta is "Spatial" interpolation.
Do you low something about it? With Spatial I didn't get results as expected honestly, maybe I configured something wrong.
Thanks and appreciate your great work here!

Post by zezo | 2020-06-02 | 11:32:49

The game uses specific interpolation algorithm. I don't think you can get the same result with another router.

It also has specific rounding of the grib data and a 2 knots wind minimum.

It does not matter much for long-time planning, but the current boat data will always differ.