CI fails with error: ValueError: time data 'latest' does not match format '%Y%m%d'

[aagbsn]

FAILED tests/test_updaters.py::test_end_to_end - ValueError: time data 'latest' does not match format '%Y%m%d'

it looks like some external data changed, but this breaks CI for unrelated changes.

[aagbsn]

per @LDiazN

data/oonipipeline/src/oonipipeline/tasks/updaters/asnmeta_updater.py

Line 18 in 17a4258

AS_ORG_MAP_URL = "https://archive.org/download/ip2country-as/all_as_org_map.json"

the dataset: https://archive.org/download/ip2country-as/all_as_org_map.json
seems to have changed

[LDiazN]

Here's a sample of the json downloaded from that URL:

{  "1": [
    [
      "Level 3 Communications, Inc.",
      "US",
      "20040604",
      "LVLT-1",
      "ARIN"
    ],
    [
      "Level 3 Parent, LLC",
      "US",
      "20180220",
      "LVLT-1",
      "ARIN"
    ]
  ],
...

And this is the expected format:

data/oonipipeline/src/oonipipeline/tasks/updaters/asnmeta_updater.py

Lines 43 to 51 in 17a4258

	rows.append(
	{
	"asn": asn,
	"org_name": v[0],
	"cc": v[1],
	"changed": changed,
	"aut_name": v[3],
	"source": v[4],
	}

The conflicting entry looks like this:

"3288": [
    [
      "MACAM-MOFET Institute, Israel",
      "",
      "20000101",
      "MOFET-AS",
      "RIPE"
    ],
    [
      "Mofet Institute",
      "",
      "20000101",
      "MOFET-AS",
      "RIPE"
    ],
    [
      "Mofet Institute",
      "IL",
      "20000101",
      "MOFET-AS",
      "RIPE"
    ],
    [
      "Zipcom Communications LTD",
      "IL",
      "20260601",
      "MOFET-AS",
      "RIPE"
    ],
    [
      "Mofet Institute",
      "IL",
      "latest",
      "MOFET-AS",
      "RIPE"
    ]
  ],

Seems like the most recent one is expressed with a "latest" in some cases, breaking this line:

data/oonipipeline/src/oonipipeline/tasks/updaters/asnmeta_updater.py

Line 42 in 17a4258

changed = datetime.strptime(v[2], "%Y%m%d").date()

[aagbsn]

'latest' isn't very descriptive, but maybe we can guess it from the current release version