mypy: merging flixbus and blablacar

[ue71603]

python /tool_scripts/script_swiss_couches.txt mylog.log all

(a) downloads flixbus
(b) downloads blablacar
(c) merges the two

problem in the merge block:

2025-03-20 11:57:01,668 - script_runner - INFO - merge_fernbusse - step: 4: conv.epip_db_to_db C:/Users/ue71603/Downloads/merge_fernbusse/03-gtfs.lmdb C:/Users/ue71603/Downloads/merge_fernbusse/04-epip.lmdb
2025-03-20 11:57:01,784 - script_runner - INFO - Copy all tables as-is 
Database Direction does not exist, creating it.
Database DataSource does not exist, creating it.
Database Operator does not exist, creating it.
Database ScheduledStopPoint does not exist, creating it.
2025-03-20 11:57:16,699 - script_runner - INFO - Copy lines, in EPIP style 
epip_line_memory
Database Line does not exist, creating it.
2025-03-20 11:57:18,015 - script_runner - INFO - Fix Quay / StopPlace locations 
2025-03-20 11:57:18,747 - script_runner - INFO - Investigate this site frame step 
epip_site_frame_memory
Database PassengerStopAssignment does not exist, creating it.
Database StopPlace does not exist, creating it.
2025-03-20 11:57:20,455 - script_runner - INFO - Service journeys 
epip_service_journey_generator
2025-03-20 11:57:20,455 - script_runner - INFO - Indexing RoutePoint to ScheduledStopPoint 
2025-03-20 11:57:20,522 - script_runner - INFO - Service journeys for now 
Database ServiceJourney does not exist, creating it.
Database ServiceJourneyPattern does not exist, creating it.
2025-03-20 11:58:08,040 - script_runner - INFO - Calendars 
2025-03-20 11:58:08,040 - script_runner - INFO - Calendar creation...
2025-03-20 11:58:09,344 - script_runner - ERROR - list index out of range Traceback (most recent call last):
  File "C:\Users\ue71603\MG_Daten\github\badger\tools\script_runner.py", line 481, in <module>
    main(
    ~~~~^
        args.script_file,
        ^^^^^^^^^^^^^^^^^
    ...<5 lines>...
        this_step=args.this_step,
        ^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "C:\Users\ue71603\MG_Daten\github\badger\tools\script_runner.py", line 417, in main
    result = load_and_run(script_name, script_args)
  File "C:\Users\ue71603\MG_Daten\github\badger\tools\script_runner.py", line 111, in load_and_run
    result = main_function(*args1)
  File "C:\Users\ue71603\MG_Daten\github\badger\conv\epip_db_to_db.py", line 106, in main
    epip_service_calendar(source_db, target_db, generator_defaults)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ue71603\MG_Daten\github\badger\transformers\epip.py", line 756, in epip_service_calendar
    + load_local(db_read, DayType, embedding=True)
      ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ue71603\MG_Daten\github\badger\netexio\dbaccess.py", line 99, in load_local
    return list(
        load_generator(db, clazz, limit, filter_id, embedding, embedded_parent, cache)
    )
  File "C:\Users\ue71603\MG_Daten\github\badger\netexio\dbaccess.py", line 477, in load_generator
    yield from load_embedded_transparent_generator(
        db, clazz, limit, filter_id, parent, cache
    )
  File "C:\Users\ue71603\MG_Daten\github\badger\netexio\dbaccess.py", line 530, in load_embedded_transparent_generator
    yield resolve_attr(obj, split)
          ~~~~~~~~~~~~^^^^^^^^^^^^
  File "C:\Users\ue71603\MG_Daten\github\badger\netexio\attributes.py", line 9, in resolve_attr
    obj = obj[name]
          ~~~^^^^^^
IndexError: list index out of range


Process finished with exit code 0

[skinkie]

I saw this error also with another feed. It might be related to embedding.

[skinkie]

The cause is that the ServiceCalendar in both deliveries has the same id. So when changing one manually everything works (after having updated the typo's in de script files). Now the MobilityData feed validator complains about timezones, I don't see any reason why this is wrong.

[ue71603]

How should we deal with this in general? In merging things don't com from the same source, so the id of generated stuff is not necessarily different. ch:1:ServiceCalendar:1 or also ch:1:SiteFrame:1 probably will always be a problem.

I am not sure that it might only apply to #20

[ue71603]

Or do we need something like: tools.merge_prepare [files]?

[ue71603]

or would this be a step in conv.netex_to_db [file1 file2] outputfile merging=True

[skinkie]

The idea in NeTEx to prevent is, is that everything has its own Codespace. The prefix part of the id. If that is guaranteed per export. It is solved. If the data is from the same source, the same object should be the same.

[ue71603]

I try using different codespaces again.

[skinkie]

I think the main issue is here that the ServiceCalendar is not using the Codespace that all other elements use.
Next to that, please don't make it a common practise to export .zip files. That is a major performance hit opposed to .xml.gz

[ue71603]

Validator. It is a rule:
https://gtfs.org/documentation/schedule/reference/#agencytxt

and we have

agency_id,agency_name,agency_url,agency_timezone,agency_lang,agency_phone,agency_fare_url,agency_email
0,BlaBlaCar Bus,http://openov.nl/,Europe/Paris,,,,
FLIXBUS-eu,FlixBus-eu,http://openov.nl/,UTC,,,,
FLIXTRAIN-eu,FlixTrain-eu,http://openov.nl/,UTC,,,,

I don't think we can ignore this. Where would we harmonise time?
Or is merging not possible?
Is also an option.

[skinkie]

I thought this was never a requirement, but even up to ten years ago this seems to be a hard requirement for the GTFS-reference. I don't think it can be done correctly when there are calendars spanning the Day Light Saving borders, because this will introduce multiple trips or trips overlapping. It is not as simple as rewriting all departures with some offset. When the above would work, a stop_timezone should be added.

It is a great assumption that an organisation exporting UTC data must have their affairs in order. We now that Flixbus hasn't.

For me converting timezones would be a completely different feature.

• guard there is no calendar spanning day light saving
• shift the departures to UTC, with respect to the provided timezone and calendar period

Writing a solution that could introduce new calendars would be a different feature, but will never work with realtime data referencing the ServiceJourney id. You can copy this comment to a new ticket and place it somewhere on the roadmap.

[ue71603]

So what do we do?
In GTFS the situation is clear

In NeTEx: Should the merge have been blocked there as well the moment it was clear, that timezones are different?

[skinkie]

Practically nothing in NeTEx indicates that a timezone is different. There are frame defaults, and there are at locale level things that indicate that there are time zones at play. The word "time zone" is not even part of NeTEx Part 2. I'll open an issue on basecamp.