dynamic column structure makes it very difficult to use the SDK in an automated form

[syed1992]

The nested data structure and the resulting dynamic columns mean that you can't just set up a standard script and expect to always work.

Recognise this is quite a difficult problem to solve and don't know of any easy way to get round it. But I think this is quite an existential question for the usefulness of the SDK. If it doesn't allow things to be automated, it won't be used at all.

[KitBurgess]

Thanks for the feedback - w.r.t the dynamic columns, are you referring to vessel.0, vessel.1, vessel.2 etc?.

Which parts of the schema wouldn't work with a standard script?

[KitBurgess]

The to_df method flattens the data structure - is this different to how you'd like to see it flattened?

[syed1992]

so for example, the case I'm working with right now at request of a client is to be able to pull up a list of vessel that were classified as floating storage a historical point in time. Once i've pulled my data, to try and clean this im then only taking my columns storage_event.1, storage_event.2 etc. However, there is no way of me knowing before i've pulled the data how many of these there will be in the dataframe. So each time I do it, i have to do df.columns.values first, look down and see how far it goes (up to 3 or 4 for example) and then ammend my script to include those columns

[syed1992]

I don't actually know how else it could be flattened (not very helpful i know). It may even be that I'm approaching this in the wrong way and there is an automated way of capturing those columns. It's just a glaring limitation that stands out to me in what I'm currently trying to do.

[KitBurgess]

I think splitting a single cargo movement into multiple rows will solve this - each row corresponding to a vessel, or an event, or a product etc.