Invalid UTF8 string when querying /item-storage/items

[lf103]

Python 3.10.6

In querying /item-storage/items, I am receiving this error output while using LDLite to get the data from my institutions Folio tenent (see attachment). I am using the sample script from the ldlite git hub to get the data and create the tables: https://github.com/library-data-platform/ldlite/blob/main/examples/folio_demo.py.

I haven't been able to get the item records or anything else after that point in the script. I have tried breaking up the queries and table creation parts into multiple cell in my Jupyter notebook and have managed to get some of the tables created but it stalls out on items (see attachment).

item-storage_items_errorOutput.txt

Test_LDLite2_script.pdf

[nassibnassar]

Which version of LDLite is this happening with? Also which operating system?

[lf103]

I'm on Windows 10. LDLite version 0.0.33 according to my env.

[nassibnassar]

If you would, please upgrade to LDLite 0.0.34 0.0.35 and try it again to see if that helps.

[nassibnassar]

As an aside, one workaround to prevent the script from stopping on that error would be to catch BaseException. So instead of:

    except (ValueError, RuntimeError) as e:
        print('folio_demo.py: error processing "' + q[1] + '"', file=sys.stderr)

Try:

    except BaseException as e:
        print('folio_demo.py: error processing "' + q[1] + '": ' + str(e), file=sys.stderr)

[lf103]

I upgraded to the latest version, 0.0.35 and changed to the suggested BaseException in the script. I am getting different errors now and not able to connect to our tenant.

I have restarted my computer but still getting the same connection problem.

Test_LDlite4_script_with_outputs.pdf

[nassibnassar]

I would restart the Jupyter Notebook server after the packages were upgraded and try it again. If you still get those connection errors, it looks to me like an issue with FOLIO which your hosting provider can probably help with.

[nassibnassar]

Also could it be that url is set to 'api url'?

[lf103]

I used our api url in actually running the script (which was the same as in the first one posted). I'll talk to hosting and see what they have to say.

[nassibnassar]

Are you able to connect to that first path, /instance-storage/instance-relationships, using some other tool, at about the same time that you get connection errors from LDLite?

The other two examples, instance-storage/instances and item-storage/items, are missing a leading slash, though I am not sure if that would cause the (somewhat different) errors you got in those cases.

[lf103]

For item-storage/items, I corrected to /item-storage/items and received this error:
LDlite_test.pdf

I am having trouble getting to the other paths in my Folio tenant. I think that a permission needs to be added to the Reports User we created for reporting. I tried to get to /instance-storage/instance-relationships using Postman and received a 503 error, which makes me think it might be about our permissions.

[nassibnassar]

On /item-storage/items, I think there might be a bug in DuckDB. (Sorry for the trouble.) LDLite has experimental support for SQLite as an alternative embedded database system; if you happen to have time to try it, you can replace

db = ld.connect_db(filename='test1_ldlite.db')

with

db = ld.experimental_connect_db_sqlite(filename='test1_ldlite_sqlite.db')

On /instance-storage/instance-relationships, I agree that it sounds like it could be a permission problem.

[lf103]

I replaced the code for the db as suggested (keeping the BaseException from earlier) and it was successful! Tables were created!

I am working with our Systems Librarian to get the permissions updated.

[nassibnassar]

If the SQLite support works well, we can remove the "experimental" caveat. The only downside I see of using SQLite is that it does not support schema namespaces and so the table names will not be compatible with Metadb naming. This means if you want to run one of the community-created queries, the table names in the query have to be changed.

[brycekbargar]

Hello @lf103 ! It's been awhile! I just wanted to make sure you knew that experimental_connect_db_sqlite was removed with the release of version 4.0.0 #76 . The transformation logic where you were seeing this error has been reworked and I believe the underlying issue has been fixed. Please let me know if you encounter this again after switching back to connect_db.