Possible Issue with utf-8 encoding under windows while reading dataset

[sstiene]

I try to load a dataset with agml under win11 with python 3.12 but get an error.

How to reproduce it:

import agml

dataset_name = 'carrot_weeds_germany'
loader = agml.data.AgMLDataLoader(dataset_name)
dataset_path = loader.dataset_root

print(f"Datensatz heruntergeladen nach: {dataset_path}")

The error is:

Traceback (most recent call last):
  File "c:\Users\stefa\HSOS\Vorbereitung Einführung in die KI (BAT) - General\Praktika\label_studio_51\agml_test.py", line 4, in <module>
    loader = agml.data.AgMLDataLoader(dataset_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\stefa\cv-env\Lib\site-packages\agml\data\loader.py", line 149, in __init__
    self._info = make_metadata(dataset, kwargs.get('meta', None))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\stefa\cv-env\Lib\site-packages\agml\data\metadata.py", line 49, in make_metadata
    return DatasetMetadata(name)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\stefa\cv-env\Lib\site-packages\agml\data\metadata.py", line 105, in __init__
    self._load_source_info(name)
  File "C:\Users\stefa\cv-env\Lib\site-packages\agml\data\metadata.py", line 148, in _load_source_info
    **load_citation_sources()[name], dataset = name)
      ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\stefa\cv-env\Lib\site-packages\agml\utils\data.py", line 38, in load_citation_sources
    return json.load(f)
           ^^^^^^^^^^^^
  File "C:\Users\stefa\AppData\Local\Programs\Python\Python312\Lib\json\__init__.py", line 293, in load
    return loads(fp.read(),
                 ^^^^^^^^^
  File "C:\Users\stefa\AppData\Local\Programs\Python\Python312\Lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 22968: character maps to <undefined>

So it seams that there is an encoding issue while reading the citation_sources under Windows. I did test it with different datasets. Same error.

[sstiene]

Adding utf-8 encoding in data.py fixed the problem:

@functools.lru_cache(maxsize` = None)
def load_citation_sources() -> dict:
    """Loads the citation sources JSON file."""
    with open(os.path.join(
            os.path.dirname(os.path.dirname(__file__)),
            '_assets/source_citations.json'), encoding="utf-8") as f:
        return json.load(f)

[masonearles]

Thanks @sstiene! Do you mind making a PR? If not, no problem. @eranario can you work this into the next release?

…

On Sun, Mar 22, 2026 at 01:15 Stefan Stiene ***@***.***> wrote: *sstiene* left a comment (Project-AgML/AgML#80) <#80 (comment)> Adding utf-8 encoding in data.py fixed the problem: @functools.lru_cache(maxsize` = None)def load_citation_sources() -> dict: """Loads the citation sources JSON file.""" with open(os.path.join( os.path.dirname(os.path.dirname(__file__)), '_assets/source_citations.json'), encoding="utf-8") as f: return json.load(f) — Reply to this email directly, view it on GitHub <#80?email_source=notifications&email_token=AG5FGPAG3UKGHHIGQV2ZQY34R6OJNA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMJQGU3TONRXG432M4TFMFZW63VKON2WE43DOJUWEZLEUVSXMZLOOSWGM33PORSXEX3DNRUWG2Y#issuecomment-4105776777>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG5FGPBJQZZZKE7DJ3U22FD4R6OJNAVCNFSM6AAAAACW2BG3P6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DCMBVG43TMNZXG4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[eranario]

@sstiene which version of AgML are you on?

I can see an existing fix here in line 64 (agml/utils/data.py) in version 0.7.4

for file_path in file_paths:
        if os.path.exists(file_path):
            with open(file_path, 'r', encoding='utf-8') as f:
                data = json.load(f)
                if isinstance(data, dict):  # Ensure it's a dictionary before merging
                    combined_data.update(data)  # Merging dictionaries (keys from the second file overwrite the first)
                else:
                    raise ValueError(f"JSON structure in {file_path} is not a dictionary")

[sstiene]

Sorry, than this was my fault. I have 0.7.0. I had to use python3.12 to make label-studio and Voxel 51 work together. And under python 3.12 venv pip install agml installed agml in version 0.7.0

[eranario]

Got it, let me know if upgrading AgML to the latest version helps.