(convagent) [zdeng@tooarrana1]/fred/oz401/zdeng% python /fred/oz401/zdeng/genagents/coser_datset.py
Generating train split: 301748 examples [00:33, 8950.63 examples/s]
Generating test split: 0 examples [00:00, ? examples/s]
Traceback (most recent call last):
File "/fred/oz339/zdeng/.conda/envs/convagent/lib/python3.10/site-packages/datasets/builder.py", line 1871, in _prepare_split_single
writer.write_table(table)
File "/fred/oz339/zdeng/.conda/envs/convagent/lib/python3.10/site-packages/datasets/arrow_writer.py", line 643, in write_table
pa_table = table_cast(pa_table, self._schema)
File "/fred/oz339/zdeng/.conda/envs/convagent/lib/python3.10/site-packages/datasets/table.py", line 2293, in table_cast
return cast_table_to_schema(table, schema)
File "/fred/oz339/zdeng/.conda/envs/convagent/lib/python3.10/site-packages/datasets/table.py", line 2241, in cast_table_to_schema
raise CastError(
datasets.table.CastError: Couldn't cast
scenario: string
topic: string
key_characters: list<item: struct<motivation: string, name: string>>
child 0, item: struct<motivation: string, name: string>
child 0, motivation: string
child 1, name: string
dialogues: list<item: struct<character: string, message: string>>
child 0, item: struct<character: string, message: string>
child 0, character: string
child 1, message: string
speaking_characters_w_env: list<item: string>
child 0, item: string
major_characters: list<item: string>
child 0, item: string
weight: double
plot: struct<chapter: string, i_chunk: int64, i_p: int64, key_characters: list<item: struct<description: string, name: string, summary: string>>, prominence: int64, state: string, summary: string>
child 0, chapter: string
child 1, i_chunk: int64
child 2, i_p: int64
child 3, key_characters: list<item: struct<description: string, name: string, summary: string>>
child 0, item: struct<description: string, name: string, summary: string>
child 0, description: string
child 1, name: string
child 2, summary: string
child 4, prominence: int64
child 5, state: string
child 6, summary: string
character_profiles: struct<Abby Abernathy: string, Abdullah Smith: string, Abraham Van Helsing: string, Adrian Ivashkov: string, Adso of Melk: string, Aelin Ashryver Whitethorn Galathynius: string, Ahab: string, Aibileen Clark: string, Alec: string, Alec d'Urberville: string, Aleksey Fürst: string, Alexei
...
er: string
child 245, Spencer Hastings: string
child 246, Starbuck: string
child 247, Steve Miller: string
child 248, Steven Conklin: string
child 249, Stubb: string
child 250, Sunny: string
child 251, Susannah Dean: string
child 252, Sydney Sage: string
child 253, Tate Collins: string
child 254, Tess Durbeyfield: string
child 255, Tessa: string
child 256, Thalia Grace: string
child 257, The Crimson King: string
child 258, Tirian: string
child 259, Tom Smith: string
child 260, Tomas: string
child 261, Tommaso de' Cavalieri: string
child 262, Tommy Barban: string
child 263, Tormund Giantsbane: string
child 264, Travis Maddox: string
child 265, Trent Kalamack: string
child 266, Trillian: string
child 267, Trina: string
child 268, Tyrion Lannister: string
child 269, Verin Mathwin: string
child 270, Victor Frankenstein: string
child 271, Walter Hartright: string
child 272, Wanda: string
child 273, Wendy Everly: string
child 274, Will Blakelee: string
child 275, Will Halloway: string
child 276, Will Parry: string
child 277, Will Tweedy: string
child 278, William of Baskerville: string
child 279, Willoughby: string
child 280, Xander Thomas Carrow: string
child 281, Yrene Towers Westfall: string
child 282, Zia Rashid: string
child 283, Zoë Nightshade: string
book: string
i_p: int64
i_c: int64
tag: string
-- schema metadata --
pandas: '{"index_columns": [], "column_indexes": [], "columns": [{"name":' + 1688
to
{'conversations': [{'from': Value(dtype='string', id=None), 'value': Value(dtype='string', id=None)}]}
because column names don't match
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/fred/oz401/zdeng/genagents/coser_datset.py", line 5, in <module>
dataset = datasets.load_dataset("Neph0s/CoSER", cache_dir="/fred/oz401/zdeng/genagents/data")
File "/fred/oz339/zdeng/.conda/envs/convagent/lib/python3.10/site-packages/datasets/load.py", line 2084, in load_dataset
builder_instance.download_and_prepare(
File "/fred/oz339/zdeng/.conda/envs/convagent/lib/python3.10/site-packages/datasets/builder.py", line 925, in download_and_prepare
self._download_and_prepare(
File "/fred/oz339/zdeng/.conda/envs/convagent/lib/python3.10/site-packages/datasets/builder.py", line 1001, in _download_and_prepare
self._prepare_split(split_generator, **prepare_split_kwargs)
File "/fred/oz339/zdeng/.conda/envs/convagent/lib/python3.10/site-packages/datasets/builder.py", line 1742, in _prepare_split
for job_id, done, content in self._prepare_split_single(
File "/fred/oz339/zdeng/.conda/envs/convagent/lib/python3.10/site-packages/datasets/builder.py", line 1873, in _prepare_split_single
raise DatasetGenerationCastError.from_cast_error(
datasets.exceptions.DatasetGenerationCastError: An error occurred while generating the dataset
All the data files must have the same columns, but at some point there are 13 new columns ({'weight', 'character_profiles', 'speaking_characters_w_env', 'tag', 'book', 'plot', 'i_c', 'dialogues', 'i_p', 'major_characters', 'topic', 'scenario', 'key_characters'}) and 1 missing columns ({'conversations'}).
This happened while the json dataset builder was generating data using
hf://datasets/Neph0s/CoSER/test/test_set.json (at revision 2446f97a077975e2cda8a7eaad20c201089c66f9)
Please either edit the data files to have matching columns, or separate them into different configurations (see docs at https://hf.co/docs/hub/datasets-manual-configuration#multiple-configurations)
Hi,
看起来huggingface测试集合有问题：