How to preprocess new dataset and generate embedding representations?

[randomFriendlyGuy]

Thanks for your excellent research.
I am trying to apply HermesSim on a new dataset, and I have successfully run lifting/pcode_lifter.py.
However, I was unable to run preprocess/preprocessing_pcode.py, which is invoked by preprocess/preprocess_all.sh.
The error indicates that several files are required, including opcodes_dict.json in pcode_acfg, pcode_tscg, pcode_sog, pcode_iscg. Even after copying these files from inputs/pcode_raw/pcode_xxx, it still throws exception.

python preprocess/preprocessing_pcode.py --freq-mode -f pkl -s new_testing -i dbs/new/features/pcode_raw_new -o inputs/pcode-new

Traceback (most recent call last):
  File "/xxxx/HermesSim/preprocess/preprocessing_pcode.py", line 478, in <module>
    main()
    ~~~~^^
  File "/xxxx/lib/python3.13/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/xxxx/lib/python3.13/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
  File "/xxxx/lib/python3.13/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/xxxx/lib/python3.13/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "/xxxx/HermesSim/preprocess/preprocessing_pcode.py", line 461, in main
    str_dict, pkl_dict = create_functions_dict(
                         ~~~~~~~~~~~~~~~~~~~~~^
        input_dir, opc_dicts, dump_str, dump_pkl)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/xxxx/HermesSim/preprocess/preprocessing_pcode.py", line 395, in create_functions_dict
    tqdm(map(process_one_file, args), total=len(args)):
    ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/xxxx/lib/python3.13/site-packages/tqdm/std.py", line 1181, in __iter__
    for obj in iterable:
               ^^^^^^^^
  File "/xxxx/HermesSim/preprocess/preprocessing_pcode.py", line 358, in process_one_file
    f_list = create_features_matrix(
        nodes, fva_data, opc_dicts[gtype], gtype, arch)
  File "/xxxx/HermesSim/preprocess/preprocessing_pcode.py", line 312, in create_features_matrix
    opcs[i] = opc_dict[ty]
              ~~~~~~~~^^^^
KeyError: 'mi_reg

I would appreciate your assistance running HermesSim on the new dataset. Could you please provide detailed guidance (or a script) on generating embedding vectors for new binaries? Thank you in advance.

[sgfvamll]

Run the script with the --training flag on the training set will produce these files. Or if you are using our model checkpoints, you should be able to find these files in the provided inputs.tar.gz.

[randomFriendlyGuy]

Thanks for your reply.
After adding the --training flag, I still met the exception.

[i] Freq_mode:  True
  0%|                                                                             | 0/1000[00:00<?, ?it/s]
Traceback (most recent call last):
  File "xxxx/HermesSim/preprocess/preprocessing_pcode.py", line 478, in <module>
    main()
    ~~~~^^
  File "xxxx/lib/python3.13/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "xxxx/lib/python3.13/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
  File "xxxx/lib/python3.13/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "xxxx/lib/python3.13/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "xxxx/HermesSim/preprocess/preprocessing_pcode.py", line 438, in main
    opc_dicts = token_mapping(
        input_dir, output_dir, freq_mode)
  File "xxxx/HermesSim/preprocess/preprocessing_pcode.py", line 188, in token_mapping
    opc_counters[gtype][ty].update([opc])
    ~~~~~~~~~~~~~~~~~~~^^^^
KeyError: 'mi_reg'

I noticed that this mi is a part of mips, and found the issue is caused by arch = f_json.split('-')[0][:-2] and here.
The method for getting the architecture name is incompatible with the new dataset. After adapting this line to my dataset, it works.

However, there is a new error while generating the embedding.
The model/core/graph_factory_base.py will load cached features directly, with suffix like '0_1_2_3', see here. However, those files are downloaded from the given link for Dataset-1, and the preprocessing_pcode.py merely produces a pickle file named graph_func_dict_opc_True.pkl.
Could you please provide some guidance on how to generate the cache files for models to embed?

Thanks for your assistance.

[sgfvamll]

These cache files are automatically generated at here.
Please provide more context if you still need help. And sorry for the late reply.

[lilian-lilifox]

The model/core/graph_factory_base.py will load cached features directly, with suffix like '0_1_2_3', see here. However, those files are downloaded from the given link for Dataset-1, and the preprocessing_pcode.py merely produces a pickle file named graph_func_dict_opc_True.pkl.
Could you please provide some guidance on how to generate the cache files for models to embed?

If we try to use preprocessing_pcode.py to generate inputs/pcode , we'll get inputs/pcode/pcode_tscg/Dataset-1_testing/graph_func_dict_opc_True.pkl, but not inputs/pcode_raw/pcode_tscg/Dataset-1_testing/graph_func_dict_opc_55.0_1_2_3.pkl as the file in the provided inputs.tar.gz. And this will cause the 3.model training step fail.

[sgfvamll]

The model/core/graph_factory_base.py will load cached features directly, with suffix like '0_1_2_3', see here. However, those files are downloaded from the given link for Dataset-1, and the preprocessing_pcode.py merely produces a pickle file named graph_func_dict_opc_True.pkl.
Could you please provide some guidance on how to generate the cache files for models to embed?

If we try to use preprocessing_pcode.py to generate inputs/pcode , we'll get inputs/pcode/pcode_tscg/Dataset-1_testing/graph_func_dict_opc_True.pkl, but not inputs/pcode_raw/pcode_tscg/Dataset-1_testing/graph_func_dict_opc_55.0_1_2_3.pkl as the file in the provided inputs.tar.gz. And this will cause the 3.model training step fail.

Sorry for that. Updating the configure json file accordingly should solve this issue ("graph_func_dict_opc_55.0.json" -> graph_func_dict_opc_True.json").