Reproducing the same RECALL@1 and MRR values

[TakahiroHaruyama]

Hi,

Thank you for publishing your research results as open source!

I got interested in your research and tried to reproduce the same RECALL@1 and MRR values using postprocess/1.generate_testing/configures/full/xm-1000-10000_Ds1.json after generating input data of Dataset-1 on my own and training/inferring. However, the values were 0.405 and 0.48, slightly different from the values shown in the paper (0.438/0.508).

Next, I trained and inferred the model in the same way using the input data you provided. The results were almost the same as those in the paper (0.445/0.515). So I compared the input files.

Provided *_55.0.pkl (extracted from inputs.tar.xz):

-rwx------@ 1 xxxx  staff  2622071486  5 24  2023 inputs/pcode_raw/pcode_sog/Dataset-1_training/graph_func_dict_opc_55.0.pkl*
-rwx------@ 1 xxxx  staff  124962329  5 25  2023 inputs/pcode_raw/pcode_sog/Dataset-1_validation/graph_func_dict_opc_55.0.pkl*
-rwx------@ 1 xxxx  staff  6088681810  5 25  2023 inputs/pcode_raw/pcode_sog/Dataset-1_testing/graph_func_dict_opc_55.0.pkl*

My generated *_True.pkl:

-rw-r--r--@ 1 xxxx  staff  4471414394  5  8 12:48 inputs/pcode_raw/pcode_sog/Dataset-1_training/graph_func_dict_opc_True.pkl
-rw-r--r--@ 1 xxxx  staff  218257863  5  8 12:51 inputs/pcode_raw/pcode_sog/Dataset-1_validation/graph_func_dict_opc_True.pkl
-rw-r--r--@ 1 xxxx  staff  10133205287  5  8 15:05 inputs/pcode_raw/pcode_sog/Dataset-1_testing/graph_func_dict_opc_True.pkl

Clearly, my input files are getting much larger. In the previous issue, you said the token mapping algorithm and parameters didn't change, so I'm not sure why there is a big difference.

Next, I compared the token mapping files.

Provided files (extracted from inputs.tar.xz):

-rwx------@ 1 haru  staff  23628234  5 24  2023 opc_counter.json*
-rwx------@ 1 haru  staff  23628201  5 24  2023 opc_occurs.json*
-rwx------@ 1 haru  staff      8209  5 24  2023 opcodes_dict.json*

My generated files:

-rw-r--r--@ 1 haru  staff  22342663  5  8 11:47 opc_counter.json
-rw-r--r--@ 1 haru  staff  22342625  5  8 11:47 opc_occurs.json
-rw-r--r--@ 1 haru  staff      7575  5  8 11:47 opcodes_dict.json

I checked the data in the files too.

In [2]: opc_cnt_ext = json.load(open('pcode_raw_extracted/pcode_sog/opc_counter_extracted.json'))
In [3]: opc_ocu_ext = json.load(open('pcode_raw_extracted/pcode_sog/opc_occurs_extracted.json'))
In [4]: opc_dict_ext = json.load(open('pcode_raw_extracted/pcode_sog/opcodes_dict_extracted.json'))
In [5]: opc_cnt = json.load(open('inputs/pcode_raw/pcode_sog/opc_counter.json'))
In [6]: opc_ocu = json.load(open('inputs/pcode_raw/pcode_sog/opc_occurs.json'))
In [7]: opc_dict = json.load(open('inputs/pcode_raw/pcode_sog/opcodes_dict.json'))
In [13]: opc_cnt['opc']['IADD']
Out[13]: 20736548
In [14]: opc_cnt_ext['opc']['IADD']
Out[14]: 21875459
In [19]: opc_ocu['opc']['ISUB']
Out[19]: 197988
In [20]: opc_ocu_ext['opc']['ISUB']
Out[20]: 198219
In [26]: opc_ocu['num_funcs']
Out[26]: 256625
In [27]: opc_ocu_ext['num_funcs']
Out[27]: 256625
In [33]: len(opc_dict.keys())
Out[33]: 422
In [34]: len(opc_dict_ext.keys())
Out[34]: 461

The extracted json files likely have more instructions than mine though the number of analyzed functions matched. So I guess there is a difference in lifting (command line option in GSAT?).

Is it possible to reproduce the same accuracy using the current version of the scripts? If not, could you tell me how to change the scripts?
I generated my input data using the shell scripts provided in the repository. No change in the lifting/preprocessing python scripts. I used gsat-1.0.jar.

[sgfvamll]

Sorry, I'm not sure about the reason for this discrepancy either. Initially, I suspected it might be due to some built-in randomness in Ghidra. However, given the significant difference in the pkl file sizes, that seems unlikely. The current command-line parameters should be the same as those used during the experiment.
If it's convenient for you, I'd appreciate it if you could further investigate whether there are significant differences between the JSON files generated on both ends—this would help me pinpoint the possible cause. And I'm wondering if you used the provided cfg_summary files.

[TakahiroHaruyama]

Thanks for reply!

I'd appreciate it if you could further investigate whether there are significant differences between the JSON files generated on both ends

I checked before asking but will check if anything has been overlooked.

And I'm wondering if you used the provided cfg_summary files.

No I created the cfg_summary files by running summarize_dataset_1.sh. I used the input files (dataset_info_csv and ACFG feature files) downloaded from here (python3 ../gdrive_download.py --features).

[TakahiroHaruyama]

I checked the difference between the json files. The biggest difference is PIECE.

So I suspected the -opt option is different, but it was discussed already in the issue #16 and both should be -opt 0.
Can you guess the cause based on the big difference of PIECE? Or can we explain the big size difference of the input .pkl files?

[sgfvamll]

Sorry. Actually I mean the json files under the path like "./dbs/Dataset-1/features/testing/pcode_raw_Dataset-1_testing" (i.e., the SOG graphs). If it's convenient, post representative "*_acfg_disasm.json" files from both sides here and I'll have a look.
I'll rerun the steps next week to see if I can reproduce your problem.

[TakahiroHaruyama]

I'm attaching the sample "*_acfg_disasm.json" files of ncat. "*_extracted.json" means files extracted from dbs.tar.xz that you provided.

lifting_artifacts.zip