Questions Regarding Reproduction of Results on EHF Dataset

[LeCongThuong]

Dear authors,

Thank you for sharing your excellent work!

I’m currently trying to reproduce the results you reported on the EHF dataset. However, I encountered some difficulties.

Since the provided codebase does not appear to support evaluation on the EHF dataset directly, I used the EHF.py script from SMPLX-er for evaluation. Unfortunately, I was not able to reproduce the results reported in your paper with any of the model weight you provided.

Here are the results I obtained:

====== EHF Evaluation ======
PA MPVPE (All): 36.33 mm
PA MPVPE (L-Hands): 12.42 mm
PA MPVPE (R-Hands): 11.91 mm
PA MPVPE (Hands): 12.17 mm
PA MPVPE (Face): 4.85 mm

MPVPE (All): 57.49 mm
MPVPE (L-Hands): 41.75 mm
MPVPE (R-Hands): 39.56 mm
MPVPE (Hands): 40.66 mm
MPVPE (Face): 17.26 mm

PA MPJPE (Body): 39.89 mm
PA MPJPE (L-Hands): 12.32 mm
PA MPJPE (R-Hands): 11.76 mm
PA MPJPE (Hands): 12.04 mm

I’d like to ask a few questions:

1. Could you please clarify which exact name of model is provided in the README.md?
2. Would it be possible for you to share the evaluation code or a complete codebase that can reproduce the EHF results reported in the paper?

Thank you for your time, and I look forward to your response.

[wqyin]

Thank you for being interested in our work.

The current released model file is a special version selected for the best in-the-wild scenario inference, while the quantitative result is not yet included in the paper. We will release the model used to report the accuracy in the paper soon.
We are still working on the code for the dataset toolbox and will release it once it is ready.

[noahcao]

Hi any update on this? When will the checkpoint and eval toolkit released for AGORA-val / AGORA-test?

[TannerGilbert]

@wqyin Thanks for the great work. Just tested the package and with the current weights I also noticed that the hands are not being tracked as expected. Any update on when new model files will be released?