Collecting trajectories in SimplerEnv to make SFT datasets

[JaehongJaehongMin]

Hello,
Thank you for sharing your great work!

I have some question about collecting trajectories in SimplerEnv-Bridge simulation.

I've fully fine tuned the original OpenVLA checkpoint with your rlds dataset you've uploaded on the issue#8 (let me call it your data).
The fully fine tuned model (please let me call it OpenVLA-Full-SFT) shows good performance! Almost same performance with your LoRA fine-tuned model on the hugging face.(Great! Thank you!)

And then I collected trajectories with this OpenVLA-Full-SFT model following your README.

The problem is here.
I fully fine-tuned the original OpenVLA checkpoint with trajectories that I collected using OpenVLA-Full-SFT (let me call it my data).
And then I've evaluated this OpenVLA-Full-SFT(w/ my dataset).
But it shows really bad performance.

In short,
OpenVLA-Full-SFT (w/ your data) works good,
OpenVLA-Full-SFT (w/ my data) works bad.

Is there anything I should be careful about when collecting trajectories or fully fine tune the original OpenVLA model?
Or could you please let me know the details about your collecting process?

[JaehongJaehongMin]

I believe there may have been some differences in how we collected the trajectories, possibly regarding the discretization of the action space.

In your paper, you mentioned that the trajectories were collected using the Octo model and in the same manner as LAPA.
After visualizing your Simpler fine-tuning data (as shown in the graph below), it appears that the actions are discretized which makes sense since Octo predicts actions in continuous space.

I have trained OpenVLA several times using my own selected data in Simpler, without discretization (I mean raw data), and observed poor performance.
This leads me to believe that preprocessing, specifically discretization is critical for SimplerEnv. I am planning to apply discretization and retrain OpenVLA accordingly.

Could you please confirm if this matches the way you prepared your fine-tuning data?

[zzj1111]

Hello, Jaehong. It seems that you does not "unnorm" your action vector. Commonly, the distribution of every dimension in action vector will in the range of [-1,1], not in [-0.03,0.03]. Please show me your SFT code and let me see what I can do for you

[JaehongJaehongMin]

Thank you for your reply!

Sorry for the confusion, but the graph above corresponds to the dataset you uploaded to Google Drive (#8). On the other hand, I’m using the official OpenVLA full fine-tune code train.py (I believe your OpenVLA-SFT-Simpler model was also trained using this code). This dataset appears to be working well.

To summarize the situation:

I currently have two datasets.

• One is your dataset from Google Drive (OpenVLA-SFT Data Collection #8) or Your Google Drive (I think you updated the link),

• The other is my own dataset, collected in Simpler using your code (specifically, the action collection part in this script). It doesn't work well with fine-tuning. I collected it one by one and checked if there is any strange behavior. I don't think it's about the data quality. You can check the trajectories in here My Google Drive.

The figure below shows the action distribution for each dimension.

• The first row corresponds to my dataset.

• The second row corresponds to yours.

From what I can tell, the main difference seems to be in the action discretization. If I recall correctly, Octo predicts actions in continuous space, whereas LAPA applies discretization as mentioned in your paper.

Could you kindly share more details on how the discretization was done for your dataset?

[zzj1111]

The discretization we used is compatible with OpenVLA's SFT code. I think if you use train.py from OpenVLA's repo to SFT, the discretization should work well.
I think the next step that may be helpful is to check some intermediate data when you run SFT. You can print the Ground Truth Action tokens and convert them to action vectors. Then manually check the distribution of action vectors-----I still guess the problem may come from "unnorm" procedure. If you find numbers in these vectors are very low(i.e. [0.01,0.02,-0,01,-0.03,0.003,-0.004,1]), the problem should be attributed to "unnorm". If so, let me know

[JaehongJaehongMin]

I'm sorry, but I don't quite understand why the 'unnorm' procedure is necessary. From what I see, both my data and your data appear to be in a similar range. You can verify this from the graph I shared earlier. Could you kindly clarify?

[Kalcy-U]

I use <model_checkpoint>/dataset_statistics.json to load norm_stats after self.vla = AutoModelForVision2Seq.from_pretrained:

        # [Hacky] Load Dataset Statistics from Disk (if passing a path to a fine-tuned model)
        if os.path.isdir(self.openvla_path):
            with open(Path(self.openvla_path) / "dataset_statistics.json", "r") as f:
                self.vla.norm_stats = json.load(f)

But it still looks bad.

[leonwu117]

Hello, jaehong. Have you sovle this problem. I also used the author's collection code directly for SFT, but found that the performance was too poor and the actions were not completed well. However, using the SFT dataset provided by the author to finetune, at least there were still successful tasks

[JaehongJaehongMin]

@leonwu117
Hello! Sorry, but I already gave up on it.
I applied various post-processing steps to my own dataset collected with the trajectory collecting code.
I also visualized the author’s trajectories and mine for the same episode, and they look very similar. So It seems like I collected them correctly.

I've also trained another VLA model (CogACT) on my data, but its performance was poor.