MeZo Forward Pass Implementation

[thistleknot]

Feature request

https://github.com/princeton-nlp/MeZO/blob/main/large_models/trainer.py

Motivation

Memory efficiency while training

Your contribution

Willing to test, train, and write-up bug reports.

[younesbelkada]

This implementation is very interesting IMO and fits the goals of PEFT library, anyone wants to give it a try?
cc @pacman100 @sayakpaul
It will be the first custom Trainer ever in PEFT, and the attached code seems quite clean. Happy to integrate it otherwise

[sayakpaul]

I would say it's better as a third-party showcase. I don't think it makes sense to add as a core feature of the library yet. Maybe, based on the adoption and community response, we could revisit this.

[Bearnardd]

Hi @younesbelkada @sayakpaul! I am also interested in this change. From my perspective there are 2 ways to implement this change. 1) It would be nice to add a showcase example how to combine peft with their trainer e.g. in the examples directory. One problem might be the fact that, if I am not mistaken you cannot simply install their code via e.g. pip and use as a package. You are forced to download their repo and use CLI which probably complicates things a bit. 2) Add a custom trainer as @younesbelkada mentioned.

[pacman100]

Hello everyone, I have read the paper today. I concur with Sayak that before investing efforts into Custom Trainer for it we need to gauge interest and performance of MeZO.

I like the approach 1 suggested by @Bearnardd to start with.

[Bearnardd]

I completely agree with your thoughts, @pacman100. Nowadays, numerous new ideas and papers emerge daily, making it impractical to dedicate time to include each one without first assessing the interest from the community and the actual performance of a particular solution. Simple example is a pretty cheap alternative :)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Aw, please implement MeZo guys, this could allow for near lossless compressed full fine tuned adaptors for true inference cost. Poor mans training would no longer have a in-superior connotation. Please @sayakpaul @pacman100 @Bearnardd ?

[thistleknot]

Yes please

[thistleknot]

I got this to work with bnb. train a 3b model with 5GB of ram using forward passes (effectively speeding up my training thanks to no need to do backward passes). I think it's worthwhile to integrate this, especially since it works alongside bnb .

[thistleknot]

from 90it/s to 60/it/s with Mezo

Nice! @thistleknot
When you say "bnb" do you mean 4bit?
When you say "from 90it/s to 60/it/s with Mezo" is 90 referring to single forward pass speeds not using mezo?

[thistleknot]

Yes and yes

…

On Sat, Aug 19, 2023, 10:22 AM Jeduh ***@***.***> wrote: Nice! @thistleknot <https://github.com/thistleknot> When you say "bnb" do you mean 4bit? When you say "from 90it/s to 60/it/s with Mezo" is 90 referring to single forward pass speeds not using mezo? — Reply to this email directly, view it on GitHub <#601 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABHKKOSK5P3ON546KWG7UXTXWDY55ANCNFSM6AAAAAAZMWBPV4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

@thistleknot would the finetuned model export as nf4? Would MeZo's predictions be made in nf4 too? How does that work?

[thistleknot]

yes and yes. the mezo function is simply a feed forward (I'm paraphrasing) compute_loss function, but it's plug and play with the lora setup with no qualms with integration that I see. I was trying to get it to work with gptq, but was unable to get a working setup, but I'm reading even if I did, it would still be limitied to a lora adapter.

So the best you can get with this without peft/lora, is simply faster training.

Shame relora isn't extended to anything other than llama.