319 add bioclip training - #320
Conversation
0d63e56 to
c21cb91
Compare
c21cb91 to
025982e
Compare
| torchvision.transforms.v2.Compose: Complete preprocessing pipeline. | ||
| """ | ||
| if architecture == "bioclip_2": | ||
| assert resize_width == 224 and resize_height == 224, "BioClip only accepts image dimensions of 224x224" |
There was a problem hiding this comment.
If that's the case, maybe we force the resize width and height to be 224 instead of depending on the user to get it right
| """ | ||
| import open_clip | ||
| from peft import LoraConfig, get_peft_model | ||
| import torch |
There was a problem hiding this comment.
should add these to the package dependencies
There was a problem hiding this comment.
looks like you also need huggingface_hub and a specific version of transformers to be compatible
|
| checkpoint = {'model': model.state_dict(), | ||
| 'stats': stats} | ||
| # save optimizer and scheduler state dicts if they are provided | ||
| if model.__class__.__name__=="BioClip": |
There was a problem hiding this comment.
do all models we use have this attribute so that this check will not error out?
There was a problem hiding this comment.
yes! all of them have this attribute and it returns the name of the class in model_architecture (so 'EfficientNet', 'ConvNeXtBase', or 'BioClip')
| return img_tensor, [file_path], [frame], torch.tensor([(height, width)]) | ||
|
|
||
|
|
||
| def get_model_transforms(resize_height, resize_width, architecture=None): |
There was a problem hiding this comment.
this can be a helper function, add an underscore at the front and remove from the init please
|
If we only need the architecture param for bioclip, maybe we want a separate function that creates a dataloader specficially for bioclip that includes the transforms instead |
2 participants
Integrates BioClip2 into the model training pipeline
Relies on features from #316 and #318 for proper functionality
model_architecture.pyTrainGeneratorso bioclip-specific transforms can be appliedget_model_transformsthat returns either default transforms or bioclip specific transformsTrainGeneratorandtrain_dataloaderload_classifier,save_classifier,load_classifier_checkpoint, and frozen epoch unfreezing logic for bioclip (especially since we only train the LoRA weights in bioclip)Does not integrate BioClip2 into classification pipeline
Closes #319