MRLib: Multimodal Recommendation Library

MRLib is an open-source, research-oriented library for multimodal recommendation.

中文文档: 🇨🇳 中文版

🎉 News

• [2026.05]🎯[Update]: We have added MSCA (accepted at WWW 2026) to MRLib, thanks to @recomall.
• [2026.04]🎯[Update]: We release the MRLib as a comprehensive benchmark and code base for mutlimodal recommendations.

---

🌟 Key Features

✨ Automatic Modality Discovery

• Zero Configuration: Automatically scans *_feat.npy and *_feat.pt files
• Flexible Integration: Supports visual, textual, audio, and other modalities
• Dynamic Loading: Loads only available modalities per dataset

💾 Intelligent Graph Caching

• Model-Specific Cache: Dedicated cache directory per model
• Parameter Validation: Verifies cache parameters match configuration
• Metadata Management: Stores graph construction parameters

📊 Real-time Visualization

• Training Metrics: Live plots of loss, and metrics
• Best Model Tracking: Auto-identifies best epoch

🔄 Continuous Updates

• Latest SOTA: Regular integration from top venues
• Active Maintenance: Bug fixes and optimizations
• Community Driven: Welcoming contributions

📚 Extensive Dataset Support

• Amazon Datasets: Baby, Sports, Clothing, Pet, Office, Toys, Beauty and etc.
• Video Datasets: TikTok and Microlens
• Custom Datasets: Clear format specifications

---

📋 Supported Models

Sorted by publication year. Reference: Awesome-Multimodal-Recommender-Systems

#	Model	Full Paper Title	Venue	Year	Link
1	VBPR	VBPR: Visual Bayesian Personalized Ranking from Implicit Feedback	AAAI	2016	link
2	MMGCN	MMGCN: Multi-modal Graph Convolution Network for Personalized Recommendation of Micro-video	ACM MM	2019	link
3	GRCN	Graph-Refined Convolutional Network for Multimedia Recommendation with Implicit Feedback	ACM MM	2020	link
4	LATTICE	Mining Latent Structures for Multimedia Recommendation	ACM MM	2021	link
5	DualGNN	DualGNN: Dual Graph Neural Network for Multimedia Recommendation	IEEE TMM	2021	link
6	SLMRec	Self-Supervised Learning for Multimedia Recommendation	IEEE TMM	2022	link
7	BM3	Bootstrap Latent Representations for Multi-modal Recommendation	WWW	2023	link
8	MMSSL	Multi-Modal Self-Supervised Learning for Recommendation	WWW	2023	link
9	FREEDOM	A Tale of Two Graphs: Freezing and Denoising Graph Structures for Multimodal Recommendation	ACM MM	2023	link
10	MGCN	Multi-View Graph Convolutional Network for Multimedia Recommendation	ACM MM	2023	link
11	DRAGON	Enhancing Dyadic Relations with Homogeneous Graphs for Multimodal Recommendation	ECAI	2023	link
12	LGMRec	LGMRec: Local and Global Graph Learning for Multimodal Recommendation	AAAI	2024	link
13	DiffMM	DiffMM: Multi-Modal Diffusion Model for Recommendation	ACM MM	2024	link
14	DAMRS	Improving Multi-modal Recommender Systems by Denoising and Aligning Multi-modal Content and User Feedback	KDD	2024	link
15	MENTOR	MENTOR: Multi-level Self-supervised Learning for Multimodal Recommendation	AAAI	2025	link
16	PGL	Mind Individual Information! Principal Graph Learning for Multimedia Recommendation	AAAI	2025	link
17	SMORE	Spectrum-based Modality Representation Fusion Graph Convolutional Network for Multimodal Recommendation	WSDM	2025	link
18	COHESION	COHESION: Composite Graph Convolutional Network with Dual-Stage Fusion for Multimodal Recommendation	SIGIR	2025	link
19	SSR	Structured Spectral Reasoning for Frequency-Adaptive Multimodal Recommendation	NeurIPS	2025	link
20	HPMRec	Hypercomplex Prompt-aware Multimodal Recommendation	CIKM	2025	link
21	LOBSTER	LOBSTER: Bilateral global semantic enhancement for multimedia recommendation	Information Fusion	2026	link
22	MSCA	Multi-view Semantic Contrastive Alignment for Multimodal Recommendation	WWW	2026	link

Models sorted by publication year. Table continuously updated with latest research.

---

🚀 Quick Start

Installation

# Clone repository
git clone https://github.com/Jinfeng-Xu/Multimodal-Recommendation-Library
cd Multimodal-Recommendation-Library

Basic Usage

# Run with default settings
python src/main.py -m HPMRec -d baby

# Specify GPU
python src/main.py -m COHESION -d sports --gpu_id 1

# Disable visualization
python src/main.py -m FREEDOM -d clothing --no-vis

Configuration

Models configured via YAML files in src/configs/model/:

# HPMRec.yaml
embedding_size: 64
feat_embed_dim: 64
n_mm_layers: 1
n_layers: [3]
knn_k: 10
mm_image_weight: 0.1
reg_weight: [0.001]
hyper_parameters: ["n_layers", "reg_weight"]

---

📁 Dataset Format

Required Files

data/
└── {dataset_name}/
    ├── inter.csv          # User-item interactions
    ├── visual_feat.npy    # Visual features (optional)
    ├── textual_feat.npy   # Text features (optional)
    └── *_feat.npy         # Other features (optional)

Interaction File Format

user_id,item_id,rating,label
0,123,5,1
1,456,4,1
2,789,5,1

Feature Files

• Format: .npy or .pt (PyTorch tensor)
• Shape: [num_items, feature_dim]
• Naming: {modality}_feat.{npy|pt}

Automatic discovery supports:

• visual_feat, image_feat
• Any custom *_feat.npy files

---

🏗️ Architecture

MRS/
├── src/
│   ├── main.py              # Entry point
│   ├── models/              # Model implementations
│   │   ├── hpmrec.py
│   │   ├── cohesion.py
│   │   └── ...
│   ├── utils/
│   │   ├── graph_cache.py   # Graph caching
│   │   ├── dataset.py       # Data processing
│   │   ├── dataloader.py    # Data loading
│   │   ├── visualization.py # Training visualization
│   │   └── quick_start.py   # Quick start utility
│   ├── configs/
│   │   └── model/           # Model configurations
│   └── log/                 # Training logs & visualizations
├── data/                    # Datasets
│   └── cache/               # Graph caches

---

📊 Performance Benchmarks (Log)

Baby dataset

Model	Recall@10	Recall@20	NDCG@10	NDCG@20	Training Time	Inference Time
VBPR	0.0372	0.0592	0.0199	0.0256	0.36s/epoch	0.63s/epoch
MMGCN	0.0414	0.0682	0.0215	0.0284	2.18s/epoch	0.62s/epoch
GRCN	0.0493	0.0790	0.0258	0.0334	1.47s/epoch	0.58s/epoch
LATTICE	0.0584	0.0893	0.0314	0.0393	0.72s/epoch	0.59s/epoch
DualGNN	0.0367	0.0592	0.0192	0.0250	4.43s/epoch	0.59s/epoch
SLMRec	0.0518	0.0774	0.0287	0.0353	2.33s/epoch	0.60s/epoch
BM3	0.0536	0.0856	0.0289	0.0371	0.61s/epoch	0.58s/epoch
MMSSL	0.0559	0.0889	0.0306	0.0391	4.64s/epoch	0.61s/epoch
FREEDOM	0.0622	0.0977	0.0337	0.0427	0.77s/epoch	0.59s/epoch
MGCN	0.0629	0.0964	0.0346	0.0433	1.15s/epoch	0.60s/epoch
DRAGON	0.0637	0.1004	0.0351	0.0445	4.48s/epoch	0.67s/epoch
LGMRec	0.0652	0.1031	0.0353	0.0450	1.47s/epoch	0.60s/epoch
DiffMM	0.0578	0.0900	0.0314	0.0397	0.91s/epoch	0.60s/epoch
DAMRS	0.0578	0.0924	0.0316	0.0406	2.56s/epoch	0.58s/epoch
MENTOR	🥈0.0670	🥉0.1048	🥈0.0362	🥈0.0459	5.70s/epoch	0.59s/epoch
PGL	0.0610	0.0960	0.0325	0.0415	0.88s/epoch	0.59s/epoch
SMORE	🥇0.0678	0.1039	🥇0.0368	🥇0.0460	1.39s/epoch	0.67s/epoch
COHESION	🥈0.0670	🥈0.1050	0.0350	0.0447	4.53s/epoch	0.59s/epoch
SSR	🥉0.0665	🥇0.1065	0.0357	🥇0.0460	9.80s/epoch	0.81s/epoch
HPMRec	0.0660	0.1024	🥉0.0360	🥉0.0453	6.15s/epoch	2.07s/epoch
LOBSTER	0.0551	0.0861	0.0296	0.0376	1.39s/epoch	0.61s/epoch

🔜 The results for the other datasets are comming soon

---

🔧 Advanced Usage

Evaluation Configuration

Default evaluation metrics include: Recall@N and NDCG@N with N = 10 or 20. Whole settings include:

• Recall, NDCG, Precision, MAP
• @5, @10, @20, @50

Custom Model Integration

1. Create model file in src/models/
2. Inherit from GeneralRecommender
3. Implement required methods:
   • __init__(self, config, dataloader)
   • forward(self, interaction)
   • calculate_loss(self, interaction)
   • full_sort_predict(self, interaction)
4. Add configuration YAML

Custom Model Integration

1. Create model file in src/models/
2. Inherit from GeneralRecommender
3. Implement required methods:
   • __init__(self, config, dataloader)
   • forward(self, interaction)
   • calculate_loss(self, interaction)
   • full_sort_predict(self, interaction)
4. Add configuration YAML

Graph Cache Management

from utils.graph_cache import GraphCacheManager

# Initialize
cache_manager = GraphCacheManager(data_path, dataset_name)

# Save graph
cache_manager.save_graph(
    model_name='MyModel',
    graph_name='item_graph',
    graph_data=graph_tensor,
    metadata={'knn_k': 10}
)

# Load graph
graph_data, metadata = cache_manager.load_graph(
    model_name='MyModel',
    graph_name='item_graph'
)

Custom Visualization

from utils.visualization import TrainingVisualizer

visualizer = TrainingVisualizer(
    model_name='HPMRec',
    dataset='baby',
    enable=True
)

# Log metrics
visualizer.log_epoch(epoch, loss, recall, ndcg)

# Save final plots
visualizer.save_plots()

---

🤝 Contributing

We welcome contributions!

Fix Bug

You can directly propose a pull request and add detailed descriptions to the comment

Add New Model

If you want to add your model to MRLib, please

• Follow existing code style
• Update documentation

---

📝 Citation

If you use MRS in your research, please cite our survey [TMM2026] MRS Survey

@article{xu2026survey,
  title={A survey on multimodal recommender systems: Recent advances and future directions},
  author={Xu, Jinfeng and Chen, Zheyu and Yang, Shuo and Li, Jinze and Wang, Wei and Hu, Xiping and Hoi, Steven and Ngai, Edith},
  journal={IEEE Transactions on Multimedia},
  year={2026},
  publisher={IEEE}
}

---

📄 License

MIT License - see LICENSE file for details.

---

🙏 Acknowledgments

• Paper list reference: Awesome-Multimodal-Recommender-Systems
• The structure of this code is based on MMRec

---

📬 Contact

• Issues: Open an Issue on GitHub
• Discussion: GitHub Discussions

---

🇨🇳 中文版