This repository contains a sanitized training-code snapshot used for the
Hiro-Pharma biomedical dialogue model project. It is derived from the
verl reinforcement learning framework and
keeps the project-specific data preprocessing, SFT, GRPO/RL, and reward
integration code needed to describe the training workflow.
The released model weights are hosted on Hugging Face:
- Model:
Hiro-Pharma - Hugging Face repository: PatSnap/Hiro-Pharma
- Base model:
Qwen/Qwen2.5-7B - License: Apache-2.0
Before using this repository or any systems trained, adapted, or served with this code, please review the project disclaimer and copyright notice.
This repository is intended to document and reproduce the training workflow at a code level. It does not include Hiro-Pharma model checkpoints, trained model weights, training logs, W&B runs, proprietary datasets, or evaluation artifacts.
The local package version in this code snapshot is 0.3.1.dev. If a released
model card records a different production training stack version, keep the model
card and GitHub release notes synchronized before public release.
Hiro-Pharma was trained for biomedical-domain dialogue use cases.
Training stages:
- Supervised fine-tuning (SFT)
- Reinforcement learning (RL), mainly using GRPO
Reward design:
- Rule-based rewards for objective or verifiable tasks
- Model-based rewards for response quality and preference-style scoring
Training framework:
verl- Ray-based distributed training
- vLLM-compatible rollout/inference components where configured
The training data is internally generated synthetic data for biomedical-domain dialogue experiments. It includes objective-question examples and question-answer dialogue examples.
The released repository does not include the training data.
The training data used for this project:
- Does not contain customer confidential data
- Does not contain real patient private or protected health information
- Is not distributed with this repository or the model weights
Users are responsible for ensuring that they have the rights and legal basis to process any data used with this code.
The open-source snapshot is cleaned for public sharing. The following classes of artifacts are intentionally excluded:
- Training logs and runtime outputs
- W&B local run directories and API keys
- Hiro-Pharma checkpoints, merged weights, and training artifacts
- Local caches such as
__pycache__,.triton, and notebook checkpoints - Internal filesystem paths and machine-specific data locations
- Private or proprietary datasets
Create a Python environment that matches your CUDA, PyTorch, Ray, vLLM, and training cluster setup, then install this repository in editable mode.
git clone <your-github-repo-url>
cd verl-0.3.1.dev
pip install -e .
pip install -r requirements.txtSome optional training paths may require additional dependencies depending on the rollout engine or distributed backend used in your environment.
The example scripts use environment variables instead of hard-coded internal paths.
Common variables:
VERL_HOME: repository root, usually$PWDDATA_HOME: local directory containing prepared training dataMODEL_HOME: local directory containing base models and reward modelsHIRO_DATA_HOME: default root for Hiro-Pharma preprocessing outputsHIRO_SFT_DATA_PATHS: colon-separated list of raw SFT dataset pathsHIRO_RL_DATA_PATH: raw RL dataset pathHIRO_OBJECTIVE_DATA_PATH: raw objective-question dataset pathWANDB_MODE: set tooffline,disabled, oronlineWANDB_API_KEY: optional; only set it in your private environmentNLTK_DATA: optional path for local NLTK resources
Example:
export VERL_HOME=$PWD
export DATA_HOME=$PWD/data
export MODEL_HOME=/path/to/models
export WANDB_MODE=offlinePrepare datasets locally, then run the preprocessing scripts with explicit input and output paths.
SFT data:
python examples/data_preprocess/biosft.py \
--data_paths ./data/synthetic_sft_1 ./data/synthetic_sft_2 \
--local_dir ./data/sftRL dialogue data:
python examples/data_preprocess/hirols.py \
--data_path ./data/hirols_raw \
--local_dir ./data/hirols_rlObjective-question data:
python examples/data_preprocess/objectqa.py \
--data_path ./data/objective_question_raw \
--local_dir ./data/objective_questionSFT example:
bash scripts/run_qwen2.5_sft.shGRPO/RL example:
bash scripts/run_qwen2.5_grpo_rm_multinode.shReview and adapt the script-level configuration before running. In particular, check model paths, dataset paths, batch sizes, rollout settings, reward-model paths, Ray cluster configuration, GPU count, and checkpoint output directories.
This code and any associated model outputs are intended for research, development, and engineering evaluation in biomedical-domain dialogue systems. They are not medical advice, clinical decision support, diagnosis, treatment guidance, prescription guidance, or a substitute for professional review. See DISCLAIMER.md for the full disclaimer.
Outputs from models trained or served with this code may be inaccurate, incomplete, unsafe, or unsuitable for clinical, regulatory, legal, or commercial decisions without appropriate human validation. Users are solely responsible for their deployment, data governance, security controls, compliance obligations, and output review processes.
Do not use this repository to process customer confidential data, trade secrets, real patient data, protected health information, unpublished proprietary materials, or third-party datasets unless you are authorized to do so and have assessed all applicable legal, privacy, data security, export control, intellectual property, and contractual obligations.
This repository is released under the Apache License 2.0. See LICENSE.
See NOTICE.md for project copyright and trademark notices.
This project is derived from the open-source verl project. Please retain
upstream notices and comply with the licenses of all third-party dependencies,
base models, reward models, and datasets used with this code.
This work builds on the verl framework for reinforcement learning from
feedback and large language model post-training.