Welcome to the project page for my research on depth-aware semantic image augmentation using generative AI. This work combines the power of 3D rendering in Blender with prompt-based synthesis in ComfyUI to generate high-quality training data under low-data conditions.
📄 Click here to view the full PDF
In computer vision, data scarcity remains a critical bottleneck—especially in domains requiring specialized datasets. This research proposed a pipeline that leverages generative tools to create semantically meaningful images for improved model training. This approach uses depth estimation, 3D modeling, and diffusion-based synthesis for image augmentation.
An original dataset is captured. Next being sent through the pipeline where depth maps are generated, 3D renders are taken, image prompts are created, and then combined together to generate new semantic images. These images are then added into the dataset and a computer vision model is trained on the "augmented" dataset.
We utilize Blender as the core 3D rendering engine for generating synthetic scenes. Key components include:
- Depth-aware mesh construction using predicted depth maps.
- Camera placement and lighting control for photorealistic rendering.
- Batch automation using Blender Python scripts for efficient rendering.
The shader node workflow is provided below:
This provides fine-grained control over perspective, lighting, and geometry—crucial for producing consistent and high-fidelity training samples.
We automate combining the depth maps and source image together to form the new 3D topology using the "render_automation.py" script.
ComfyUI is used to synthesize photorealistic images from prompts generated based on object labels and scene templates. We utilize the below workflow to generate our images:
An example model training script can be viewed in the "model_training.ipynb" file.
Below are some samples images generated of cocoa plants. We used both Local and Hybrid variety plants for this test and noticed minimal artifacting:
Additionaly, in more quantitative metrics like LPIPS and CLIP Similarity we achieve the scores below:
These scores indicate the we have generated highly semantically related yet novel pieces of training data to use in our dataset.