SMERVisual

SMERVisual is a Python package designed for explainable machine learning using the Self Model Entities Related (SMER) method. It provides tools for explainable classification of images with LLM-generated text descriptions, which are then analyzed using the SMER explanation technique.

The package supports both OpenAI API models and local language models, offering flexibility in model selection.

---

Installation

Install SMERVisual using pip:

pip install smer-visual

---

Features

Image Description

The image_description function generates text descriptions for images using either OpenAI models or local language models. Key features include:

• Support for OpenAI models like gpt-4o-mini and local models.
• Customizable prompts for generating concise and informative descriptions.

Description Embeddings

The get_description_embeddings function computes embeddings for image descriptions using OpenAI or local models.

Explainable Classification

The classify_with_logreg function performs logistic regression-based classification on image datasets while computing SMER values for explainability. Key features include:

• Aggregation of embeddings for classification.
• Computation of feature importance for each word in the description.
• Support for AOPC (Area Over the Perturbation Curve) analysis.

Visualization

• Important Words: The plot_important_words function visualizes the most important words across images.
• AOPC Curves: The plot_aopc function compares the explainability of SMER and LIME methods by plotting AOPC curves.
• Bounding Boxes: The BoundingBoxGenerator class overlays bounding boxes on images, highlighting critical words identified in classification.

---

Usage Example

Image Description and Embeddings

from smer_visual.smer import image_description, get_description_embeddings

# Generate image descriptions
descriptions = image_description(
    model="gpt-4o-mini",
    data_folder="path/to/images",
    api_key="your_openai_api_key"
)

# Generate embeddings for descriptions
embeddings_df = get_description_embeddings(
    descriptions=descriptions,
    embedding_model="text-embedding-ada-002",
    api_key="your_openai_api_key"
)

Explainable Classification

from smer_visual.smer import classify_with_logreg, aggregate_embeddings
from sklearn.linear_model import LogisticRegression
import numpy as np

# Prepare data
embeddings_df["aggregated_embedding"] = embeddings_df["embedding"].apply(aggregate_embeddings)
X_train = np.stack(embeddings_df["aggregated_embedding"].values)
y_train = embeddings_df["label"]

# Train a logistic regression model
logreg_model = LogisticRegression()
logreg_model.fit(X_train, y_train)

# Perform classification and compute feature importance
aopc_df, updated_dataset = classify_with_logreg(embeddings_df, X_train, logreg_model)

Visualization

from smer_visual.smer import plot_important_words, plot_aopc

# Plot important words
plot_important_words(updated_dataset)

# Plot AOPC curves
plot_aopc(aopc_df, logreg_model, max_k=5)

Bounding Box Generation

from smer_visual.smer import save_bounding_box_images
results = save_bounding_box_images(
    input_path="data/",
    output_folder="output",
    df_top_words=top_words_df,
    model_id="IDEA-Research/grounding-dino-base",
    box_threshold = 0.5,
    text_threshold = 0.4
)

---

Why Use SMERVisual?

• Explainable AI – Provides insight into model decision-making.
• Model-Agnostic – Compatible with OpenAI APIs and open-source models.
• Zero-Shot Detection – No additional training data required.
• Easy Integration – Simple API for seamless use with existing machine learning workflows.

---

Contributing

Contributions are welcome. If you’d like to contribute, follow these steps:

1. Fork the repository.
2. Create a new branch (git checkout -b feature-branch).
3. Commit changes (git commit -m "Add new feature").
4. Push to the branch (git push origin feature-branch).
5. Open a pull request.

---

License

SMERVisual is released under the MIT License.

---