A text classification system that categorizes news articles into 17 different categories using a fine-tuned large language model.
This project takes news articles and figures out what category they belong to - things like sports, politics, weather, crime, and so on. It uses Mistral-7B, which is a pretty capable open-source language model, and fine-tunes it specifically for this classification task.
The interesting part is that I didn't just throw the model at the data and hope for the best. The original dataset was quite messy, so a good chunk of the work went into cleaning it up and making sure the labels actually made sense.
I started with a dataset of about 5,100 news articles, each labeled with one of 17 categories. Sounds straightforward, right? Well, not quite.
When I actually looked at the data, I found:
- Duplicate articles with different labels
- Non-English text mixed in
- Articles where the label just didn't match the content at all
- Categories that were quite vague and overlapped with each other
So before I could even think about training a model, I had to deal with all of this.
I built a pipeline that:
- Removes boilerplate text (like "click here to subscribe" type stuff)
- Filters out non-English content
- Finds and removes near-duplicate articles
- Uses a simple TF-IDF model to flag potentially mislabeled samples
After all this, I was left with about 2,300 clean samples - less than half of what I started with. That tells you something about the quality of the original data.
For the actual classification, I used:
- Mistral-7B as the base model
- QLoRA for efficient fine-tuning (4-bit quantization + low-rank adapters)
- A chunking strategy to handle long articles that don't fit in the model's context window
I also experimented with combining TF-IDF predictions with the LLM outputs, which gave a small but consistent improvement.
The final model achieves:
- 85% accuracy across 17 categories
- 0.82 macro F1 score
Some categories work really well:
- Environmental news: 100% F1 (the model gets these perfectly)
- Sports: 97% F1
- Labour news: 97% F1
Others are more challenging:
- Social: 44% F1 (only 73 training samples after cleaning)
- Arts: 62% F1 (overlaps with lifestyle, religion, etc.)
The errors the model makes are usually reasonable - like confusing a tornado story between "disaster" and "weather", which honestly could go either way.
A few things became clear during this project:
-
Data quality matters more than model size. I could have used a bigger model, but fixing the labels had a much bigger impact.
-
Some categories are just ambiguous. When even humans would disagree on whether something is "social" or "human interest", you can't expect a model to do better.
-
Statistical testing is important. The difference between 84% and 85% accuracy looks meaningful, but with only 234 validation samples, it's not statistically significant. Being honest about this is better than overstating results.
-
Cleaning data is unglamorous but necessary. I spent more time on data cleaning than on model training, and that's probably how it should be.
├── data_cleaning.ipynb # Data preprocessing pipeline
├── mistral_classifier_06012026_fixed.ipynb # Main training & evaluation
├── requirements.txt # Python dependencies
├── tfidf_assets/
│ └── meta.json # TF-IDF configuration
└── images/ # Visualizations
├── confusion_matrices.png
├── class_imbalance_analysis.png
└── ...
- Python 3.10+
- CUDA-capable GPU (I used an RTX 5070, but anything with 8GB+ VRAM should work)
- About 20GB of disk space for the model weights
# Clone the repo
git clone https://github.com/yourusername/llmTextClassifier.git
cd llmTextClassifier
# Create virtual environment
python -m venv .venv
source .venv/bin/activate # or .venv\Scripts\activate on Windows
# Install dependencies
pip install -r requirements.txt- Start with
data_cleaning.ipynbif you have raw data to process - Then run
mistral_classifier_06012026_fixed.ipynbfor training and evaluation
The trained LoRA adapter weights are not included in the repo (they're about 100MB), but you can regenerate them by running the training cells.
The model classifies articles into these 17 categories:
| Category | Description |
|---|---|
| arts | Music, theatre, visual arts, etc. |
| crime | Criminal activities, court cases |
| disaster | Natural or man-made disasters |
| economy | Financial news, markets, trade |
| education | Schools, universities, learning |
| environmental | Climate, pollution, conservation |
| health | Medical news, public health |
| humanInterest | Feel-good stories, unusual events |
| labour | Unions, employment, workplace issues |
| lifestyle | Fashion, food, travel, hobbies |
| politics | Government, elections, policy |
| religion | Faith, religious institutions |
| science | Research, discoveries, technology |
| social | Community issues, social movements |
| sport | Athletic events, teams, players |
| unrest | Protests, conflicts, civil disorder |
| weather | Forecasts, climate events |
- The model struggles with ambiguous categories (social, arts, humanInterest)
- Validation set is small (234 samples), so metrics have wide confidence intervals
- Some category boundaries are inherently fuzzy
MIT License - do whatever you want with it.
- Mistral AI for the base model
- Hugging Face for transformers and PEFT libraries
- The scikit-learn team for making data preprocessing so much easier