FactGuard

A fake news classifier that reads the style, not the story.

Paste a headline and body text, and a Random Forest trained on the TF-IDF profile of function words labels the article truthful or fake, with a confidence score. Built with scikit-learn, NLTK, pandas, and Streamlit.

Note: FactGuard is an undergraduate machine learning project. It judges how an article is written, not whether its claims are true, and it was trained on American political news from 2015 to 2018. Read its verdicts accordingly.

Live app: factguard.streamlit.app

Contents

• What it does
• Where it started
• The route to the final model
• How a story is classified
• Results
• The datasets
• The notebooks
• The app
• Getting started
• Deploying
• Project structure
• Limitations
• Future work
• License

What it does

You give FactGuard the title and body of a news article. It merges and tokenizes the text, keeps only the function words (about, however, those, and roughly 1,150 others), and feeds their TF-IDF profile to a Random Forest. Back comes a verdict, truthful or fake, with the model's confidence and an expandable list of the exact stopwords it based the decision on.

That last part matters. Because the model never sees names, places, or topic words, it cannot cheat by recognizing subjects it saw during training. It is classifying prose style, the way sentences are glued together.

Where it started

FactGuard was a group project from my undergraduate years, built as a first serious pass at applied NLP. The brief we set ourselves was modest: given an article's title and text, estimate the chance it is fake. The whole build lives in ten numbered notebooks (originally run on Google Colab), and the finished model is served by a small Streamlit app. A slide deck from the original presentation is included at docs/factguard-technical-presentation.pdf if you want the long version with figures.

What kept the project interesting was that the obvious approaches worked suspiciously well, and chasing down why turned out to be the real lesson.

The route to the final model

Exploratory analysis turned up something odd right away: in this dataset, fake and real articles barely overlap in the percentage of capital letters in their titles. Fake headlines shout; wire copy does not. A one-line heuristic thresholding on title capitalization scores 0.98 accuracy (notebook 04). It is also worthless as a gatekeeper, for the same reason naive spam filters were: the moment publishers stop typing in caps, it is beaten.

So we moved to actual models. Naive Bayes over a bag of words reached 0.91 on the test split, a Random Forest over the same features 0.94, and switching to TF-IDF pushed the Random Forest to 0.95 (notebooks 05 to 07). Those numbers look great until you notice a problem with the data: every single truthful article came from one outlet, Reuters. We had stripped the literal (Reuters) dateline during preprocessing, but the worry remained that the model had learned to recognize Reuters house style rather than truthfulness.

The worry was justified. Tested on stories from outside the dataset (notebook 08), the model did barely better than a coin flip. That result was honestly disheartening at the time, and it is also the most instructive thing in the repository.

Two changes fixed it. First, the vocabulary was cut down to stopwords only, so the classifier works from each article's function-word profile instead of its subject matter. Persons, organizations, dates, and jargon all disappear, and with them most of the source-specific giveaways. Second, politics articles from The Guardian were mixed into the truthful class, half Reuters and half Guardian, with the two classes balanced exactly (13,957 articles each in the training split) (notebook 09). The retrained model scores 0.97 on its held-out test split and, more importantly, behaves sensibly on stories it has never seen (notebook 10).

How a story is classified

Normalization does a fair amount of work before the model sees anything:

• URLs collapse to a {link} placeholder and Twitter handles to @twitter-handle, so links count as a stylistic event rather than an identity.
• Text is lowercased, except words written entirely in capitals.
• Digits, the (Reuters) dateline, possessive 's tokens, and single characters (apart from i and !) are stripped.
• Weekday and month names are dropped, since they date the article rather than describe its style.

The pipeline itself is a scikit-learn Pipeline of CountVectorizer, TfidfTransformer, and RandomForestClassifier. The vectorizer's vocabulary was learned from stopword-filtered training text, so at prediction time every token outside that vocabulary is simply ignored.

flowchart LR
    A["Title + body text"] --> B["Tokenize and normalize"]
    B --> C["Vocabulary filter<br/>(stopwords only)"]
    C --> D["TF-IDF weighting"]
    D --> E["Random Forest"]
    E --> F["Verdict + confidence"]

Loading

The stopword list is a 1,160-word expanded set (in data/gist_stopwords.txt), several times the size of NLTK's default English list, with via, eu, and uk removed because they carry topical meaning in political news.

Results

Test-split accuracy for the final iteration in each modeling notebook:

Notebook	Model	Test accuracy
04	Title-capitalization heuristic	0.98
05	Naive Bayes, bag of words	0.91
06	Random Forest, bag of words	0.94
07	Random Forest, TF-IDF	0.95
09	Random Forest, TF-IDF, stopword vocabulary + Guardian data	0.97

Read the table with care. The heuristic at the top is the least trustworthy entry in it, and every number comes from a held-out split of the same dataset, which flatters any model. The Kaggle-only TF-IDF model (07) posts 0.95 here yet fell apart on outside articles. The final model (09) posts 0.97 with per-class precision and recall between 0.97 and 0.98, and it is the only one that held up when tested beyond the training distribution, which was the point.

The datasets

Primary: ISOT fake and real news. Published on Kaggle by Clément Bisaillon (v1, March 2020) and collected by the ISOT Research Lab at the University of Victoria. Truthful articles were crawled from Reuters; fake articles come from outlets flagged by PolitiFact and Wikipedia. Around 45,000 articles as published, mostly US political and world news from 2015 to 2018, of which roughly 38,600 survive cleaning and deduplication. Details are in the lab's dataset documentation, and the dataset is on Kaggle.

The ISOT authors ask that the following papers be cited when their data is used:

1. Ahmed H, Traore I, Saad S. "Detecting opinion spams and fake news using text classification", Journal of Security and Privacy, Volume 1, Issue 1, Wiley, January/February 2018.
2. Ahmed H, Traore I, Saad S. (2017) "Detection of Online Fake News Using N-Gram Analysis and Machine Learning Techniques". In: Traore I., Woungang I., Awad A. (eds) Intelligent, Secure, and Dependable Systems in Distributed and Cloud Environments. ISDDC 2017. Lecture Notes in Computer Science, vol 10618. Springer, Cham (pp. 127-138).

Supplemental: Guardian news. Published on Kaggle by Sameed Hayat (v1, June 2019), scraped from The Guardian. It holds over 52,000 articles across sections, of which only the politics section (about 12,650 articles) is used here, to match the political skew of the primary set. Available on Kaggle.

The notebooks

The build is documented start to finish in notebooks/, numbered in the order they were written:

Notebook	What it covers
01	Download, clean, and explore the ISOT dataset
02	Tokenization and normalization of the primary set
03	Download and clean the Guardian dataset
04	The title-capitalization heuristic
05	Naive Bayes over bag-of-words features
06	Random Forest over bag-of-words features
07	Random Forest over TF-IDF features
08	Testing the Kaggle-only model on outside stories (where it fails)
09	The final model: stopword vocabulary, Guardian-balanced data
10	Testing the final model on outside stories

The notebooks were written for Google Colab against a mounted Drive folder, so their file paths reflect that environment. Rerunning them needs a Kaggle API token; data/kaggle.json is a placeholder showing the expected format.

The app

The Streamlit app is a single page. A form takes the article title and body, and submitting it returns a verdict card: truthful in green or fake in red, a confidence bar, and a short reminder that this is a style judgement. An expander below lists every stopword token the model actually received, which makes the "style, not substance" idea concrete: you can see that the classifier decided from words like would, those, and however. The sidebar summarizes the method and links back here.

Both trained pipelines ship in models/: rf_tfidf_guardian_model.pkl is the deployed final model, and rf_tfidf_model.pkl is the Kaggle-only predecessor, kept for comparison.

Getting started

The hosted copy at factguard.streamlit.app needs no setup. To run it locally you need Python 3.11 (older 3.x versions likely work, but 3.11 is what the dev container uses):

git clone https://github.com/rsvptr/factguard.git
cd factguard
pip install -r requirements.txt
streamlit run streamlit_app.py

Then open http://localhost:8501. The NLTK tokenizer data (punkt and punkt_tab) downloads automatically on first launch.

Opening the repository in GitHub Codespaces also works with zero setup: the dev container installs the dependencies and starts the app on port 8501 by itself.

Deploying

The app runs on Streamlit Community Cloud, which is how factguard.streamlit.app is hosted. Point a new Streamlit Cloud app at this repository with streamlit_app.py as the entrypoint and it deploys as is: requirements.txt supplies the Python packages and nltk.txt tells the platform which NLTK corpora to fetch. No secrets, database, or environment variables are involved.

Project structure

factguard/
├── .devcontainer/
│   └── devcontainer.json         Codespaces setup: installs deps, launches the app
├── .streamlit/
│   └── config.toml               Dark theme for the app
├── data/
│   ├── gist_stopwords.txt        The 1,160-word expanded stopword list
│   └── kaggle.json               Placeholder Kaggle API token for the notebooks
├── docs/
│   └── factguard-technical-presentation.pdf
├── models/
│   ├── rf_tfidf_guardian_model.pkl   Deployed final model (ISOT + Guardian)
│   └── rf_tfidf_model.pkl            Kaggle-only predecessor, kept for comparison
├── notebooks/                    The full build, notebooks 01 through 10
├── model_helperfunctions.py      Tokenization, normalization, and evaluation helpers
├── streamlit_app.py              The web app
├── nltk.txt                      NLTK corpora for Streamlit Cloud
└── requirements.txt

One structural constraint worth knowing: model_helperfunctions.py must keep its name and stay at the repository root. The pickled pipelines store references to functions in that module (passthrough and identity_function) and resolve them by import when the model is loaded.

Limitations

• The model judges style, not truth. A fabricated story written in calm wire-service prose will pass, and a real story written breathlessly can fail.
• The training data is US political news from 2015 to 2018, with just two outlets (Reuters and The Guardian) supplying every truthful article. Anything from another beat or era is off-distribution.
• English only.
• The stopword profile survives topic changes, but not deliberate style mimicry. Text generated by a modern LLM in a newsroom's house style would likely sail through.
• The models are pickled scikit-learn pipelines, so loading them under a very different scikit-learn version may produce warnings or errors. The pinned minimums in requirements.txt are known to work.

Future work

Three things would help, in rough order of effort:

1. More labeled sources. All truthful articles trace back to two outlets, and the provenance of the fake ones is unknown. Widening both classes is tedious labeling work, which is exactly why it was skipped, but it is the single biggest robustness win available.
2. Source as a feature. Collecting the URL of each story and folding source reputation into the classification, instead of relying on prose style alone.
3. A deep learning successor. The Random Forest reads surface style, which was defensible in a world where fake news was written by people in a hurry. Current LLMs can imitate any outlet's style on demand, and catching that likely takes a model of meaning rather than of function words.

License

MIT. See LICENSE.