This project is a Hinglish (Hindi-English mixed) language modeling pipeline built from scratch. It processes Hinglish text, analyzes its structure, and builds an N-gram based language model for tasks like text prediction or completion.
- Hinglish dataset loading from
.txtfiles - Text cleaning and normalization
- Tokenization that handles Devanagari and English scripts
- Language identification (Hindi/English/Mixed)
- Data visualization (word count, character count, language mix)
- Word cloud generation
- Train-test splitting and vocabulary analysis
- Rare word handling via
<UNK>token - Custom N-gram language model with Kneser-Ney and Laplace smoothing
- Perplexity scoring to evaluate predictions
#🚀 Running the Code
- Load and clean the Hinglish data
2.Analyze the text and visualize
3.Tokenize, split and prepare training data
4.Build and train the N-gram model
5.Save the processed data for further use
6.Character Count Distribution
7.Language Composition (Pie Chart)
8.WordCloud for Hinglish Words