feat: Enhance NER pipeline with robust imports and interpretability

[michaWorku]

This commit addresses the 'No module named utils' import error and significantly improves the pipeline's stability, error handling, and introduces model interpretability features.

Key changes include:

• NER Models (src/models/ner_trainer.py, src/models/model_evaluator.py):

  • Corrected module import paths (e.g., from conll_parser to src.utils.conll_parser) to resolve ImportError: No module named 'utils'.
  • Improved sys.path handling within modules for consistent discovery.
  • Integrated SHAP and LIME interpretability analysis into model_evaluator.py to provide insights into model predictions and behavior.
  • Enhanced data handling in ner_trainer.py for robustness with empty datasets and edge cases in metric computation.
  • Added dummy data generation in __main__ blocks for easier local testing of individual model components.

• Pipeline Orchestration (scripts/run_pipeline.py):

  • Streamlined sys.path setup to ensure the src directory is consistently recognized as a top-level package.
  • Integrated asyncio.run for proper execution of the asynchronous Telegram scraper.
  • Improved error handling with detailed tracebacks for better debugging.
  • Added logic to automatically create a dummy config/channels_to_crawl.txt if it's missing, simplifying initial setup.
  • Ensured DataFrame column types and NaN values are handled robustly across stages (e.g., message_text, tokens, message_id, views).

• Data Ingestion (src/data_ingestion/telegram_scraper.py):

  • Updated the scraper to accept a message_limit parameter, replacing a hardcoded value.
  • Modified to read channel configurations from config/channels_to_crawl.txt.

• Configuration & Data Management:

  • Replaced the .csv extension for channels_to_crawl with a .txt file, reflecting a simpler, line-delimited list of channels.
  • Updated .gitignore to reflect new directory structures and ignored files, including data/predicted and data/analytics.

These updates make the pipeline more reliable, easier to use and debug, and provide valuable insights into the NER model's decision-making process for better micro-lending assessments.