Feature/retrieval optimization

[farshad68]

Hey, I've resolved the merge conflicts between the optimization and the retrieval_optimization branch. Could you review the changes and verify your feature functionality remains intact?

Here is a summary of how the conflicts were handled:

Pipeline Execution (Main): I kept your old_loaders.json_loader ingestion logic, but I rejected the commented-out search and generation blocks. The active asyncio search and generation logic from the baseline is retained.

Search Logging & Dependency Injection (search.py):

I kept your profilingLogger additions and the user_config/opt_config dependency injections in search_authorized (critical for the optimizer).

Ingestion ETA (ingestion.py): I completely accepted your progress bar, batch tracking, and perf_counter logic.

Dataset Loading (tools.py): I rejected the custom pandas CSVDataset wrapper because it hardcoded paths. Instead, I kept the native HuggingFace datasets.load_dataset logic, but I added your sep=';' argument to ensure your specific European CSV formats still parse correctly.

SMAC Optimizer Fix (retrieval_optimization.py): Your optimization script was crashing with a RuntimeError because the tight ForbiddenAndConjunction rules caused SMAC to discard all random initial configurations. I added an initial_design injection to force SMAC to evaluate the default config so it can always start successfully.

Please pull the latest branch, run a test optimization, and let me know if the logging or CSV ingestion behaves as you expect.

[carowa292]

Hi,

I went through your points and ran the optimization successfully.

Dataset Loading (tools.py):
Adopted your approach — we stay with datasets.load_dataset (no
pandas), but added sep=';' so our semicolon-delimited CSVs parse
correctly.

main.py → debug.py:
I renamed main.py to debug.py. We only use it as a scratch script to
manually test ingestion/search/generate.

old_loaders.py:
Deleted — no longer used anywhere.

pyproject.toml:
Added bert-score and scikit-learn (were missing after
the merge).

Logging:
Works