Student Name: Sat Chidananda
This project implements an inference-time reasoning agent designed to solve complex problems across multiple domains (Math, Coding, Common Sense, etc.). The agent uses a multi-strategy architecture that dynamically selects the best reasoning technique based on the complexity of the input question.
- Multi-Strategy Routing: Automatically switches between "Chain of Thought" reasoning and direct answering based on question complexity heuristics.
- Self-Consistency: Utilizes majority voting on multiple reasoning paths for intermediate-complexity problems to improve accuracy.
- Robust Extraction: Implements strict output formatting parsing to ensure answers are extracted cleanly (e.g.,
Final Answer: <val>) without reasoning chatter. - Parallel Processing: The generation script utilizes multi-threading to process the large test dataset efficiently (~1 hour runtime).
To verify the agent's performance on the development set, please follow these steps:
- Ensure you are connected to the ASU Network or VPN (Required for API access).
- Python 3.8+ is installed.
- Clone the repo:
git clone https://github.com/163264128/NLP-Final-Project.git - Be in root directory:
cd NLP-Final-Project
pip install -r requirements.txt
OPENAI_API_KEY=cse476
API_BASE=http://10.4.58.53:41701/v1
MODEL_NAME=bens_model
Run python src/evaluate.py
Run python src/generate_answer.py