Skip to content

fix : apricot test_generated comment bash error - #22

Merged
haddybhaiya merged 4 commits into
mainfrom
phs4-fixs
May 5, 2026
Merged

fix : apricot test_generated comment bash error#22
haddybhaiya merged 4 commits into
mainfrom
phs4-fixs

Conversation

@haddybhaiya

Copy link
Copy Markdown
Member

This pull request improves the formatting and reliability of generated test code by introducing a normalization function to clean up LLM outputs, updating prompt instructions, and adding dedicated tests for the new normalization logic.

Test Output Normalization:

  • Added a private function _normalize_generated_tests in test_generator.py to strip markdown code fences, language specifiers, and shell command lines (like pytest ...) from LLM-generated test code. This ensures only valid Python test code is returned.
  • Updated the generate_tests function to use _normalize_generated_tests instead of manual string manipulation.

Prompt and Output Formatting:

  • Enhanced the prompt in generate_tests to explicitly instruct the LLM to avoid markdown code fences and shell commands in its output.
  • In main.py, changed the final output format to wrap suggested tests in a bash code block, improving readability.

Testing:

  • Added new tests in test_test_generator_formatting.py to verify that _normalize_generated_tests correctly removes code fences and bash headers, and preserves valid Python code.

@github-actions

github-actions Bot commented May 4, 2026

Copy link
Copy Markdown

Automated Code Review Comment:

Bugs

  • None

Improvements

  • The code quality has improved with the addition of the _normalize_generated_tests function, which removes markdown code fences and shell commands from the generated test content.
  • The generate_tests function now uses the _normalize_generated_tests function to normalize the generated test content.

Suggestions

  • Consider adding more test cases to cover different scenarios for the _normalize_generated_tests function.
  • The generate_tests function could be improved by adding more error handling and logging to handle exceptions.
  • It's a good practice to follow PEP 8 for Python code formatting, but the code looks well-formatted.
  • The code looks good and follows best practices, but it would be beneficial to add more documentation to explain the purpose of each function and how they are used.

Code looks good ✅


Suggested Tests

def test_empty_diff():
    assert True

@haddybhaiya

Copy link
Copy Markdown
Member Author

added bash for tests to be written in generated_test.py , working IMO

@haddybhaiya haddybhaiya left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

waiting for approvals , please check carefully!

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR aims to make LLM-generated tests cleaner before they are written/commented by the agent, mainly by normalizing fenced output and adding focused tests for that normalization path.

Changes:

  • Added _normalize_generated_tests and routed generate_tests() through it to strip code fences and leading pytest command lines.
  • Added targeted tests for normalization behavior in a new test module.
  • Changed the PR comment formatting for generated tests and replaced the checked-in tests/test_generated.py contents.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
tests/test_test_generator_formatting.py Adds unit tests for normalization of generated test output.
tests/test_generated.py Replaces prior contents with a trivial placeholder test.
agent/main.py Changes how suggested tests are rendered in the final PR comment.
agent/llm/test_generator.py Introduces output normalization and updates the LLM prompt rules.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread agent/main.py

# 9. Combine output
final_output = f"{review}\n\n---\n\n### Suggested Tests\n{tests}"
final_output = f"{review}\n\n---\n\n### Suggested Tests\n```bash\n{tests}\n```"
Comment thread tests/test_generated.py
Comment on lines +1 to +2
def test_empty_diff():
assert True No newline at end of file

@pleasingsunlight pleasingsunlight left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR aims at improving formatting for tests via introducing new functions to normalize tests and confirm the same via test_test_generated_formatting.py .

Upon proper review of the changes made, the intention is fulfilled as seen from the comment by 'github-actions-bot' ;

Tho there is still room for improvement here, especially when it comes to test coverage, preventing over-stripping of valid lines.

It is safe to merge the fixes to main.

  • A few more tries would serve to better assess the improved formatting of the agent.

Hence for now :
We ball ! 🏐 😄 ~~~

@deoxyforge

Copy link
Copy Markdown
Collaborator

Everthing seems fine to me go ahead and merge this pr

@deoxyforge deoxyforge left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Go ahead everything looks good

@haddybhaiya
haddybhaiya merged commit 77b00cc into main May 5, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants