add dependency-update workflow template (fixes #683)#756
add dependency-update workflow template (fixes #683)#756Rimsha2535 wants to merge 3 commits intomainfrom
Conversation
Rimsha2535
requested a deployment
to
manual-approval
GitHub Actions
Waiting
kratz00
changed the title
|
|
||
| - name: Audit Dependencies | ||
| id: audit-dependencies | ||
| run: poetry run -- nox -s dependency:audit |
There was a problem hiding this comment.
We can ask the users of the python-toolbox what they'd prefer.
When I'd written that we perform a check by running poetry run -- nox -s dependency:audit, I had thought we could check to see if there are vulnerabilities detected or not. If there were vulnerabilities, then we'd proceed with updating the dependencies. Otherwise, we would skip the update.
One way to do this would be to check the length of the produced JSON;
# this will both print the results & output them to a json file
poetry run -- nox -s dependency:audit | tee vulnerabilities.json
LENGTH=$(jq 'length' vulnerabilities.json)
echo "count=$LENGTH" >> $GITHUB_OUTPUTIn the next step, where we run update-dependencies, we can add an if-statement
if: steps.audit-dependencies.outputs.count > 0| - name: Create Pull Request | ||
| id: create-pull-request | ||
| if: steps.check-for-poetry-lock-changes.outputs.changed == 'true' | ||
| uses: peter-evans/create-pull-request@v7 |
There was a problem hiding this comment.
In general, we try not to use third-party GitHub actions. This is because they can pose a security risk. Currently, we still use the a few third-party actions, like ravsamhq/notify-slack-action, as there isn't a GitHub equivalent.
So for the "Create Pull Request", we'd prefer to use the GitHub provided commands, so this should look mostly the same as:
https://github.com/exasol/project-keeper/blob/main/.github/workflows/dependencies_update.yml#L120
Though, we do not need to have as many initial checks as is provided in:
https://github.com/exasol/project-keeper/blob/main/.github/workflows/dependencies_update.yml#L132
Rimsha2535
requested a deployment
to
manual-approval
GitHub Actions
Waiting
Rimsha2535
requested a deployment
to
manual-approval
GitHub Actions
Waiting
|
| @@ -0,0 +1,60 @@ | |||
| name: Dependency Update | |||
There was a problem hiding this comment.
(note to future self)
When we've aligned on exasol/toolbox/templates/github/workflows/dependency-update.yml, we should execute poetry run -- nox -s workflow:generate -- all so this it up to date.
We might also want to merge this to a development branch which needs to be created.
| dependency-update | ||
| ================= | ||
|
|
||
| This workflow updates the project dependencies using Poetry. |
There was a problem hiding this comment.
This description is rather nice and kudos for updating the documentation in a correct way!
That's seriously not an easy feat the first time around (and many of still make mistakes 😅 ).
And you got all the double `s around the variables correctly done! 🥳 👏 🪅
(^ But would just emphasize this lots, as it's great, even though there's most text later. 😅)
We should move it though, as we're adding a new GitHub workflow and not a GitHub action.
Please do the following:
- Remove this file (
doc/github_actions/dependency_update.rst) - Undo the addition in
doc/github_actions/github_actions.rstreferencing this file - Add a brief entry for the
dependency-update.ymlworkflow in this table (alphabetically sorted):- https://exasol.github.io/python-toolbox/main/user_guide/features/github_workflows/index.html#workflows
- which locally is located in
doc/user_guide/features/github_workflows/index.rst
- Then add a new subsection (like
Pull Request) in the sectionCI Actions- https://exasol.github.io/python-toolbox/main/user_guide/features/github_workflows/index.html#ci-actions
- which locally is also located in
doc/user_guide/features/github_workflows/index.rst
For step 3., try to create a short description similar in nature to the other entries in the table. This is just a short statement of what the workflow does.
For step 4., you can look at the other sections for an idea, but try to fill in the details for these points:
- under what conditions is workflow triggered
- what do we want to achieve with this workflow?
- what do the workflow do?
For the last two points, you can reuse parts of the text you'd already written in doc/github_actions/dependency_update.rst 😊
| It first runs a dependency audit via ``nox -s dependency:audit`` and then updates the dependencies using ``poetry update``. | ||
| If the ``poetry.lock`` file changes, a pull request is created automatically. | ||
|
|
||
| Example Usage |
There was a problem hiding this comment.
I'd omit the Example Usage in this case, as we already have instruction on this page: https://exasol.github.io/python-toolbox/main/user_guide/features/github_workflows/index.html. While you correctly wrote out the instructions, we're moving away from using the tbx commands to instead use:
poetry run -- nox -s workflow:generate -- all
# or for an individual workflow
poetry run -- nox -s workflow:generate -- dependency-update| jobs: | ||
| dependency-update: | ||
| name: Dependency Update | ||
| runs-on: "ubuntu-24.04" |
There was a problem hiding this comment.
Ah, sorry, this due to a miscommunication and lack of verbosity on my end 😅.
- When files are in
exasol/toolbox/templates/github/workflows/, we want to use the Jinja variables still. - We use the command
poetry run -- nox -s workflow:generate -- dependency-updateto fill in these Jinja variables and automatically write/update the file to/in.github/workflows/dependency-update.yml - Then, the tricky part for your PR is that we want to test this as well, so the we have to make one manual change to
.github/workflows/dependency-update.yml.
So in this template file (step 1.) :
- line 8:
workflow_call:would be removed - line 13:
"ubuntu-24.04"should be"(( os_version ))" - line 29:
"3.10"should be"(( minimum_python_version ))" - line 30:
"2.3.0"should be"(( dependency_manager_version ))"
After doing step 2, then in .github/workflows/dependency-update.yml, we'd manually add (step 3):
- line 8:
workflow_call:
|
|
||
| file_list = run_command(["ls", ".github/workflows"]).stdout.splitlines() | ||
| assert len(file_list) == 13 | ||
| assert len(file_list) == 14 |
There was a problem hiding this comment.
Also great that you got all the tests to work! 🫶
| - name: Check for poetry.lock Changes | ||
| id: check-for-poetry-lock-changes | ||
| if: steps.audit-dependencies.outputs.count > 0 | ||
| shell: bash |
There was a problem hiding this comment.
It doesn't hurt, but we could remove the shell: bash in this and later steps.
In our case, the OS version should always be Linux-based runners, so it's not needed.
| if: steps.check-for-poetry-lock-changes.outputs.changed == 'true' && github.ref == 'refs/heads/main' | ||
| shell: bash | ||
| run: | | ||
| branch_name="dependency-update/poetry-lock" |
There was a problem hiding this comment.
This should probably be something like:
branch_name="dependency-update/$(date "+%Y%m%d%H%M%S")"
While ideally, a PR is created, and we react within a week. Sometimes that doesn't happen. By adding the date, we can ensure that we have unique branch names should more than 1 PR be open for the dependency updater over a time period.
| GH_TOKEN: ${{ github.token }} | ||
| shell: bash | ||
| run: | | ||
| gh pr create \ |
There was a problem hiding this comment.
While we definitely should only have main as the default branch, some of our Python repositories are using the antiquated master.
| gh pr create \ | |
| BASE_BRANCH=$(gh repo view --json defaultBranchRef -q .defaultBranchRef.name) | |
| gh pr create \ | |
| --base "$BASE_BRANCH" \ |
2 participants
Fixes #683
Checklist
Note: If any of the items in the checklist are not relevant to your PR, just check the box.
For any Pull Request
Is the following correct:
When Changes Were Made
Did you:
When Preparing a Release
Have you:
Notes