-
Notifications
You must be signed in to change notification settings - Fork 0
Feature/air 191 #37
base: main
Are you sure you want to change the base?
Feature/air 191 #37
Changes from all commits
84b1d6a
89ceab0
4b6a27d
82aa1f9
d569a58
f495eff
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -11,6 +11,9 @@ inputs: | |
| repository: | ||
| description: "app or service name" | ||
| required: true | ||
| aws_account_id: | ||
| description: aws account id | ||
| required: true | ||
| make-directory: | ||
| description: directory of makefile | ||
| required: false | ||
|
|
@@ -31,10 +34,11 @@ runs: | |
| run: | | ||
| echo "build zip..." | ||
| make -C ${{ inputs.make-directory }} build | ||
| # TODO: Will delete this step in next PR. | ||
| - name: Push to lambda registry s3 | ||
| id: push-zip | ||
| shell: bash | ||
| run: | | ||
| echo "push zip..." | ||
| aws s3 cp ${{ inputs.repository }}.zip ${{ inputs.registry }}/${{ inputs.env }}/${{ inputs.repository }}.zip | ||
| # aws s3 cp ${{ inputs.repository }}.zip ${{ inputs.registry }}/${{ inputs.env }}/${{ inputs.repository }}.zip | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. why remove s3 cp. this action is push Lambda zip
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We are not using this step. |
||
| echo "::set-output name=zip_uri::${{ inputs.registry }}/${{ inputs.env }}" | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,99 @@ | ||
| name: copy ecr image | ||
| description: copy a container image from source env to destination env | ||
|
|
||
| inputs: | ||
| src_env: | ||
| description: ecr source environment | ||
| required: true | ||
| dst_env: | ||
| description: ecr destination environment | ||
| required: true | ||
| profile: | ||
| description: cicd profle | ||
| required: true | ||
| package_ref: | ||
| description: container image tag to copy | ||
| required: true | ||
|
|
||
|
|
||
| outputs: | ||
| tag: | ||
| description: image tag | ||
| value: ${{ steps.erc-image.outputs.tag }} | ||
|
|
||
| # assume caller has connect to AWS account | ||
| runs: | ||
| using: "composite" | ||
| steps: | ||
| - name: Get given application settings for ${{ inputs.src_env }} env | ||
| id: app_settings_src | ||
| uses: Iterative-Scopes/cicd-framework/actions/cicdcfg@feature/air-191 | ||
| with: | ||
| env: ${{ inputs.src_env }} | ||
| role: deploy | ||
| profile: ${{ inputs.profile }} | ||
|
|
||
| - name: Configure AWS credentials | ||
| id: aws_configure_src | ||
| uses: aws-actions/configure-aws-credentials@v1 | ||
| with: | ||
| role-to-assume: arn:aws:iam::${{ steps.app_settings_src.outputs.aws_account }}:role/${{ steps.app_settings_src.outputs.aws_role }} | ||
| role-duration-seconds: ${{ steps.app_settings_src.outputs.aws_role_duration }} | ||
| aws-region: ${{ steps.app_settings_src.outputs.aws_region }} | ||
| audience: ${{ steps.app_settings_src.outputs.aws_audience }} | ||
|
|
||
| - name: Get triggering commit sha | ||
| id: get_commit_sha | ||
| uses: Iterative-Scopes/cicd-framework/actions/get-commit-sha@feature/air-191 | ||
|
|
||
| - name: Login to Amazon ECR (${{ inputs.src_env }}) | ||
| id: login_ecr_src | ||
| uses: aws-actions/amazon-ecr-login@v1 | ||
| env: | ||
| AWS_REGION: ${{ steps.app_settings_src.outputs.aws_region }} | ||
|
|
||
| - name: Pull image from ECR (${{ inputs.src_env }}) | ||
| id: pull_erc_image | ||
| env: | ||
| REMOTE_REGISTRY: ${{ steps.app_settings_src.outputs.aws_account }}.dkr.ecr.${{ steps.app_settings_src.outputs.aws_region }}.amazonaws.com/${{ steps.app_settings_src.outputs.app_repository }} | ||
| IMAGE_TAG: ${{ steps.get_commit_sha.outputs.sha }} | ||
| shell: bash | ||
| run: | | ||
| echo "Pulling image $REMOTE_REGISTRY:$IMAGE_TAG" | ||
| docker pull $REMOTE_REGISTRY:$IMAGE_TAG | ||
| echo "::set-output name=src_registry::"$REMOTE_REGISTRY"" | ||
|
|
||
| - name: Get given application settings for ${{ inputs.dst_env }} env | ||
| id: app_settings_dst | ||
| uses: Iterative-Scopes/cicd-framework/actions/cicdcfg@feature/air-191 | ||
| with: | ||
| env: ${{ inputs.dst_env }} | ||
| role: deploy | ||
| profile: ${{ inputs.profile }} | ||
|
|
||
| - name: Configure AWS credentials | ||
| id: aws_configure_dst | ||
| uses: aws-actions/configure-aws-credentials@v1 | ||
| with: | ||
| role-to-assume: arn:aws:iam::${{ steps.app_settings_dst.outputs.aws_account }}:role/${{ steps.app_settings_dst.outputs.aws_role }} | ||
| role-duration-seconds: ${{ steps.app_settings_dst.outputs.aws_role_duration }} | ||
| aws-region: ${{ steps.app_settings_dst.outputs.aws_region }} | ||
| audience: ${{ steps.app_settings_dst.outputs.aws_audience }} | ||
|
|
||
| - name: Login to Amazon ECR (${{ inputs.dst_env }}) | ||
| id: login_ecr_dst | ||
| uses: aws-actions/amazon-ecr-login@v1 | ||
| env: | ||
| AWS_REGION: ${{ steps.app_settings_dst.outputs.aws_region }} | ||
|
|
||
| - name: Push image to ECR (${{ inputs.dst_env }}) | ||
| id: push_erc_image | ||
| env: | ||
| SRC_REGISTRY: ${{ steps.pull_erc_image.outputs.src_registry }} | ||
| REMOTE_REGISTRY: ${{ steps.app_settings_dst.outputs.aws_account }}.dkr.ecr.${{ steps.app_settings_dst.outputs.aws_region }}.amazonaws.com/${{ steps.app_settings_dst.outputs.app_repository }} | ||
| IMAGE_TAG: ${{ steps.get_commit_sha.outputs.sha }} | ||
| shell: bash | ||
| run: | | ||
| echo "Pushing image to $REMOTE_REGISTRY:$IMAGE_TAG" | ||
| docker tag $SRC_REGISTRY:$IMAGE_TAG $REMOTE_REGISTRY:$IMAGE_TAG | ||
| docker push $REMOTE_REGISTRY:$IMAGE_TAG |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| name: fetch triggering commit sha | ||
| description: fetch triggering commit sha | ||
|
|
||
| outputs: | ||
| sha: | ||
| description: commit sha | ||
| value: ${{ steps.commit_sha.outputs.sha }} | ||
|
|
||
| # assume caller has connect to AWS account | ||
| runs: | ||
| using: "composite" | ||
| steps: | ||
| - name: Get commit sha | ||
| id: commit_sha | ||
| shell: bash | ||
| run: | | ||
| if [ "$GITHUB_EVENT_NAME" == "pull_request" ]; then | ||
| COMMIT_SHA=$(cat $GITHUB_EVENT_PATH | jq -r .pull_request.head.sha) | ||
| else | ||
| COMMIT_SHA=${{ github.sha }} | ||
| fi | ||
| echo "set output sha ["$COMMIT_SHA"]" | ||
| echo "::set-output name=sha::"$COMMIT_SHA"" |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| # This file is the sharable action with build,unittest, integration test, code coverage reports. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. please remove this function out. we don't want to have infra provisioning is performed from GitHub Action
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We need to use serverless setup to deploy lambda. I discussed this with @johnseq-iterative and clearly mentioned that infrastructure provisioning code should not be part of this pipeline. If you see the serverless templates here only the lambda specific roles and events are getting created from the template. We are expecting the other infra should already be in place before we run this pipeline. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @is-ttang Avik and I did discuss this. My understanding is the plan is to keep infra+code provisioning together until we are ready to separate them out. That keeps the CICD responsibility squarely on one team while we wait to e.g. move to gitlab. Does that work as an interim solution? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sorry -- reading this more closely. @is-achakraborty this makes sense. We'll split out as much as we can from our templates and create out of band. That's fine.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes. but, you can run the SLS provisioning separately. it requires have more privilege for GitHub to access AWS to provision. Second, you just need to run ONCE, NOT run lambda update every time. At moment, GitHub OpenID connect scope is locked down to ECR publish and Lambda S3 write
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| # The sharable action can be used across applications by referring this action in the github workflow. | ||
|
|
||
| name: Setup serverless environment | ||
| description: Setup serverless environment | ||
|
|
||
| runs: | ||
| using: "composite" | ||
| steps: | ||
| - name: Install serverless | ||
| shell: bash | ||
| run: | | ||
| npm install -g serverless@3.24.1 | ||
| serverless --version | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this change is to break the pipeline in training-pipeline
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you give me the repo reference where it is getting used
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could you make NOT require? so NOT require to make any change in the training-pipeline (repo)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
use the sharble action from https://github.com/Iterative-Scopes/training-pipeline/tree/main/.github/workflows and https://github.com/Iterative-Scopes/ml-compute-framework/tree/main/.github/workflows