Skip to content
This repository was archived by the owner on Jul 8, 2024. It is now read-only.
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 14 additions & 4 deletions actions/build-ecr-image/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,20 @@ inputs:
aws_region:
description: aws region
required: true
aws_account_id:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this change is to break the pipeline in training-pipeline

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you give me the repo reference where it is getting used

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you make NOT require? so NOT require to make any change in the training-pipeline (repo)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

description: aws account id
required: true
repository:
description: aws ecr repository
required: true
make_cmd:
description: make command
default: container
required: true
make-directory:
description: directory of makefile
required: false
default: $GITHUB_WORKSPACE

outputs:
tag:
Expand All @@ -27,17 +34,20 @@ runs:
uses: aws-actions/amazon-ecr-login@v1
env:
AWS_REGION: ${{ inputs.aws_region }}

- name: Get triggering commit sha
id: get_commit_sha
uses: Iterative-Scopes/cicd-framework/actions/get-commit-sha@feature/air-191

- name: Build, tag, and push image to Amazon ECR
id: erc-image
env:
REMOTE_REGISTRY: ${{ steps.login-ecr.outputs.registry }}/${{ inputs.repository }}
IMAGE_TAG: ${{ github.ref_name }}
REMOTE_REGISTRY: ${{ inputs.aws_account_id }}.dkr.ecr.${{ inputs.aws_region }}.amazonaws.com/${{ inputs.repository }}
Comment thread
is-achakraborty marked this conversation as resolved.
IMAGE_TAG: ${{ steps.get_commit_sha.outputs.sha }}
shell: bash
run: |
echo "image tag [$IMAGE_TAG]"
make ${{ inputs.make_cmd }} remotereg="$REMOTE_REGISTRY" tag="$IMAGE_TAG"
make ${{ inputs.make_cmd }}test
make -C ${{ inputs.make-directory }} ${{ inputs.make_cmd }} remotereg="$REMOTE_REGISTRY" tag="$IMAGE_TAG"
docker push $REMOTE_REGISTRY:$IMAGE_TAG
echo "set output tag ["$IMAGE_TAG"]"
echo "::set-output name=tag::"$IMAGE_TAG""
9 changes: 7 additions & 2 deletions actions/build-lambda-package/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,9 @@ inputs:
description: directory of makefile
required: false
default: $GITHUB_WORKSPACE
aws_account_id:
description: aws account id
required: true

outputs:
package_ref:
Expand Down Expand Up @@ -41,15 +44,17 @@ runs:
- name: Build Lambda container
id: build-ecr
if: ${{ steps.app_settings.outputs.app_registry == 'ecr' }}
uses: Iterative-Scopes/cicd-framework/actions/build-ecr-image@main
uses: Iterative-Scopes/cicd-framework/actions/build-ecr-image@feature/air-191
with:
aws_region: ${{ steps.app_settings.outputs.aws_region }}
aws_account_id: ${{ steps.app_settings.outputs.aws_account }}
repository: ${{ steps.app_settings.outputs.app_repository }}
make-directory: ${{ inputs. make-directory }}

- name: Build Lambda zip
id: build-zip
if: contains(${{ steps.app_settings.outputs.app_registry}}, "s3:::")
uses: Iterative-Scopes/cicd-framework/actions/build-lambda-zip@main
uses: Iterative-Scopes/cicd-framework/actions/build-lambda-zip@feature/air-191
with:
repository: ${{ steps.app_settings.outputs.app_repository }}
registry: ${{ steps.app_settings.outputs.app_registry }}
Expand Down
6 changes: 5 additions & 1 deletion actions/build-lambda-zip/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,9 @@ inputs:
repository:
description: "app or service name"
required: true
aws_account_id:
description: aws account id
required: true
make-directory:
description: directory of makefile
required: false
Expand All @@ -31,10 +34,11 @@ runs:
run: |
echo "build zip..."
make -C ${{ inputs.make-directory }} build
# TODO: Will delete this step in next PR.
- name: Push to lambda registry s3
id: push-zip
shell: bash
run: |
echo "push zip..."
aws s3 cp ${{ inputs.repository }}.zip ${{ inputs.registry }}/${{ inputs.env }}/${{ inputs.repository }}.zip
# aws s3 cp ${{ inputs.repository }}.zip ${{ inputs.registry }}/${{ inputs.env }}/${{ inputs.repository }}.zip

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why remove s3 cp. this action is push Lambda zip

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are not using this step. sls would take care of building the zip and pushing that to s3.

echo "::set-output name=zip_uri::${{ inputs.registry }}/${{ inputs.env }}"
3 changes: 3 additions & 0 deletions actions/cicdcfg/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@ outputs:
app_name:
description: "app or service name"
value: ${{ steps.get-settings.outputs.app_name }}
app_deployer:
description: "application deployer (Default: none)"
value: ${{ steps.get-settings.outputs.app_deployer }}
app_registry:
description: "app or service name"
value: ${{ steps.get-settings.outputs.app_registry }}
Expand Down
Binary file modified actions/cicdcfg/dist/cicdcfg-0.3.0.tar.gz
Binary file not shown.
3 changes: 3 additions & 0 deletions actions/cicdcfg/src/cicdcfg/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,9 @@ def get_settings(env, role, profile, outfile):
file.write(f"aws_role={cfg['role'][role]['name']}\n")
file.write(f"aws_role_duration={cfg['role'][role]['duration']}\n")
file.write(f"app_name={cfg['application']['name']}\n")
deployer = cfg["application"]["deployer"] if "deployer" in cfg["application"] else "none"
print(f"Debug-Here->> [{deployer}]")
file.write(f"app_deployer={deployer}\n")
file.write(f"app_registry={cfg['application']['package']['registry']}\n")
file.write(f"app_repository={cfg['application']['package']['repository']}\n")
file.write(f"app_runtime={cfg['application']['runtime']['name']}\n")
Expand Down
99 changes: 99 additions & 0 deletions actions/copy-ecr-image/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
name: copy ecr image
description: copy a container image from source env to destination env

inputs:
src_env:
description: ecr source environment
required: true
dst_env:
description: ecr destination environment
required: true
profile:
description: cicd profle
required: true
package_ref:
description: container image tag to copy
required: true


outputs:
tag:
description: image tag
value: ${{ steps.erc-image.outputs.tag }}

# assume caller has connect to AWS account
runs:
using: "composite"
steps:
- name: Get given application settings for ${{ inputs.src_env }} env
id: app_settings_src
uses: Iterative-Scopes/cicd-framework/actions/cicdcfg@feature/air-191
with:
env: ${{ inputs.src_env }}
role: deploy
profile: ${{ inputs.profile }}

- name: Configure AWS credentials
id: aws_configure_src
uses: aws-actions/configure-aws-credentials@v1
with:
role-to-assume: arn:aws:iam::${{ steps.app_settings_src.outputs.aws_account }}:role/${{ steps.app_settings_src.outputs.aws_role }}
role-duration-seconds: ${{ steps.app_settings_src.outputs.aws_role_duration }}
aws-region: ${{ steps.app_settings_src.outputs.aws_region }}
audience: ${{ steps.app_settings_src.outputs.aws_audience }}

- name: Get triggering commit sha
id: get_commit_sha
uses: Iterative-Scopes/cicd-framework/actions/get-commit-sha@feature/air-191

- name: Login to Amazon ECR (${{ inputs.src_env }})
id: login_ecr_src
uses: aws-actions/amazon-ecr-login@v1
env:
AWS_REGION: ${{ steps.app_settings_src.outputs.aws_region }}

- name: Pull image from ECR (${{ inputs.src_env }})
id: pull_erc_image
env:
REMOTE_REGISTRY: ${{ steps.app_settings_src.outputs.aws_account }}.dkr.ecr.${{ steps.app_settings_src.outputs.aws_region }}.amazonaws.com/${{ steps.app_settings_src.outputs.app_repository }}
IMAGE_TAG: ${{ steps.get_commit_sha.outputs.sha }}
shell: bash
run: |
echo "Pulling image $REMOTE_REGISTRY:$IMAGE_TAG"
docker pull $REMOTE_REGISTRY:$IMAGE_TAG
echo "::set-output name=src_registry::"$REMOTE_REGISTRY""

- name: Get given application settings for ${{ inputs.dst_env }} env
id: app_settings_dst
uses: Iterative-Scopes/cicd-framework/actions/cicdcfg@feature/air-191
with:
env: ${{ inputs.dst_env }}
role: deploy
profile: ${{ inputs.profile }}

- name: Configure AWS credentials
id: aws_configure_dst
uses: aws-actions/configure-aws-credentials@v1
with:
role-to-assume: arn:aws:iam::${{ steps.app_settings_dst.outputs.aws_account }}:role/${{ steps.app_settings_dst.outputs.aws_role }}
role-duration-seconds: ${{ steps.app_settings_dst.outputs.aws_role_duration }}
aws-region: ${{ steps.app_settings_dst.outputs.aws_region }}
audience: ${{ steps.app_settings_dst.outputs.aws_audience }}

- name: Login to Amazon ECR (${{ inputs.dst_env }})
id: login_ecr_dst
uses: aws-actions/amazon-ecr-login@v1
env:
AWS_REGION: ${{ steps.app_settings_dst.outputs.aws_region }}

- name: Push image to ECR (${{ inputs.dst_env }})
id: push_erc_image
env:
SRC_REGISTRY: ${{ steps.pull_erc_image.outputs.src_registry }}
REMOTE_REGISTRY: ${{ steps.app_settings_dst.outputs.aws_account }}.dkr.ecr.${{ steps.app_settings_dst.outputs.aws_region }}.amazonaws.com/${{ steps.app_settings_dst.outputs.app_repository }}
IMAGE_TAG: ${{ steps.get_commit_sha.outputs.sha }}
shell: bash
run: |
echo "Pushing image to $REMOTE_REGISTRY:$IMAGE_TAG"
docker tag $SRC_REGISTRY:$IMAGE_TAG $REMOTE_REGISTRY:$IMAGE_TAG
docker push $REMOTE_REGISTRY:$IMAGE_TAG
35 changes: 24 additions & 11 deletions actions/deploy-lambda/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,17 @@ inputs:
description: lambda package ref from lambda build job
required: true
default: "null"
make-directory:
description: directory of makefile
required: false
default: $GITHUB_WORKSPACE

runs:
using: "composite"
steps:
- name: Get given application settings
id: app_settings
uses: Iterative-Scopes/cicd-framework/actions/cicdcfg@main
uses: Iterative-Scopes/cicd-framework/actions/cicdcfg@feature/air-191
with:
env: ${{ inputs.env }}
role: deploy
Expand All @@ -28,7 +32,7 @@ runs:
shell: bash
run: |
echo LAMBDA_NAME=${{ inputs.env }}-${{ steps.app_settings.outputs.app_name }} >> $GITHUB_ENV
if [ $LAMBDA_NAME == "null" || ${{ inputs.package_ref }} == "null" ];
if [[ $LAMBDA_NAME == "null" || ${{ inputs.package_ref }} == "null" ]];
then
echo "Missing package ref [${{ inputs.package_ref }}] or lambda name: [$LAMBDA_NAME]";
exit 128;
Expand All @@ -44,27 +48,36 @@ runs:
audience: ${{ steps.app_settings.outputs.aws_audience }}

- name: Update container lambda - get image uri with given image tag
if: ${{ steps.app_settings.outputs.app_registry == 'ecr' }}
if: ${{ steps.app_settings.outputs.app_registry == 'ecr' &&
steps.app_settings.outputs.app_deployer != 'sls' }}
uses: Iterative-Scopes/cicd-framework/actions/get-ecr-image-uri@main
id: get-image-uri
with:
with:
aws_region: ${{ steps.app_settings.outputs.aws_region }}
repository: ${{ steps.app_settings.outputs.app_repository }}
image_tag: ${{ inputs.package_ref }}
image_tag: ${{ inputs.package_ref }}

- name: Update container lambda - update lambda with image uri
if: ${{ steps.app_settings.outputs.app_registry == 'ecr' &&
steps.get-image-uri.outputs.uri }}
if: ${{ steps.app_settings.outputs.app_registry == 'ecr' &&
steps.get-image-uri.outputs.uri &&
steps.app_settings.outputs.app_deployer != 'sls' }}
shell: bash
run: |
echo ${{ steps.get-image-uri.outputs.uri }}
aws lambda update-function-code --function-name ${{ env.LAMBDA_NAME }} --image-uri ${{ steps.get-image-uri.outputs.uri }}
exit 0

- name: Update lambda from S3 registry
if: contains(${{ steps.app_settings.outputs.app_registry}}, "s3:::")
if: ${{ startsWith(steps.app_settings.outputs.app_registry, 's3:') &&
steps.app_settings.outputs.app_deployer != 'sls' }}
shell: bash
run: |
echo "update lambda with [${{ inputs.package_ref }}]"
aws lambda update-function-code --dry-run --function-name ${{ steps.app_settings.outputs.app_repository}}-${{ inputs.env }}-lookup_patient_data --s3-bucket dev-is-phi-infrastructure --s3-key serverless/dev/${{ steps.app_settings.outputs.app_repository}}.zip
exit 0
aws lambda update-function-code --dry-run --function-name ${{ steps.app_settings.outputs.app_repository}}-${{ inputs.env }}-lookup_patient_data --s3-bucket dev-is-phi-infrastructure --s3-key serverless/dev/${{ steps.app_settings.outputs.app_repository}}.zip
exit 0

- name: Deploy lambda
if: ${{ steps.app_settings.outputs.app_deployer == 'sls' }}
shell: bash
run: |
make -C ${{ inputs.make-directory }} deploy stage="${{ inputs.env }}" tag="${{ inputs.package_ref }}"
23 changes: 23 additions & 0 deletions actions/get-commit-sha/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
name: fetch triggering commit sha
description: fetch triggering commit sha

outputs:
sha:
description: commit sha
value: ${{ steps.commit_sha.outputs.sha }}

# assume caller has connect to AWS account
runs:
using: "composite"
steps:
- name: Get commit sha
id: commit_sha
shell: bash
run: |
if [ "$GITHUB_EVENT_NAME" == "pull_request" ]; then
COMMIT_SHA=$(cat $GITHUB_EVENT_PATH | jq -r .pull_request.head.sha)
else
COMMIT_SHA=${{ github.sha }}
fi
echo "set output sha ["$COMMIT_SHA"]"
echo "::set-output name=sha::"$COMMIT_SHA""
14 changes: 14 additions & 0 deletions actions/setup-serverless/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# This file is the sharable action with build,unittest, integration test, code coverage reports.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please remove this function out. we don't want to have infra provisioning is performed from GitHub Action

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to use serverless setup to deploy lambda. I discussed this with @johnseq-iterative and clearly mentioned that infrastructure provisioning code should not be part of this pipeline. If you see the serverless templates here only the lambda specific roles and events are getting created from the template. We are expecting the other infra should already be in place before we run this pipeline.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@is-ttang Avik and I did discuss this. My understanding is the plan is to keep infra+code provisioning together until we are ready to separate them out. That keeps the CICD responsibility squarely on one team while we wait to e.g. move to gitlab. Does that work as an interim solution?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry -- reading this more closely. @is-achakraborty this makes sense. We'll split out as much as we can from our templates and create out of band. That's fine.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. but, you can run the SLS provisioning separately. it requires have more privilege for GitHub to access AWS to provision. Second, you just need to run ONCE, NOT run lambda update every time. At moment, GitHub OpenID connect scope is locked down to ECR publish and Lambda S3 write

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Separately means out of the pipeline? If yes then I think no meaning in using this pipeline anymore.
Yes we need the required AWS permissions for the pipeline to work. We can sit together and decide on the minimal subset of permission required.

# The sharable action can be used across applications by referring this action in the github workflow.

name: Setup serverless environment
description: Setup serverless environment

runs:
using: "composite"
steps:
- name: Install serverless
shell: bash
run: |
npm install -g serverless@3.24.1
serverless --version