Skip to content

S3 for Public Feed Downloads#652

Merged
binh-dam-ibigroup merged 18 commits into
devfrom
use-s3-for-public-feed-downloads
Jun 11, 2026
Merged

S3 for Public Feed Downloads#652
binh-dam-ibigroup merged 18 commits into
devfrom
use-s3-for-public-feed-downloads

Conversation

@binh-dam-ibigroup

@binh-dam-ibigroup binh-dam-ibigroup commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Checklist

  • Appropriate branch selected (all PRs must first be merged to dev before they can be merged to master)
  • Any modified or new methods or classes have helpful JavaDoc and code is thoroughly commented
  • The description lists all applicable issues this PR seeks to resolve
  • The description lists any configuration setting(s) that differ from the default settings
  • All tests and CI builds passing

Description

This PR adds support for the use of S3 bucket for public feed hosting, even if S3 is not used for storing feed versions for internal DataTools use.

Behind the scenes, support is added for alternate S3 credentials, region, and bucket for the MTCFeedResource and FeedUpdater that, if specified, override the application's ones:

extensions
  mtc
    s3_region
    s3_bucket
    s3_credentials_file

@br648 br648 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some minor nits. Also, this might be a good time to migrate to the newer and latest versions of Amazon AWS dependencies e.g. com.amazonaws -> software.amazon.awssdk

Comment thread src/main/java/com/conveyal/datatools/common/utils/aws/S3Utils.java Outdated
Comment thread src/main/java/com/conveyal/datatools/manager/jobs/FeedUpdater.java Outdated
Comment thread src/main/java/com/conveyal/datatools/manager/jobs/PublishProjectFeedsJob.java Outdated
@br648 br648 assigned binh-dam-ibigroup and unassigned br648 Jun 5, 2026
@binh-dam-ibigroup

Copy link
Copy Markdown
Contributor Author

Also, this might be a good time to migrate to the newer and latest versions of Amazon AWS dependencies

We have an internal backlog ticket for that!

@miles-grant-ibigroup

Copy link
Copy Markdown
Contributor

Took a look at the code but I'm a bit confused what this does -- does this allow for project-specific s3 deploying? Or is this just an extension to the mtc extension that allows for the use of the existing s3 deployment code

@binh-dam-ibigroup

Copy link
Copy Markdown
Contributor Author

does this allow for project-specific s3 deploying? Or is this just an extension to the mtc extension that allows for the use of the existing s3 deployment code

It allows separate AWS credentials for DT and the MTC extension, so that DT can upload the public feeds elsewhere. This PR does not cover project-specific publishing.

@miles-grant-ibigroup miles-grant-ibigroup left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very clean! I'm hoping we do the project-specific html deployment at some point but for now this is a clean solution

@binh-dam-ibigroup binh-dam-ibigroup merged commit 66d5567 into dev Jun 11, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants