Oxbow

Oxbow is a project to take an existing storage location which contains Apache Parquet files into a Delta Lake table. It is intended to run both as an AWS Lambda or as a command line application.

The project is named after Oxbow lakes to keep with the lake theme.

Table of Contents

• Using
  • Command Line
  • Lambda
• Development
  • Design
• Licensing

Using

Command Line

Executing cargo build --release from a clone of this repository will build the command line binary oxbow which can be used directly to convert a directory full of .parquet files into a Delta table.

This is an in place operation and will convert the specified table location into a Delta table!

Simple local files

% oxbow --table ./path/to/my/parquet-files

Files on AWS

% export AWS_REGION=us-west-2
% export AWS_SECRET_ACCESS_KEY=xxxx
# Set other AWS environment variables
% oxbow --table s3://my-bucket/prefix/to/parquet

Lambda

The Oxbow project includes several AWS Lambda functions for different Delta Lake operations, each designed for specific use cases.

Available Lambdas

Lambda	Purpose	Documentation
oxbow	Main lambda for converting parquet files to Delta Lake tables	See oxbow lambda docs
file-loader	Converts structured data files to parquet format for Delta Lake ingestion	See file-loader docs
group-events	Groups S3 events to reduce lock contention for high-concurrency scenarios	See group-events docs
sqs-ingest	Ingests JSON data from SQS into Delta Lake tables	See sqs-ingest docs
webhook	Receives JSONL webhook payloads and writes to Delta Lake tables	See webhook docs
parquet-concat	Concatenates multiple Parquet files to optimize oxbow ingestion	See parquet-concat docs
cdf-to-csv	Converts Delta Lake change data feed to CSV format	See cdf-to-csv docs
glue-create	Creates AWS Glue table entries for new Delta Lake tables	See glue-create docs
glue-sync	Syncs AWS Glue Data Catalog schemas with Delta Lake table schemas	See glue-sync docs

Deployment

The deployment/ directory contains the necessary Terraform to provision the function, a DynamoDB table for locking, S3 bucket, and IAM permissions.

After configuring the necessary authentication for Terraform, the following steps can be used to provision:

cargo lambda build --release --output-format zip --bin oxbow-lambda
terraform init
terraform plan
terraform apply

ℹ️	Terraform configures the Lambda to run with the smallest amount of memory allowed. For bucket locations with massive .parquet files, this may need to be tuned.

Advanced

To help ameliorate concurrency challenges for Delta Lake on AWS with the DynamoDb lock, the deployment/ directory also contains an "advanced" pattern which uses the group-events Lambda to help serialize S3 Bucket Notifications into an AWS SQS FIFO with Message Group IDs.

To build all the necessary code locally for the Advanced pattern, please run make build-release

Development

Building and testing can be done with cargo: cargo test.

In order to deploy this in AWS Lambda, it must first be built with the cargo lambda command line tool, e.g.:

cargo lambda build --release --output-format zip

This will produce the file: target/lambda/oxbow-lambda/bootstrap.zip which can be uploaded direectly in the web console, or referenced in the Terraform (see deployment.tf).

Design

Command Line

When running oxbow via command line it is a one time operation. It will take an existing directory or location full of .parquet files and create a Delta table out of it.

Lambda

When running oxbow inside of a AWS Lambda function it should be configured with an S3 Event Trigger and create new commits to a Delta Lake table any time a .parquet file is added to the bucket/prefix.

Licensing

This repository is licensed under the AGPL 3.0. If your organization is interested in re-licensing this function for re-use, contact me via email for commercial licensing terms: rtyler@buoyantdata.com