About

Tutorial for creating a manual MPI cluster on Google Cloud using Terraform and gcloud CLI.

Prerequisites

• gcloud CLI installed and configured
• Terraform installed
• Google Cloud project ID configured in .env file

Configuration

Create a .env file in the project root with your configuration:

cp .env.example .env

Edit .env with your values, then load the variables:

source .env

Environment Variables

• GCP_PROJECT_ID - Your Google Cloud Project ID
• GCP_ZONE - Google Cloud Zone (e.g., us-west1-a)
• GCP_REGION - Google Cloud Region (e.g., us-west1)
• CLUSTER_NAME - Cluster name tag
• MPIUSER - MPI user name
• MASTER_NODE - Master node name
• WORKER_NODES - Comma-separated worker node names
• NUM_WORKERS - Number of worker nodes
• MACHINE_TYPE - VM machine type (e.g., e2-micro)

Quick Start

1. Create VM Instances

cd terraform
terraform init
terraform apply -auto-approve

Wait 60 seconds for the startup scripts to complete and the cluster to be ready.

2. Copy Your MPI Program to Master

gcloud compute scp /path/to/local/dir mpiuser@master:/home/mpiuser/ --zone=$GCP_ZONE --project=$GCP_PROJECT_ID --recurse

3. Connect to Master

gcloud compute ssh mpiuser@master --zone=$GCP_ZONE --project=$GCP_PROJECT_ID

4. Compile Your Program

On the master node, compile your MPI program:

mpicc -o /home/mpiuser/<your_program> /home/mpiuser/<your_program>.c

5. Copy Executable to Workers

From the master, copy the compiled executable to all worker nodes:

for i in 1 2 3; do
  scp /home/mpiuser/<your_program> worker-$i:/home/mpiuser/
done

6. Run Your MPI Program

Execute your MPI program across the cluster:

mpirun -np 2 --hostfile /home/mpiuser/hfile /home/mpiuser/<your_program>

Where:

• -np 2 specifies the number of processes
• --hostfile /home/mpiuser/hfile specifies the host file with cluster nodes
• /home/mpiuser/<your_program> is the path to your compiled executable

Cleanup

To destroy the cluster and free resources:

cd terraform
terraform destroy -auto-approve