How to deploy locally (not AWS)

[jlaura]

Apologies for the likely naive question. I have a local, test STAC-API compliant instance running on in a docker cluster (minio for the holding STAC compliant JSON files, Apache, etc.) and then have STAC-Browser serving these files out in the nice web instance.

I am interested in testing sat-api without pushing to AWS. So I have also set up an ES cluster and pushed the stac metadata into that. How do I go about deploying the sat-api locally?

My current attempts to use a simple docker-compose file with the API are failing and I am just guessing at the environment variables and the ports that I need to populate/expose.

version: '3'

services:
  satapi:
    image: 'satutils/sat-api:develop'
    ports:
      - '9380:80'
    environment:
      STAC_ID: THEMIS
      ES_HOST: smalls:9300
    networks:
      - default

[drewbo]

@jlaura not a naive question at all. A while back this was an express API (easier to run locally) but that's deprecated now that we've moved to API Gateway. Because we aren't using SAM or serverless, there aren't great options to run this locally out of the box. Likely the "easiest" (but not that easy) way to run the API locally is by following the deploy instructions here until the deploy step and then running kes compile instead of kes deploy. This will give you a cloudformation template which you can use with localstack. There's a chance @matthewhanson is working on updated deployment steps which simplify this a bit as well

[matthewhanson]

Hi @jlaura , you might want to check out the new STAC-API: https://github.com/stac-utils/stac-api
It's a completely refactored version of sat-api, and is compliant with the latest version of STAC, 0.9.0.
I have not deployed it locally, but it uses serverless so should be easier to set up.

[jlaura]

@matthewhanson Thanks for the link. I have been able to make some more progress getting this running locally.

I wanted to post some info in case it is of use for anyone else trying to set this up locally.

Minio(S3) / Apache are running to store / serve the STAC compliant metadata, COGs, and thumbnail jpegs using the following docker-compose. Note that I am also using traefik locally, but this can be removed / omitted.

Minio/Apache docker-compose.yml

version: '3'

services:
  apache:
    image: 'bitnami/apache:latest'
    ports:
      - '9080:8080'
      - '9443:8443'
    volumes:
      - ../single_image/:/app
    labels:
      - "traefik.enable=true"
      - "traefik.frontend.priority=600"
      - "traefik.frontend.rule=Method:GET"
      - "traefik.frontend.rule=PathPrefixStrip:/stac"
      - "traefik.backend=apache"
      - "traefik.docker.network=traefik-public"
    networks:
      - default
      - traefik-public

  minio:
    image: 'minio/minio:latest'
    ports:
      - 9000:9000
    volumes:
      - ./minio_data/:/data
    labels:
      - "traefik.enable=true"
      - "traefik.frontend.priority=600"
      - "traefik.frontend.rule=Method:GET"
      - "traefik.frontend.rule=PathPrefixStrip:/minio"
      - "traefik.backend=minio"
      - "traefik.docker.network=traefik-public"
    networks:
      - default
      - traefik-public
    command: server /data
    environment:
      MINIO_ACCESS_KEY: minio
      MINIO_SECRET_KEY: minio123

networks:
  traefik-public:
    external : true

We then need an elasticsearch backend to get the metadata pushed into. I am running this in yet another docker-compose:

ElasticSearch docker-compose.yml

                                                                                     1,1           All
version: '3'

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.8.7
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    ports:
      - "9300:9200"
  kibana:
    image: docker.elastic.co/kibana/kibana:6.3.2
    ports:
      - "9601:5601"

I am currently working on getting the ingest and api lambdas exposed as well in this setup. Luckily, lambci offers containers that I think will work.

Lambda docker-compose.yml

version: '3'
services:
  lambda_ingest:
    image: lambci/lambda:nodejs12.x
    environment:
      DOCKER_LAMBDA_STAY_OPEN : 1
      ES_HOST : http://smalls:9300/
    ports:
      - "9001:9001"
    networks:
      - default
    volumes:
      - ./lambdas/ingest/dist:/var/task:ro,delegated
      - ./libs:/opt
    command:  index.handler

  lambda_api:
    image: lambci/lambda:nodejs12.x
    environment:
      DOCKER_LAMBDA_STAY_OPEN : 1
      ES_HOST : http://smalls:9300/
      SATAPI_URL: http://smalls:9002/
    ports:
      - "9002:9001"
    networks:
      - default
    volumes:
      - ./lambdas/api/dist:/var/task:ro,delegated
    command:  index.handler

Using curl, I can do an ingest without a problem: curl -d '{"url":"http://127.0.0.1/stac:9000/themis/test/collection.json"}' http://127.0.0.1:9001/2015-03-31/functions/myfunction/invocations and can see in the ES logs that things are being written. It is also possible to see the return on the call to the api function: curl http://127.0.0.1:9002/2015-03-31/functions/myfunction/invocations.

Next step is to test sat-search. I am running into a few issues here that I wonder if others might have insight:

• The API_URL needs to be set to some other address than the default. I tried using the environment variable and using an os.environ setter before import of sat search, but no joy. I had to modify the config.py. The variables were being set properly in search.py, but somewhere deeper in (urllib and ssl) the API URL was defaulting back.
• The stac/search endpoint is 404ing. I don't know if this is caller error or setup on the lambci side where each container needs to be a different container. I think the former since the determine endpoint function in the stac-api should be handling dispatching. Are stac-search and the stac-api in sync with each other for how URLs are being built? I'm on the stac 0.9.0 compliant branch and the git repo dev for stac-search.

Let me know if this info is helpful or if you all prefer users to focus on doing an AWS deploy.

[danielklim]

@drewbo @jlaura A bit late to the party but I am putting the finishing touches on a SAT-API stack running on a local openfaas /kubernetes cluster with a local ES/KB deployment: sat-api-openfaas. My notes aren't finished yet but it does appear to work per spec with the Sentinel-5P dataset hosted on AWS. Please let me know if you have any questions -- thanks!