AWS

The CRDS client can be configured to read files from Amazon’s S3 service. The STScI AWS environment currently hosts files in the following buckets:

Environment	S3 Bucket Name	S3 Prefix
HST OPS	hst-crds-cache-ops
HST TEST	hst-crds-cache-test
ROMAN OPS	stpubdata	/roman/crds
ROMAN TEST	stpubdata-tst	/roman/crds/test
ROMAN INT	stpubdata-tst	/roman/crds/int

Tip

Running $ source crds_s3_set {observatory} {environment} sets all the necessary environment variables necessary to configure CRDS to read files from S3. See “Configuration” section below for more details.

Configuring CRDS to use S3

The CRDS client must be configured with environment variables to read files from S3 buckets. The exact configuration depends on the observatory. CRDS provides a convenience wrapper script crds_s3_set to automatically set the requisite environment vars depending on the observatory and use case inputs. The exact variables required can be found by running the script, viewing the source code, or looking at the examples below.

When CRDS detects that S3 access is enabled via the CRDS_MODE=s3 environment variable (alongside a few other environment variables), it will automatically use the S3 buckets for downloading mapping and reference files instead of the HTTP-based CRDS server.

Prerequisites

1. The boto3 and awscli packages must be installed in the CRDS environment to enable S3 access. This can be done via pip when installing CRDS by specifying the aws extra dependencies:

$ pip install crds[aws]

2. Roman Ops is publicly available as an AWS OpenData bucket - no AWS credentials are required to access it.

3. Other buckets require valid AWS credentials with access to the appropriate bucket: the compute environment must be configured with AWS credentials that have been granted access to the appropriate bucket. This is typically done by configuring the AWS CLI with aws configure or by setting the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables. HST buckets are accessible to STScI internal users only, and at this time STScI does not host any public S3 buckets for JWST CRDS access.

Configuration

You can configure your environment for AWS/S3 manually or by using the crds_s3_set script. Examples for both approaches are below:

ROMANHST

The s3 buckets for Roman only contain mappings and references from the last 5 contexts. The CRDS cache for Roman Ops is publicly accessible in the Open Data bucket and does not require an AWS account or credentials. If you do not want to use the latest context, you will need to manually set the CRDS_CONTEXT environment variable as well.

Using the crds_s3_set script to automatically set environment variables:

# Set CRDS_PATH to your local cache directory (will be created if it doesn't exist)
# Defaults to /tmp/crds_cache if not set in advance
$ export CRDS_PATH=/path/to/local/cache
# source crds_s3_set <observatory> <environment>
$ source crds_s3_set roman ops

If setting manually, the equivalent environment variables would be:

$ export CRDS_PATH=/path/to/local/cache
$ export CRDS_MAPPING_URI=s3://stpubdata/roman/crds/mappings/roman
$ export CRDS_DOWNLOAD_MODE=plugin
$ export CRDS_DOWNLOAD_PLUGIN='crds_s3_get ${FILENAME} -d ${OUTPUT_PATH} -s ${FILE_SIZE} -c ${FILE_SHA1SUM}'
$ export CRDS_S3_ENABLED=1
$ export CRDS_S3_BUCKET=stpubdata
$ export CRDS_REF_SUBDIR_MODE=flat
$ export CRDS_SERVER_URL=https://roman-crds-serverless.stsci.edu
$ export CRDS_CONFIG_URI=s3://stpubdata/roman/crds/config/roman
$ export CRDS_S3_PREFIX=/roman/crds
$ export CRDS_OBSERVATORY=roman
$ export CRDS_REFERENCE_URI=s3://stpubdata/roman/crds/references/roman
$ export CRDS_MODE=s3
$ export CRDS_S3_RETURN_URI=0

Fetching CRDS Files from S3

Once the environment is configured for S3 access, CRDS commands such as crds sync and crds bestrefs as well as mission-specific calibration pipeline commands will automatically fetch files from S3 as needed. The context defaults to latest if not specified explicitly (same behavior as other modes). Here are some example commands:

# Run romancal pipeline (will download necessary mappings/references from S3)
$ strun romancal.pipeline.ExposurePipeline l1_img.asdf

# Alternatively: Sync all files from the S3 bucket to the local cache
$ crds sync --all