Getting Started | Campus#

Start Here!#

• Interested in Elm, but not a user yet? Start with the "Prerequsites" section below.

• Just got an account? Start with the "Accessing your Elm Bucket" section

• Need help choosing a tool to transfer data? Start with the "Getting Data In and Out of Elm" Section

Prerequisites#

To start using Elm, you will need:

• an active SUNet ID,

  What is a SUNet ID?

  A SUNet ID is a unique 3-8 character account name that identifies you as a member of the Stanford community, with access to the Stanford University Network of computing resources and services. Not to be confused with University ID (a 8-digit number that appears on your Stanford ID Card), your SUNet ID is a permanent and visible part of your Stanford identity and often appears in your Stanford email address (eg. sunetid@stanford.edu).

  SUNet IDs are not managed by Research Computing. For more information, see https://accounts.stanford.edu/

  SUNet ID service levels and external collaborators

  Base-level service is sufficient for Elm accounts. External collaborators, or users without a SUNet ID, can be sponsored by a PI a get a sponsored SUNet ID at no cost. Please see the sponsorship page for more information.

• Belong to an Elm Stanford Workgroup, as a member or as an administrator. This grants access to an Elm instance or to an Elm instance bucket.

• Familiarity with the concepts and terms used throughout our documentation

Optional, but Highly Recommended#

• A Sherlock Account
• Some familiarity with Unix/Linux command-line environments
• Familiarity with the S3 API

How much does Elm cost?#

Please visit the University IT rates page for the most up-to-date information on the cost of Elm storage.

Order a new Elm bucket or make a change to an existing one#

Please see our orders page for more information.

Buckets and Naming Conventions#

When you create a bucket on Elm’s Campus partition, you must choose a bucket name that follows specific guidelines:

• Between 3 - 63 characters
• Only lowercase letters, numbers, and hyphens (-) are allowed.
• No spaces, periods, or special characters.
• Bucket names must not end with the suffix -s3alias. This suffix is reserved for access point alias names.
• Example: sunetid, lab123-backup, etc.

Accessing your Elm Bucket#

Because Elm is designed for infrequently accessed data, it's reasonable to expect both new and exiting users may need a refresher from time to time on how to access their storage bucket.

All you need to provide is the name of the storage bucket you'd like to access and the highest possible risk classification of data stored within it.

Getting Data In and Out of Elm#

You can manage your Elm bucket in two primary ways:

The first step to gaining access to an Elm instance or to an Elm instance bucket is be added to one of the Elm Stanford Workgroups.

To determine the workgroups associated with your bucket, please refer to the Elm Workgroup & URL Helper tool above.

If you are a student and need access to a dedicated Elm instance then your PIs will need to have added your SUNet ID to the primary workgroup, elm:p-<project>, and optionally to have added you to the uploader or editor workgroups to give you permission beyond reading buckets.

Once you are a member or administrator of one or more of these workgroups you may access the associated MinIO console. When you navigate to the console URL you should see a login page that offers a Stanford OpenID Connect button:

If you click on the Stanford OpenID Connect button you will be redirected the Stanford OIDC Service. The first time you visit this page you will be asked to confirm the release of your elm:* entitlements (Stanford Workgroups) to the Stanford Research Computing Elm MinIO Service.

• You may select "Ask me again at next login", in which case you will be prompted to release your entitlements to the Elm service the next time you log in.
• You may select "Ask me again if information to be provided to this service changes", in which case you will not be prompted again unless either something changes on the Elm side or your list of Elm entitlements changes.
• You may select "Do not ask me again" in which case you're approving to always release your entitlement details to all services.

After selecting your option, click the Accept button to continue.

Once you click the Accept button you should be directed back to the Elm instance and logged in. If you land on a page that simply states Invalid Login then something has gone wrong. Either you weren't a member of the required workgroups for that Elm instance, or something went wrong with our login process. Please reach out to us for assistance.

Upon success you should see one of two possible types of landing page. If you are a normal workgroup member, not an administrator, you will see the standard user landing page, which allows you to browse S3 buckets, create new Access Keys tied to your account, view (but not modify) administrative details about Buckets, and Sign out:

If you are an administrator of your workgroup, you will see a landing page that lists some additional options for administering the Elm instance:

While you can use the console UI to manage objects in buckets, browsing and downloading objects, or managing objects if you are a member of an uploader or editor workgroup, the expectation is that most users will want to configure a local S3 client to manage the process of downloading and uploading objects. The advantage to using S3 is that it can be run from a terminal in the background, and that it won't time out after 12 hours.

S3 Client Setup#

Once you are logged into the console you may use the Access Keys button on the left-hand side of the screen to generate a new S3 Access Key / Secret Key pair.

Once you click on the Access Keys button you will be directed to a page that lists any existing keys, and that allows you to create new keys or to delete existing keys.

If you click on the Create access key button, on the right-hand side of the page, a new dialogue pop-up will allow you to specify details about the Access Key / Secret Key pair you are creating. By default the Access Key and Secret Key fields will be filled in with cryptographically strong values. It is strongly recommended that you not try to change these values, e.g., to make them easier to remember. In order to use these keys with a client you'll need to be saving the values to a local file, so there's little point to trying to make the values something you can remember.

Additional options allow you to set a time the Access Key will expire, or to restrict what the Access Key is allowed to do by defining a custom AWS IAM Policy that restricts permissions beyond what your account allows by default:

After you click on the Create button you will be given an option to download the credentials you've created. This will be the only chance you have to download the Secret Key. If you lose the Secret Key value you will need to create a new Access Key / Secret Key pair.

After you've downloaded your new Access Key / Secret Key pair you can set up an S3 client. There are many S3 client options available, the ones we've tested are:

• MinIO Client
• AWS Command Line Interface
• Rclone

MinIO Client Setup#

The MinIO Client, called mc, provides the most feature compatible Elm client, as it's built by the same people who build the S3 service used by Elm.

mc can be installed on Linux, MacOS, or Windows. The Quickstart guide offers directions on how to install the client on these platforms.

As part of the installation process you'll take the Access Key / Secret Key pair you generated and add it to the mc configuration using a command:

mc alias set ALIAS HOSTNAME ACCESS_KEY SECRET_KEY

as a concrete example, let's say we wanted to add a key for the https://campus.elm.stanford.edu:9000/ service and I had generated an Access Key / Secret Key pair

HtLLtW6JnYlvl7RzU5OT / CYPxhg4enYu2XosCSJaeNwGyDGsBwa9ArpxfZLzM

I could add a my-elm alias:

$ mc alias set my-elm https://campus.elm.stanford.edu:9000 HtLLtW6JnYlvl7RzU5OT CYPxhg4enYu2XosCSJaeNwGyDGsBwa9ArpxfZLzM
Added `my-elm` successfully.

The Elm service benefits from using larger part sizes when uploading multi-part objects, when using mc put you must use it with its --part-size flag set to 5GiB when uploading files:

$ mc put --part-size 5GiB file.dat my-elm/bucket/file.dat

AWS Command Line Interface Setup#

AWS Command Line Interface, called aws, is the official Amazon S3 client, and Elm ought to be fully compatible with it. To successfully get the AWS CLI working, you should select a minimum of AWS version v2.13.0 or newer of the module instead of the default version. Please note that version 2.27.49 is available on Sherlock.

aws can be installed on Linux, MacOS, or Windows. The AWS CLI Getting Started guide provides directions on installing the client.

As part of the installation process you'll also need to follow the additional MinIO provided directions in AWS CLI with MinIO Server.

As an example, if you want to use the profile name my-elm:

$ aws configure --profile my-elm
AWS Access Key ID [None]: HtLLtW6JnYlvl7RzU5OT
AWS Secret Access Key [None]: CYPxhg4enYu2XosCSJaeNwGyDGsBwa9ArpxfZLzM
Default region name [None]: us-east-1
Default output format [None]:

$ aws configure --profile my-elm set s3.signature_version s3v4

The MinIO aws directions point out the requirement to set the endpoint URL, otherwise aws will assume you're trying to connect to AWS S3 services instead of Elm services. If you've setup a distinct profile then you may want to configure that profile to automatically use the Elm endpoint URL:

$ aws configure --profile my-elm set endpoint_url https://campus.elm.stanford.edu:9000

The Elm service benefits from using larger part sizes when uploading multi-part objects, you also must configure your aws client to use a chunk size of 5GiB:

$ aws configure --profile my-elm set s3.multipart_chunksize 5GiB

Rclone Setup#

RClone is another popular tool that offers some support for S3 services like Elm.

rclone can be installed on Linux, MacOS, or Windows. The install directions provide sections that detail multiple ways to install rclone across each of those platforms.

The S3 MinIO configuration instructions outline an example profile:

[minio]
type = s3
provider = Minio
env_auth = false
access_key_id = HtLLtW6JnYlvl7RzU5OT
secret_access_key = CYPxhg4enYu2XosCSJaeNwGyDGsBwa9ArpxfZLzM
region = us-east-1
endpoint = https://campus.elm.stanford.edu:9000
location_constraint =
server_side_encryption =

You'll need to set access_key_id / secret_access_key to the Access Key / Secret Key values you downloaded, and set the endpoint to the URL for your Elm instance. Using the example profile name of my-elm and configuring it to use https://campus.elm.stanford.edu:9000:

[my-elm]
type = s3
provider = Minio
access_key_id = HtLLtW6JnYlvl7RzU5OT
secret_access_key = CYPxhg4enYu2XosCSJaeNwGyDGsBwa9ArpxfZLzM
endpoint = https://campus.elm.stanford.edu:9000

You can configure rclone by calling it with the command config. An example of this setup is below, with some of the verbose output elided using ellipses:

$ rclone config
...
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> n
name> my-elm
...
Storage> s3
...
provider> Minio
...
env_auth> false
...
access_key_id> HtLLtW6JnYlvl7RzU5OT
...
secret_access_key> CYPxhg4enYu2XosCSJaeNwGyDGsBwa9ArpxfZLzM
...
region>
...
endpoint> https://campus.elm.stanford.edu:9000
...
location_constraint>
...
acl>
...
server_side_encryption>
...
sse_kms_key_id>
Edit advanced config?
y) Yes
n) No (default)
y/n> n
--------------------
[my-elm]
type = s3
provider = Minio
access_key_id = HtLLtW6JnYlvl7RzU5OT
secret_access_key = CYPxhg4enYu2XosCSJaeNwGyDGsBwa9ArpxfZLzM
endpoint = https://campus.elm.stanford.edu:9000
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:

Name                 Type
====                 ====
my-elm               s3

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

By default rclone attempts to create buckets automatically. Unless you have extended permissions as an administrator you won't have permission to create buckets, and will need to supply the flag --s3-no-check-bucket when calling rclone to avoid a permission denied error.

The Elm service benefits from using larger part sizes when uploading multi-part objects, you must set the flag --s3-chunk-size to 5Gi:

$ rclone --s3-no-check-bucket --s3-chunk-size 5Gi copyto file.dat my-elm:bucket/file.dat

$ rclone --s3-no-check-bucket --s3-chunk-size 5Gi copy ./dir/ my-elm:bucket/dir/

Alternatively when you configure rclone, or reconfigure it, you can opt to go into the Edit advanced config section and set the chunk_size option to 5Gi:

$ rclone config
...
Edit advanced config?
y) Yes
n) No (default)
y/n> y
...
Option upload_cutoff.
Cutoff for switching to chunked upload.
Any files larger than this will be uploaded in chunks of chunk_size.
The minimum is 0 and the maximum is 5 GiB.
...
chunk_size> 5Gi
...
Edit advanced config?
y) Yes
n) No (default)
y/n> n
--------------------
[my-elm]
type = s3
provider = Minio
access_key_id = HtLLtW6JnYlvl7RzU5OT
secret_access_key = CYPxhg4enYu2XosCSJaeNwGyDGsBwa9ArpxfZLzM
endpoint = https://campus.elm.stanford.edu:9000
chunk_size = 5Gi
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:

Name                 Type
====                 ====
my-elm               s3

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

Globus for Elm#

Globus is a data transfer and management service that simplifies the process of transferring large datasets across different storage systems, including cloud storage. Read more about Globus in general. Please refer to the S3 Client Setup section above to setup your Access Key/Secret Key pair to configure the Globus Connector.

Elm is an on-prem cloud-like service, and a Globus Mapped Collection using Globus S3 Connector is managed separately from Elm itself. For more information on setting up Globus and accessing your bucket on Elm, refer to https://globus.stanford.edu/cloud/elm.html

elm_archive#

elm_archive is a utility designed to facilitate the archival of data from Sherlock to Elm storage using Globus. In particular, this tool is useful for handling and packing many small files. For additional information on how to use elm_archive, review Archiving data to Elm from Sherlock.

For more information on how to use elm_archive, refer to https://docs.elm.stanford.edu/faq/#moving-data