Getting Started#

Prerequisites#

To start using Elm, you will need:

an active SUNet ID,

What is a SUNet ID?

A SUNet ID is a unique 3-8 character account name that identifies you as a member of the Stanford community, with access to the Stanford University Network of computing resources and services. Not to be confused with University ID (a 8-digit number that appears on your Stanford ID Card), your SUNet ID is a permanent and visible part of your Stanford identity and often appears in your Stanford email address (eg. sunetid@stanford.edu).

SUNet IDs are not managed by Research Computing. For more information, see https://accounts.stanford.edu/

SUNet ID service levels and external collaborators

Base-level service is sufficient for Elm accounts. External collaborators, or users without a SUNet ID, can be sponsored by a PI a get a sponsored SUNet ID at no cost. Please see the sponsorship page for more information.
Belong to an Elm Stanford Workgroup, as a member or as an administrator, that grants access to an Elm instance or to an Elm instance bucket.
Familiarity with the concepts and terms used throughout our documentation

Optional, but highly recommended#

A Sherlock Account
Some familiarity with [Unix/Linux command-line environments][url_unix]
Familiarity with the S3 API

How to request a new Elm Space#

Elm is currently under development and is not taking new storage requests.

How to access an existing Elm Space#

The first step to gaining access to an Elm instance or to an Elm instance bucket is be added to one of the Elm Stanford Workgroups.

If you are a PI and you requested a dedicated Elm instance then part of that setup should have been adding you as a member and administrator to a dedicated elm:p-* prefixed workgroup (e.g., elm:p-hnc for the Human Neural Circuitry program).

For dedicated Elm instances we create the following workgroups for a given project:

elm:p-<project>
elm:p-<project>/administrator
elm:p-<project>-uploader
elm:p-<project>-editor

If you are a student and need access to a dedicated Elm instance then your PIs will need to have added your SUNet ID to the primary workgroup, elm:p-<project>, and optionally to have added you to the uploader or editor workgroups to give you permission beyond reading buckets.

The following workgroups and Elm instances are currently supported:

elm:p-bil (https://bil.elm.stanford.edu:9001/ console, https://bil.stanford.edu:9000/ S3)
elm:p-hnc (https://hnc.elm.stanford.edu:9001/ console, https://hnc.stanford.edu:9000/ S3)
elm:p-srcc (https://srcc.elm.stanford.edu:9001/ console, https://srcc.elm.stanford.edu:9000/ S3)

Once you are a member or administrator of one or more of these workgroups you may access it's console. When you navigate to the console URL you should see a login page that offers a Stanford OpenID Connect button:

If you click on the Stanford OpenID Connect button you will be redirected the Stanford OIDC Service. The first time you visit this page you will be asked to confirm the release of your elm:* entitlements (Stanford Workgroups) to the Stanford Research Computing Elm MinIO Service.

You may select "Ask me again at next login", in which case you will be prompted to release your entitlements to the Elm service the next time you log in.
You may select "Ask me again if information to be provided to this service changes", in which case you will not be prompted again unless either something changes on the Elm side or your list of Elm entitlements changes.
You may select "Do not ask me again" in which case you're approving to always release your entitlement details to all services.

After selecting your option, click the Accept button to continue.

Once you click the Accept button you should be directed back to the Elm instance and logged in. If you land on a page that simply states Invalid Login then something has gone wrong. Either you weren't a member of the required workgroups for that Elm instance, or something went wrong with our login process. Please reach out to us for assistance.

Upon success you should see one of two possible types of landing page. If you are a normal workgroup member, not an administrator, you will see the standard user landing page, which allows you to browse S3 buckets, create new Access Keys tied to your account, view (but not modify) administrative details about Buckets, and Sign out:

If you are an administrator of your workgroup, you will see a landing page that lists some additional options for administering the Elm instance:

Console Session Expiration

Login sessions on the console expire after 12 hours. If you haven't signed out of the console 12 hours after your login, you will be forced to login again.

While you can use the console UI to manage objects in buckets, browsing and downloading objects, or managing objects if you are a member of an uploader or editor workgroup, the expectation is that most users will want to configure a local S3 client to manage the process of downloading and uploading objects. The advantage to using S3 is that it can be run from a terminal in the background, and that it won't time out after 12 hours.

S3 Client Setup#

Once you are logged into the console you may use the Access Keys button on the left-hand side of the screen to generate a new S3 Access Key / Secret Key pair.

Once you click on the Access Keys button you will be directed to a page that lists any existing keys, and that allows you to create new keys or to delete existing keys.

If you click on the Create access key button, on the right-hand side of the page, a new dialogue pop-up will allow you to specify details about the Access Key / Secret Key pair you are creating. By default the Access Key and Secret Key fields will be filled in with cryptographically strong values. It is strongly recommended that you not try to change these values, e.g., to make them easier to remember. In order to use these keys with a client you'll need to be saving the values to a local file, so there's little point to trying to make the values something you can remember.

Additional options allow you to set a time the Access Key will expire, or to restrict what the Access Key is allowed to do by defining a custom AWS IAM Policy that restricts permissions beyond what your account allows by default:

After you click on the Create button you will be given an option to download the credentials you've created. This will be the only chance you have to download the Secret Key. If you lose the Secret Key value you will need to create a new Access Key / Secret Key pair.

After you've downloaded your new Access Key / Secret Key pair you can set up an S3 client. There are many S3 client options available, the ones we've tested are:

MinIO Client
AWS Command Line Interface
Rclone

MinIO Client Setup#

The MinIO Client, called mc, provides the most feature compatible Elm client, as it's built by the same people who build the S3 service used by Elm.

mc can be installed on Linux, MacOS, or Windows. The Quickstart guide offers directions on how to install the client on these platforms.

As part of the installation process you'll take the Access Key / Secret Key pair you generated and add it to the mc configuration using a command:

mc alias set ALIAS HOSTNAME ACCESS_KEY SECRET_KEY

as a concrete example, let's say we wanted to add a key for the https://test.elm.stanford.edu:9000/ service and I had generated an Access Key / Secret Key pair

HtLLtW6JnYlvl7RzU5OT / CYPxhg4enYu2XosCSJaeNwGyDGsBwa9ArpxfZLzM

I could add a my-elm alias:

$ mc alias set my-elm https://test.elm.stanford.edu:9000 HtLLtW6JnYlvl7RzU5OT CYPxhg4enYu2XosCSJaeNwGyDGsBwa9ArpxfZLzM
Added `hnc-elm` successfully.

The Elm service benefits from using larger part sizes when uploading multi-part objects, if you're able to we request you use mc put with its --part-size flag set to 5GiB when uploading files:

$ mc put --part-size 5GiB file.dat my-elm/bucket/file.dat

AWS Command Line Interface Setup#

AWS Command Line Interface, called aws, is the official Amazon S3 client, and Elm ought to be fully compatible with it.

aws can be installed on Linux, MacOS, or Windows. The AWS CLI Getting Started guide provides directions on installing the client.

As part of the installation process you'll also need to follow the additional MinIO provided directions in AWS CLI with MinIO Server.

As an example, if you want to use the profile name my-elm:

$ aws configure --profile my-elm
AWS Access Key ID [None]: HtLLtW6JnYlvl7RzU5OT
AWS Secret Access Key [None]: CYPxhg4enYu2XosCSJaeNwGyDGsBwa9ArpxfZLzM
Default region name [None]: us-east-1
Default output format [None]:

$ aws configure --profile my-elm set s3.signature_version s3v4

The MinIO aws directions point out the requirement to set the endpoint URL, otherwise aws will assume you're trying to connect to AWS S3 services instead of Elm services. If you've setup a distinct profile then you may want to configure that profile to automatically use the Elm endpoint URL:

$ aws configure --profile my-elm set endpoint_url https://test.elm.stanford.edu:9000

The Elm service benefits from using larger part sizes when uploading multi-part objects, if you're able to we request that you also configure your aws client to use a chunk size of 5GiB:

$ aws configure --profile my-elm set s3.multipart_chunksize 5GiB

Rclone Setup#

RClone is another popular tool that offers some support for S3 services like Elm.

rclone can be installed on Linux, MacOS, or Windows. The install directions provide sections that detail multiple ways to install rclone across each of those platforms.

The S3 MinIO configuration instructions outline an example profile:

[minio]
type = s3
provider = Minio
env_auth = false
access_key_id = USWUXHGYZQYFYFFIT3RE
secret_access_key = MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
region = us-east-1
endpoint = http://192.168.1.106:9000
location_constraint =
server_side_encryption =

You'll need to set access_key_id / secret_access_key to the Access Key / Secret Key values you downloaded, and set the endpoint to the URL for your Elm instance. Using the example profile name of my-elm and configuring it to use https://test.elm.stanford.edu:9000:

[my-elm]
type = s3
provider = Minio
access_key_id = HtLLtW6JnYlvl7RzU5OT
secret_access_key = CYPxhg4enYu2XosCSJaeNwGyDGsBwa9ArpxfZLzM
endpoint = http://test.elm.stanford.edu:9000

You can configure rclone by calling it with the command config. An example of this setup is below, with some of the verbose output elided using ellipses:

$ rclone config
...
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> n
name> my-elm
...
Storage> s3
...
provider> Minio
...
env_auth> false
...
access_key_id> HtLLtW6JnYlvl7RzU5OT
...
secret_access_key> CYPxhg4enYu2XosCSJaeNwGyDGsBwa9ArpxfZLzM
...
region>
...
endpoint> https://test.elm.stanford.edu:9000
...
location_constraint>
...
acl>
...
server_side_encryption>
...
sse_kms_key_id>
Edit advanced config?
y) Yes
n) No (default)
y/n> n
--------------------
[my-elm]
type = s3
provider = Minio
access_key_id = HtLLtW6JnYlvl7RzU5OT
secret_access_key = CYPxhg4enYu2XosCSJaeNwGyDGsBwa9ArpxfZLzM
endpoint = https://test.elm.stanford.edu:9000
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:

Name                 Type
====                 ====
my-elm               s3

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

By default rclone attempts to create buckets automatically. Unless you have extended permissions as an administrator you won't have permission to create buckets, and will need to supply the flag --s3-no-check-bucket when calling rclone to avoid a permission denied error.

The Elm service benefits from using larger part sizes when uploading multi-part objects, if you're able to we request you set the flag --s3-chunk-size to 5Gi:

$ rclone --s3-no-check-bucket --s3-chunk-size 5Gi copyto file.dat my-elm:bucket/file.dat

$ rclone --s3-no-check-bucket --s3-chunk-size 5Gi copy ./dir/ my-elm:bucket/dir/

Alternatively when you configure rclone, or reconfigure it, you can opt to go into the Edit advanced config section and set the chunk_size option to 5Gi:

$ rclone config
...
Edit advanced config?
y) Yes
n) No (default)
y/n> y
...
Option upload_cutoff.
Cutoff for switching to chunked upload.
Any files larger than this will be uploaded in chunks of chunk_size.
The minimum is 0 and the maximum is 5 GiB.
...
chunk_size> 5Gi
...
Edit advanced config?
y) Yes
n) No (default)
y/n> n
--------------------
[my-elm]
type = s3
provider = Minio
access_key_id = HtLLtW6JnYlvl7RzU5OT
secret_access_key = CYPxhg4enYu2XosCSJaeNwGyDGsBwa9ArpxfZLzM
endpoint = https://test.elm.stanford.edu:9000
chunk_size = 5Gi
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:

Name                 Type
====                 ====
my-elm               s3

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

How much does Elm cost?#

Please visit the University IT rates page for the most up-to-date information on the cost of Elm storage.