Skip to content

docs

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

docs

README.md

Single-Device Setup

This section describes the setup process for Single-Device Mode. In this mode, the user runs all of the code on their own machine and uploads directly to their configured cloud storage provider. Only this machine may interact with the archive.

Single-Device Mode

First, create an account with a cloud object storage provider. Create a configuration, which we'll call utahfs.yaml, with the following contents:

storage-provider:
  s3-app-id: # Generated app id goes here.
  s3-app-key: # Generated app key goes here.
  s3-bucket: # Chosen bucket name.
  s3-url: # URL of service.
  s3-region: # Name of region.
  # # Alternatively, remove the above s3-* keys and setup Backblaze B2 access:
  # b2-key-id:
  # b2-app-key:
  # b2-bucket:
  # # Or finally, use GCS:
  # gcs-bucket-name:

keep-metadata: true

# password: password123
# archive: true
# oram: true

and fill in the items under storage-provider with the credentials you got from your storage provider. Setup instructions for several storage providers are available at:

The user can optionally set the encryption password in the configuration file by uncommenting the line password: password123 and replacing password123 with their actual password. Setting the password in the configuration file makes it easier to use strong passwords, like those generated by running $ openssl rand -hex 16. If that line is left out, the user will be prompted for their password every time the client starts up. The password will be set the first time the client runs and can't be changed once set.

Archive mode is enabled by uncommenting the line archive: true and may be enabled/disabled as the user desires over time. Oblivious RAM mode is enabled by uncommenting the line oram: true but must be the same over the lifetime of the archive.

After this, the next step is to actually run the UtahFS client. Install it, if you haven't already:

$ go get github.com/cloudflare/utahfs/cmd/utahfs-client

and run it with:

$ utahfs-client -cfg ./utahfs.yaml -mount ./utahfs

where ./utahfs.yaml is the path to the configuration file, and ./utahfs is the path to the directory to mount. The directory to mount must already exist and be empty.

You're done! Please be sure to read the note on locally stored data.

Multi-Device Setup

This section describes the setup process for Multi-Device Mode. In this mode, the user needs to run a server process on trusted hardware. For example, a Raspberry Pi or Intel NUC on their LAN are good candidates. The users' devices interact only with the server, and the server handles uploads / downloads with the cloud storage provider. This allows many users to work on the same archive at once and can improve user-perceived performance.

Multi-Device Mode

First, create an account with a cloud object storage provider. Setup instructions for several storage providers are available at:

Then create a file for the server configuration on the server, which we'll call utahfs-server.yaml, with the following contents:

storage-provider:
  s3-app-id: # Generated app id goes here.
  s3-app-key: # Generated app key goes here.
  s3-bucket: # Chosen bucket name.
  s3-url: # URL of service.
  s3-region: # Name of region.
  # # Alternatively, remove the above s3-* keys and setup Backblaze B2 access:
  # b2-acct-id:
  # b2-app-key:
  # b2-bucket:
  # b2-url:
  # # Or finally, use GCS:
  # gcs-bucket-name:

keep-metadata: true

# oram:
#   key: # Generate with: `openssl rand -hex 16`

transport-key: # Generate with: `openssl rand -hex 16`

and fill in the items under storage-provider with the credentials you got from your storage provider. Generate a transport key with the command given above, and fill that in as well. The transport key is a shared secret between the client and server that allows them to authenticate with each other.

If you're going to be using Oblivious RAM, uncomment the two lines starting with oram: and generate a new key there too.

Install the server, if you haven't already:

$ go get github.com/cloudflare/utahfs/cmd/utahfs-server

and run it with:

$ utahfs-server -cfg ./utahfs-server.yaml

This process needs to be running at all times, so consider setting it up with a systemd service or something similar.

Second, create a file for the client configuration, which we'll call utahfs.yaml, with the following contents:

remote-server:
  url: https://server:3002/
  transport-key: # Fill with the transport key generated previously.

# password: password123
# archive: true
# oram: true

Set url to be the URL of the server, and transport-key to be the same value that you generated for the server's config. Refer to the section on Single-Device Setup for information about password, archive, and oram.

Finally, install the UtahFS client, if you haven't already:

$ go get github.com/cloudflare/utahfs/cmd/utahfs-client

and run it with:

$ utahfs-client -cfg ./utahfs.yaml -mount ./utahfs

where ./utahfs.yaml is the path to the configuration file, and ./utahfs is the path to the directory to mount. Again, the directory to mount must already exist and be empty.

You're done! Please be sure to read the note on locally stored data.

Appendix

Important Note on Locally Stored Data

UtahFS maintains some locally stored data, either on the user's computer in Single-Device mode or on the server in Multi-Device mode. Many users have been confused over the purpose of this data, so this section briefly describes how it's used.

The default location for local data in Single-Device mode is a hidden .utahfs directory, or a utahfs-data directory in Multi-Device mode. This directory usually has four files:

  1. cache - Commonly accessed data blocks are cached here for faster access. After it reaches a configured maximum size (based on the disk-cache-size config setting), it evicts blocks on an LRU basis.
  2. metadata - All metadata blocks (like inodes, or global state) are cached here for faster access. This is enabled by the keep-metadata config setting, and does not have a maximum size.
  3. pin.json - This keeps a small amount of cryptographic information that can help detect if the storage provider has tampered with our archive.
  4. wal - This contains a WAL, or Write-Ahead Log, where changes to the files in the archive are buffered before being pushed to the storage provider in the background. It has a maximum size (based on the max-wal-size config setting), after which new changes will block on the storage provider accepting previous writes.

Out of all of these, the most important one is #4 (the WAL) because it stores data that hasn't been uploaded yet. You can see the number of blocks in the WAL by going to the metrics server (http://localhost:3001/metrics in Single-Device mode, or http://localhost:3003/metrics on the server in Multi-Device mode) and checking the value of the local_wal_size metric. This metric must be zero, indicating that all changes have been uploaded, before you can safely delete the local data folder.

See the Advanced Configuration document for more information about the config settings mentioned above and other fine-tuning.