Skip to content

Commit f680119

Browse files
dltjdltj
authored
Merge branch 'bep:master' into master
2 parents ad3c4eb + a67c241 commit f680119

36 files changed

+1862
-628
lines changed

‎.circleci/config.yml‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,14 +2,13 @@ version: 2
22
jobs:
33
build:
44
docker:
5-
- image: bepsays/ci-goreleaser:1.14.3
5+
- image: bepsays/ci-goreleaser:1.21900.20200
66
working_directory: /src
77
steps:
88
- checkout
99
- run:
1010
command: |
1111
go mod download
12-
goreleaser
1312
workflows:
1413
version: 2
1514
release:
@@ -18,6 +17,6 @@ workflows:
1817
context: org-global
1918
filters:
2019
tags:
21-
only: /v[0-9]+(\.[0-9]+)*(-.*)*/
20+
only: /disabled-v[0-9]+(\.[0-9]+)*(-.*)*/
2221
branches:
2322
ignore: /.*/

‎.github/dependabot.yml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# See https://docs.github.com/en/github/administering-a-repository/configuration-options-for-dependency-updates#package-ecosystem
2+
version: 2
3+
updates:
4+
- package-ecosystem: "gomod"
5+
directory: "/"
6+
schedule:
7+
interval: "daily"

‎.github/workflows/test.yml‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
on:
2+
push:
3+
branches: [ master ]
4+
pull_request:
5+
name: Test
6+
permissions:
7+
contents: read
8+
jobs:
9+
test:
10+
strategy:
11+
matrix:
12+
go-version: [1.19.x,1.20.x]
13+
platform: [macos-latest, ubuntu-latest, windows-latest]
14+
runs-on: ${{ matrix.platform }}
15+
steps:
16+
- name: Install Go
17+
uses: actions/setup-go@v4
18+
with:
19+
go-version: ${{ matrix.go-version }}
20+
- name: Install staticcheck
21+
run: go install honnef.co/go/tools/cmd/staticcheck@latest
22+
shell: bash
23+
- name: Install golint
24+
run: go install golang.org/x/lint/golint@latest
25+
shell: bash
26+
- name: Update PATH
27+
run: echo "$(go env GOPATH)/bin" >> $GITHUB_PATH
28+
shell: bash
29+
- name: Checkout code
30+
uses: actions/checkout@v3
31+
- name: Fmt
32+
if: matrix.platform != 'windows-latest' # :(
33+
run: "diff <(gofmt -d .) <(printf '')"
34+
shell: bash
35+
- name: Vet
36+
run: go vet ./...
37+
- name: Staticcheck
38+
run: staticcheck ./...
39+
#- name: Lint
40+
# run: golint ./...
41+
- name: Test
42+
env:
43+
S3DEPLOY_TEST_KEY: ${{ secrets.S3DEPLOY_TEST_KEY }}
44+
S3DEPLOY_TEST_SECRET: ${{ secrets.S3DEPLOY_TEST_SECRET }}
45+
run: go test -race ./... -coverpkg=./... -coverprofile=coverage.txt -covermode=atomic
46+
- name: Upload coverage
47+
if: success() && matrix.platform == 'ubuntu-latest' && !github.event.pull_request.head.repo.fork && github.actor != 'dependabot[bot]'
48+
run: |
49+
curl https://keybase.io/codecovsecurity/pgp_keys.asc | gpg --no-default-keyring --keyring trustedkeys.gpg --import # One-time step
50+
curl -Os https://uploader.codecov.io/latest/linux/codecov
51+
curl -Os https://uploader.codecov.io/latest/linux/codecov.SHA256SUM
52+
curl -Os https://uploader.codecov.io/latest/linux/codecov.SHA256SUM.sig
53+
gpgv codecov.SHA256SUM.sig codecov.SHA256SUM
54+
shasum -a 256 -c codecov.SHA256SUM
55+
chmod +x codecov
56+
./codecov

‎.travis.yml‎

Lines changed: 0 additions & 16 deletions
This file was deleted.

‎README.md‎

Lines changed: 99 additions & 63 deletions
Original file line numberDiff line numberDiff line change
@@ -1,115 +1,147 @@
11
# s3deploy
22

3+
[![Project status: active – The project has reached a stable, usable state and is being actively developed.](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active)
34
[![GoDoc](https://godoc.org/github.com/bep/s3deploy?status.svg)](https://godoc.org/github.com/bep/s3deploy)
4-
[![Build Status](https://travis-ci.org/bep/s3deploy.svg)](https://travis-ci.org/bep/s3deploy)
5-
[![Build status](https://ci.appveyor.com/api/projects/status/rofnubomdtpv5ht4?svg=true)](https://ci.appveyor.com/project/bep/s3deploy)
5+
[![Test](https://github.com/bep/s3deploy/actions/workflows/test.yml/badge.svg)](https://github.com/bep/s3deploy/actions/workflows/test.yml)
66
[![Go Report Card](https://goreportcard.com/badge/github.com/bep/s3deploy)](https://goreportcard.com/report/github.com/bep/s3deploy)
77
[![codecov](https://codecov.io/gh/bep/s3deploy/branch/master/graph/badge.svg)](https://codecov.io/gh/bep/s3deploy)
88
[![Release](https://img.shields.io/github/release/bep/s3deploy.svg?style=flat-square)](https://github.com/bep/s3deploy/releases/latest)
99

1010
A simple tool to deploy static websites to Amazon S3 and CloudFront with Gzip and custom headers support (e.g. "Cache-Control"). It uses ETag hashes to check if a file has changed, which makes it optimal in combination with static site generators like [Hugo](https://github.com/gohugoio/hugo).
1111

12+
* [Install](#install)
13+
* [Configuration](#configuration)
14+
* [Flags](#flags)
15+
* [Routes](#routes)
16+
* [Global AWS Configuration](#global-aws-configuration)
17+
* [Example IAM Policy](#example-iam-policy)
18+
* [CloudFront CDN Cache Invalidation](#cloudfront-cdn-cache-invalidation)
19+
* [Example IAM Policy With CloudFront Config](#example-iam-policy-with-cloudfront-config)
20+
* [Background Information](#background-information)
21+
* [Alternatives](#alternatives)
22+
* [Stargazers over time](#stargazers-over-time)
23+
1224
## Install
1325

1426
Pre-built binaries can be found [here](https://github.com/bep/s3deploy/releases/latest).
1527

16-
**s3deploy** is a [Go application](https://golang.org/doc/install), so you can also get and build it yourself via `go get`:
28+
**s3deploy** is a [Go application](https://golang.org/doc/install), so you can also install the latest version with:
1729

1830
```bash
19-
go get -u -v github.com/bep/s3deploy
31+
go install github.com/bep/s3deploy/v2@latest
2032
```
2133

2234
To install on MacOS using Homebrew:
2335

2436
```bash
2537
brew install bep/tap/s3deploy
2638
```
39+
40+
**Note** The brew tap above currently stops at v2.8.1; see [this issue](https://github.com/bep/s3deploy/issues/312) for more info.
2741

2842
Note that `s3deploy` is a perfect tool to use with a continuous integration tool such as [CircleCI](https://circleci.com/). See [this](https://mostlygeek.com/posts/hugo-circle-s3-hosting/) for a tutorial that uses s3deploy with CircleCI.
2943

30-
## Use
44+
## Configuration
3145

32-
```bash
33-
Usage of s3deploy:
34-
-V print version and exit
35-
-bucket string
36-
destination bucket name on AWS
37-
-config string
38-
optional config file (default ".s3deploy.yml")
39-
-distribution-id string
40-
optional CDN distribution ID for cache invalidation
41-
-force
42-
upload even if the etags match
43-
-h help
44-
-key string
45-
access key ID for AWS
46-
-max-delete int
47-
maximum number of files to delete per deploy (default 256)
48-
-public-access
49-
set public ACL on uploaded objects, defaults to private if not set.
50-
-path string
51-
optional bucket sub path
52-
-quiet
53-
enable silent mode
54-
-region string
55-
name of AWS region
56-
-secret string
57-
secret access key for AWS
58-
-source string
59-
path of files to upload (default ".")
60-
-try
61-
trial run, no remote updates
62-
-v enable verbose logging
63-
-workers int
64-
number of workers to upload files (default -1)
46+
### Flags
47+
48+
The list of flags from running `s3deploy -h`:
49+
50+
```
51+
-V print version and exit
52+
-acl string
53+
provide an ACL for uploaded objects. to make objects public, set to 'public-read'. all possible values are listed here: https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html#canned-acl (default "private")
54+
-bucket string
55+
destination bucket name on AWS
56+
-config string
57+
optional config file (default ".s3deploy.yml")
58+
-distribution-id value
59+
optional CDN distribution ID for cache invalidation, repeat flag for multiple distributions
60+
-endpoint-url url
61+
optional AWS endpoint URL override
62+
-force
63+
upload even if the etags match
64+
-h help
65+
-ignore string
66+
regexp pattern for ignoring files
67+
-key string
68+
access key ID for AWS
69+
-max-delete int
70+
maximum number of files to delete per deploy (default 256)
71+
-path string
72+
optional bucket sub path
73+
-public-access
74+
DEPRECATED: please set -acl='public-read'
75+
-quiet
76+
enable silent mode
77+
-region string
78+
name of AWS region
79+
-secret string
80+
secret access key for AWS
81+
-source string
82+
path of files to upload (default ".")
83+
-try
84+
trial run, no remote updates
85+
-v enable verbose logging
86+
-workers int
87+
number of workers to upload files (default -1)
6588
```
6689

67-
### Notes
90+
The flags can be set in one of (in priority order):
6891

69-
- The `key` and `secret` command flags can also be set with environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
70-
- The `region` flag is the AWS API name for the region where your bucket resides. See the table below or the [AWS Regions](https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region) documentation file for an up-to-date version.
92+
1. As a flag, e.g. `s3deploy -path public/`
93+
1. As an OS environment variable prefixed with `S3DEPLOY_`, e.g. `S3DEPLOY_PATH="public/"`.
94+
1. As a key/value in `.s3deploy.yml`, e.g. `path: "public/"`
95+
1. For `key` and `secret` resolution, the OS environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` (and `AWS_SESSION_TOKEN`) will also be checked. This way you don't need to do any special to make it work with [AWS Vault](https://github.com/99designs/aws-vault) and similar tools.
96+
7197

72-
Bucket region | API value | Bucket region | API value
73-
------------- | --------- | ------------- | ----------
74-
Canada (Central) | `ca-central-1` | Asia Pacific (Mumbai) | `ap-south-1`
75-
US East (Ohio) | `us-east-2` | Asia Pacific (Seoul) | `ap-northeast-2`
76-
US East (N. Virginia) | `us-east-1` | Asia Pacific (Singapore) | `ap-southeast-1`
77-
US West (N. California) | `us-west-1` | Asia Pacific (Sydney) | `ap-southeast-2`
78-
US West (Oregon) | `us-west-2` | Asia Pacific (Tokyo) | `ap-northeast-1`
79-
EU (Frankfurt) | `eu-central-1` | China (Beijing) | `cn-north-1`
80-
EU (Ireland) | `eu-west-1` | China (Ningxia) | `cn-northwest-1`
81-
EU (London) | `eu-west-2`
82-
EU (Paris) | `eu-west-3`
83-
South America (São Paulo) | `sa-east-1`
98+
Environment variable expressions in `.s3deploy.yml` on the form `${VAR}` will be expanded before it's parsed:
8499

85-
## Global AWS Configuration
100+
```yaml
101+
path: "${MYVARS_PATH}"
102+
max-delete: "${MYVARS_MAX_DELETE@U}"
103+
```
86104
87-
See https://docs.aws.amazon.com/sdk-for-go/api/aws/session/#hdr-Sessions_from_Shared_Config
105+
Note the special `@U` (_Unquoute_) syntax for the int field.
88106

89-
The `AWS SDK` will fall back to credentials from `~/.aws/credentials`.
107+
### Routes
108+
109+
The `.s3deploy.yml` configuration file can also contain one or more routes. A route matches files given a regexp. Each route can apply:
90110

91-
If you set the `AWS_SDK_LOAD_CONFIG` enviroment variable, it will also load shared config from `~/.aws/config` where you can set the global `region` to use if not provided etc.
111+
`header`
112+
: Header values, the most notable is probably `Cache-Control`. Note that the list of [system-defined metadata](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html#object-metadata) that S3 currently supports and returns as HTTP headers when hosting a static site is very short. If you have more advanced requirements (e.g. security headers), see [this comment](https://github.com/bep/s3deploy/issues/57#issuecomment-991782098).
92113

93-
## Advanced Configuration
114+
`gzip`
115+
: Set to true to gzip the content when stored in S3. This will also set the correct `Content-Encoding` when fetching the object from S3.
94116

95-
Add a `.s3deploy.yml` configuration file in the root of your site. Example configuration:
117+
Example:
96118

97119
```yaml
98120
routes:
99121
- route: "^.+\\.(js|css|svg|ttf)$"
100-
# cache static assets for 20 years
122+
# cache static assets for 1 year.
101123
headers:
102-
Cache-Control: "max-age=630720000, no-transform, public"
124+
Cache-Control: "max-age=31536000, no-transform, public"
103125
gzip: true
104126
- route: "^.+\\.(png|jpg)$"
105127
headers:
106-
Cache-Control: "max-age=630720000, no-transform, public"
128+
Cache-Control: "max-age=31536000, no-transform, public"
107129
gzip: false
108130
- route: "^.+\\.(html|xml|json)$"
109131
gzip: true
110132
```
111133

112134

135+
136+
137+
## Global AWS Configuration
138+
139+
See https://docs.aws.amazon.com/sdk-for-go/api/aws/session/#hdr-Sessions_from_Shared_Config
140+
141+
The `AWS SDK` will fall back to credentials from `~/.aws/credentials`.
142+
143+
If you set the `AWS_SDK_LOAD_CONFIG` environment variable, it will also load shared config from `~/.aws/config` where you can set the global `region` to use if not provided etc.
144+
113145
## Example IAM Policy
114146

115147
```json
@@ -182,11 +214,15 @@ Note that CloudFront allows [1,000 paths per month at no charge](https://aws.ama
182214

183215
## Background Information
184216

185-
If you're looking at `s3deploy` then you've probably already seen the [`aws s3 sync` command](http://docs.aws.amazon.com/cli/latest/reference/s3/sync.html) - this command has a sync-strategy that is not optimised for static sites, it compares the **timestamp** and **size** of your files to decide whether to upload the file.
217+
If you're looking at `s3deploy` then you've probably already seen the [`aws s3 sync` command](https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html) - this command has a sync-strategy that is not optimised for static sites, it compares the **timestamp** and **size** of your files to decide whether to upload the file.
186218

187219
Because static-site generators can recreate **every** file (even if identical) the timestamp is updated and thus `aws s3 sync` will needlessly upload every single file. `s3deploy` on the other hand checks the etag hash to check for actual changes, and uses that instead.
188220

189221
## Alternatives
190222

191223
* [go3up](https://github.com/alexaandru/go3up) by Alexandru Ungur
192224
* [s3up](https://github.com/nathany/s3up) by Nathan Youngman (the starting-point of this project)
225+
226+
## Stargazers over time
227+
228+
[![Stargazers over time](https://starchart.cc/bep/s3deploy.svg)](https://starchart.cc/bep/s3deploy)

‎appveyor.yml‎

Lines changed: 0 additions & 8 deletions
This file was deleted.

‎go.mod‎

Lines changed: 32 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,38 @@
11
module github.com/bep/s3deploy/v2
22

3-
go 1.14
3+
go 1.18
44

55
require (
6-
github.com/aws/aws-sdk-go v1.15.27
6+
github.com/aws/aws-sdk-go-v2 v1.18.1
7+
github.com/aws/aws-sdk-go-v2/credentials v1.13.22
8+
github.com/aws/aws-sdk-go-v2/service/cloudfront v1.26.7
9+
github.com/aws/aws-sdk-go-v2/service/s3 v1.35.0
10+
github.com/bep/helpers v0.4.0
11+
github.com/dsnet/golib/memfile v1.0.0
12+
github.com/frankban/quicktest v1.14.5
13+
github.com/oklog/ulid/v2 v2.1.0
14+
github.com/peterbourgon/ff/v3 v3.3.0
15+
github.com/rogpeppe/go-internal v1.10.1-0.20230506080601-b93e0026e77b
16+
golang.org/x/sync v0.3.0
17+
golang.org/x/text v0.10.0
18+
gopkg.in/yaml.v2 v2.4.0
19+
)
20+
21+
require (
22+
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.10 // indirect
23+
github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.34 // indirect
24+
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.28 // indirect
25+
github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.26 // indirect
26+
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.9.11 // indirect
27+
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.1.29 // indirect
28+
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.28 // indirect
29+
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.14.3 // indirect
30+
github.com/aws/smithy-go v1.13.5 // indirect
731
github.com/davecgh/go-spew v1.1.1 // indirect
8-
github.com/dsnet/golib v0.0.0-20171103203638-1ea166775780
9-
github.com/gopherjs/gopherjs v0.0.0-20181003052733-bf5fc7d38140 // indirect
10-
github.com/jtolds/gls v4.2.1+incompatible // indirect
11-
github.com/pmezard/go-difflib v1.0.0 // indirect
12-
github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d // indirect
13-
github.com/smartystreets/goconvey v0.0.0-20180222194500-ef6db91d284a // indirect
14-
github.com/stretchr/testify v1.2.2
15-
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d // indirect
16-
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f
17-
golang.org/x/text v0.3.0
18-
gopkg.in/yaml.v2 v2.2.1
32+
github.com/google/go-cmp v0.5.9 // indirect
33+
github.com/jmespath/go-jmespath v0.4.0 // indirect
34+
github.com/kr/pretty v0.3.1 // indirect
35+
github.com/kr/text v0.2.0 // indirect
36+
golang.org/x/sys v0.5.0 // indirect
37+
golang.org/x/tools v0.6.0 // indirect
1938
)

0 commit comments

Comments
 (0)