Support GCS bulk delete with serialized single DELETE requests

[harshavardhana]

GCS's S3 API compatibility layer does not support the multi-object delete API. This change automatically detects GCS endpoints and falls back to single DELETE requests for bulk delete operations.

Fixes #2181

[harshavardhana]

@dbishop PTAL if this helps

[dbishop]

@harshavardhana lgtm! I'm not sure the best way to get a warp binary with this incorporated. Is there an easy way for me to clone this repo, switch to this branch, and then have my warp repo's git tree reference my local minio-go tree instead of the real one upstream? Or is there a more idiomatic way to test an upstream golang library's PR branch in a downstream utility?

[harshavardhana]

@harshavardhana lgtm! I'm not sure the best way to get a warp binary with this incorporated. Is there an easy way for me to clone this repo, switch to this branch, and then have my warp repo's git tree reference my local minio-go tree instead of the real one upstream? Or is there a more idiomatic way to test an upstream golang library's PR branch in a downstream utility?

@dbishop you need to use go mod edit -replace directive however I just wanted to show what is going to be implemented when it detects a GCS endpoint so hoping this is sufficient.

[dbishop]

@harshavardhana ,
Thanks for the pointer. I got a warp binary built from our fork's "main', with our GCS-delete hack reverted, successfully pointed to a local minio-go tree on your branch here. I ran it in GCP against GCS and the deletes worked fine!

(note: there's also a patch active here, that I haven't submitted yet, that ensures that multiple clients don't try to delete the same objects when there's more than one client and --list-existing is supplied)

# warp delete --tls --bucket=[redacted] --region=us-east4 --host=storage.googleapis.com:443 --concurrent=8 --list-existing --objects=0 --noclear
# ...

Report: DELETE (29 reqs). Ran Duration: 9s, starting 17:21:41 PST
 * Objects per request: 96. Concurrency: 16. Warp Instances: 2.
 * Average: 220.26 obj/s (± 40.79) (9s)
 * Reqs: Avg: 3557.8ms, 50%: 3707.3ms, 90%: 4670.9ms, 99%: 4670.9ms, Fastest: 591.1ms, Slowest: 4519.0ms, StdDev: 677.3ms

Throughput by client:
Client cpu-large-002-jKDh throughput: 202.02 obj/s (± 22.52) (5s)
 * Reqs: Avg: 3408.2ms, 50%: 3707.3ms, 90%: 4161.3ms, 99%: 4161.3ms, Fastest: 591.1ms, Slowest: 4140.5ms, StdDev: 827.5ms

Client cpu-large-003-rY0C throughput: 108.03 obj/s (± 18.31) (9s)
 * Reqs: Avg: 3718.0ms, 50%: 3707.3ms, 90%: 4670.9ms, 99%: 4670.9ms, Fastest: 2946.6ms, Slowest: 4519.0ms, StdDev: 444.1ms


Throughput, split into 5 x 1s:
 * Fastest: 428.20 obj/s (1s, starting 17:21:41 PST)
 * 50% Median: 424.77 obj/s (1s, starting 17:21:40 PST)
 * Slowest: 343.63 obj/s (1s, starting 17:21:43 PST)