go-getter

go-getter is a Go library maintained by HashiCorp that downloads files or directories from a wide variety of sources using a single URL-like string as input. Instead of writing separate download logic for Git, Mercurial, HTTP, Amazon S3, and Google Cloud Storage, callers pass one string and go-getter’s detectors and getters figure out the right protocol, forcing it with proto:: syntax when the URL is ambiguous.

The library is best known as the download engine inside Terraform (for fetching modules) and Nomad (for fetching job artifacts), which is why its URL format supports extras like subdirectory extraction (//path), checksum verification, and automatic archive unpacking directly as query parameters on the source string.

What You Get

• A Client type with Get, GetAny, and GetFile for directory, auto-detected, and single-file downloads
• Built-in getters for file, git, hg, http/https, S3, and GCS sources
• A pluggable Getter/Detector/Decompressor interface set for adding custom protocols or archive formats
• Checksum verification and automatic archive extraction built into the download path
• A small cmd/go-getter CLI for testing URL strings without writing Go code

Common Use Cases

• Fetching Terraform-style modules from Git, HTTP, or registry sources via one source string
• Downloading versioned release artifacts (binaries, archives) with checksum verification
• Pulling a single subdirectory out of a larger Git repo or archive
• Installing from S3/GCS buckets using the same client API as Git or HTTP sources

Under The Hood

Architecture The Client struct in client.go orchestrates the whole download: its Get() method detects the protocol via Detect(), strips forced-getter syntax (scheme::url), splits off any subdirectory component via SourceDirSubdir, extracts checksum/archive query parameters, and dispatches to a Getter implementation selected from the package-level Getters map by URL scheme. Each protocol (file, git, hg, http, s3, gcs) implements the Getter interface (Get, GetFile, ClientMode, SetClient) in its own get_*.go file, cleanly separating protocol logic from orchestration. Detector implementations (detect_*.go) are a separate strategy layer that rewrites shorthand sources like “github.com/user/repo” into fully qualified URLs before the client sees them. Adding a new protocol means implementing Getter and registering it in the Getters map without touching Client.Get(), though that single method itself is dense, handling subdir, checksum, and archive logic inline.

Tech Stack A standard Go module with no web framework or ORM — this is a pure library plus a thin cmd/go-getter CLI wrapper. Git and Mercurial support shells out to the local git/hg binaries via os/exec; HTTP uses github.com/hashicorp/go-cleanhttp for a hardened default client. Supporting packages include github.com/hashicorp/go-version for semver comparisons, github.com/bgentry/go-netrc for .netrc parsing, github.com/ulikunitz/xz and github.com/klauspost/compress for archive decompression, and github.com/cheggaaa/pb for progress bars. Cloud protocols pull in the official SDKs — aws-sdk-go-v2 for S3 and cloud.google.com/go/storage for GCS — with golang.org/x/oauth2 and google.golang.org/api handling auth. CI runs via GitHub Actions across dedicated build, test, and copyright-header workflows.

Code Quality The repo carries dozens of _test.go files covering nearly every getter/detector/decompressor combination (get_git_test.go, get_s3_test.go, decompress_zip_test.go, and more), backed by real archive fixtures under test-fixtures/ and testdata/. Error handling is idiomatic Go — errors are returned and wrapped with fmt.Errorf/%w rather than swallowed or panicked on. Naming is consistent throughout: the get_<proto>.go / detect_<proto>.go / decompress_<format>.go pattern makes the protocol matrix easy to navigate even without deep prior context.

API Design The standout design choice is overloading a single string to express what would otherwise be a structured config object: proto::url//subdir?option=value encodes protocol forcing, subdirectory selection, and protocol-specific options (checksum, archive format, git ref) all in one input — which is exactly why Terraform can expose a single “source” argument to end users. Top-level shortcuts (Get, GetAny, GetFile) cover the common case with zero boilerplate, while the underlying Client struct exposes full customization of Detectors/Getters/Decompressors for advanced callers. The tradeoff is that the magic-string format isn’t self-documenting from the types alone — options like forced protocols and checksum query parameters have to be learned from the README rather than discovered via autocomplete.