Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,21 +58,21 @@ command menu.

## Output

Response bodies go to stdout; status, headers, timing, warnings, and errors go
to stderr. This keeps pipelines clean:
Response bodies go to stdout. Status, headers, timing, warnings, and errors go
to stderr. This separation keeps pipelines clean:

```sh
fetch example.com/api | jq .
```

Terminal output is formatted automatically. Redirected output is unformatted by
default, and binary responses are protected from accidental terminal output.
By default, `fetch` formats terminal output and does not format redirected
output. It blocks accidental binary output to a terminal.
See [Output Formatting](docs/output-formatting.md) for pager, color, binary,
clipboard, and file behavior.

Use `--har request.har` to record the final HTTP exchange as a HAR 1.2 sidecar
without changing normal response output. HAR files can contain credentials,
cookies, and bodies and should be treated as sensitive data.
cookies, and bodies. Treat HAR files as sensitive data.

## Documentation

Expand Down
87 changes: 55 additions & 32 deletions docs/advanced-features.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@ UDP DNS queries advertise EDNS(0) and retry truncated responses over TCP.

### DNS over TCP

Use the `tcp://` scheme for plain DNS over TCP. This avoids UDP truncation
entirely because responses are streamed with a 2-byte length prefix.
Use the `tcp://` scheme for plain DNS over TCP. TCP uses a 2-byte length prefix
and prevents UDP truncation.

```sh
fetch --dns-server tcp://1.1.1.1 example.com
Expand All @@ -39,8 +39,8 @@ fetch --dns-server tcp://1.1.1.1:53 example.com
### DNS over TLS (DoT)

Use the `tls://` or `dot://` scheme for DNS over TLS. The default port is 853.
Both IP addresses and hostnames are accepted; hostnames are resolved with the
system resolver and used for TLS server name verification.
You can specify an IP address or a hostname. The system resolver resolves
hostnames. `fetch` uses the hostname to verify the TLS server name.

```sh
fetch --dns-server tls://1.1.1.1 example.com
Expand All @@ -50,7 +50,8 @@ fetch --dns-server tls://dns.google:853 example.com

### DNS over QUIC (DoQ)

Use the `quic://` or `doq://` scheme for DNS over QUIC. The default port is 853. Both IP addresses and hostnames are accepted.
Use the `quic://` or `doq://` scheme for DNS over QUIC. The default port is 853.
You can specify an IP address or a hostname.

```sh
fetch --dns-server quic://1.1.1.1 example.com
Expand Down Expand Up @@ -83,9 +84,17 @@ fetch --inspect-dns example.com
fetch --inspect-dns --dns-server https://1.1.1.1/dns-query example.com
```

Request-only CLI flags warn that no HTTP request will be sent and those flags have no effect when used with `--inspect-dns`; config-file defaults do not trigger this warning.
Request-only CLI flags have no effect with `--inspect-dns`. They cause a warning
that `fetch` does not send an HTTP request. Configuration-file defaults do not
cause this warning.

The output shows the resolver backend, A, AAAA, CNAME, TXT, MX, NS, SOA, SRV, CAA, SVCB, and HTTPS records when present, address count, record count, lookup duration, and per-record TTLs. UDP DNS inspection advertises EDNS(0) and retries truncated UDP responses over TCP; if TCP fallback cannot complete the lookup, `fetch` warns that the results are incomplete and exits with a non-zero status.
The output identifies the resolver and shows available A, AAAA, CNAME, TXT, MX,
NS, SOA, SRV, CAA, SVCB, and HTTPS records. It also shows each record TTL, the
address and record counts, and the lookup duration.

UDP inspection advertises EDNS(0). It retries a truncated UDP response with TCP.
If the TCP retry fails, `fetch` warns that the results are incomplete and exits
with a nonzero status.

### Configuration File

Expand Down Expand Up @@ -116,8 +125,8 @@ fetch --proxy http://proxy.example.com:8080 example.com
fetch --proxy https://secure-proxy.example.com:8443 example.com
```

HTTPS proxy TLS is configured separately from origin TLS. The proxy handshake
uses platform verification, and origin `--ca-cert`, `--cert`/`--key`, and
`fetch` configures HTTPS proxy TLS separately from origin TLS. The proxy
handshake uses platform verification. Origin `--ca-cert`, `--cert`/`--key`, and
`--insecure` settings do not apply to the proxy.

### SOCKS5 Proxy
Expand Down Expand Up @@ -145,10 +154,9 @@ export NO_PROXY="localhost,127.0.0.1,192.168.0.0/16,.internal.com"
fetch example.com # Uses proxy from environment
```

Proxy variables also support lowercase forms: `http_proxy`, `https_proxy`,
`all_proxy`, and `no_proxy`. Uppercase names are checked before lowercase names
for each variable, except uppercase `HTTP_PROXY` is ignored when
`REQUEST_METHOD` is set.
Proxy variables also have lowercase forms: `http_proxy`, `https_proxy`,
`all_proxy`, and `no_proxy`. `fetch` checks each uppercase name before its
lowercase name. It ignores uppercase `HTTP_PROXY` if `REQUEST_METHOD` is set.

Proxy precedence is: an explicit `--proxy` or configured `proxy = ...` value,
then scheme-specific environment variables (`HTTP_PROXY` for HTTP requests and
Expand All @@ -171,7 +179,8 @@ proxy = socks5://internal-proxy:1080

### `--unix`

Connect via Unix domain socket instead of TCP. Available on Unix-like systems only.
Connect through a Unix domain socket instead of TCP. This option is available
only on Unix-like systems.

### Docker API

Expand All @@ -187,7 +196,8 @@ fetch --unix /var/run/myservice.sock http://localhost/api/status
fetch --unix ~/myapp.sock http://localhost/health
```

**Note**: The hostname in the URL is ignored when using Unix sockets; the socket path determines the destination.
With a Unix socket, the socket path sets the destination. `fetch` ignores the
hostname in the URL.

## HTTP Versions

Expand Down Expand Up @@ -216,7 +226,7 @@ the same automatic direct HTTPS path. Prompt fresh HTTPS/SVCB results are tried
before cached entries, while cached entries can race slower HTTPS-record
discovery so a learned `Alt-Svc` alternative can be used on later requests.

Setting `--http 1`, `--http 2`, or `--http 3` forces that protocol; it does not
Setting `--http 1`, `--http 2`, or `--http 3` forces that protocol. It does not
set a version cap. Use `--http 1` or `--http 2` to opt out of automatic HTTP/3.

### HTTP/1.1
Expand Down Expand Up @@ -267,7 +277,7 @@ By default, `fetch` negotiates the best available version:
usable
4. Races QUIC setup against TCP/TLS when a usable `h3` candidate is discovered
before TCP/TLS wins
5. Otherwise offers HTTP/2 via ALPN
5. Otherwise, offers HTTP/2 with ALPN
6. Falls back to HTTP/1.1 if needed

## TLS Configuration
Expand Down Expand Up @@ -335,7 +345,9 @@ Output includes:

Expiry is color-coded: red if expired or less than 7 days remaining, yellow if less than 30 days, green otherwise.

Request-only CLI flags (e.g. `--data`, `--timing`, `--grpc`) warn that no HTTP request will be sent and those flags have no effect when used with `--inspect-tls`; config-file defaults do not trigger this warning.
Request-only CLI flags, such as `--data`, `--timing`, and `--grpc`, have no
effect with `--inspect-tls`. They cause a warning that `fetch` does not send an
HTTP request. Configuration-file defaults do not cause this warning.

`--dns-server` applies to TLS inspection too, so certificate diagnostics can use
the same UDP or DNS-over-HTTPS resolver override as normal requests. When
Expand Down Expand Up @@ -401,9 +413,10 @@ Compression modes:
Output files receive decoded/decompressed bodies by default too. Use
`--compress off` for byte-for-byte downloads of `.gz`, `.br`, or `.zst` assets.

For SSE (`text/event-stream`) responses in `auto` mode, `fetch` retries without
`Accept-Encoding` when the server replies with compressed content. This avoids
common buffering behavior that prevents events from appearing as they arrive.
For SSE (`text/event-stream`) responses in `auto` mode, `fetch` retries
compressed responses to `GET` and `HEAD` requests without `Accept-Encoding`.
For other methods, it keeps the compressed response and gives a warning. For
immediate SSE streaming with another method, use `--compress off`.

Using `off` is useful when:

Expand Down Expand Up @@ -495,14 +508,17 @@ streams.

### `--connect-timeout SECONDS`

Set a timeout for just the connection phase (DNS resolution, TCP connect, TLS handshake):
Set a timeout for the connection phase. This phase includes DNS resolution, the
TCP connection, and the TLS handshake:

```sh
fetch --connect-timeout 5 example.com
fetch --connect-timeout 5 --timeout 30 example.com # Both timeouts
```

This is useful for fast-failing on unreachable hosts while allowing large responses to transfer slowly. The connect timeout is independent of `--timeout` — both can be set simultaneously, and `--timeout` still caps the entire request.
Use this option to stop connection attempts to unreachable hosts quickly. A
large response can continue to transfer after the connect timeout. You can set
both timeout options. The `--timeout` value still limits the complete request.

### Configuration File

Expand Down Expand Up @@ -585,7 +601,7 @@ fetch --session staging https://staging.example.com/login

### Configuration File

Set session names per-host so you don't need `--session` every time:
Set session names for each host to omit `--session` from subsequent commands:

```ini
# Global default session
Expand All @@ -611,7 +627,8 @@ Sessions are stored as JSON in the user's cache directory:
- **Expired cookies**: Cookies with an explicit expiry in the past are filtered out on load.
- **Session cookies** (no explicit expiry): Persist across invocations since the session is explicitly named.
- **Cookie domain matching**: Delegated to the Rust cookie store, which implements RFC 6265 behavior.
- **Atomic writes**: Session files are written atomically (temp file + rename) to avoid corruption.
- **Atomic writes**: `fetch` writes a temporary file and then renames it. This
operation prevents file corruption.
- **Name validation**: Only `[a-zA-Z0-9_-]` characters are allowed to prevent path traversal.

## Debugging Network Issues
Expand All @@ -626,25 +643,31 @@ fetch --har request.har https://api.example.com/users
fetch -o response.json --har request.har https://api.example.com/users
```

The destination is reserved before network I/O and atomically installed after
the response completes. It honors `--clobber`. Redirect, retry, and
authentication challenge exchanges are not included; only the final exchange
is recorded. Captures larger than 16 MiB are counted but omitted from the HAR.
`fetch` reserves the destination before network I/O. It installs the file
atomically after the response. The operation obeys `--clobber`. The HAR records
only the final exchange. It does not include redirect, retry, or authentication
challenge exchanges. For captures larger than 16 MiB, it records the size but
omits the content.

HAR files may contain authorization headers, cookies, request bodies, and
response bodies. Store and share them as sensitive data.

### Timing Waterfall

`--timing` (or `-T`) displays a timing waterfall chart after the response, showing how time was spent across DNS resolution, TCP connection setup, TLS handshake, time to first byte, and body download:
`--timing` (or `-T`) displays a timing waterfall chart after the response. The
chart shows DNS resolution, TCP connection, TLS handshake, time to first byte,
and body-download phases:

```sh
fetch --timing https://example.com
```

The chart adapts to the request: TLS is omitted for plaintext HTTP, HTTP/3 reports connection setup as QUIC, and connection phases are omitted when an existing pooled connection is reused. Combine with `-vvv` for both inline debug text and the waterfall summary.
The chart omits TLS for plaintext HTTP. It shows the HTTP/3 connection phase as
QUIC. If `fetch` reuses a pooled connection, the chart omits the connection
phases. Combine this option with `-vvv` to show debug text and the waterfall
summary.

Can also be configured in the [configuration file](configuration.md):
You can also enable the chart in the [configuration file](configuration.md):

```ini
timing = true
Expand Down
12 changes: 6 additions & 6 deletions docs/agent-skill.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ fetch --install-skill pi --scope project
| `gemini` | `~/.gemini/skills/fetch` | `.gemini/skills/fetch` |
| `pi` | `~/.pi/agent/skills/fetch` | `.pi/skills/fetch` |

`all` means the five locations in the table; it does not probe for or write to
`all` means the five locations in the table. It does not probe for or write to
other agent directories.

## Preview Changes
Expand Down Expand Up @@ -91,11 +91,11 @@ when you intend to remove a locally changed copy.

The skill workflow:

- operates only on the selected user or project destinations;
- does not download files;
- does not edit agent configuration files;
- detects modified installations before replacement or removal; and
- uses locked, atomic filesystem operations.
- Operates only on the selected user or project destinations.
- Does not download files.
- Does not edit agent configuration files.
- Detects modified installations before replacement or removal.
- Uses locked, atomic file operations.

## See Also

Expand Down
26 changes: 16 additions & 10 deletions docs/authentication.md
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,7 @@ ca-cert = /path/to/api-ca.crt
- Certificates and keys must be in PEM format
- TLS requests reject a private key without a client certificate, including values from config files or `--from-curl`
- Encrypted private keys are not supported
- Combined PEM files should have the certificate before the key
- In a combined PEM file, put the certificate before the key.

### Example: Self-Signed Certificates

Expand Down Expand Up @@ -232,10 +232,13 @@ If you need multiple authentication headers, use `-H` for additional headers.

## Security Considerations

1. **Avoid embedding secrets in scripts** - Use environment variables or secure vaults
2. **Protect configuration files** - Set appropriate file permissions (`chmod 600`)
3. **Use HTTPS** - Never send credentials over unencrypted HTTP
4. **Rotate credentials regularly** - Especially API keys and tokens
1. **Do not put secrets in scripts.** Use environment variables or secure
vaults.
2. **Protect configuration files.** Set applicable file permissions, such as
`chmod 600`.
3. **Use HTTPS.** Do not send credentials over unencrypted HTTP.
4. **Replace credentials regularly.** This is especially important for API keys
and tokens.

### Secure Credential Handling

Expand All @@ -257,7 +260,7 @@ fetch --bearer "$(cat /run/secrets/api-token)" example.com

- Verify credentials are correct
- Check if the authentication method matches what the server expects
- Ensure tokens haven't expired
- Make sure that the tokens are not expired.

### 403 Forbidden

Expand All @@ -266,15 +269,18 @@ fetch --bearer "$(cat /run/secrets/api-token)" example.com

### Certificate Errors with mTLS

- Verify certificate and key match: `openssl x509 -noout -modulus -in cert.crt | openssl md5` should match `openssl rsa -noout -modulus -in key.key | openssl md5`
- Check certificate expiration: `openssl x509 -noout -dates -in cert.crt`
- Ensure the CA certificate is correct for the server
- Make sure that the certificate and key match. The output of
`openssl x509 -noout -modulus -in cert.crt | openssl md5` must match the
output of `openssl rsa -noout -modulus -in key.key | openssl md5`.
- Check the certificate expiration:
`openssl x509 -noout -dates -in cert.crt`.
- Make sure that the CA certificate is correct for the server.

### AWS SigV4 Errors

- Verify `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are set
- Check the region and service name are correct
- Ensure your credentials have the required IAM permissions
- Make sure that your credentials have the necessary IAM permissions.
- Verify system clock is accurate (signatures are time-sensitive)

## See Also
Expand Down
Loading
Loading