No download limit - unused option MaxReadBeforeTruncate

[vbanos]

gowarc has an option MaxReadBeforeTruncate with a default limit of 1GB. The problem is that this option isn’t used anywhere in gowarc and Zeno. Also, there was never such an option in net/http.

https://github.com/internetarchive/gowarc/blob/master/client.go#L139

This is a critical issue. A large target URL will OOM Zeno because when you use net/http to download a URL and you invoke io.ReadAll() it reads all in memory.

[vbanos]

One way to handle this is to use net/http + io.LimitReader like in the following example:

package main

import (
    "fmt"
    "io"
    "net/http"
)

func main() {
    resp, err := http.Get("https://example.com/largefile")
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()

    const maxBytes = 1 << 20 // 1 MB
    limited := io.LimitReader(resp.Body, maxBytes)

    data, err := io.ReadAll(limited)
    if err != nil {
        panic(err)
    }

    fmt.Printf("Read %d bytes\n", len(data))
}

The problem is that our request is prepared in https://github.com/internetarchive/gowarc/blob/master/client.go but it is executed in Zeno https://github.com/internetarchive/Zeno/blob/main/internal/pkg/archiver/archiver.go.

One possible solution would be:

1. Remove MaxReadBeforeTruncate from gowarc
2. Add the option to Zeno.
3. Use io.LimitReader in Zeno.

We also need to add the Truncate header in the WARC files we create when this occurs.
https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/#warc-truncated

[yzqzss]

gowarc has an option MaxReadBeforeTruncate with a default limit of 1GB. The problem is that this option isn’t used anywhere in gowarc and Zeno. Also, there was never such an option in net/http.

MaxReadBeforeTruncate is an unimplemented feature that I guess was originally intended to write responses that are too large for the remaining free size in a single WARC file to the next WARC file.

---

This is a critical issue. A large target URL will OOM Zeno because when you use net/http to download a URL and you invoke io.ReadAll() it reads all in memory.

We rarely use io.ReadAll() in Zeno.

One is for extracting outlinks from text/* aggressively by using extractor.LinkRegex or extractor.LinkRegexStrict.
The remaining two are sitespecific related and can be ignored.

Additionally, we have implemented a MIME check in Zeno's archiver/body.go to only read resp.Body such as text/* and PDF that require post-processing into spooledBuff (in-memory buffer + on-disk buffer) and replace the original resp.Body in place.

https://github.com/internetarchive/Zeno/blob/6456525a298d8f891e93ccec267d768e25901550/internal/pkg/archiver/body.go#L47-L53

For other binary responses we are not concerned with, they are simply read and discarded in Zeno. (Note: Although we discard the resp.Body in Zeno, warclib correctly writes the response data stream from the conn to the WARC file as-is. ref: warc.customDialer.wrapConnection)

https://github.com/internetarchive/Zeno/blob/6456525a298d8f891e93ccec267d768e25901550/internal/pkg/archiver/body.go#L77-L80

---

Therefore, I believe that the potential OOM risk caused by io.ReadAll() is currently small.

[yzqzss]

So, three things:

1. warclib: The ability to truncate the response and write it across multiple WARC files.
2. Zeno: Improve extractLinksFromPage() to process in chunks.
3. Zeno/warclib: Implement a "download limit". This is actually in my GSoC plan 3. :) [GSoC] Progress Tracking Zeno#322