custom class MultiMemberGzipReader and other tweaks - #16

Open

nlevitt wants to merge 14 commits into

internetarchive:masterfrom

nlevitt:cdx-writer-big-update

Commits on Mar 22, 2016

invalidate content_file of previous record when reading next one
nlevitt
committed
27fb081
port over raj hacks from old cdx-writer branch to accept arcs missing filedesc header, and split line only on space character
nlevitt
committed
98341bb

Commits on Mar 28, 2016

new class MultiMemberGzipReader for better member-at-a-time gzip parsing (fixes fundamental problem with old approach, that a read on a short gzip member would read into the following member)
nlevitt
committed
fb47352
oops travis has no such version of python "dev"
nlevitt
committed
7803c27

Commits on Mar 29, 2016

Commits on Mar 30, 2016

never snarf content as a side effect of looking at record.content_type
nlevitt
committed
13fc74d
improve error message
nlevitt
committed
9c346e0
remove unusued SPLIT thing
nlevitt
committed
2d3cbc5

Commits on Sep 2, 2016

Commits on Jan 26, 2017

fix bug spinning forever in case gzip input stream ends in the middle of the data section, and add tests for incomplete gzip input
nlevitt
committed
cf89841