Compression

[DirkReiners]

Depending on the type of the large files, it might make sense to compress them before they are sent and stored. compressjs has a nice array of algorithms and is pure JS, so it should be fairly easy to integrate.

The tricky part is deciding which compressor top use for what. It would be best if it was possible to add that info to the git filter, not sure how to do that.

[tiesselune]

Actually, 2 filters are used in git: clean and smudge that can be configured to an arbitrary command.

It's currently configured in git-jam init, in the git-jam.js file, in the jamInit() function:

    return gitUtils.config('filter.jam.clean','git-jam filter-clean')
    .then(function(){
        return gitUtils.config('filter.jam.smudge','git-jam filter-smudge')
    })

This will alias git-jam filter-something as the jam filter used in your .gitattributes.

*.png filter=jam -crlf

So if you define another filter like jamzip like this:

    return gitUtils.config('filter.jamzip.clean','git-jam filter-clean-zip')
    .then(function(){
        return gitUtils.config('filter.jamzip.smudge','git-jam filter-smudge-zip')
    })

Then two commands filter-clean-zip and filter-smudge-zip and a command like git-jam filter-zip "*.png" that adds *.png filter=jamzip -crlfinto your .gittatributes file, it would actually be cleaned and smudged by your new custom function.

So it would be completely possible to implement, possibly using the .jamconfig file to use different algorithm without defining so many new filter functions...

[tiesselune]

So I have thought about this and I got a few ideas :

Push filter vs. gitattributes

Git uses smudge/clean on git add command, so compressing large files directly into the filters (that create the sha-1 version of the files) would slow the add command, which is not great.

Instead, I suggest the feature be implemented on push and pull commands, and not on filters.

The idea is, along with reworking push/pull interface, piping the files to be uploaded/downloaded into another filter that will actually compress/decompress the files before push and after pull.

This avoids cluttering gittatributes and slowing down add/checkout operations.

CompressJS vs. GZip

It seems compressJS is not so fast (which seems logical since it's pure JS for heavy compression, see the benchmarks), so it would not be a good candidate performance-wise.

The problem is not generating heavy dependencies, which was the whole point of git-jam.

Since GZip is commonly installed on most Unix systems (or is easy to install, anyway) and comes with Git Bash on Windows systems, it would not create a heavy dependency for most users, and git-jam would handle the absence of gzip by falling back to uncompressed mode.

Plus, it seems like a good compromise between compression quality and compression time, whichever the file type, so it seems like a good starting point for quick-and-simple feature integration.

Piping

Like I mentionned, reworking push/pull interface so that files are piped into custom filters before being push/pulled would allow for compression using gzip, but such a system could allow anyone to write their own pipes, using other compression systems for instance, and extend the system, just like adding new backends.

I'm thinking about implementing that soon enough.