Skip to content

Multithreading & ZSTD_c_nbWorkers parameter #10

Description

@nono303

Playing with Multithreading support and tuning ZSTD_c_nbWorkers (cf. https://facebook.github.io/zstd/zstd_manual.html#chapter4)

Select how many threads will be spawned to compress in parallel.
When nbWorkers >= 1, triggers asynchronous mode when invoking ZSTD_compressStream*() :
ZSTD_compressStream*() consumes input and flush output if possible, but immediately gives back control to caller,
while compression is performed in parallel, within worker thread(s).
(note : a strong exception to this rule is when first invocation of ZSTD_compressStream2() sets ZSTD_e_end :
 in which case, ZSTD_compressStream2() delegates to ZSTD_compress2(), which is always a blocking call).
More workers improve speed, but also increase memory usage.
Default value is `0`, aka "single-threaded mode" : no worker is spawned,
compression is performed inside Caller's thread, and all invocations are blocking */

I've benched and compared compression efficiencies between mod_zstd and zstd.exe cmd line.
Here are the results on a 16 core machine (with a 156MiB XML file)

  • zstd -15 -f 156.kml -o 156.zst}
ZSTD_NBTHREADS=0	20.70'
ZSTD_NBTHREADS=1	19.38'
ZSTD_NBTHREADS=2	11.61'
ZSTD_NBTHREADS=16	4.1'
  • Mod_zstd with ZSTD_c_compressionLevel=15
ZSTD_c_nbWorkers=0	21.002994'
ZSTD_c_nbWorkers=1	6.658820'
ZSTD_c_nbWorkers=2	6.414540'
ZSTD_c_nbWorkers=16	6.748982

✅ For zstd.exe, time seems coherent and relative ty to the number of thread.
❓ For mod_zstd implementation, it sounds like multithreading parameter give on/off results :

  • off with 0
  • on with any other values.

Does I missed something?

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedExtra attention is neededquestionFurther information is requested

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions