qzip cannot decompress gzip with QAT HW

[lianghouxu]

use qzip to decompress created with 'qzip -O gzip' GZIP file, only the 1st round to use QAT_HW, then all fall back to QAT_SW.
[root@spr02 mysqlbackup]# qzip -d -k sbtest-0619.sql.gzip.gz
Reading input file sbtest-0619.sql.gzip.gz (536870912 Bytes)
Decompressing...
id1=31, id2=139, cm=8, flag=0
std GZIP
Reading input file sbtest-0619.sql.gzip.gz (536870912 Bytes)
Decompressing...
id1=119, id2=88, cm=5, flag=34
no QZ
Reading input file sbtest-0619.sql.gzip.gz (536870912 Bytes)
Decompressing...
id1=142, id2=115, cm=120, flag=89
no QZ
Reading input file sbtest-0619.sql.gzip.gz (536870912 Bytes)
Decompressing...
id1=112, id2=200, cm=129, flag=235
no QZ

[XingHongChenIntel]

hello, lianghou, thanks for your question. could you please show the more detailed info, especailly for driver version and if you use the intree driver(using yum/rpm to install) or OOT driver(using source code to build). if you use the OOT driver, could you please share the driver config file. and it will be better if you can also share the test file "sbtest-0619.sql.gzip.gz" .

I have used the OOT driver which version is "QAT20_1.0.50", and test data "calgray" to test on "centos 8/kernel 5.18" platform.
and the test result is fine.

[lianghouxu]

Hi, XingHong, I used OOT driver QAT20_1.0.50, too, and qzip v1.1.2, also built from source code.

The issue should be related to QATzip.

The steps to reproduce the issue:

1. Get a 10G file, e.g. 10gfile.sql
2. qzip -O gzip 10gfile.sql
3. qzip -d 10gfile.sql.gz, then use pcm-accel -qat to monitor QAT performance.

I added some debug info in qatzip_gzip.c, we can see that only the first 512M data can be detected as std gzip correctly, other data all detected incorrectly.

[root@spr02 mysqlbackup]# qzip -d -k sbtest-10g.sql.gz
Reading input file sbtest-10g.sql.gz (536870912 Bytes)
Decompressing...
id1=31, id2=139, cm=8, flag=0
std GZIP
Reading input file sbtest-10g.sql.gz (536870912 Bytes)
Decompressing...
id1=176, id2=68, cm=133, flag=91
no QZ
Reading input file sbtest-10g.sql.gz (536870912 Bytes)
Decompressing...
id1=178, id2=16, cm=26, flag=48
no QZ
Reading input file sbtest-10g.sql.gz (536870912 Bytes)
Decompressing...
id1=233, id2=120, cm=67, flag=198

[XingHongChenIntel]

yes, you are right, I have reproduce this issue, and create jira ticket https://jira.devtools.intel.com/browse/QATAPP-30505 to track this issue

[Yogaraj-Alamenda]

This is a historical legacy issue, In fact, the current software behavior conforms to the logic of the current code. but we believe the existing code logic is not inappropriate.

It's all about how we find the complete compressed gzip frame when we decompress the gzip format in qatzip.

In "checkHeader" function, we are using below code to get the compressed size and uncompressed size.
"
case DEFLATE_GZIP:
qzFooter = (StdGzF_T *)(findStdGzipFooter(src_ptr, src_avail_len));

    compressed_sz = (unsigned char *)qzFooter - src_ptr - stdGzipHeaderSz();
    uncompressed_sz = qzFooter->i_size;
    if ((unsigned char *)qzFooter == src_ptr + src_avail_len - stdGzipFooterSz()) {
        isEndWithFooter = 1;
    }
    break;

"
But the src_avail_len maybe not contain the complete the gzip src frame. which means, we couldn't find the gzip Footer in src buffer.
For other code format, we will return the "QZ_DATA_ERROR" to qzip to present the "incomplete source buffer" issue. but for Gzip, qatzip will fallback to sw directly, because zlib could decompress the incomplete buffer by stream mode.

Since QATzip cannot differentiate between the incomplete GZIP block in the end of input buffer from the last complete GZIP block in the buffer, we have to fallback to SW for decompression and differentiate above two conditions with return code of Z_OK from Z_STREM_END.
The issue seems bit complex to fix it in qatzip due to access of footer by different methods. We are keeping on hold for now due to limitations.