Skip to content

Python SHR calculation can overflow #306

@Allan-xmos

Description

@Allan-xmos

WHen calculating the maximum output of the filter, the SHR calcualtion can overflow:

# (the 0x7FFFFF00 is the largest value a 1-block Stage1 filter can output)
max_samp_out = np.dot( np.int64( 0x7FFFFF00) * np.sign(self.coefs),
np.round(self.coefs ).astype(np.int64) )
# 32-bit VPU multiplications have a built-in 30-bit right-shift
max_samp_out >>= 30
# We can do floor(log2()-30) or ceil(log2()-31) and the result is basically
# the same.
frac_shr = np.log2(np.abs(max_samp_out)) - 30
self.shr = np.int32(np.floor(frac_shr))

This is probably to do with the dot product occuring before the >>30 and overflowing int64 range. Reproduce this and update the calulations so they don't overflow.

Note this is quite pessamistic in it's assumption of the worst case PDM input, we can probably get away with a smaller shift

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions