Skip to content

Inconsistent fixed point rounding #489

Description

@GoogleCodeExporter
static INLINE OPJ_INT32 opj_int_fix_mul(OPJ_INT32 a, OPJ_INT32 b) {
    OPJ_INT64 temp = (OPJ_INT64) a * (OPJ_INT64) b ;
    temp += 4096;
    assert((temp >> 13) <= (OPJ_INT64)0x7FFFFFFF);
    assert((temp >> 13) >= (-(OPJ_INT64)0x7FFFFFFF - (OPJ_INT64)1));
    return (OPJ_INT32) (temp >> 13);
}

The above code will round 1.5f up to 2.0f, and -1.5f up to -1.0f.

(when I use floats, I mean their fixed point equivalent)



Original issue reported on code.google.com by boxe...@gmail.com on 30 Apr 2015 at 2:38

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions