Skip to content

real::erf() inaccuracy #2

@samhocevar

Description

@samhocevar

The current real::erf() implementation uses a Taylor series that does not converge and thus has precision issues.

  • The affected range is [7,19].
  • The approximate error is 10^(-n²/1.13).

That is to say, erf(8) will have a precision error of 10^(-8²/1.13) = 2.3e-57, e.g. the 57th decimal will be inaccurate.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions