Improve error message: RecursionError

[leezu]

pyflakes may fail with a RecursionError when run on large directories (e.g. a conda install) by accident. Below excerpt shows the current error message.

It would be more helpful to show which file / module caused the RecursionError.

Thanks!

  File "/var/lib/jenkins/workspace/gluon-nlp-lint/conda/lint/lib/python3.6/site-packages/pyflakes/checker.py", line 863, in handleNode

    handler(node)

  File "/var/lib/jenkins/workspace/gluon-nlp-lint/conda/lint/lib/python3.6/site-packages/pyflakes/checker.py", line 816, in handleChildren

    self.handleNode(node, tree)

  File "/var/lib/jenkins/workspace/gluon-nlp-lint/conda/lint/lib/python3.6/site-packages/pyflakes/checker.py", line 863, in handleNode

    handler(node)

  File "/var/lib/jenkins/workspace/gluon-nlp-lint/conda/lint/lib/python3.6/site-packages/pyflakes/checker.py", line 816, in handleChildren

    self.handleNode(node, tree)

  File "/var/lib/jenkins/workspace/gluon-nlp-lint/conda/lint/lib/python3.6/site-packages/pyflakes/checker.py", line 863, in handleNode

    handler(node)

  File "/var/lib/jenkins/workspace/gluon-nlp-lint/conda/lint/lib/python3.6/site-packages/pyflakes/checker.py", line 816, in handleChildren

    self.handleNode(node, tree)

  File "/var/lib/jenkins/workspace/gluon-nlp-lint/conda/lint/lib/python3.6/site-packages/pyflakes/checker.py", line 863, in handleNode

    handler(node)

  File "/var/lib/jenkins/workspace/gluon-nlp-lint/conda/lint/lib/python3.6/site-packages/pyflakes/checker.py", line 816, in handleChildren

    self.handleNode(node, tree)

  File "/var/lib/jenkins/workspace/gluon-nlp-lint/conda/lint/lib/python3.6/site-packages/pyflakes/checker.py", line 863, in handleNode

    handler(node)

  File "/var/lib/jenkins/workspace/gluon-nlp-lint/conda/lint/lib/python3.6/site-packages/pyflakes/checker.py", line 816, in handleChildren

    self.handleNode(node, tree)

  File "/var/lib/jenkins/workspace/gluon-nlp-lint/conda/lint/lib/python3.6/site-packages/pyflakes/checker.py", line 863, in handleNode

    handler(node)

  File "/var/lib/jenkins/workspace/gluon-nlp-lint/conda/lint/lib/python3.6/site-packages/pyflakes/checker.py", line 816, in handleChildren

    self.handleNode(node, tree)

  File "/var/lib/jenkins/workspace/gluon-nlp-lint/conda/lint/lib/python3.6/site-packages/pyflakes/checker.py", line 855, in handleNode

    if self.futuresAllowed and not (isinstance(node, ast.ImportFrom) or

  File "/var/lib/jenkins/workspace/gluon-nlp-lint/conda/lint/lib/python3.6/site-packages/pyflakes/checker.py", line 541, in futuresAllowed

    for scope in self.scopeStack):

RecursionError: maximum recursion depth exceeded

"""


The above exception was the direct cause of the following exception:


Traceback (most recent call last):

  File "/var/lib/jenkins/workspace/gluon-nlp-lint/conda/lint/bin/flake8", line 11, in <module>

    load_entry_point('flake8==3.5.0', 'console_scripts', 'flake8')()

  File "/var/lib/jenkins/workspace/gluon-nlp-lint/conda/lint/lib/python3.6/site-packages/flake8/main/cli.py", line 16, in main

    app.run(argv)

  File "/var/lib/jenkins/workspace/gluon-nlp-lint/conda/lint/lib/python3.6/site-packages/flake8/main/application.py", line 396, in run

    self._run(argv)

  File "/var/lib/jenkins/workspace/gluon-nlp-lint/conda/lint/lib/python3.6/site-packages/flake8/main/application.py", line 384, in _run

    self.run_checks()

  File "/var/lib/jenkins/workspace/gluon-nlp-lint/conda/lint/lib/python3.6/site-packages/flake8/main/application.py", line 310, in run_checks

    self.file_checker_manager.run()

  File "/var/lib/jenkins/workspace/gluon-nlp-lint/conda/lint/lib/python3.6/site-packages/flake8/checker.py", line 319, in run

    self.run_parallel()

  File "/var/lib/jenkins/workspace/gluon-nlp-lint/conda/lint/lib/python3.6/site-packages/flake8/checker.py", line 288, in run_parallel

    for ret in pool_map:

  File "/var/lib/jenkins/workspace/gluon-nlp-lint/conda/lint/lib/python3.6/multiprocessing/pool.py", line 347, in <genexpr>

    return (item for chunk in result for item in chunk)

  File "/var/lib/jenkins/workspace/gluon-nlp-lint/conda/lint/lib/python3.6/multiprocessing/pool.py", line 735, in next

    raise value

RecursionError: maximum recursion depth exceeded

[jayvdb]

I suspect that this is a duplicate of the quite old #231

PRs fixing this were provided two years ago #68 and #70 . None of the other committers have reviewed them yet.

Could you test either of those to see if they fix your problem?

If not, could you find which file in https://github.com/dmlc/gluon-nlp is causing the problem?

[leezu]

Thanks @jayvdb . Actually no file in gluon-nlp is causing this issue. Instead, our CI server was set up to install a conda environment in a subdirectory of the checked-out sources (i.e. in /path/gluon-nlp/conda) and naively enabling pyflakes on the CI caused it to try validating all files in the conda env in addition to the source of gluon-nlp.
As above RecursionError does not provide information about which file is causing the failure, it was not immediately obvious what was causing the issue.

[asottile]

agreed -- this is likely a duplicate of #231 -- the recursion error isn't actually about directory depth but about expression complexity