VSCode debugging freezes or goes into loop for large Pandas DataFrames

[ashrauma]

Environment data

• VS Code version: 1.57.1
• Extension version (available under the Extensions sidebar): v2020.11.371526539
• OS and version: Windows_NT x64 10.0.18363
• Python version (& distribution if applicable, e.g. Anaconda): 3.7.5
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): virtualenv
• Relevant/affected Python packages and their versions: pandas==1.3.0
• Relevant/affected Python-related VS Code extensions and their versions: Pylance==v2020.12.0
• Value of the python.languageServer setting: Pylance

[NOTE: If you suspect that your issue is related to the Microsoft Python Language Server (python.languageServer: 'Microsoft'), please download our new language server Pylance from the VS Code marketplace to see if that fixes your issue]

Expected behaviour

Debug normally

Actual behaviour

Loops into debug at a line

Steps to reproduce:

[NOTE: Self-contained, minimal reproducing code samples are extremely helpful and will expedite addressing your issue]

1. Please see VSCode debugging freezes or goes into loop for large Pandas DataFrames vscode#130615

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

XXX

[int19h]

In short, the problem is that the debugger is trying to visualize the dataframe in the Locals window by calling repr() on it; but, because max_rows and max_columns were set to not have any limit, it's taking too long. It doesn't actually hang - if you wait long enough, you'll see that the spinner in Locals eventually goes away, and you can see the dataframe there. But, due to the way VSCode itself works, the Locals window is refreshed every time you make a step, and so it quickly becomes unusable.

We already have some code to special-case the standard Python collection types (which have the same problem). I suppose we could do the same for dataframe. However, this would still not solve the problem in general, because there may be other types for which repr() takes too long to compute - and we don't know that in advance. For some other languages/frameworks, this sort of thing is handled by computing the representation in a background thread that can be abandoned if needed - but this doesn't work for Python due to GIL.

[int19h]

@fabioz It looks like pydevd already has some special handling for dataframes - is it supposed to handle this case, and is just not kicking in for some reason?

[fabioz]

@fabioz It looks like pydevd already has some special handling for dataframes - is it supposed to handle this case, and is just not kicking in for some reason?

@int19h, as far as I can see, there's no special handling for dataframes (there's only special support for django.forms.Form and numpy.ndarray), so, this is something new (what's available for pandas is being able to collect a pandas dataframe as a xml, but it's not applicable to debugpy and the DAP protocol).

but this doesn't work for Python due to GIL.

I haven't really checked what's the case for how pandas does it, but if it's a C-extension, this isn't always an issue as it could potentially release GIL when doing heavy computation (given how slow it seems to be, it may be possible that it's even done in Python).

So, computing in a thread is something we could investigate in the future... as a note, there is already some support for that when not using the DAP -- it does have some issues, for instance, when dealing with UI frameworks accessing UI objects can only be done in the main thread and doing otherwise can crash the application, so, it'd probably need to be opt-in instead of opt-out, but it's something that's not implemented for the DAP at all right now (also, it'd be non-blocking, but then there could be lots of things being computed in the background, which may not be ideal).

Anyways, for this specific issue the best approach is probably doing a special handling (as is done for numpy and django forms).

[fabioz]

As a note, I'm working on this.

Pandas repr is really slow, even on the regular case (without that customization). Also, the debugger will right now iterate and provide all the methods, but then, each method ends up calling repr again so that it will end up appearing as:

<bound method NDFrame.__contains__ of              0            1            2            3            4   \
0   Test String  Test String  Test String  Test String  Test String   
1   Test String  Test String  Test String  Test String  Test String   
2   Test String  Test String  Test String  Test String  Test String   
3   Test String  Test String  Test String  Test String  Test String   
4   Test String  Test String  Test String  Test String  Test String   
5   Test String  Test String  Test String  Test String  Test String   
6   Test String  Test String  Test String  Test String  Test String   
7   Test String  Test String  Test String  Test String  Test String   
8   Test String  Test String  Test String  Test String  Test String   
9   Test String  Test String  Test String  Test String  Test String   
10  Test String  Test String  Test String  Test String  Test String   
11  Test String  Test String  Test String  Test String  Test String   
12  Test String  Test String  T...

So, the experience of debugging with Pandas is awful even on the good case because its repr is so slow.

I'm working on customizing it so that such info is not collected and a custom repr is used in the debugger.

[zmbc]

In the debug console, if one accidentally clicks on a large Pandas Series (which is easy to do), it starts to try to compute some repr (of the underlying numpy arrays?) that is O(n), and this causes the debugger to freeze. It's annoying how easy it is to do this by accident!