Python-JS two-way communication

[thorwhalen]

We want to be able to take a "JS component" and use it from python, in a jupyter notebook.

By "JS component" I mean "some JS+HTML+CSS that defines an element, view, or full 'app' whose functionality can be reused in various places".

The idea is to be able to facilitate reuse between front-end dev and back-end dev
(in particular, analysts and ML engineers).
We'd like, for instance, an analyst working within a notebook to import a python interface
to some rich data-explorer component written in ReactJS, feed it what it needs (e.g. a data source/target),
and then use the component's features fully, enabling the work done in it to then be
used back in the notebook.

For example, saw we take a forced-directed graph or tsne viz tool that visualizes data, enabling us to view, filter and select data points.
We'd like our analysts to be able to provide their data, or point to a source where the data can be found,
and then once they find an select the data they want to work with, using the features of the tool,
be able to extract that data and get it into some python collection that they can then continue their analysis with.

Here's a proof-of-concept solution for the example described above, along with other instances of this class.

Use some persisted data store and a platform-independent format as the middleware for py and js communication.

• From python, we write the data to a DB or files in a JS-friendly format (like json or csv).
• JS can then read this data from that source and do it's work
• When we want to communicate some data back to python, we do so again, through the "shared" data store

We could already get far by automating the strategy above as much as possible:
Not obliging the user to set up the data store themselves or have to define the codecs
necessary to communicate to the data middleware (at least not for the most common data types an analyst might use),
or cache management (to not clutter storage with ad hoc data).
Perhaps this approach could be further extended to allow some in-memory data store to be used,
which would improve communication speed.

We'd like to be able to do this with any JS component, but only through a specific normalized interface.
That is, the automation of the PY-JS communication setup will be done only for very specific (but open-closed)
python and JS interfaces. To use a third-party JS component, we will need to wrap it into a facade that
is compliant with our normalized interface that we will rely on, and the python interface will be totally determined by this.
This facade needs to be open-closed though.
That is, it needs to be able to accomodate any "lines of communication" between python and the JS component.

Note: Some solutions work in some environments, and not in others.
Typically, I've seen some things work in my browser-based notebook (what is launched by jupyter notebook in terminal), but not in VScode notebook.
We don't want to target covering all environments, but at least browser-based and VScode, which should cover a lot of users.

[thorwhalen]

ChatGPT's feedback

I asked chatGPT to read the above and do to "Give me an overview of the architecture of such a system, breaking the work that needs to be done into tasks that you will describe and suggest some tools (and possibly design patterns) that can be used to complete them". Note that the answer should be taken with a grain of salt. Some of the suggestions are dubious, and I have not told it about py2http, http2py, and http2js, which we'll use for the middleware.

Certainly! The document you provided outlines a vision for integrating JS components, particularly those written in ReactJS, into a Jupyter notebook environment, facilitating seamless communication between the front-end and back-end. Here's an overview of the architecture and the tasks involved:

Architecture Overview:

1. Python Interface:

   • Acts as the primary interface for analysts and ML engineers.
   • Provides methods to send data to and receive data from the JS component.

2. Middleware Data Store:

   • A persistent storage solution that acts as a bridge between Python and JS.
   • Stores data in a platform-independent format (e.g., JSON, CSV).

3. JS Component Facade:

   • A normalized interface that wraps third-party JS components.
   • Ensures that any JS component can communicate with the Python interface through the middleware.

4. JS Component:

   • The actual visualization or tool (e.g., forced-directed graph, tsne viz).
   • Interacts with the data provided by the Python interface and allows user interaction.

Tasks Breakdown:

1. Define the Python Interface:

   • Task: Design a Python class or module that provides methods for data input/output and interaction with the JS component.
   • Tools: Python's standard libraries.
   • Patterns: Factory Pattern (for creating instances of different JS components).

2. Set Up the Middleware Data Store:

   • Task: Choose and set up a persistent storage solution.
   • Tools: SQLite (for lightweight DB), or flat files (JSON, CSV), or in-memory solutions like Redis.
   • Patterns: Singleton Pattern (to ensure a single instance of the data store).

3. Develop the JS Component Facade:

   • Task: Design a standardized JS interface that can wrap any third-party JS component, ensuring it communicates correctly with the middleware.
   • Tools: JavaScript (ES6+), possibly TypeScript for stricter type definitions.
   • Patterns: Facade Pattern (to provide a simplified interface), Adapter Pattern (to adapt third-party components to the facade).

4. Integration of JS Components:

   • Task: Integrate the actual JS components, ensuring they work within the Jupyter environment.
   • Tools: ReactJS, D3.js (for visualizations), Webpack or Parcel (for bundling).
   • Patterns: Component Pattern (React's core principle).

5. Automate Data Communication:

   • Task: Automate the process of data transfer between Python and JS, ensuring minimal setup for the end-user.
   • Tools: AJAX or Fetch API (for asynchronous data retrieval), Python libraries like pandas (for data manipulation).
   • Patterns: Observer Pattern (to watch for data changes and synchronize).

6. Environment Compatibility:

   • Task: Ensure the solution works in both browser-based Jupyter and VSCode notebooks.
   • Tools: Jupyter extensions, VSCode extensions.
   • Patterns: Strategy Pattern (to switch between different environment-specific implementations).

7. Documentation and Testing:

   • Task: Provide comprehensive documentation and tests.
   • Tools: Jupyter's nbconvert (for documentation), pytest (for Python testing), Jest (for JS testing).
   • Patterns: Unit Testing, Integration Testing.

Conclusion:

The envisioned system aims to bridge the gap between front-end and back-end development within a Jupyter notebook environment, promoting reusability and collaboration. By breaking down the architecture into specific tasks and leveraging appropriate tools and design patterns, the system can provide a seamless experience for both analysts and ML engineers.