[No Ticket] Update python library requirements

[C1KB-tensileai]

What is this and why are we doing it?

This consolidates and updates the Python modules and packages required for the T-MSIS Analytic File (TAF) project. Specific versions are used when necessary to align with the Databricks 13.3 LTS runtime. The Databricks CLI has been removed since it is no longer actively supported.

What are the security implications from this change?

This change should not introduce any new security implications as we are removing a lot of unnecessary modules and packages.

How did I test this?

I tested this change locally by re-creating my Python virtual environment. No issues were encountered during that test.

Terminal Output

(.venv) scleeton@BLD:/mnt/e/Projects/T-MSIS-Analytic-File-Generation-Python$ pip install -r requirements.txt 
Collecting autoflake
  Using cached autoflake-2.2.1-py3-none-any.whl (32 kB)
Collecting bandit
  Downloading bandit-1.7.6-py3-none-any.whl (123 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 123.5/123.5 KB 2.1 MB/s eta 0:00:00
Collecting black
  Downloading black-23.12.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.7/1.7 MB 15.7 MB/s eta 0:00:00
Collecting flake8
  Downloading flake8-7.0.0-py2.py3-none-any.whl (57 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.6/57.6 KB 9.6 MB/s eta 0:00:00
Collecting isort
  Downloading isort-5.13.2-py3-none-any.whl (92 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 92.3/92.3 KB 15.3 MB/s eta 0:00:00
Collecting numpy==1.21.5
  Using cached numpy-1.21.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (15.9 MB)
Collecting pandas==1.4.4
  Using cached pandas-1.4.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.6 MB)
Collecting pyspark==3.4.0
  Using cached pyspark-3.4.0.tar.gz (310.8 MB)
  Preparing metadata (setup.py) ... done
Collecting pytest
  Downloading pytest-7.4.4-py3-none-any.whl (325 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 325.3/325.3 KB 28.4 MB/s eta 0:00:00
Collecting setuptools==63.4.1
  Using cached setuptools-63.4.1-py3-none-any.whl (1.2 MB)
Collecting wheel==0.37.1
  Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Collecting python-dateutil>=2.8.1
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting pytz>=2020.1
  Using cached pytz-2023.3.post1-py2.py3-none-any.whl (502 kB)
Collecting py4j==0.10.9.7
  Using cached py4j-0.10.9.7-py2.py3-none-any.whl (200 kB)
Collecting pyflakes>=3.0.0
  Downloading pyflakes-3.2.0-py2.py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.7/62.7 KB 13.6 MB/s eta 0:00:00
Collecting tomli>=2.0.1
  Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
Collecting PyYAML>=5.3.1
  Using cached PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (705 kB)
Collecting stevedore>=1.20.0
  Using cached stevedore-5.1.0-py3-none-any.whl (49 kB)
Collecting rich
  Downloading rich-13.7.0-py3-none-any.whl (240 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 240.6/240.6 KB 24.5 MB/s eta 0:00:00
Collecting GitPython>=3.1.30
  Using cached GitPython-3.1.40-py3-none-any.whl (190 kB)
Collecting mypy-extensions>=0.4.3
  Using cached mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)
Collecting packaging>=22.0
  Using cached packaging-23.2-py3-none-any.whl (53 kB)
Collecting click>=8.0.0
  Using cached click-8.1.7-py3-none-any.whl (97 kB)
Collecting platformdirs>=2
  Downloading platformdirs-4.1.0-py3-none-any.whl (17 kB)
Collecting typing-extensions>=4.0.1
  Downloading typing_extensions-4.9.0-py3-none-any.whl (32 kB)
Collecting pathspec>=0.9.0
  Downloading pathspec-0.12.1-py3-none-any.whl (31 kB)
Collecting mccabe<0.8.0,>=0.7.0
  Using cached mccabe-0.7.0-py2.py3-none-any.whl (7.3 kB)
Collecting pycodestyle<2.12.0,>=2.11.0
  Using cached pycodestyle-2.11.1-py2.py3-none-any.whl (31 kB)
Collecting iniconfig
  Using cached iniconfig-2.0.0-py3-none-any.whl (5.9 kB)
Collecting exceptiongroup>=1.0.0rc8
  Downloading exceptiongroup-1.2.0-py3-none-any.whl (16 kB)
Collecting pluggy<2.0,>=0.12
  Using cached pluggy-1.3.0-py3-none-any.whl (18 kB)
Collecting gitdb<5,>=4.0.1
  Using cached gitdb-4.0.11-py3-none-any.whl (62 kB)
Collecting six>=1.5
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting pbr!=2.1.0,>=2.0.0
  Downloading pbr-6.0.0-py2.py3-none-any.whl (107 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 107.5/107.5 KB 22.4 MB/s eta 0:00:00
Collecting markdown-it-py>=2.2.0
  Using cached markdown_it_py-3.0.0-py3-none-any.whl (87 kB)
Collecting pygments<3.0.0,>=2.13.0
  Downloading pygments-2.17.2-py3-none-any.whl (1.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 28.3 MB/s eta 0:00:00
Collecting smmap<6,>=3.0.1
  Using cached smmap-5.0.1-py3-none-any.whl (24 kB)
Collecting mdurl~=0.1
  Using cached mdurl-0.1.2-py3-none-any.whl (10.0 kB)
Using legacy 'setup.py install' for pyspark, since package 'wheel' is not installed.
Installing collected packages: pytz, py4j, wheel, typing-extensions, tomli, smmap, six, setuptools, PyYAML, pyspark, pygments, pyflakes, pycodestyle, pluggy, platformdirs, pbr, pathspec, packaging, numpy, mypy-extensions, mdurl, mccabe, isort, iniconfig, exceptiongroup, click, stevedore, python-dateutil, pytest, markdown-it-py, gitdb, flake8, black, autoflake, rich, pandas, GitPython, bandit
  Attempting uninstall: setuptools
    Found existing installation: setuptools 59.6.0
    Uninstalling setuptools-59.6.0:
      Successfully uninstalled setuptools-59.6.0
  Running setup.py install for pyspark ... done
Successfully installed GitPython-3.1.40 PyYAML-6.0.1 autoflake-2.2.1 bandit-1.7.6 black-23.12.1 click-8.1.7 exceptiongroup-1.2.0 flake8-7.0.0 gitdb-4.0.11 iniconfig-2.0.0 isort-5.13.2 markdown-it-py-3.0.0 mccabe-0.7.0 mdurl-0.1.2 mypy-extensions-1.0.0 numpy-1.21.5 packaging-23.2 pandas-1.4.4 pathspec-0.12.1 pbr-6.0.0 platformdirs-4.1.0 pluggy-1.3.0 py4j-0.10.9.7 pycodestyle-2.11.1 pyflakes-3.2.0 pygments-2.17.2 pyspark-3.4.0 pytest-7.4.4 python-dateutil-2.8.2 pytz-2023.3.post1 rich-13.7.0 setuptools-63.4.1 six-1.16.0 smmap-5.0.1 stevedore-5.1.0 tomli-2.0.1 typing-extensions-4.9.0 wheel-0.37.1

Should there be new or updated documentation for this change? (Be specific.)

The documentation in this repository should be sufficient to help users install the Python modules and packages required for this project.

PR Checklist

• The JIRA ticket number and a short description is in the subject line
• My code follows any applicable style guides
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation

[C1KB-tensileai]

@a-pfeiffer FYI, this is the change we discussed last week to consolidate the requirements. I removed a lot of unnecessary modules/libraries that probably won't be necessary to continue support and development on this project.

In this PR's description, I noted that the Databricks CLI is no longer supported in Python. Instructions to install or update the Databricks CLI that is currently supported can be found here.

This change may be moot when #111 is merged as we will no longer rely on the use of a requirements file to install Python modules/libraries.

[a-pfeiffer]

@XiaoBarry and @alihashmi-wh Hi Xiao and Ali, can either of you review this and add an approval as you see fit? Thanks!