Add support for Spark 4.0.0

[kevinwallimann]

[m8719-github]

@kevinwallimann I have a working prototype for Spark 4.0.1 but the code is no longer backwards compatible with earlier versions of Spark due to major public API change in Spark 4 (particularly in Column).

I can create a draft PR for review, but a major version bump is likely required.

[moiseving]

hey @m8719-github any chance this could get a release in the foreseeable future?

[m8719-github]

hey @m8719-github any chance this could get a release in the foreseeable future?

I am in the process of updating for Spark 4.1.0, I'll create a PR once I am done (should be couple of days); it'll need a major version bump because of backwards compatibility breakage.

[tashoyan]

Is there a chance that the support for Spark 4 will be implemented?

[moiseving]

heya @m8719-github is the 4.1 update still in the works?

[petitbonney]

I tried some contrib: #373