Make JBang preprocessor pick up scripts dynamic

[ferenc-csaky]

Currently, if the JBangPreprocessor matches to a UDF script it will compile it no matter what. This means even if a UDF script is not used by the project.

Since the script compilation is a very time consuming actions, we should make this behavior smarter, analyzing if the project actually uses the UDF and only compile it if it is necessary.

[mbroecheler]

We used to have "smart" preprocessors in DataSQRL but decided to get rid of them because they introduced subtle bugs and were hard to maintain. Also, when something went wrong, it was very hard to figure out what happened.

Generally, the idea is for preprocessor to be "dumb". Do one thing predictably to set up the build dir for the compiler. All the smarts should be in the compiler. Adjusting preprocessors depending on what the compiler will do invites trouble.

My 2 cents:
The key issue here is that jbang compilation takes a few seconds which feels like eternity when you are waiting. I suggest we address that root cause. The use case where you have multiple SQRL projects in the same directory and only one needs the UDF is a bit contrived to begin with (e..g. in datasqrl-examples). In most cases, those would end up in separate directories.

To make Jbang faster I suggest:

• we cache the flink dependency all UDFs need in the docker image so we save the download. The actual compilation for most UDFs is really fast. In particular with poor internet connection it can take a long time to retrieve the dependency and it requires internet for all UDFs.
• to speed things up, the user can copy the UDF jar that gets generated by jbang into the same folder as the function definition. We then add a check to the preprocessor: if a [function-name].jar already exists, don't compile. That's still "dumb enough" to be easy to follow and provides for a number of ways to speed things up: you can run jbang separately, you can copy the jar over from the last time you compiled, etc. Most of the generated jars are very small so you can check them into git. This addresses the most common case: while you are iterating on your UDF and SQRL script at the same time, you want to recompile the jar every time. Once the UDF is "stable" you can copy the jar in and make it effectively "static" to speed up compilations.

We can start with the latter since it is really easy to implement in the preprocessor.