Add support for delta lake

[imor]

delta-kernel-rs recently added write support for delta tables. Try out the new crate and see if delta lake support can be added in a sink.

[abduldjafar]

Nice. Thanks alot @imor

[abduldjafar]

and btw , i've bit experience in rust. may i try to take this one?

[imor]

Certainly, you can take a look at these examples from delta-kernel-rs: https://github.com/delta-io/delta-kernel-rs/blob/main/kernel/tests/write.rs

[abduldjafar]

or we can try use this one

[abduldjafar]

I’m working with this one. Right now already sink from postgres to deltalake. But the deltalake still in local

[abduldjafar]

Is it ok if I work in my fork firts? Or i should work with another branch in this repo

[abduldjafar]

and is this tool has slack or discord group for discussion? thanks

[imor]

I’m working with this one. Right now already sink from postgres to deltalake. But the deltalake still in local

Local is fine for initial implementation and testing.

Is it ok if I work in my fork firts? Or i should work with another branch in this repo

You won't have commit access to this repo, so you'd need to fork it and work in a branch on your fork. Don't worry, you'll be able to open a pull request from your fork's branch which can be merged in this repo.

and is this tool has slack or discord group for discussion? thanks

No, not at the moment. Feel free to ask your questions in this thread though. I'll try to answer them.

[abhi-airspace-intelligence]

We are implementing delta table support for internal use. We will upstream this when ready. uni-intelligence#1

[abduldjafar]

We are implementing delta table support for internal use. We will upstream this when ready. uni-intelligence#1

Hi @abhi-airspace-intelligence . any update for this?

[abhiaagarwal]

Hey @abduldjafar (personal account), I ended up moving it in-tree since there were too many breaking API changes at the time and the arrow version conflicts with the iceberg implementation made it impossible to put it in the etl-destinations crate. I can check with my team to see if I can release it as a standalone crate!