Skip to content

setup clean base for Pietro#51

Merged
piotor87 merged 15 commits into
masterfrom
clean-base-for-Pietro
Jun 25, 2026
Merged

setup clean base for Pietro#51
piotor87 merged 15 commits into
masterfrom
clean-base-for-Pietro

Conversation

@vincent-octo

@vincent-octo vincent-octo commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

This is a PR that sets up a clean python sub-project engine inside our existing codebase.

TODO

  • Base structure that takes the Parquet file from the intake stage, get the columns from config, and outputs a Parquet file.

  • Parallelize the processing step

  • Put intake config into shared config

    I think it's better to have the shared config file (src/kanta/config.py) have only the variables that are meant for cross-module use.
    Other variables, e.g. chunking knobs, should belong to their own module.
    That way the shared config file doesn't turn into a gigantic mess.
    🤷

  • logging

How it works

  • The shared config is at the root of our package at src/kanta/config.py.
  • The engine module is made available by having the file src/kanta/engine/__init__.py
  • In src/kanta/engine/__main__.py the engine imports the injection module and the shared config with:
    from kanta import config
    from kanta.engine import injection
    
    if __name__ == '__main__':
        injection.fake_run(config.EXAMPLE_VAR)
  • Running python3 -m kanta.engine will call src/kanta/engine/__main__.py

@vincent-octo

Copy link
Copy Markdown
Contributor Author

@piotor87 Here is an example on how to have a clean setup.
Let me know if you have questions or want to discuss =)

@vincent-octo vincent-octo force-pushed the clean-base-for-Pietro branch from 06203a5 to d0c1c53 Compare June 12, 2026 08:31
@piotor87 piotor87 linked an issue Jun 16, 2026 that may be closed by this pull request
If the role of a variable is a cross-module contract, then it belongs
to kanta.config.
Otherwise it belongs to their own module (e.g. engine knobs belong into
kanta.engine).
@vincent-octo vincent-octo force-pushed the clean-base-for-Pietro branch from 1387eaf to d61593b Compare June 17, 2026 14:38
@piotor87 piotor87 marked this pull request as ready for review June 25, 2026 11:42
@piotor87 piotor87 merged commit 67a5378 into master Jun 25, 2026
1 check passed
@piotor87 piotor87 linked an issue Jun 25, 2026 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pipeline structural changes Cross-script references

2 participants