hof/flow -> airflow

[verdverm]

It would be great to transform workflows to other frameworks, airflow as an example, to scale out. There by, hof flow becomes a way to quickly develop the ETL, re: better CUE ergonomics for the ETL twiddling / iteration

Probably helpful if we could snapshot task i/o too, for both iteration and determining what code to generate. Ideally flow can have a standardize record format to feed into hof generators for target workflow frameworks. Might also be helpful for non-workflow functions, like API handlers when generating CRUD+

[gedw99]

Totally agree …

have a look at:

conduit

https://github.com/ConduitIO/conduit

anyquery

https://github.com/julien040/anyquery/

these are golang based CDC that track changes in many data stores …

I am looking at the best way to integrate these with hof

would appreciate any feedback you have .

see my github profile whee we can chat on signal