Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ditch Dagster #2523

Closed
soininen opened this issue Jan 26, 2024 · 5 comments · Fixed by spine-tools/spine-engine#136
Closed

Ditch Dagster #2523

soininen opened this issue Jan 26, 2024 · 5 comments · Fixed by spine-tools/spine-engine#136

Comments

@soininen
Copy link
Contributor

As @manuelma puts it in #2522:

We are hooked on an older version, using API that is deprecated in the most recent version. My feeling is we could replicate the bit of functionality that we use from that old dagster with not too much effort (but I might be wrong)?

I have the same feeling. We are not really using much of Dagster's functionality anyway and getting rid of it would fix issues with dependencies and Python versions such as #2522 and #2517
@soininen
Copy link
Contributor Author

soininen commented Apr 9, 2024

A Python package to look into as a possible Dagster replacement: Joblib.

@jkiviluo
Copy link
Member

jkiviluo commented Apr 9, 2024

Joblib looks cool

@manuelma
Copy link
Collaborator

manuelma commented Apr 9, 2024 •
edited
Loading

The main features of joblib don't look very well aligned with our requirements though (unless I'm missing something). All we need is a system to replace deprecated dagster's PipelineDefinition, SolidDefinition, etc. In other words, we need to be able to (i) define a pipeline (set of jobs where the output of some is the input of some others) and (ii) a function that given a pipeline returns a list of all the jobs that are ready to execute. It should be straightforward to implement and much easier to maintain in the long-run.

@manuelma manuelma mentioned this issue Apr 20, 2024
Merged
5 tasks
ptsavol added a commit that referenced this issue Apr 22, 2024
@ptsavol
Get spine-engine from jumpster branch
f7619ab 
Re Issue #2523
This was referenced Apr 22, 2024
Merged
Merged
soininen added a commit to spine-tools/spine-engine that referenced this issue Apr 25, 2024
@soininen
Fix stop engine message
7d1b6b7 
Execution direction was accidentally converted to string in
SpineEngine._stop_item()

Re spine-tools/Spine-Toolbox#2523
ptsavol pushed a commit to spine-tools/spine-engine that referenced this issue Apr 25, 2024
@manuelma
Jumpster (#136)
243280a 
- Replace dagster with jumpster

Re spine-tools/Spine-Toolbox#2523
ptsavol added a commit to spine-tools/spine-items that referenced this issue Apr 25, 2024
@ptsavol
Changes related to ditching Dagster (#204)
18d8a92 
- Use ExecutionDirection enum instead of strings

Re spine-tools/Spine-Toolbox#2523
ptsavol added a commit that referenced this issue Apr 25, 2024
@ptsavol
Changes related to ditching Dagster (#2715)
535a6ae 
- Fix animations and item icons on Design View and headless mode
- Update CHANGELOG

Re Issue #2523
@ptsavol
Copy link
Member

ptsavol commented Apr 25, 2024

This is done. Nice work @manuelma. Dagster is no longer a requirement in 0.8-dev branch. If you plan on hopping between 0.8-dev and master branches you may want to keep dagster in your toolbox environment until these branches have been merged.

@ptsavol ptsavol closed this as completed Apr 25, 2024
@ptsavol ptsavol mentioned this issue Apr 25, 2024
Closed
@manuelma
Copy link
Collaborator

Thank you @PekkaSavolainen - to be fair you did most of it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Jumpster spine-tools/spine-engine
4 participants
@manuelma @soininen @jkiviluo @ptsavol