Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

⚡️Optimize xircuits workflow save file #170

Merged
merged 5 commits into from
Jul 12, 2022

Conversation

AdrySky
Copy link
Contributor

@AdrySky AdrySky commented Jul 6, 2022

Description

This will optimize which metadata should changes when saving/detecting changes. I took the liberty to update all examples metadata so that it's easier to see which metadata changes after saving.

Currently, it's not possible to change the layers( nodesLayer and linksLayer) as everytime xircuits deserialized(detecting changes), it create the layers by default so it's hard to control that metadata. However, we can still control the nodes and links inside the layers. This PR will only update the metadata based on what have changes on the canvas like node position change, adding a link and etc. To summarize, this optimization will only update necessary changes beside the layers' id.

Pull Request Type

  • Xircuits Core (Jupyterlab Related changes)
  • Xircuits Canvas (Custom RD Related changes)
  • Xircuits Component Library
  • Testing Automation
  • Documentation
  • Others (Please Specify)

Type of Change

  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Tests

I recommended using VScodes's Source Control to see the metadata changes.

  1. Sanity check
    1. Open any example
    2. Directly save xircuits
    3. Open VSCode and click Source Control on left sidebar
    4. Click on the changed example's xircuits file
    5. You should see only the layers' id have been changed (It's not possible to changed that for now)
  2. Move a node and see the changes
    1. Open any example
    2. Move any node
    3. Save xircuits
    4. Open VSCode and click Source Control on left sidebar
    5. Click on the changed example's xircuits file
    6. See what have changed.
    7. The changed metadata should be the links points' position, node's position and its ports position. Include the layers id.

Tested on?

  • Windows
  • Linux Ubuntu
  • Centos
  • Mac
  • Others (State here -> xxx )

@AdrySky AdrySky added the enhancement New feature or request label Jul 6, 2022
@AdrySky AdrySky requested a review from MFA-X-AI July 6, 2022 07:41
@AdrySky AdrySky self-assigned this Jul 6, 2022
Copy link
Member

@MFA-X-AI MFA-X-AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've tried 3 variants:

  1. sample.xircuits: the minimum workflow with a HelloHyperparameter component + hyperparameter.
  2. KerasModelPredict.xircuits
  3. image_classification_template.xircuits the largest workflow we have so far with ~ 21 components 36 hyperparameters 70+ links.

For these, I simply apply a simple drag of a component, save it, then run git diff --stat. Here are the results:


Before:

sample.xircuits
12KB | 61 +-
1 file changed, 32 insertions(+), 29 deletions(-)

 KerasModelPredict.xircuits
26KB | 188 +++++++++++++++++++-----------------
 1 file changed, 99 insertions(+), 89 deletions(-)

image_classification_template.xircuits
248 KB | 2160 ++++++++++++-----------
1 file changed, 1116 insertions(+), 1044 deletions(-)

After:

 examples/untitled.xircuits | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)


 examples/KerasModelPredict.xircuits | 120 +++++++++++++++++++-----------------
 1 file changed, 65 insertions(+), 55 deletions(-)

examples/image_classification_template.xircuits | 1288 ++++++++++++-----------
 1 file changed, 680 insertions(+), 608 deletions(-)

The PR optimized the needed changes by half, amazing!

One thing I think we can further optimize is if we can save only 3 significant figures. Currently even though we do not move that particular object, I've noticed sometimes the s.f. of xy coordinates shifts by a few micro coordinates.

Consider a part of this json:

"x": 2713.8753255842857,
"y": 43.29165854974872

When saving, it shifted to:

"x": 2713.8750631877942,
"y": 43.29170544782238

If we can somehow save and load only 3 s.f., I think this can be even more optimized.

@AdrySky
Copy link
Contributor Author

AdrySky commented Jul 12, 2022

I've noticed sometimes the s.f. of xy coordinates shifts by a few micro coordinates.

Yup, noticed that too while working on it. So I rounded it to 3 nearest decimal point.
Not sure should I update again the examples.

Copy link
Member

@MFA-X-AI MFA-X-AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the new changes, we can see

 examples/untitled.xircuits | 32 ++++++++++++++++----------------
 1 file changed, 16 insertions(+), 16 deletions(-)

 examples/KerasModelPredict.xircuits | 58 ++++++++++++++++++-------------------
 1 file changed, 29 insertions(+), 29 deletions(-)

 examples/image_classification_template.xircuits | 106 ++++++++++++------------
 1 file changed, 53 insertions(+), 53 deletions(-)

Tabling the changes would give:

Original v1 v2
61 22 32
188 120 58
2160 1288 106

The new change is crazy efficient for larger canvases! Great work, this will greatly reduce the changes on .xircuits for every release.

@MFA-X-AI MFA-X-AI merged commit 646301c into master Jul 12, 2022
@MFA-X-AI MFA-X-AI deleted the adry/optimize-data-when-deserializing branch July 12, 2022 09:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants