Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Datatx pull model #2052

Merged
merged 21 commits into from
Mar 16, 2022
Merged

Datatx pull model #2052

merged 21 commits into from
Mar 16, 2022

Conversation

redblom
Copy link
Contributor

@redblom redblom commented Sep 8, 2021

This PR implements CS3 tx (pull model) (cs3org/cs3apis#143) using rclone as data transfer tool.
The following commands are available: transfer-create, transfer-list, transfer-get-status, transfer-retry, transfer-cancel

Create a transfer (sender):

#  ./cmd/reva/reva transfer-create -grantee 3000989c-6gh5-4cec-a9ce-e876tf7de452 -granteeType user -idp cesnet.cz /home/data/big-data
+--------------------------------------+-------------+--------------------------------------+-----------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+-------------+--------------------------------------+---------------------+-------------------------------+-------------------------------+
| #                                    | OWNER.IDP   | OWNER.OPAQUEID                       | RESOURCEID                                                                                          | PERMISSIONS                                                                                                                                                                                                     | TYPE              | GRANTEE.IDP | GRANTEE.OPAQUEID                     | SHARETYPE           | CREATED                       | UPDATED                       |
+--------------------------------------+-------------+--------------------------------------+-----------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+-------------+--------------------------------------+---------------------+-------------------------------+-------------------------------+
| 775b47bd-4cca-4470-86f7-0424b7a998ab | surfsara.nl | 4029579c-6ad5-4cec-a9ce-e843f77de452 | storage_id:"123e4567-e89b-12d3-a456-426655440000" opaque_id:"fileid-jimmie%2Fdata%2Fdata/big-data"  | permissions:<create_container:true delete:true get_path:true initiate_file_download:true initiate_file_upload:true list_container:true list_file_versions:true move:true restore_file_version:true stat:true >  | GRANTEE_TYPE_USER | cesnet.cz   | 3000989c-6gh5-4cec-a9ce-e876tf7de452 | SHARE_TYPE_TRANSFER | 2021-09-06 10:52:47 +0000 UTC | 2021-09-06 10:52:47 +0000 UTC |
+--------------------------------------+-------------+--------------------------------------+-----------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+-------------+--------------------------------------+---------------------+-------------------------------+-------------------------------+

Discover and accept the transfer (receiver):

# ./cmd/reva/reva ocm-share-list-received
+--------------------------------------+-------------+--------------------------------------+----------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+-------------+--------------------------------------+-------------------------------+-------------------------------+----------------------+---------------------+
| #                                    | OWNER.IDP   | OWNER.OPAQUEID                       | RESOURCEID                                                                                               | PERMISSIONS                                                                                                                                                       | TYPE              | GRANTEE.IDP | GRANTEE.OPAQUEID                     | CREATED                       | UPDATED                       | STATE                | SHARETYPE           |
+--------------------------------------+-------------+--------------------------------------+----------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+-------------+--------------------------------------+-------------------------------+-------------------------------+----------------------+---------------------+
| 40ff72c6-c30c-4380-827d-5e2d3970b57c | surfsara.nl | 4029579c-6ad5-4cec-a9ce-e843f77de452 | storage_id:"123e4567-e89b-12d3-a456-426655440000" opaque_id:"fileid-jimmie%2Fdata%2Fbig-data"       | permissions:<get_path:true get_quota:true initiate_file_download:true list_grants:true list_container:true list_file_versions:true list_recycle:true stat:true >  | GRANTEE_TYPE_USER | cesnet.cz   | 3000989c-6gh5-4cec-a9ce-e876tf7de452 | 2021-09-06 10:52:47 +0000 UTC | 2021-09-06 10:52:47 +0000 UTC | SHARE_STATE_PENDING  | SHARE_TYPE_TRANSFER |
+--------------------------------------+-------------+--------------------------------------+----------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+-------------+--------------------------------------+-------------------------------+-------------------------------+----------------------+---------------------+

# ./cmd/reva/reva ocm-share-update-received -state accepted 40ff72c6-c30c-4380-827d-5e2d3970b57c
OK

Receiver discovers the transfer ID needed to get the status of, retry, or cancel the transfer:

# ./cmd/reva/reva transfer-list -shareId 40ff72c6-c30c-4380-827d-5e2d3970b57c
+--------------------------------------+--------------------------------------+
| SHAREID.OPAQUEID                     | ID.OPAQUEID                          |
+--------------------------------------+--------------------------------------+
| 40ff72c6-c30c-4380-827d-5e2d3970b57c | 76508e0e-eda1-43fc-9b4d-c6b526b5a75c |
+--------------------------------------+--------------------------------------+

Get the transfer status:

# ./cmd/reva/reva transfer-get-status -txId 76508e0e-eda1-43fc-9b4d-c6b526b5a75c
+--------------------------------------+--------------------------------------+----------------+-----------------------------------+
| SHAREID.OPAQUEID                     | ID.OPAQUEID                          |         STATUS | CTIME                             |
+--------------------------------------+--------------------------------------+----------------+-----------------------------------+
| 40ff72c6-c30c-4380-827d-5e2d3970b57c | 76508e0e-eda1-43fc-9b4d-c6b526b5a75c | STATUS_TRANSFER_COMPLETE | Mon Sep 6 10:53:08 +0000 UTC 2021 |
+--------------------------------------+--------------------------------------+----------------+-----------------------------------+

Retry a failed transfer:

# ./cmd/reva/reva transfer-retry -txId 76508e0e-eda1-43fc-9b4d-c6b526b5a75c
+--------------------------------------+--------------------------------------+---------------------+-------------------------------+
| SHAREID.OPAQUEID                     | ID.OPAQUEID                          |              STATUS | CTIME                         |
+--------------------------------------+--------------------------------------+---------------------+-------------------------------+
| 40ff72c6-c30c-4380-827d-5e2d3970b57c | 76508e0e-eda1-43fc-9b4d-c6b526b5a75c | STATUS_TRANSFER_NEW | 2021-09-06 10:53:08 +0000 UTC |
+--------------------------------------+--------------------------------------+---------------------+-------------------------------+

Cancel a running transfer:

# ./cmd/reva/reva transfer-cancel -txId f82396c4-386b-484f-856c-b75936980f1e
+--------------------------------------+--------------------------------------+---------------------------+-----------------------------------+
| SHAREID.OPAQUEID                     | ID.OPAQUEID                          |                    STATUS | CTIME                             |
+--------------------------------------+--------------------------------------+---------------------------+-----------------------------------+
| a940df85-9023-4cf8-809a-4c33748f2c66 | f82396c4-386b-484f-856c-b75936980f1e | STATUS_TRANSFER_CANCELLED | Wed Sep 1 11:56:25 +0000 UTC 2021 |
+--------------------------------------+--------------------------------------+---------------------------+-----------------------------------+

@redblom redblom requested a review from labkode as a code owner September 8, 2021 08:27
@redblom redblom mentioned this pull request Sep 8, 2021
@redblom redblom changed the title [WIP] Datatx pull model Datatx pull model Sep 27, 2021
@redblom
Copy link
Contributor Author

redblom commented Sep 27, 2021

@labkode @ishank011 please review

@labkode
Copy link
Member

labkode commented Dec 13, 2021

@redblom the PR is ready, can you just rebase on top of master and then we merge it with the cs3apis changes?

@redblom redblom mentioned this pull request Dec 13, 2021
@redblom redblom requested review from glpatcern and a team as code owners December 13, 2021 10:56
@redblom
Copy link
Contributor Author

redblom commented Dec 13, 2021

@redblom the PR is ready, can you just rebase on top of master and then we merge it with the cs3apis changes?

@labkode : Done!

glpatcern
glpatcern previously approved these changes Feb 14, 2022
Copy link
Member

@glpatcern glpatcern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As this work was already reviewed and it was simply pending the CI, I'm approving it.

ishank011
ishank011 previously approved these changes Mar 16, 2022
Copy link
Contributor

@ishank011 ishank011 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One comment and then we can merge

go.mod Outdated Show resolved Hide resolved
@redblom redblom dismissed stale reviews from ishank011 and glpatcern via 02754ff March 16, 2022 09:49
@ishank011 ishank011 merged commit 559fccb into cs3org:master Mar 16, 2022
@lgtm-com
Copy link

lgtm-com bot commented Mar 16, 2022

This pull request introduces 1 alert when merging 02754ff into dcb9b01 - view on LGTM.com

new alerts:

  • 1 for Wrapped error is always nil

Daniel-WWU-IT pushed a commit to Daniel-WWU-IT/reva that referenced this pull request Mar 16, 2022
@micbar micbar mentioned this pull request May 5, 2022
45 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants