Add remote sync capability - POC #442

Chiff · 2023-04-03T19:58:16Z

This is still a work in progress, however, I'd like to receive feedback.

With this feature, users should be able to use CS on multiple devices, while we would sync their data (basically share DataStore.kt) on Google Drive. Each user will be using their own drive with their own credentials without storing their data by CS.

Usage:
0. User will set up the https://console.cloud.google.com/ project with GoogleDriveAPI enabled and will create OAuth2 web application credentials with an allowed redirect to "https://chiff.github.io/cloudstream-sync/google-drive" (this URL should be on CS repo) or URL they provide themselves

User will visit settings and will add GoogleDrive account, providing their clientId and secret
They will be redirected to google and they would confirm they want to grant access to provided URL (and by proxy CS app) so it can manage GoogleDrive
When everything ends successfully they will be logged in via CS AccountManager
new BackupAPI interface will observe changes in DataStore and will upload new Data. This BackupAPI will also periodically download data from the drive (or another provider) and save it into DataStore

At the current state of PR this "works" but I need to work out how should we correctly sync data.

…d strings

C10udburst · 2023-04-08T08:12:54Z

app/src/main/java/com/lagradost/cloudstream3/syncproviders/providers/GoogleDriveApi.kt

+    override val requiresSecret = true
+    override val requiresClientId = true
+    override val defaultFilenameValue = "cloudstreamapp-sync-file"
+    override val defaultRedirectUrl =   "https://chiff.github.io/cloudstream-sync/google-drive"


I assume Google doesn't allow to directly set the redirect url to cloudstreamapp://oauth/google-drive?

Tldr; Yes you are right.

Took a while to figure this out. There are two viable approaches I am aware of.

Easiest for the user would be for cloudstream to create google console project, create an android oauth client and we would use in app login which is really smooth to use and looks really cool. However this may need to pay some kind of subscription because there is a quota for free. Also, there may be something like a privacy concern for users.

The other way (the one i implemented) is to create web client which allows redirects only to https origins and that's the reason for this redirect url. I tried deep links for smthn like https://my-non-existent.url/cloudstream/etc but I wasn't able to make it work. I think it is good compromise to create redirect url with cloudstream's gh-pages where source code is still visible for those concerned. Again, this method may seem more plausible for privacy concerned users as they have their own project under control.

Note: I will also create a guide to create projects and some explaining what it does. Im busy for next week and come back to implementation later the other on. I will still be able to answer any questions and if there is a need for
a conversation I created feature request in discord called "bidirectional device sync"

…multiple devices are online

…d improve data restore (pt.2)

…as really changed (true -> true is not a change)

…i is active

…ad scheduler is throttling job

Chiff · 2023-05-08T18:27:33Z

hey @C10udburst im marking the PR as ready to review. can you please fork https://github.com/Chiff/cloudstream-sync into cloudstream and replace one url in help.html pointing to https://chiff.github.io/cloudstream-sync/google-drive. When its ready i will also point app to correct urls

C10udburst · 2023-05-12T21:24:01Z

hey @C10udburst im marking the PR as ready to review. can you please fork Chiff/cloudstream-sync into cloudstream and replace one url in help.html pointing to https://chiff.github.io/cloudstream-sync/google-drive. When its ready i will also point app to correct urls

Done! https://recloudstream.github.io/cloudstream-sync/google-drive/

Co-Authored-By: Martin Filo <[email protected]>

Chiff · 2023-05-13T17:33:52Z

Done! https://recloudstream.github.io/cloudstream-sync/google-drive/

Great, thanks :) i have also updated https://github.com/Chiff/cloudstream-sync to contain new url

LagradOst · 2023-05-23T11:59:37Z

Actually really nice code, and it works. But I have some minor issues.

The icon does not have the icon attr so it will not show on flashbang
Backups are not incompatible and that is unacceptable, this wont be merged until it is fixed. but should be an easy fix
Very nice text tutorial, however it was quite a long process to set it up along with minor issues like a single / fucking it up or me not being a testing user stopping it from working. So a video would be really recommended for going over the process of setting it up or some form of semiofficial appid
I have not actually tested this on several devices, but many users would like to pair tv and phone for example, is it possible to make it easier for a tv user to fill in the data besides typing the bigass id? Also how good are cross device sync, does it break it completely?

Chiff · 2023-05-28T12:34:00Z

Hi @LagradOst ,

thank you very much for looking into this and identifying the problems and suggesting solutions. it is very helpful! :) I will make sure that I will test the icons and backup compatibility.

The first two are easy to fix and will be done.

with the other ones, there are two "problems" and I identify them as 1. performance and 2. user friendliness/ease of use

I cannot test performance on TV because I don't have one :D but I was considering the performance of the operations I added such as comparing JSONs, modifying preferences, and calling the methods I added. I aimed to use my sync methods as least as I could do with implementing scheduling and throttling logic. However, it would be nice if someone would try this on TV, and if there are any issues we can fix them.
That configuration is not really the most optimal solution and I am aware of that. When thinking of the implementation I considered using some free DB or API. Google provides some DBs but using file sync is "easier" because we can reuse the existing backup API of CS and that's why I went with this solution (DB could still be configured later and I think there is a sufficient foundation to work on).
- For now, let's stick with file sync and using Google. With that, we have multiple options and what was implemented is not the most user-friendly but necessary because there is no infrastructure we can share those keys.
- Let's talk about moving access tokens first. Because every user has to create their own project we cannot move access tokens into the app hence using an n-code pin with TVs because every installation of the CS is standalone and does not know about other installations of the same user. Potentially we could do some network discovery but I think it's over-engineering. So with the current setup users should hope that their TV is able to share a clipboard with their phone or laptop.
- There is still HOPE! :) I would suggest another way to log in as an alternative feature to sync with google drive. Users will have an option to "DIY" (as it is right now - own Google project, custom redirect URL) and the other would be provided by the CS app. When I was considering possible implementations I stumbled across Google's "one-tap login" feature. I tried it and it worked seamlessly. However (😞) I wasn't able to bind it to a custom Google console project. Then, after your comment, I found out that there is a library that can do it while still using Google's Android API. This would mean that you or any CS member who manages these things would create one project for all CS users and they would just use a one-tap login with no setup required! But it has a caveat... there is a user limit on the API. I'm not sure what is your stance on payments for the app but this would be whole another problem we would need to tackle or alternatively, some kind of sponsorship to pay for Google's services.

If there is any discussion needed I would be happy to talk it out through Discord (Chiffie#4015)

NiceSapien · 2023-06-07T12:02:37Z

For the TV's test, you can try downloading a emulator on your PC or connect a keyboard to your existing Android device and try navigating through the keyboard, but without using the alphabets and letters as they are not available to TVs.

Moreover, this setup looks very complex. If you could add some provider, other than GDrive would be much cooler. Otherwise it's great for now👍

…ersions

NiceSapien · 2023-06-19T17:56:43Z

Is this complete?

# Conflicts: # app/src/main/java/com/lagradost/cloudstream3/ui/player/GeneratorPlayer.kt # app/src/main/res/values/strings.xml

LagradOst · 2023-07-31T23:05:59Z

This will need to me moved to viewbindings

Chiff · 2023-07-31T23:19:22Z

I'm not sure what are you talking about. Is that the new merge conflicts?

During this weekend I plan to look on the url shortening for easier sharing on TVs and then I'll try to resolve the conflicts.

If it is indeed connected to the conflicts, can you share some commit hashes so I can orient myself better? Tbh I'll need to study a little on that matter

LagradOst · 2023-07-31T23:22:03Z

I'm not sure what are you talking about. Is that the new merge conflicts?

During this weekend I plan to look on the url shortening for easier sharing on TVs and then I'll try to resolve the conflicts.

If it is indeed connected to the conflicts, can you share some commit hashes so I can orient myself better? Tbh I'll need to study a little on that matter

TLDR: ALL the ui code needs to be changed as cloudstream moved from synthetic to viewbindings. so I wont merge any code that has synthetic. and yes that is why you have 1000 merge conflicts

LagradOst · 2023-07-31T23:23:32Z

check out #507 for more info

Chiff · 2023-07-31T23:25:54Z

Thanks for the info, will change that :)

Chiff · 2023-09-17T17:11:29Z

I don't have time for this anymore. I will leave my repo visible and if anyone wants to continue the implementation.

CranberrySoup · 2023-09-17T17:53:40Z

I will try to continue it 🫂

WellCodeIsDelicious · 2023-09-17T18:15:00Z

Good soup 🙏

Chiff added 6 commits April 3, 2023 21:43

feat: add remote sync capability - POC

1122137

feat: add remote sync capability - fix build.gradle.kts

100d3da

feat: add remote sync capability - fix build.gradle.kts, remove unuse…

d50fbe5

…d strings

feat: add remote sync capability - update scheduling logic

5ec4439

feat: add remote sync capability - improve downloading

92578e5

feat: add remote sync capability - update notes

b77aecf

C10udburst reviewed Apr 8, 2023

View reviewed changes

Chiff and others added 12 commits April 21, 2023 23:28

Merge branch 'master' into feature/remote-sync

3d53ac6

feat: add remote sync capability - refactor and improve syncing when …

f4dfd2f

…multiple devices are online

feat: add remote sync capability - refactor and improve syncing when …

e1e039b

…multiple devices are online

feat: add remote sync capability - refactor, refresh ui on restore an…

fe36b69

…d improve data restore (pt.2)

feat: add remote sync capability - schedule updates only when value w…

7145bef

…as really changed (true -> true is not a change)

Merge branch 'recloudstream:master' into feature/remote-sync

78b0cad

feat: add remote sync capability - run upload on activity destroy

842ac5f

feat: add remote sync capability - update sync meta only if backup ap…

13978f8

…i is active

feat: add remote sync capability - fix file exists check

de3019b

feat: add remote sync capability - fix sync/restore logic

dee51d8

feat: add remote sync capability - downloader will now wait when uplo…

0405cdc

…ad scheduler is throttling job

feat: add remote sync capability - add info button in login dialog

13d9cee

Chiff marked this pull request as ready for review May 8, 2023 18:27

feat: add remote sync capability - add info button in login dialog

2557d82

Co-Authored-By: Martin Filo <[email protected]>

C10udburst requested a review from LagradOst May 23, 2023 08:20

nikidziuba approved these changes May 23, 2023

View reviewed changes

fix: add remote sync capability - fix icon for flashbang theme

4b28bf9

fix: add remote sync capability - fix icon backup import from older v…

05705b1

…ersions

LagradOst mentioned this pull request Jun 17, 2023

Add Firebase with Google Authenication to Backup Data #487

Closed

3 tasks

Merge branch 'master' into feature/remote-sync

0fb2e68

# Conflicts: # app/src/main/java/com/lagradost/cloudstream3/ui/player/GeneratorPlayer.kt # app/src/main/res/values/strings.xml

ccandour mentioned this pull request Aug 12, 2023

Provide account sync functionality. #549

Open

3 tasks

Chiff closed this Sep 17, 2023

CranberrySoup mentioned this pull request Sep 22, 2023

Remote sync #623

Open

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add remote sync capability - POC #442

Add remote sync capability - POC #442

Chiff commented Apr 3, 2023

C10udburst Apr 8, 2023

Chiff Apr 8, 2023 •

edited

Loading

Chiff commented May 8, 2023

C10udburst commented May 12, 2023

Chiff commented May 13, 2023

LagradOst commented May 23, 2023

Chiff commented May 28, 2023 •

edited

Loading

NiceSapien commented Jun 7, 2023

NiceSapien commented Jun 19, 2023

LagradOst commented Jul 31, 2023

Chiff commented Jul 31, 2023

LagradOst commented Jul 31, 2023 •

edited

Loading

LagradOst commented Jul 31, 2023

Chiff commented Jul 31, 2023

Chiff commented Sep 17, 2023

CranberrySoup commented Sep 17, 2023

WellCodeIsDelicious commented Sep 17, 2023

Add remote sync capability - POC #442

Add remote sync capability - POC #442

Conversation

Chiff commented Apr 3, 2023

C10udburst Apr 8, 2023

Choose a reason for hiding this comment

Chiff Apr 8, 2023 • edited Loading

Choose a reason for hiding this comment

Chiff commented May 8, 2023

C10udburst commented May 12, 2023

Chiff commented May 13, 2023

LagradOst commented May 23, 2023

Chiff commented May 28, 2023 • edited Loading

NiceSapien commented Jun 7, 2023

NiceSapien commented Jun 19, 2023

LagradOst commented Jul 31, 2023

Chiff commented Jul 31, 2023

LagradOst commented Jul 31, 2023 • edited Loading

LagradOst commented Jul 31, 2023

Chiff commented Jul 31, 2023

Chiff commented Sep 17, 2023

CranberrySoup commented Sep 17, 2023

WellCodeIsDelicious commented Sep 17, 2023

Chiff Apr 8, 2023 •

edited

Loading

Chiff commented May 28, 2023 •

edited

Loading

LagradOst commented Jul 31, 2023 •

edited

Loading