Proton API Bridge

Thanks to Proton open sourcing proton-go-api and the web, iOS, and Android client codebases, we don't need to completely reverse engineer the APIs by observing the web client traffic!

proton-go-api provides the basic building blocks of API calls and error handling, such as 429 exponential back-off, but it is pretty much just a barebone interface to the Proton API. For example, the encryption and decryption of the Proton Drive file are not provided in this library.

This codebase, Proton API Bridge, bridges the gap, so software like rclone can be built on top of this quickly. This codebase handles the intricate tasks before and after calling Proton APIs, particularly the complex encryption scheme, allowing developers to implement features for other software on top of this codebase.

Currently, only Proton Drive APIs are bridged, as we are aiming to implement a backend for rclone.

Sidenotes

We are using a fork of the proton-go-api, as we are adding quite some new code to it. We are actively rebasing on top of the master branch of the upstream, as we will try to commit back to the upstream once we feel like the code changes are stable.

Unit testing and linting

golangci-lint run && go test -race -failfast -v ./...

Drive APIs

In collaboration with Azimjon Pulatov, in memory of our good old days at Meta, London, in the summer of 2022.

Thanks to Anson Chen for the motivation and some initial help on various matters!

Currently, the development are split into 2 versions. V1 supports the features required by rclone, such as file listing. As the unit and integration tests from rclone have all been passed, we would stabilize this and then move onto developing V2. V2 will bring in optimizations and enhancements, esp. supporting thumbnails. Please see the list below.

V1

Features

• Log in to an account without 2FA using username and password
• Obtain keyring
• Cache access token, etc. to be able to reuse the session
  • Bug: 403: Access token does not have sufficient scope - used the wrong newClient function
• Volume actions
  • List all volumes
• Share actions
  • Get all shares
  • Get default share
• Fix context with proper propagation instead of using ctx everywhere
• Folder actions
  • List all folders and files within the root folder
    • BUG: listing directory - missing signature when there are more than 1 share -> we need to check for the "active" folder type first
  • List all folders and files recursively within the root folder
  • Delete
  • Create
  • (Feature) Update
  • (Feature) Move
• File actions
  • Download
    • Download empty file
    • Improve large file download handling
    • Properly handle large files and empty files (check iOS codebase)
      • esp. large files, where buffering in-memory will screw up the runtime
    • Check signature and hash
  • Delete
  • Upload
    • Handle empty file
    • Parse mime type
    • Add revision
    • Modified time
    • Handle failed / interrupted upload
  • List file metadata
• Duplicated file name handling: 422: A file or folder with that name already exists (Code=2500, Status=422)
• Init ProtonDrive with config passed in as Map
• Remove all log.Fatalln and use proper error propagation (basically remove HandleError and we go from there)
• Integration tests
  • Remove drive demo code
  • Create a Drive struct to encapsulate all the functions (maybe?)
  • Move comments to proper places
  • Modify shouldRejectDestructiveActions()
  • Refactor
• Reduce config options on caching access token
• Remove integration test safeguarding

TODO

• address go dependencies
  • Fixed by doing the following in the go-proton-api repo to bump to use the latest commit
    • go get github.com/ProtonMail/go-proton-api@ea8de5f674b7f9b0cca8e3a5076ffe3c5a867e01
    • go get github.com/ProtonMail/gluon@fb7689b15ae39c3efec3ff3c615c3d2dac41cec8
• Remove mail-related apis (to reduce dependencies)
• Make a "super class" and expose all necessary methods for the outside to call
• Add 2FA login
• Fix the function argument passing (using pointers)
• Handle account with
  • multiple addresses
  • multiple keys per addresses
• Update RClone's contribution.md file
• Remove delete all's hardcoded string
• Point to the right proton-go-api branch
  • Run go get github.com/henrybear327/go-proton-api@dev to update go mod
• Pass in AppVersion as a config option
• Proper error handling by looking at the return code instead of the error string
  • Duplicated folder name handling: 422: A file or folder with that name already exists (Code=2500, Status=422)
  • Not found: ERROR RESTY 422: File or folder was not found. (Code=2501, Status=422), Attempt 1
  • Failed upload: Draft already exists on this revision (Code=2500, Status=409)
• Fix file upload progress -> If the upload failed, please Replace file. If the upload is still in progress, replacing it will cancel the ongoing upload
• Concurrency control on file encryption, decryption, and block upload

Known limitations

• No thumbnails, respecting accepted MIME types, max upload size, can't init Proton Drive, etc.
• Assumptions
  • only one main share per account
  • only operate on active links

V2

• Support thumbnail
• Potential bugs
  • Confirm the HMAC algorithm -> if you create a draft using integration test, and then use the web frontend to finish the upload (you will see overwrite pop-up), and then use the web frontend to upload again the same file, but this time you will have 2 files with duplicated names
  • Might have missing signature issues on some old accounts, e.g. GetHashKey on rootLink might fail -> currently have a quick patch, but might need to double check the behavior
  • Double check the attrs field parsing, esp. for size
  • Double check the attrs field, esp. for size
• Crypto-related operations, e.g. signature verification, still needs to cross check with iOS or web open source codebase
• Mimetype detection by using the file content itself, or Google content sniffer
• Remove e.g. proton.link related exposures in the function signature (this library should abstract them all)
• Improve documentation
• Go through Drive iOS source code and check the logic control flow
• File
  • Parallel download / upload -> enc/dec is expensive
  • Filename encoding
• Commit back to proton-go-api and switch to using upstream (make sure the tag is at the tip though)
• Support legacy 2-password mode
• Proton Drive init (no prior Proton Drive login before -> probably will have no key, volume, etc. to start with at all)
• linkID caching -> would need to listen to the event api though
• Integration tests
  • Check file metadata
  • Try to check if all functions are used at least once so we know if it's functioning or not
• Handle accounts with multiple shares
• Use CI to run integration tests
• Some error handling from here MAX_NAME_LENGTH, TIMEOUT
• Mimetype restrictions
• Address TODO and FIXME

Questions

• rclone's folder / file rename detection? -> just implement the interface and rclone will deal with the rest!

Notes

• Due to caching, functions using ...ByID needs to perform protonDrive.removeLinkIDFromCache(linkID, false) in order to get the latest data!