[Tour Of Beam] persistence_key for Pg::SaveSnippet #24287

eantyshev · 2022-11-21T14:26:32Z

This is to reuse user's snippets on Playground side

generate a random persistence_key so it would be hard to guess and rewrite via Pg::SaveSnippet for malicious third party
simulate Pg::GetSnippet call in integration_tests, to check how snippets are updated
fix trigger type and paths in ToB Examples CI

adresses #22691

spec

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

eantyshev · 2022-11-21T19:10:11Z

.github/workflows/tour_of_beam_examples_ci.yml

@@ -16,10 +16,10 @@
 name: Tour Of Beam Examples CI

 on:
-  push:
+  pull_request:
    paths:


this prevented ToB Examples CI from running

eantyshev · 2022-11-21T19:13:23Z

learning/tour-of-beam/backend/internal/service/persistence_key.go

+		[]string{os.Getenv("PERSISTENCE_KEY_SALT"), sdk.String(), unitId, uid},
+		"|")
+	_, err := h.Write([]byte(plainKey))
+	if err != nil {


failing to write to in-memory structure is bad enough to panic

github-actions · 2022-11-22T09:04:58Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @damccorm for label build.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

olehborysevych · 2022-11-22T15:13:23Z

learning/tour-of-beam/backend/README.md

 - set environment variables:
   * PROJECT_ID: GCP id
   * REGION: the region, "us-central1" fe
 - existing setup of Playground backend in a project
+ - create a secret `persistence_key_salt` in Secret Manager


@eantyshev what are the requirements for the key salt value?

thanks for bringing up, added to README

learning/tour-of-beam/backend/README.md

olehborysevych · 2022-11-22T15:25:59Z

learning/tour-of-beam/backend/README.md

-  --trigger-http --set-env-vars="DATASTORE_PROJECT_ID=$PROJECT_ID,GOOGLE_PROJECT_ID=$PROJECT_ID"
+  --trigger-http \
+  --set-env-vars="DATASTORE_PROJECT_ID=$PROJECT_ID,GOOGLE_PROJECT_ID=$PROJECT_ID" \
+  --set-secrets 'PERSISTENCE_KEY_SALT=persistence_key_salt:latest"


can we have some kind of fallback in case we will decide to go without a salt from secrets? just asking ))

If we don't set PERSISTENCE_KEY_SALT in postUserCode CF, then empty salt is used, and everything works but with some minor security risks:
It would be possible to calculate the persistence_key for another user as sha256(SDK, unitID, userID)
But Firebase User ID isn't exposed too, unless someone is already eavesdropping on another user's session. Quite a minor risk, maybe

But, if at some point we decide to set PERSISTENCE_KEY_SALT non-empty, there'll be no way to keep user progresses, so better to decide now

olehborysevych

LGTM (except one nit regarding README)

olehborysevych · 2022-11-23T12:07:08Z

learning/tour-of-beam/backend/README.md

@@ -101,7 +103,9 @@ gcloud datastore indexes create ./internal/storage/index.yaml
 for endpoint in getSdkList getContentTree getUnitComplete getUserProgress postUnitComplete postUserCode; do


@eantyshev could you please also fix getUnitComplete -> getUnitContent here please
and also TOB_LEARNING_PATH -> TOB_LEARNING_ROOT below

olehborysevych · 2022-11-23T15:29:16Z

R: @damccorm

github-actions · 2022-11-23T15:30:23Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

damccorm

Just one question, otherwise LGTM

learning/tour-of-beam/backend/internal/service/persistence_key.go

eantyshev · 2022-11-27T17:20:59Z

After some thought, decided to get rid of service-wide secret completely
First, having a system-wide secret is not a good practice owasp guidelines for storing passwords

Second, we don't have to re-caclulate persistence_key every call: we can calculate it once and store in tb_user_progress entity.
It's not only safer, but also removes over-complication in generatePersistenceKey, as it doesn't need to depend on (SDK, unitID, userID), if we choose a secure random source.

damccorm

SGTM

* restore CI * gen fix * mock_helper * tests * persistent_key * nit * README * README * review * no secrets

restore CI

9e8359f

github-actions bot added the build label Nov 21, 2022

gen fix

d339639

github-actions bot added the learning label Nov 21, 2022

eantyshev added 4 commits November 21, 2022 17:43

mock_helper

a5a2f90

tests

3e10462

persistent_key

de05a32

nit

3e7e7b6

eantyshev commented Nov 21, 2022

View reviewed changes

eantyshev marked this pull request as ready for review November 22, 2022 09:01

github-actions bot added the Next Action: Reviewers label Nov 22, 2022

README

90f0f55

olehborysevych reviewed Nov 22, 2022

View reviewed changes

README

c37c06f

eantyshev requested a review from olehborysevych November 23, 2022 11:10

olehborysevych approved these changes Nov 23, 2022

View reviewed changes

review

b56daae

damccorm reviewed Nov 23, 2022

View reviewed changes

learning/tour-of-beam/backend/internal/service/persistence_key.go Outdated Show resolved Hide resolved

no secrets

e9eb95c

eantyshev requested review from damccorm and olehborysevych and removed request for damccorm and olehborysevych November 28, 2022 06:24

damccorm approved these changes Nov 28, 2022

View reviewed changes

damccorm merged commit 0fe001e into apache:master Nov 28, 2022

ruslan-ikhsan pushed a commit to ruslan-ikhsan/beam that referenced this pull request Nov 30, 2022

[Tour Of Beam] persistence_key for Pg::SaveSnippet (apache#24287)

6ec3125

* restore CI * gen fix * mock_helper * tests * persistent_key * nit * README * README * review * no secrets

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Tour Of Beam] persistence_key for Pg::SaveSnippet #24287

[Tour Of Beam] persistence_key for Pg::SaveSnippet #24287

eantyshev commented Nov 21, 2022 •

edited

Loading

eantyshev Nov 21, 2022

eantyshev Nov 21, 2022 •

edited

Loading

github-actions bot commented Nov 22, 2022

olehborysevych Nov 22, 2022

eantyshev Nov 23, 2022

olehborysevych Nov 22, 2022

eantyshev Nov 22, 2022

olehborysevych left a comment

olehborysevych Nov 23, 2022

eantyshev Nov 23, 2022

olehborysevych commented Nov 23, 2022

github-actions bot commented Nov 23, 2022

damccorm left a comment

eantyshev commented Nov 27, 2022 •

edited

Loading

damccorm left a comment

		@@ -101,7 +103,9 @@ gcloud datastore indexes create ./internal/storage/index.yaml
		for endpoint in getSdkList getContentTree getUnitComplete getUserProgress postUnitComplete postUserCode; do

[Tour Of Beam] persistence_key for Pg::SaveSnippet #24287

[Tour Of Beam] persistence_key for Pg::SaveSnippet #24287

Conversation

eantyshev commented Nov 21, 2022 • edited Loading

GitHub Actions Tests Status (on master branch)

eantyshev Nov 21, 2022

Choose a reason for hiding this comment

eantyshev Nov 21, 2022 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Nov 22, 2022

olehborysevych Nov 22, 2022

Choose a reason for hiding this comment

eantyshev Nov 23, 2022

Choose a reason for hiding this comment

olehborysevych Nov 22, 2022

Choose a reason for hiding this comment

eantyshev Nov 22, 2022

Choose a reason for hiding this comment

olehborysevych left a comment

Choose a reason for hiding this comment

olehborysevych Nov 23, 2022

Choose a reason for hiding this comment

eantyshev Nov 23, 2022

Choose a reason for hiding this comment

olehborysevych commented Nov 23, 2022

github-actions bot commented Nov 23, 2022

damccorm left a comment

Choose a reason for hiding this comment

eantyshev commented Nov 27, 2022 • edited Loading

damccorm left a comment

Choose a reason for hiding this comment

eantyshev commented Nov 21, 2022 •

edited

Loading

eantyshev Nov 21, 2022 •

edited

Loading

eantyshev commented Nov 27, 2022 •

edited

Loading