(BQ Python) Pass project field from options or parameter when writing with dynamic destinations

[ahmedabu98]

Currently dynamic destinations cannot be passed in as <DATASET>.<TABLE>. For single destinations, the project ID is added from pipeline options earlier in the workflow. This isn't applied to dynamic destinations because the actual destinations are determined later.

These changes will provide the project ID at the moments when they are needed, ie. when submitting:

1. load jobs
2. schema update jobs
3. copy jobs
4. delete table operations

These changes will also give precedence to the project parameter in WriteToBigQuery over the project found in pipeline options. The reasoning is that a user may have one pipeline writing to tables in different projects. Reverted this part because it's not backwards compatible.

[ahmedabu98]

Run Python 3.8 PostCommit

[github-actions]

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @yeandy for label python.
R: @Abacn for label io.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[ahmedabu98]

Run Python 3.8 PostCommit

[codecov]

Codecov Report

Merging #23011 (a8925fb) into master (e83192c) will decrease coverage by 0.05%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master   #23011      +/-   ##
==========================================
- Coverage   73.66%   73.61%   -0.06%     
==========================================
  Files         713      716       +3     
  Lines       94970    95228     +258     
==========================================
+ Hits        69960    70100     +140     
- Misses      23709    23827     +118     
  Partials     1301     1301              

Flag	Coverage Δ
python	83.39% <100.00%> (-0.12%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
sdks/python/apache_beam/io/gcp/bigquery.py	74.24% <ø> (ø)
...s/python/apache_beam/io/gcp/bigquery_file_loads.py	87.85% <100.00%> (+0.14%)	⬆️
sdks/python/apache_beam/typehints/__init__.py	77.77% <0.00%> (-22.23%)	⬇️
...ks/python/apache_beam/runners/worker/statecache.py	89.69% <0.00%> (-6.47%)	⬇️
...python/apache_beam/runners/worker/worker_status.py	74.66% <0.00%> (-5.05%)	⬇️
...dks/python/apache_beam/metrics/monitoring_infos.py	92.50% <0.00%> (-4.50%)	⬇️
sdks/python/apache_beam/internal/pickler.py	92.00% <0.00%> (-3.46%)	⬇️
sdks/python/apache_beam/dataframe/schemas.py	96.62% <0.00%> (-1.05%)	⬇️
...hon/apache_beam/runners/worker/bundle_processor.py	93.17% <0.00%> (-0.38%)	⬇️
...ks/python/apache_beam/runners/worker/sdk_worker.py	88.62% <0.00%> (-0.32%)	⬇️
... and 25 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[ahmedabu98]

Run Python 3.8 PostCommit

[ahmedabu98]

Run Python 3.8 PostCommit

[ahmedabu98]

R: @johnjcasey

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

[ahmedabu98]

Run Python 3.8 PostCommit

[ahmedabu98]

Run Python 3.8 PostCommit

[ahmedabu98]

Run Python 3.8 PostCommit