Exporter: Incremental export of notebooks, SQL objects and some other resources

[alexott]

Changes

This draft PR is providing implementation of incremental export of notebooks, SQL objects (dashboards, alerts, queries) and some other resources.

Example of usage:

./terraform-provider-databricks exporter -debug -directory output -skip-interactive \ 
  -listing sql-queries,sql-dashboards,sql-alerts 
  -incremental -updated-since '2023-07-24T00:00:00Z'

Current limitations

• when modified resource has dependencies they are exported as well. No check (yet) if it's modified or not (we don't store last modified date for dependent objects yet). Also, all resources that don't have last modified date will be exported.
• there is no way for detecting deletion of objects - workaround would be to do a periodic full export.

Implementation

Two new options were added - -incremental to trigger incremental behavior, and -updated-since to specify a timestamp (see TODOs).

When incremental mode is set, then list operations of objects having last modified timestamp will do the check and emit only objects that are modified at or after a given timestamp.

When all resources are exported, the new set of resources is merged with following objects:

• existing TF resources are loaded from the corresponding source files, and that resources that aren't a part of new export are added to corresponding hclwrite.File object (see context.updateExportedWithIncrementals function)
• existing variables are also merged with new variables (see context.generateVariables function)
• existing import.sh is analyzed, and its content is merged with new resources.

TODOs

• Add tests
• Make sure that all resources that have "last modified date" are exported (global init scripts, IP Access Lists, etc.)
• store last run timestamp in some configuration file inside the output diretory and load from it if -updated-since isn't provided
• (to discuss) should we store last modified date in some configuration file that will be loaded? Like, to avoid re-downloading of notebooks if DLT pipeline is modified and notebook isn't...

Tests

No tests yet, will be added as we proceed with code review

• make test run locally
• relevant change in docs/ folder
• covered with integration tests in internal/acceptance
• relevant acceptance tests are passing
• using Go SDK

[codecov-commenter]

Codecov Report

Merging #2563 (5551c11) into master (53e2035) will decrease coverage by 0.55%.
The diff coverage is 68.20%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2563      +/-   ##
==========================================
- Coverage   88.17%   87.63%   -0.55%     
==========================================
  Files         145      145              
  Lines       12071    12264     +193     
==========================================
+ Hits        10644    10747     +103     
- Misses        937     1012      +75     
- Partials      490      505      +15     

Files Changed	Coverage Δ
exporter/model.go	91.66% <ø> (ø)
pipelines/resource_pipeline.go	92.37% <ø> (ø)
sql/api/query.go	75.40% <ø> (ø)
workspace/resource_notebook.go	88.88% <ø> (ø)
exporter/importables.go	86.74% <60.60%> (-2.19%)	⬇️
exporter/context.go	78.06% <69.67%> (-5.77%)	⬇️
exporter/util.go	86.41% <70.00%> (-1.88%)	⬇️
exporter/command.go	81.39% <100.00%> (+0.67%)	⬆️
sql/resource_sql_alerts.go	33.69% <100.00%> (+1.47%)	⬆️
sql/resource_sql_dashboard.go	89.09% <100.00%> (+0.41%)	⬆️
... and 1 more

[nfx]

can you also parse durations, e.g. 1w, 1M, 1d, 1h - the equivalents of time.Duration.String.

[alexott]

Maybe, let see if it will be required - I want to get feedback from customer on UX. But really, the duration here doesn't matter much, maybe except the case when it's scheduled from cron, but then it should be provided via updated-since parameter, not from file

[pietern]

I recommend making incremental the default mode and doing a full export with updatedSinceMs == 0. That would save quite a bit of branching and debugging easier.

[pietern]

Not closed in the err path (may be problematic on Windows).

defer shFile.close()

[alexott]

but shFile will be non-nil only if err == nil

Question is answered

[alexott]

@mgyucht, can you please start a new merge? For some reason, one of the tests began to fail. It's now fixed