Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bulk Load CDK: Simply Interface & Add Check #45369

Merged
merged 1 commit into from
Sep 10, 2024

Conversation

johnny-schmidt
Copy link
Contributor

What

  • Simplify destination interface. Got rid of StreamLoaderFactory. Client must implement Destination (now called DestinationWrite).
  • Added DesitnationCheck (no default implementation yet)

@johnny-schmidt johnny-schmidt requested a review from a team as a code owner September 10, 2024 20:18
Copy link

vercel bot commented Sep 10, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment
Name Status Preview Comments Updated (UTC)
airbyte-docs ⬜️ Ignored (Inspect) Visit Preview Sep 10, 2024 10:44pm

@octavia-squidington-iii octavia-squidington-iii added the CDK Connector Development Kit label Sep 10, 2024
@@ -19,7 +19,9 @@ import kotlinx.coroutines.sync.withLock
* TODO: Some degree of logging/monitoring around how accurate we're actually being?
*/
@Singleton
class MemoryManager(availableMemoryProvider: AvailableMemoryProvider) {
class MemoryManager(
private val availableMemoryProvider: AvailableMemoryProvider
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This caused a micronaut bug -- the value wasn't getting injected without val

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

weird...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah live and learn

@Singleton
@Requires(property = Operation.PROPERTY, value = "check")
@Requires(env = ["destination"])
class CheckOperation(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any reason why we wouldn't want an abstract class here, with the check() as an abstract function

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The @Requires aren't inherited. It would force each implementor to specify the conditions under which their implementation is run.

@@ -19,7 +19,9 @@ import kotlinx.coroutines.sync.withLock
* TODO: Some degree of logging/monitoring around how accurate we're actually being?
*/
@Singleton
class MemoryManager(availableMemoryProvider: AvailableMemoryProvider) {
class MemoryManager(
private val availableMemoryProvider: AvailableMemoryProvider
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

weird...

*/
interface Destination {
interface DestinationWrite {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not in love with the name. DestinationWriteOperation?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NVM, There s a WriteOperation down below. Begs the question why we need both though

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See previous comment.

suspend fun teardown(succeeded: Boolean = true) {}
}

@Singleton
@Secondary
class DefaultDestination(private val streamLoaderFactory: StreamLoaderFactory) : Destination {
class DefaultDestinationWrite : DestinationWrite {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need a default that crashes? Might be worth explaining why

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't hurt to tell the implementor what's missing 🤷

@johnny-schmidt johnny-schmidt force-pushed the issue-9746/simplify-interface-add-check branch from fa5f5d8 to 8630887 Compare September 10, 2024 22:43
@johnny-schmidt johnny-schmidt merged commit b6825ee into master Sep 10, 2024
29 checks passed
@johnny-schmidt johnny-schmidt deleted the issue-9746/simplify-interface-add-check branch September 10, 2024 22:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CDK Connector Development Kit
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants