feat: add of offload threshold property in namespace configuration (#109

) * feat: add of offload threshold property in namespace configuration * doc(readme): add of doc to the new offload_threshold_size_mb property in the namespace_config * fix: using int instead of int64 to marshal and unmarshal OffloadThresoldSizeInMb * doc: updated doc to include new namespace threshold param --------- Co-authored-by: Mattias Santoro <[email protected]>
streamnative · May 1, 2024 · 9f30205 · 9f30205
1 parent 969d5ef
commit 9f30205
Show file tree

Hide file tree

Showing 12 changed files with 152 additions and 41 deletions.
diff --git a/README.md b/README.md
@@ -281,6 +281,7 @@ namespace_config nested schema
 | `replication_clusters`           | List of replication clusters for the namespace | No       |
 | `schema_compatibility_strategy`  | Set schema compatibility strategy              | No       |
 | `schema_validation_enforce`      | Enable or disable schema validation            | No       |
+| `offload_threshold_size_in_mb`   | Set topic offload threshold size in MB         | No       |
 
 The `schema_compatibility_strategy` can take the following values:
 

diff --git a/docs/index.md b/docs/index.md
@@ -8,17 +8,19 @@ description: |-
 
 # pulsar Provider
 
-<!-- schema generated by tfplugindocs -->
 
+
+
+
+<!-- schema generated by tfplugindocs -->
 ## Schema
 
 ### Optional
 
 - `api_version` (String) Api Version to be used for the pulsar admin interaction
 - `audience` (String) The OAuth 2.0 resource server identifier for the Pulsar cluster
 - `client_id` (String) The OAuth 2.0 client identifier
-- `issuer_url` (String) The OAuth 2.0 URL of the authentication provider which allows the Pulsar client to obtain an
-  access token
+- `issuer_url` (String) The OAuth 2.0 URL of the authentication provider which allows the Pulsar client to obtain an access token
 - `key_file_path` (String) The path of the private key file
 - `scope` (String) The OAuth 2.0 scope(s) to request
 - `tls_allow_insecure_connection` (Boolean) Boolean flag to accept untrusted TLS certificates

diff --git a/docs/resources/cluster.md b/docs/resources/cluster.md
@@ -8,22 +8,23 @@ description: |-
 
 # pulsar_cluster (Resource)
 
-<!-- schema generated by tfplugindocs -->
 
+
+
+
+<!-- schema generated by tfplugindocs -->
 ## Schema
 
 ### Required
 
 - `cluster` (String) Name of the cluster
-- `cluster_data` (Block Set, Min: 1, Max: 1) Specific configs of this cluster (
-  see [below for nested schema](#nestedblock--cluster_data))
+- `cluster_data` (Block Set, Min: 1, Max: 1) Specific configs of this cluster (see [below for nested schema](#nestedblock--cluster_data))
 
 ### Read-Only
 
 - `id` (String) The ID of this resource.
 
 <a id="nestedblock--cluster_data"></a>
-
 ### Nested Schema for `cluster_data`
 
 Required:

diff --git a/docs/resources/function.md b/docs/resources/function.md
@@ -0,0 +1,65 @@
+---
+# generated by https://github.com/hashicorp/terraform-plugin-docs
+page_title: "pulsar_function Resource - terraform-provider-pulsar"
+subcategory: ""
+description: |-
+  
+---
+
+# pulsar_function (Resource)
+
+
+
+
+
+<!-- schema generated by tfplugindocs -->
+## Schema
+
+### Required
+
+- `name` (String) The name of the function.
+- `namespace` (String) The namespace of the function.
+- `tenant` (String) The tenant of the function.
+
+### Optional
+
+- `auto_ack` (Boolean) Whether to automatically acknowledge messages processed by the function.
+- `classname` (String) The class name of the function.
+- `cleanup_subscription` (Boolean) Whether to clean up subscription when the function is deleted.
+- `cpu` (Number) The CPU that needs to be allocated per function instance
+- `custom_runtime_options` (String) The custom runtime options of the function.
+- `custom_schema_inputs` (Map of String) The custom schema inputs of the function.
+- `custom_schema_outputs` (Map of String) The custom schema outputs of the function.
+- `custom_serde_inputs` (Map of String) The custom serde inputs of the function.
+- `dead_letter_topic` (String) The dead letter topic of the function.
+- `disk_mb` (Number) The disk that need to be allocated per function instance
+- `forward_source_message_property` (Boolean) Whether to forward source message property to the function output message.
+- `go` (String) The path to the go file.
+- `input_type_classname` (String) The input type class name of the function.
+- `inputs` (Set of String) The input topics of the function.
+- `jar` (String) The path to the jar file.
+- `log_topic` (String) The log topic of the function.
+- `max_message_retries` (Number) The maximum number of times that a message will be retried when the function is configured with `EFFECTIVELY_ONCE` processing guarantees.
+- `output` (String) The output topic of the function.
+- `output_schema_type` (String) The output schema type of the function.
+- `output_serde_classname` (String) The output serde class name of the function.
+- `output_type_classname` (String) The output type class name of the function.
+- `parallelism` (Number) The parallelism of the function.
+- `processing_guarantees` (String) The processing guarantees (aka delivery semantics) applied to the function. Possible values are `ATMOST_ONCE`, `ATLEAST_ONCE`, and `EFFECTIVELY_ONCE`.
+- `py` (String) The path to the python file.
+- `ram_mb` (Number) The RAM that need to be allocated per function instance
+- `retain_key_ordering` (Boolean) Whether to retain key ordering when the function is restarted after failure.
+- `retain_ordering` (Boolean) Whether to retain ordering when the function is restarted after failure.
+- `secrets` (String) The secrets of the function.
+- `skip_to_latest` (Boolean) Whether to skip to the latest position when the function is restarted after failure.
+- `subscription_name` (String) The subscription name of the function.
+- `subscription_position` (String) The subscription position of the function. Possible values are `LATEST`, `EARLIEST`, and `CUSTOM`.
+- `timeout_ms` (Number) The timeout of the function in milliseconds.
+- `topics_pattern` (String) The input topics pattern of the function. The pattern is a regex expression. The function consumes from all topics matching the pattern.
+- `user_config` (Map of String) User-defined config key/values
+
+### Read-Only
+
+- `id` (String) The ID of this resource.
+
+
diff --git a/docs/resources/namespace.md b/docs/resources/namespace.md
@@ -8,8 +8,11 @@ description: |-
 
 # pulsar_namespace (Resource)
 
-<!-- schema generated by tfplugindocs -->
 
+
+
+
+<!-- schema generated by tfplugindocs -->
 ## Schema
 
 ### Required
@@ -36,7 +39,6 @@ description: |-
 - `id` (String) The ID of this resource.
 
 <a id="nestedblock--backlog_quota"></a>
-
 ### Nested Schema for `backlog_quota`
 
 Required:
@@ -46,8 +48,8 @@ Required:
 - `policy` (String)
 - `type` (String)
 
-<a id="nestedblock--dispatch_rate"></a>
 
+<a id="nestedblock--dispatch_rate"></a>
 ### Nested Schema for `dispatch_rate`
 
 Required:
@@ -67,7 +69,6 @@ Required:
 - `rate_period_seconds` (Number)
 
 <a id="nestedblock--namespace_config"></a>
-
 ### Nested Schema for `namespace_config`
 
 Optional:
@@ -78,21 +79,22 @@ Optional:
 - `max_consumers_per_topic` (Number)
 - `max_producers_per_topic` (Number)
 - `message_ttl_seconds` (Number)
+- `offload_threshold_size_in_mb` (Number)
 - `replication_clusters` (List of String)
 - `schema_compatibility_strategy` (String)
 - `schema_validation_enforce` (Boolean)
 
-<a id="nestedblock--permission_grant"></a>
 
+<a id="nestedblock--permission_grant"></a>
 ### Nested Schema for `permission_grant`
 
 Required:
 
 - `actions` (Set of String)
 - `role` (String)
 
-<a id="nestedblock--persistence_policies"></a>
 
+<a id="nestedblock--persistence_policies"></a>
 ### Nested Schema for `persistence_policies`
 
 Required:
@@ -102,8 +104,8 @@ Required:
 - `bookkeeper_write_quorum` (Number)
 - `managed_ledger_max_mark_delete_rate` (Number)
 
-<a id="nestedblock--retention_policies"></a>
 
+<a id="nestedblock--retention_policies"></a>
 ### Nested Schema for `retention_policies`
 
 Required:

diff --git a/docs/resources/sink.md b/docs/resources/sink.md
@@ -8,18 +8,18 @@ description: |-
 
 # pulsar_sink (Resource)
 
-<!-- schema generated by tfplugindocs -->
 
+
+
+
+<!-- schema generated by tfplugindocs -->
 ## Schema
 
 ### Required
 
-- `archive` (String) Path to the archive file for the sink. It also supports
-  url-path [http/https/file (file protocol assumes that file already exists on worker host)] from which worker can
-  download the package
+- `archive` (String) Path to the archive file for the sink. It also supports url-path [http/https/file (file protocol assumes that file already exists on worker host)] from which worker can download the package
 - `auto_ack` (Boolean) Whether or not the framework will automatically acknowledge messages
-- `cleanup_subscription` (Boolean) Whether the subscriptions the functions created/used should be deleted when the
-  functions was deleted
+- `cleanup_subscription` (Boolean) Whether the subscriptions the functions created/used should be deleted when the functions was deleted
 - `name` (String) The sink's name
 - `namespace` (String) The sink's namespace
 - `tenant` (String) The sink's tenant
@@ -32,19 +32,21 @@ description: |-
 - `custom_runtime_options` (String) A string that encodes options to customize the runtime
 - `custom_schema_inputs` (Map of String) The map of input topics to Schema types or class names (as a JSON string)
 - `custom_serde_inputs` (Map of String) The map of input topics to SerDe class names (as a JSON string)
+- `dead_letter_topic` (String) Name of the dead topic where the failing messages will be sent
 - `disk_mb` (Number) The disk that need to be allocated per sink instance (applicable only to Docker runtime)
 - `input_specs` (Block Set) The map of input topics specs (see [below for nested schema](#nestedblock--input_specs))
 - `inputs` (Set of String) The sink's input topics
+- `max_redeliver_count` (Number) Maximum number of times that a message will be redelivered before being sent to the dead letter topic
+- `negative_ack_redelivery_delay_ms` (Number) The negative ack message redelivery delay in milliseconds
 - `parallelism` (Number) The sink's parallelism factor
-- `processing_guarantees` (String) Define the message delivery semantics, default to ATLEAST_ONCE (ATLEAST_ONCE,
-  ATMOST_ONCE, EFFECTIVELY_ONCE)
-- `ram_mb` (Number) The RAM that need to be allocated per sink instance (applicable only to the process and Docker
-  runtimes)
+- `processing_guarantees` (String) Define the message delivery semantics, default to ATLEAST_ONCE (ATLEAST_ONCE, ATMOST_ONCE, EFFECTIVELY_ONCE)
+- `ram_mb` (Number) The RAM that need to be allocated per sink instance (applicable only to the process and Docker runtimes)
+- `retain_key_ordering` (Boolean) Sink consumes and processes messages in key order
 - `retain_ordering` (Boolean) Sink consumes and sinks messages in order
-- `subscription_name` (String) Pulsar source subscription name if user wants a specific subscription-name for
-  input-topic consumer
-- `subscription_position` (String) Pulsar source subscription position if user wants to consume messages from the
-  specified location (Latest, Earliest). Default to Earliest.
+- `secrets` (String) The map of secretName to an object that encapsulates how the secret is fetched by the underlying secrets provider
+- `sink_type` (String) The sinks's connector provider
+- `subscription_name` (String) Pulsar source subscription name if user wants a specific subscription-name for input-topic consumer
+- `subscription_position` (String) Pulsar source subscription position if user wants to consume messages from the specified location (Latest, Earliest). Default to Earliest.
 - `timeout_ms` (Number) The message timeout in milliseconds
 - `topics_pattern` (String) TopicsPattern to consume from list of topics under a namespace that match the pattern
 
@@ -53,7 +55,6 @@ description: |-
 - `id` (String) The ID of this resource.
 
 <a id="nestedblock--input_specs"></a>
-
 ### Nested Schema for `input_specs`
 
 Required:

diff --git a/docs/resources/source.md b/docs/resources/source.md
@@ -8,33 +8,45 @@ description: |-
 
 # pulsar_source (Resource)
 
-<!-- schema generated by tfplugindocs -->
 
+
+
+
+<!-- schema generated by tfplugindocs -->
 ## Schema
 
 ### Required
 
-- `archive` (String) The path to the NAR archive for the Source. It also supports
-  url-path [http/https/file (file protocol assumes that file already exists on worker host)] from which worker can
-  download the package
+- `archive` (String) The path to the NAR archive for the Source. It also supports url-path [http/https/file (file protocol assumes that file already exists on worker host)] from which worker can download the package
 - `destination_topic_name` (String) The Pulsar topic to which data is sent
 - `name` (String) The source's name
 - `namespace` (String) The source's namespace
 - `tenant` (String) The source's tenant
 
 ### Optional
 
+- `batch_builder` (String) BatchBuilder provides two types of batch construction methods, DEFAULT and KEY_BASED.
 - `classname` (String) The source's class name if archive is file-url-path (file://)
+- `compression_type` (String) Set the compression type for the producer. By default, message payloads are not compressed. Supported compression types are: LZ4, ZLIB, ZSTD, SNAPPY and NONE
 - `configs` (String) User defined configs key/values (JSON string)
+- `consumer_crypto_failure_action` (String) The desired action if consumer fail to decrypt data, one of FAIL, DISCARD, CONSUME
 - `cpu` (Number) The CPU that needs to be allocated per source instance (applicable only to Docker runtime)
+- `crypto_key_reader_classname` (String) The classname for the crypto key reader that can be used to access the keys in the keystore
+- `crypto_key_reader_config` (String) The config for the crypto key reader that can be used to access the keys in the keystore
+- `custom_runtime_options` (String) A string that encodes options to customize the runtime, see docs for configured runtime for details
 - `deserialization_classname` (String) The SerDe classname for the source
 - `disk_mb` (Number) The disk that need to be allocated per source instance (applicable only to Docker runtime)
+- `encryption_keys` (Set of String) One or more public keys to encrypt data key. It can be used to encrypt data key with multiple keys.
+- `max_pending_messages` (Number) The maximum size of a queue holding pending messages
+- `max_pending_messages_across_partitions` (Number) The maximum number of pending messages across partitions
 - `parallelism` (Number) The source's parallelism factor
-- `processing_guarantees` (String) Define the message delivery semantics, default to ATLEAST_ONCE (ATLEAST_ONCE,
-  ATMOST_ONCE, EFFECTIVELY_ONCE)
-- `ram_mb` (Number) The RAM that need to be allocated per source instance (applicable only to the process and Docker
-  runtimes)
+- `processing_guarantees` (String) Define the message delivery semantics, default to ATLEAST_ONCE (ATLEAST_ONCE, ATMOST_ONCE, EFFECTIVELY_ONCE)
+- `producer_crypto_failure_action` (String) The desired action if producer fail to encrypt data, one of FAIL, SEND
+- `ram_mb` (Number) The RAM that need to be allocated per source instance (applicable only to the process and Docker runtimes)
 - `runtime_flags` (String) User defined configs key/values (JSON string)
+- `schema_type` (String) The schema type (either a builtin schema like 'avro', 'json', etc.. or custom Schema class name to be used to encode messages emitted from the source
+- `secrets` (String) The map of secretName to an object that encapsulates how the secret is fetched by the underlying secrets provider
+- `use_thread_local_producers` (Boolean) Whether to use thread local producers
 
 ### Read-Only
 

diff --git a/docs/resources/tenant.md b/docs/resources/tenant.md
@@ -8,8 +8,11 @@ description: |-
 
 # pulsar_tenant (Resource)
 
-<!-- schema generated by tfplugindocs -->
 
+
+
+
+<!-- schema generated by tfplugindocs -->
 ## Schema
 
 ### Required

diff --git a/docs/resources/topic.md b/docs/resources/topic.md
@@ -8,8 +8,11 @@ description: |-
 
 # pulsar_topic (Resource)
 
-<!-- schema generated by tfplugindocs -->
 
+
+
+
+<!-- schema generated by tfplugindocs -->
 ## Schema
 
 ### Required
@@ -30,16 +33,15 @@ description: |-
 - `id` (String) The ID of this resource.
 
 <a id="nestedblock--permission_grant"></a>
-
 ### Nested Schema for `permission_grant`
 
 Required:
 
 - `actions` (Set of String)
 - `role` (String)
 
-<a id="nestedblock--retention_policies"></a>
 
+<a id="nestedblock--retention_policies"></a>
 ### Nested Schema for `retention_policies`
 
 Required: