v24.3.1

[piyushredpanda]

Features

• Added support for Iceberg Topics (various improvements below)
• New REST API for mounting/unmounting topics by @mmaslankaprv in #23167
• adds rpk cluster storage topic mount, unmount, list-mount, status-mount, cancel-mount by @gene-redpanda in #23575
• Add leadership pinning: ability to set preferred racks for topic partition leaders. To configure, set redpanda.leaders.preference topic config property or default_leaders_preference cluster config property. by @ztlpn in #23691
• Enable node_local_core_assignment feature by default by @ztlpn in #23453
• Adds Schema Registry support for the JavaScript Data Transforms SDK by @oleiman in #21491
• Adds list-mountable to allow listing mountable topics by @gene-redpanda in #23924
• Adds the topic property delete.retention.ms, as well as the cluster property tombstone_retention_ms. Configuring these allow for the removal of tombstone records in compacted topics with tiered storage disabled in redpanda. by @WillemKauf in #23662
• Schema Registry: Support normalize=true by @BenPope in #22519
• Schema Registry: added support for the "verbose" query parameter on the schema compatibility checker endpoint by @pgellert in #22877
• Schema Registry: verbose compatibility error reporting is now supported for JSON as well by @pgellert in #23208
• #17984 Adds a new broker configuration transaction_max_timeout_ms. The configuration controls the maximum allowed user set timeout for transactions. If a client requested transaction timeout exceeds this configuration, the broker will return
  an error during transactional producer initialization. This guardrail prevents hanging transactions from blocking consumer progress. The default value is 15mins. by @bharathv in #21504
• rpk: Add rpk registry mode to manage the schema registry mode. by @r-vasquez in #22675
• rpk: supports triggering on-demand partition balancer by @daisukebe in #22855
• Added support for using PKCS#12 files for TLS services by @michael-redpanda in #21313
• Adds admin API endpoint for enterprise feature info GET /v1/features/enterprise by @oleiman in #23314
• A new metric (cluster_features_enterprise_license_expiry_sec) is added for easier monitoring of the enterprise license's expiry time. by @pgellert in #23367
• After the cluster is first formed, a trial license is automatically loaded to provide an evaluation period of enterprise features. by @pgellert in #23893

Improvements

• --regex flag in rpk topic describe now supports internal topics. by @r-vasquez in #23487
• A number of optimizations to local storage compaction. by @WillemKauf in #23380
• Add an LRU caching layer to Rust transform SDK Schema Registry client by @oleiman in #19859
• Add support for differentiating tombstone records from empty-string value records in rpk produce and rpk consume. by @WillemKauf in #23264
• Added support for Metadata API v8 by @michael-redpanda in #22669
• Added vectorized_kafka_rpc_connections_rejected_rate_limit metric which counts incoming Kafka connections rejected due to the connection rate limit (if set), analogously to the existing vectorized_kafka_rpc_connections_rejected metric which counts rejected connections due to the hitting the open connection limit. by @travisdowns in #22803
• Adds a shard label to some consumer group metrics. by @ballard26 in #23339
• Adds support for setting schema registry connection parameters in the rpk stanza of redpanda.yaml. by @andrewstucki in #24017
• Adds the cloud_storage_backend::oracle value, and helps the s3_client properly configure for OCI storage. by @WillemKauf in #22902
• Adds the ability to configure Node UUID and ID overrides at broker startup. by @oleiman in #22972
• Allow rpk cluster self-test start to run, even in a cluster with mixed versions of redpanda (before and after cloudcheck addition in 24.2.x). by @WillemKauf in #21370
• Allows DeleteRecords requests from Kafka clients or rpk topic trim-prefix to be called with truncation_offset <= start_offset without returning an error. The request is instead treated as a no-op. by @WillemKauf in #22905
• Allows the self-test to be completely compatible with a mixed version cluster, in the case of a rolling upgrade. by @WillemKauf in #22831
• Deprecate leader_balancer_mode cluster config property. by @ztlpn in #23780
• Implements @redpanda-data/transform-sdk-sr.SchemaFormat for the WASM Transforms JS module by @oleiman in #23164
• Improve handling of boolean property values during a CreateTopics request by making parsing case-insensitive. by @WillemKauf in #23682
• Improve handling of boolean values during a CreateTopics request by no longer silently ignoring an invalid value, instead throwing a configuration error. by @WillemKauf in #23682
• Improve handling of certain invalid topic configuration parameters that would lead to a timeout failure instead of a graceful error code during a CreateTopics request. by @WillemKauf in #23682
• Improve property configuration descriptions. by @Deflaimun in #23347
• Minimizes data loss in recovery scenarios by @mmaslankaprv in #24071
• Reduce the memory overhead of many small segments. by @rockwotj in #22962
• Return core assignments from health report in /v1/cluster/partitions admin API output. by @ztlpn in #22695
• Schema Registry: 5 new compatibility checks are added for protobuf (ONEOF_FIELD_REMOVED, MULTIPLE_FIELDS_MOVED_TO_ONEOF, REQUIRED_FIELD_{ADDED,REMOVED}, FIELD_NAMED_TYPE_CHANGED, MESSAGE_REMOVED) by @pgellert in #22798
• Schema Registry: Improve AVRO Normalization by @BenPope in #22519
• Schema Registry: now reports more specific error messages for Avro and Protobuf schemas when they are incompatible with earlier schemas. by @pgellert in #22958
• Set the default value of topic_partitions_reserve_shard0 to zero. This means that we no longer weight shard 0 as if it has 2 more partitions than it actually has, leading to more even partition distribution in cases where the total number of partitions is close to the vCPU count. by @travisdowns in #22841
• The command line is now printed to the log at startup by the Redpanda process. by @travisdowns in #22826
• Upgrade data transforms tinygo compiler to version 0.34.0 by @rockwotj in #23969
• #17682 Schema Registry: Remove spurious log entry: No syntax specified for the proto file by @BenPope in #22633
• #21536 rpk topic describe-storage can be used now with internal topics. by @r-vasquez in #22338
• #22333 rpk debug bundle: include the result of uname -a by @JFlath in #22334
• #22666 Allows users to query the value of a cluster property with rpk cluster config get using either the original property name, or any of its aliases. Whereas before, rpk cluster config get using a property's aliased name would return a Property {} not found result. by @WillemKauf in #22674
• #23038 Schema Registry: Tolerate null metadata and ruleSet when registering or querying a schema by @BenPope in #23044
• rpk registry schema get will now fail if is incorrectly invoked using the --print-schema and --format flag. by @r-vasquez in #23886
• fixed large allocation in Raft implementation by @mmaslankaprv in #24002
• fixed spurious warning log entries that may be altering users by @mmaslankaprv in #22752
• optimized memory usage with large core count nodes by @mmaslankaprv in #22977
• rpk cluster license info now shows if your loaded license is expired by @r-vasquez in #23636
• rpk cluster license info: now includes more information about your license status and possible violations. by @r-vasquez in #23744
• rpk connect will be downloaded on the first run. by @r-vasquez in #23117
• rpk will now warn you when you are running enterprise features without a license. by @r-vasquez in #23472
• rpk: redpanda admin brokers list exposes Host/Port/Rack/UUID additionally by @daisukebe in #23595
• rpk: add --format JSON/YAML support to rpk topic list. by @gene-redpanda in #23806
• rpk: introduce rpk connect install, rpk connect upgrade, and rpk connect uninstall, manual ways to manage rpk connect versioning. by @r-vasquez in #23117
• tbd by @nvartolomei in #23381
• with this improvement the force reconfigured partitions will be backfilled from Tiered Storage even if they lost all local data by @mmaslankaprv in #22908

---

This discussion was created from the release v24.3.1.

[Deflaimun]

Find out what's new in docs at https://docs.redpanda.com/current/get-started/whats-new/