citus packages include columnar and thus conflict with hydra

[Vonng]

What's wrong?

It seems that Hydra's columnar is using the same name as citus columnar

This may introduce a problem: you can not install hydra and citus packages simultaneously.

It's okay to choose one on the database CREATE EXTENSION level, but conflict during installation may be unnecessary.

  file /usr/pgsql-15/share/extension/columnar--10.0-3--10.1-1.sql conflicts between attempted installs of hydra_15-1.0.0-1.el8.x86_64 and citus_15-12.0.0-1PGDG.rhel8.x86_64
  file /usr/pgsql-15/share/extension/columnar--10.1-1--10.2-1.sql conflicts between attempted installs of hydra_15-1.0.0-1.el8.x86_64 and citus_15-12.0.0-1PGDG.rhel8.x86_64
  file /usr/pgsql-15/share/extension/columnar--10.2-2--10.2-3.sql conflicts between attempted installs of hydra_15-1.0.0-1.el8.x86_64 and citus_15-12.0.0-1PGDG.rhel8.x86_64
  file /usr/pgsql-15/share/extension/columnar--10.2-3--10.2-4.sql conflicts between attempted installs of hydra_15-1.0.0-1.el8.x86_64 and citus_15-12.0.0-1PGDG.rhel8.x86_64
  file /usr/pgsql-15/share/extension/columnar--9.5-1--10.0-1.sql conflicts between attempted installs of hydra_15-1.0.0-1.el8.x86_64 and citus_15-12.0.0-1PGDG.rhel8.x86_64
  file /usr/pgsql-15/include/server/citus_version.h conflicts between attempted installs of citus_15-devel-12.0.0-1PGDG.rhel8.x86_64 and hydra_15-1.0.0-1.el8.x86_64

[wuputah]

Hi there- the core of the problem here Hydra is a fork of Citus columnar so it makes sense there is an overlap here, and not something I think we can solve. It would be nice if Citus would separate the packages you are using there into Citus distributed and Citus columnar, that way you could still have Citus distributed without Citus columnar.

That said, you should be to overwrite the Citus columnar files with Hydra files without issue.

[amalek215]

Given this issue how would one test side by side Citus columnar and Hydra columnar?

Or more importantly how would one migrate from Citus columnar to Hydra?

In my use case going from Citus columnar to heap (thereby loosing all compression) and removing Citus before installing Hydra is not an option due to space constraints (would require an additional 400TB).

[wuputah]

Maintaining compatibility with or migrating from Citus Columnar directly (i.e. physically) within a single database is not a supported use case, so while it might work with a bit of fiddling, it's not something we are trying to support. The data should move through a logical backup or, as you suggested, could be done through heap tables. Because of the name/file conflicts, the two extensions cannot be installed side-by-side in the same database, so they can only be compared via separate installations of Postgres.

would require an additional 400TB

That's a lot of data! What is your compressed data size?

[amalek215]

That's a lot of data! What is your compressed data size?

67TB at a 9.18 compression rate for ~29,000 tables (plus an additional 20% compression from ZFS lz4 file system compression)

As excited as I am to test Hydra if there is no migration that does not include full removal of Citus / decompression of all our data then it's a nonstarter for us.

[wuputah]

You can vote on issues using reactions (like 👍 ), but posting "+1" isn't polite Github behavior. I've removed these to discourage this.

[tunatoksoz]

I wish hydra columnar extension was originally prefixed with hydra_. That'd have prevented the collision. I wonder if there's a path to rename one or the other. I am in the same boat with @Vonng

[winrid]

@wuputah so you can't officially scale Hydra horizontally with Citus? That would be a powerful combination!