[exporter/clickhouse] create database fail if clickhouse cluster #36540

maniac1978 · 2024-11-26T08:23:28Z

Component(s)

exporter/clickhouse

What happened?

Description

OTel collector can't start with clickhouse exporter when clickhouse cluster in config and the database doesn't exist.
OTel collector starts successfully if there is a standalone clickhouse server in the configuration, the database is successfully created.
If clickhouse cluster:
Error: cannot start pipelines: create database: code: 81, message: Database otel does not exist
collector server run finished with error: cannot start pipelines: create database: code: 81, message: Database otel does not exist

Steps to Reproduce

Describe the connection to the clickhouse cluster in the exporter.

Expected Result

Successful database creation

Actual Result

OTel collector can't start with error: Database otel does not exist

Collector version

0.114.0

Environment information

Environment

OS: Ubuntu 22.04.5 LTS
Compiler(if manually compiled): go version go1.22.1 linux/amd64

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        cors:
          allowed_origins:
            - "https://*.somedomain.com:3000"
            - "https://*.somedomain.com"
        endpoint: 0.0.0.0:4318
        tls:
          cert_file: /etc/otelcol/cert.crt
          key_file: /etc/otelcol/privkey.key

processors:
  batch:

exporters:
  clickhouse:
    endpoint: tcp://192.168.201.97:9000,tcp://192.168.201.98:9000,tcp://192.168.201.99:9000
    username: user
    password: password
    database: otel
    logs_table_name: otel_logs
    traces_table_name: otel_traces
    metrics_table_name: otel_metrics
    cluster_name: tlmtr_dev_cluster
    table_engine:
      name: ReplicatedMergeTree

service:

  pipelines:

    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [clickhouse]

Log output

Nov 26 08:15:46 mapp-dev-tlmtr1 systemd[1]: Started OpenTelemetry Collector.
Nov 26 08:15:46 mapp-dev-tlmtr1 otelcol[18967]: 2024-11-26T08:15:46.640Z#011info#[email protected]/service.go:166#011Setting up own telemetry...
Nov 26 08:15:46 mapp-dev-tlmtr1 otelcol[18967]: 2024-11-26T08:15:46.640Z#011info#011telemetry/metrics.go:70#011Serving metrics#011{"address": "localhost:8888", "metrics level": "Normal"}
Nov 26 08:15:46 mapp-dev-tlmtr1 otelcol[18967]: 2024-11-26T08:15:46.659Z#011info#[email protected]/service.go:238#011Starting otelcol...#011{"Version": "", "NumCPU": 2}
Nov 26 08:15:46 mapp-dev-tlmtr1 otelcol[18967]: 2024-11-26T08:15:46.659Z#011info#011extensions/extensions.go:39#011Starting extensions...
Nov 26 08:15:46 mapp-dev-tlmtr1 otelcol[18967]: 2024-11-26T08:15:46.662Z#011error#011graph/graph.go:426#011Failed to start component#011{"error": "create database: code: 81, message: Database otel does not exist", "type": "Exporter"
, "id": "clickhouse"}
Nov 26 08:15:46 mapp-dev-tlmtr1 otelcol[18967]: 2024-11-26T08:15:46.662Z#011info#[email protected]/service.go:303#011Starting shutdown...
Nov 26 08:15:46 mapp-dev-tlmtr1 otelcol[18967]: 2024-11-26T08:15:46.662Z#011info#011extensions/extensions.go:66#011Stopping extensions...
Nov 26 08:15:46 mapp-dev-tlmtr1 otelcol[18967]: 2024-11-26T08:15:46.662Z#011info#[email protected]/service.go:317#011Shutdown complete.
Nov 26 08:15:46 mapp-dev-tlmtr1 otelcol[18967]: Error: cannot start pipelines: create database: code: 81, message: Database otel does not exist
Nov 26 08:15:46 mapp-dev-tlmtr1 otelcol[18967]: 2024/11/26 08:15:46 collector server run finished with error: cannot start pipelines: create database: code: 81, message: Database otel does not exist
Nov 26 08:15:46 mapp-dev-tlmtr1 systemd[1]: otelcol.service: Main process exited, code=exited, status=1/FAILURE
Nov 26 08:15:46 mapp-dev-tlmtr1 systemd[1]: otelcol.service: Failed with result 'exit-code'.
Nov 26 08:15:51 mapp-dev-tlmtr1 systemd[1]: Stopped OpenTelemetry Collector.

Additional context

I tried to connect clickhouse clusters with versions:
ClickHouse server version 23.8.9.54 (official build) with Zookeeper.
ClickHouse server version 24.8.7.41 (official build) with ClickHouseKeeper.

github-actions · 2024-11-26T08:23:47Z

Pinging code owners:

exporter/clickhouse: @hanjm @dmitryax @Frapschen @SpencerTorres

See Adding Labels via Comments if you do not have permissions to add labels yourself.

maniac1978 · 2024-11-26T08:45:08Z

It looks like I was in a hurry to claim that the database is being created with a standalone clickhouse server.
With version 0.114.0, the database is not created at all, even with a standalone clickhouse server.

The database was successfully created with version otelcol 0.95.
But I had to go to the latest version, because 0.95 when connecting to the clickhouse cluster, gave an error "* '' has invalid keys: table_engine".

And I did not check the creation of the database on version 0.114.0 with standalone clickhouse.
Now I tried to connect it to a standalone clickhouse server and got the same error: collector server run finished with error: cannot start pipelines: create database: code: 81, message: Database otel does not exist.

So it was in this issue: #23664

maniac1978 · 2024-11-26T09:33:09Z

I ran the otelcol version 0.95 with the same configuration and the database was created on standalone clickhouse server.

SpencerTorres · 2024-11-27T05:12:13Z

Hey! Thanks for submitting this. Could be a duplicate of #35713, check that thread and see if you can provide similar debugging info and I can look into it

maniac1978 · 2024-11-28T12:03:31Z

The problem is not the clickhouse cluster.
Otelcol v0.114.0 cannot create a database even on a standalone clickhouse server.

clickhouse-server.err.log:

2024.11.28 11:56:25.949781 [ 19974 ] {} <Error> ServerErrorHandler: Code: 81. DB::Exception: Database otel does not exist. (UNKNOWN_DATABASE), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000d176d5b
1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000007dabdec
2. DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x0000000007dab92b
3. DB::DatabaseCatalog::assertDatabaseExists(String const&) const @ 0x0000000010bfafab
4. DB::Context::setCurrentDatabase(String const&) @ 0x0000000010b51e12
5. DB::TCPHandler::runImpl() @ 0x00000000125236e7
6. DB::TCPHandler::run() @ 0x000000001253a238
7. Poco::Net::TCPServerConnection::start() @ 0x00000000153190e7
8. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015319579
9. Poco::PooledThread::run() @ 0x00000000152e6521
10. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000152e4add
11. ? @ 0x00007ffa18282ac3
12. ? @ 0x00007ffa18314850
 (version 24.8.7.41 (official build))

And absolutely nothing in the query_log.

/etc/otelcol/config.yaml:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        cors:
          allowed_origins:
            - "https://*.somedomain.com:3000"
            - "https://*.somedomain.com"
        endpoint: 0.0.0.0:4318
        tls:
          cert_file: /etc/otelcol/cert.crt
          key_file: /etc/otelcol/privkey.key

processors:
  batch:

exporters:
  clickhouse:
    endpoint: tcp://127.0.0.1:9000
    username: admin
    password: password
    database: otel
    logs_table_name: otel_logs
    traces_table_name: otel_traces
    metrics_table_name: otel_metrics

service:

  pipelines:

    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [clickhouse]

maniac1978 · 2024-11-28T12:09:14Z

I ran the 0.95 collector with the same configuration, it created the otel database.
After that, I started the collector 0.114 and it started successfully.

maniac1978 · 2024-12-03T11:27:29Z

Hey! Thanks for submitting this. Could be a duplicate of #35713, check that thread and see if you can provide similar debugging info and I can look into it

I built version 0.107, early ones are no longer available for download. And I also get the error "Database otel does not exist" even with an standalone clickhouse.
Version 0.95 runs without errors and creates the "otel" database.

maniac1978 · 2024-12-06T06:15:05Z

I built version 0.115.
"Database otel does not exist" (((

maniac1978 · 2024-12-12T12:36:52Z

What am I doing wrong?

I downloaded cmd/builder/v0.115.1
I updated go to go1.23.4.
I ran "make build".
And got ocb version 0.115.0
If I change the version (OTELCOL_BUILDER_VERSION ?= 0.115.0) in the Makefile, build does not work.

How do I create a collector that will create a base in clickhouse ?

maniac1978 added bug Something isn't working needs triage New item requiring triage labels Nov 26, 2024

github-actions bot added the exporter/clickhouse label Nov 26, 2024

github-actions bot mentioned this issue Dec 3, 2024

Weekly Report: 2024-11-26 - 2024-12-03 #36628

Closed

github-actions bot mentioned this issue Dec 10, 2024

Weekly Report: 2024-12-03 - 2024-12-10 #36739

Open

pimeys mentioned this issue Dec 10, 2024

fix: update otel-collector and manually create the otel database grafbase/grafbase#2447

Merged

github-actions bot mentioned this issue Dec 17, 2024

Weekly Report: 2024-12-10 - 2024-12-17 #36867

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[exporter/clickhouse] create database fail if clickhouse cluster #36540

[exporter/clickhouse] create database fail if clickhouse cluster #36540

maniac1978 commented Nov 26, 2024 •

edited

Loading

github-actions bot commented Nov 26, 2024

maniac1978 commented Nov 26, 2024 •

edited

Loading

maniac1978 commented Nov 26, 2024 •

edited

Loading

SpencerTorres commented Nov 27, 2024

maniac1978 commented Nov 28, 2024 •

edited

Loading

maniac1978 commented Nov 28, 2024

maniac1978 commented Dec 3, 2024

maniac1978 commented Dec 6, 2024

maniac1978 commented Dec 12, 2024

[exporter/clickhouse] create database fail if clickhouse cluster #36540

[exporter/clickhouse] create database fail if clickhouse cluster #36540

Comments

maniac1978 commented Nov 26, 2024 • edited Loading

Component(s)

What happened?

Description

Steps to Reproduce

Expected Result

Actual Result

Collector version

Environment information

Environment

OpenTelemetry Collector configuration

Log output

Additional context

github-actions bot commented Nov 26, 2024

maniac1978 commented Nov 26, 2024 • edited Loading

maniac1978 commented Nov 26, 2024 • edited Loading

SpencerTorres commented Nov 27, 2024

maniac1978 commented Nov 28, 2024 • edited Loading

maniac1978 commented Nov 28, 2024

maniac1978 commented Dec 3, 2024

maniac1978 commented Dec 6, 2024

maniac1978 commented Dec 12, 2024

maniac1978 commented Nov 26, 2024 •

edited

Loading

maniac1978 commented Nov 26, 2024 •

edited

Loading

maniac1978 commented Nov 26, 2024 •

edited

Loading

maniac1978 commented Nov 28, 2024 •

edited

Loading