Need help to resolve the error :- got unrecoverable error in primary and no secondary error_class=Fluent::UnrecoverableError error="failed to execute columnify command. stdout= stderr=Failed to close columnifier: interface conversion: interface {} is nil, not string2024/07/09 22:25:41 Failed to write: interface conversion: interface {} is nil, not string\n status=#<Process::Status: pid 1191 exit 1>"

[wasifshareef]

HI,
I am trying to use columnify to generate output in parquet format to send data to azure using azure plugin,however i am getting error when i run the fluentd container with below error . I will appreciate if someone can help me here.

2024-07-09 22:25:41 +0000 [warn]: #0 got unrecoverable error in primary and no secondary error_class=Fluent::UnrecoverableError error="failed to execute columnify command. stdout= stderr=Failed to close columnifier: interface conversion: interface {} is nil, not string
2024/07/09 22:25:41 Failed to write: interface conversion: interface {} is nil, not string\n status=#<Process::Status: pid 1191 exit 1>"
  2024-07-09 22:25:41 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-azurestorage-gen2-0.3.5/lib/fluent/plugin/out_azurestorage_gen2.rb:834:in `compress'
  2024-07-09 22:25:41 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-azurestorage-gen2-0.3.5/lib/fluent/plugin/out_azurestorage_gen2.rb:165:in `write'
  2024-07-09 22:25:41 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.3/lib/fluent/plugin/output.rb:1225:in `try_flush'
  2024-07-09 22:25:41 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.3/lib/fluent/plugin/output.rb:1538:in `flush_thread_run'
  2024-07-09 22:25:41 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.3/lib/fluent/plugin/output.rb:510:in `block (2 levels) in start'
  2024-07-09 22:25:41 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

[wasifshareef]

here are the schema files i am trying to use
cat schemazeek.avsc
{
"type": "record",
"name": "LogRecord",
"fields": [
{"name": "_node_name", "type": "string"},
{"name": "ts", "type": "float"},
{"name": "uid", "type": "string"},
{"name": "id.orig_h", "type": "string"},
{"name": "id.orig_p", "type": "int"},
{"name": "id.resp_h", "type": "string"},
{"name": "id.resp_p", "type": "int"},
{"name": "id.vlan", "type": "float"},
{"name": "proto", "type": "string"},
{"name": "duration", "type": "float"},
{"name": "orig_bytes", "type": "float"},
{"name": "resp_bytes", "type": "float"},
{"name": "conn_state", "type": "string"},
{"name": "local_orig", "type": "boolean"},
{"name": "local_resp", "type": "boolean"},
{"name": "missed_bytes", "type": "int"},
{"name": "orig_pkts", "type": "int"},
{"name": "orig_ip_bytes", "type": "int"},
{"name": "resp_pkts", "type": "int"},
{"name": "resp_ip_bytes", "type": "int"},
{"name": "tailed_path", "type": "string"},
{"name": "beat.hostname", "type": "string"},
{"name": "tag", "type": "string"},
{"name": "clientkey", "type": "string"},
{"name": "consumer_key", "type": "string"},
{"name": "service", "type": "string"},
{"name": "history", "type": "string"},
{"name": "tunnel_parents", "type": "string"}
]
}

++++++

{
"type": "record",
"name": "LogRecord",
"fields": [
{"name": "timestamp", "type": "string"},
{"name": "flow_id", "type": "int"},
{"name": "in_iface", "type": "string"},
{"name": "event_type", "type": "string"},
{"name": "src_ip", "type": "string"},
{"name": "src_port", "type": "float"},
{"name": "dest_ip", "type": "string"},
{"name": "dest_port", "type": "float"},
{"name": "proto", "type": "string"},
{"name": "snmp", "type": "string"},
{"name": "beat.hostname", "type": "string"},
{"name": "tag", "type": "string"},
{"name": "clientkey", "type": "string"},
{"name": "consumer_key", "type": "string"},
{"name": "alert", "type": "string"},
{"name": "flow", "type": "string"},
{"name": "payload_printable", "type": "string"},
{"name": "stream", "type": "float"},
{"name": "app_proto", "type": "string"},
{"name": "netflow", "type": "string"},
{"name": "tcp", "type": "string"},
{"name": "icmp_type", "type": "float"},
{"name": "icmp_code", "type": "float"}
]
}

Thanks,