[Bug]: BigQueryAvroUtils don't handle Avro FLOAT type correcty.

[an2x]

What happened?

#32482 attempted to improve BQ <> Avro schema and type conversion logic, but it still seems to not handle FLOAT type properly (though I think it wasn't working before this refactoring either).

Example code:

  public static void main(String[] args) {
    String avroSchemaString =
        "{\n"
            + "    \"type\": \"record\",\n"
            + "    \"name\": \"MySchema\",\n"
            + "    \"namespace\": \"com.company\",\n"
            + "    \"fields\": [\n"
            + "        {\n"
            + "            \"name\": \"ff\",\n"
            + "            \"type\": \"float\"\n"
            + "        }\n"
            + "    ]\n"
            + "}";

    Schema avroSchema = new Schema.Parser().parse(avroSchemaString);
    org.apache.beam.sdk.schemas.Schema beamSchema = AvroUtils.toBeamSchema(avroSchema);
    TableSchema bqSchema = BigQueryUtils.toTableSchema(beamSchema);

    System.out.println("AVRO SCHEMA:\n" + avroSchema + "\n");
    System.out.println("BEAM SCHEMA:\n" + beamSchema + "\n");
    System.out.println("BQ SCHEMA:\n" + bqSchema + "\n");

    Record record = new GenericRecordBuilder(avroSchema).set("ff", 123.123f).build();

    System.out.println("RECORD:\n" + BigQueryAvroUtils.convertGenericRecordToTableRow(record, bqSchema) + "\n");
  }

Result:

AVRO SCHEMA:
{"type":"record","name":"MySchema","namespace":"com.company","fields":[{"name":"ff","type":"float"}]}

BEAM SCHEMA:
Fields:
Field{name=ff, description=, type=FLOAT NOT NULL, options={{}}}
Encoding positions:
{ff=0}
Options:{{}}UUID: e7c3c4c1-aeef-4521-8582-e60611562bfd

BQ SCHEMA:
GenericData{classInfo=[fields, foreignTypeInfo], {fields=[GenericData{classInfo=[categories, collation, dataPolicies, defaultValueExpression, description, fields, foreignTypeDefinition, maxLength, mode, name, policyTags, precision, rangeElementType, roundingMode, scale, type], {mode=REQUIRED, name=ff, type=FLOAT64}}]}}

Exception in thread "main" java.lang.UnsupportedOperationException: Unexpected Avro field schema type FLOAT for field named ff
	at org.apache.beam.sdk.io.gcp.bigquery.BigQueryAvroUtils.convertRequiredField(BigQueryAvroUtils.java:353)
	at org.apache.beam.sdk.io.gcp.bigquery.BigQueryAvroUtils.getTypedCellValue(BigQueryAvroUtils.java:283)
	at org.apache.beam.sdk.io.gcp.bigquery.BigQueryAvroUtils.convertGenericRecordToTableRow(BigQueryAvroUtils.java:262)
	at org.apache.beam.sdk.io.gcp.bigquery.BigQueryAvroUtils.convertGenericRecordToTableRow(BigQueryAvroUtils.java:247)
	at org.apache.beam.sdk.io.gcp.bigquery.BigQueryAvroUtils.main(BigQueryAvroUtils.java:489)

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[damccorm]

@RustedBones @ahmedabu98 @Abacn FYI

[RustedBones]

Thanks for the detailed report. I may have overlooked this case if this wasn't tested. I can have a look next week

[RustedBones]

I see the issue. The conversions in BigQueryAvroUtils are biased towards BQ read: BQ FLOAT -> Avro double.

On write/load, we should support Avro float -> BQ -> FLOAT