Package io.debezium.converters

Class CloudEventsConverter

java.lang.Object
io.debezium.converters.CloudEventsConverter
All Implemented Interfaces:
org.apache.kafka.connect.storage.Converter
public class CloudEventsConverter extends Object implements org.apache.kafka.connect.storage.Converter
Implementation of Converter that express schemas and objects with CloudEvents specification. The serialization format can be Json or Avro.

The serialization format of CloudEvents is configured with cloudevents.serializer.type option.

The serialization format of the data attribute in CloudEvents is configured with cloudevents.data.serializer.type option.

Configuration options of the underlying converters can be passed through using the json and avro prefixes, respectively.

There are two modes for transferring CloudEvents as Kafka messages: structured and binary. In the structured content mode, event metadata attributes and event data are placed into the Kafka message value section using an event format. In the binary content mode, the value of the event data is placed into the Kafka message's value section as-is, with the content-type header value declaring its media type; all other event attributes are mapped to the Kafka message's header section.

Since Kafka converters has not support headers yet, right now CloudEvents converter use structured mode as the default.

  • Field Details

    • EXTENSION_NAME_PREFIX

      private static final String EXTENSION_NAME_PREFIX
      See Also:

    • TX_ATTRIBUTE_PREFIX

      private static final String TX_ATTRIBUTE_PREFIX
      See Also:

    • CONFLUENT_AVRO_CONVERTER_CLASS

      private static final String CONFLUENT_AVRO_CONVERTER_CLASS
      Instantiated reflectively to avoid hard dependency to Avro converter.
      See Also:

    • CONFLUENT_SCHEMA_REGISTRY_URL_CONFIG

      private static final String CONFLUENT_SCHEMA_REGISTRY_URL_CONFIG
      See Also:

    • APICURIO_AVRO_CONVERTER_CLASS

      private static String APICURIO_AVRO_CONVERTER_CLASS

    • APICURIO_SCHEMA_REGISTRY_URL_CONFIG

      private static final String APICURIO_SCHEMA_REGISTRY_URL_CONFIG
      See Also:

    • DATA_SCHEMA_SUFFIX

      private static final String DATA_SCHEMA_SUFFIX
      Suffix appended to schema names of data schema in case of Avro/Avro, to keep data schema and CE schema apart from each other
      See Also:

    • LOGGER

      private static final org.slf4j.Logger LOGGER

    • CONVERT_TO_CONNECT_METHOD

      private static Method CONVERT_TO_CONNECT_METHOD

    • providers

      @Immutable private static Map<String,CloudEventsProvider> providers

    • ceSerializerType

      private SerializerType ceSerializerType

    • dataSerializerType

      private SerializerType dataSerializerType

    • jsonCloudEventsConverter

      private final org.apache.kafka.connect.json.JsonConverter jsonCloudEventsConverter

    • jsonCloudEventsConverterConfig

      private org.apache.kafka.connect.json.JsonConverterConfig jsonCloudEventsConverterConfig

    • jsonHeaderConverter

      private org.apache.kafka.connect.json.JsonConverter jsonHeaderConverter

    • jsonDataConverter

      private final org.apache.kafka.connect.json.JsonConverter jsonDataConverter

    • enableJsonSchemas

      private boolean enableJsonSchemas

    • jsonDeserializer

      private final org.apache.kafka.connect.json.JsonDeserializer jsonDeserializer

    • avroConverter

      private org.apache.kafka.connect.storage.Converter avroConverter

    • schemaRegistryUrls

      private List<String> schemaRegistryUrls

    • schemaNameAdjuster

      private SchemaNameAdjuster schemaNameAdjuster

    • extensionAttributesEnable

      private boolean extensionAttributesEnable

    • cloudEventsSchemaName

      private String cloudEventsSchemaName

    • metadataSource

      private CloudEventsConverterConfig.MetadataSource metadataSource

    • cloudEventsValidator

      private final CloudEventsValidator cloudEventsValidator

  • Constructor Details

    • CloudEventsConverter

      public CloudEventsConverter()

    • CloudEventsConverter

      public CloudEventsConverter(org.apache.kafka.connect.storage.Converter avroConverter)

  • Method Details

    • configure

      public void configure(Map<String,?> configs, boolean isKey)
      Specified by:
      configure in interface org.apache.kafka.connect.storage.Converter

    • configureConverterType

      protected Map<String,String> configureConverterType(boolean isKey, Map<String,String> config)

    • fromConnectData

      public byte[] fromConnectData(String topic, org.apache.kafka.connect.data.Schema schema, Object value)
      Specified by:
      fromConnectData in interface org.apache.kafka.connect.storage.Converter

    • fromConnectData

      public byte[] fromConnectData(String topic, org.apache.kafka.common.header.Headers headers, org.apache.kafka.connect.data.Schema schema, Object value)
      Specified by:
      fromConnectData in interface org.apache.kafka.connect.storage.Converter

    • lookupCloudEventsProvider

      private static CloudEventsProvider lookupCloudEventsProvider(org.apache.kafka.connect.data.Struct source)
      Lookup the CloudEventsProvider implementation for the source connector.

    • getSource

      private org.apache.kafka.connect.data.Struct getSource(org.apache.kafka.connect.data.Struct record, org.apache.kafka.common.header.Headers headers)

    • convertToCloudEventsFormatWithDataAsAvro

      private org.apache.kafka.connect.data.SchemaAndValue convertToCloudEventsFormatWithDataAsAvro(String topic, RecordParser parser, CloudEventsMaker maker)
      Creates a CloudEvents wrapper, converting the "data" to Avro.

    • getSchemaIdFromAvroMessage

      private String getSchemaIdFromAvroMessage(byte[] serializedData)
      Obtains the schema id from the given Avro record. They are prefixed by one magic byte, followed by an int for the schema id.

    • toConnectData

      public org.apache.kafka.connect.data.SchemaAndValue toConnectData(String topic, byte[] value)
      Specified by:
      toConnectData in interface org.apache.kafka.connect.storage.Converter

    • reconvertData

      private org.apache.kafka.connect.data.SchemaAndValue reconvertData(String topic, com.fasterxml.jackson.databind.JsonNode data, SerializerType dataType, Boolean enableSchemas)

    • convertToCloudEventsFormat

      private org.apache.kafka.connect.data.SchemaAndValue convertToCloudEventsFormat(RecordParser parser, CloudEventsMaker maker, org.apache.kafka.connect.data.Schema dataSchemaType, String dataSchema, Object serializedData)

    • ceValueFromStruct

      private void ceValueFromStruct(org.apache.kafka.connect.data.Struct struct, org.apache.kafka.connect.data.Schema schema, CloudEventsConverter.CEValueBuilder ceValueBuilder, Function<String,String> nameMapper)

    • ceSchemaFromSchema

      private void ceSchemaFromSchema(org.apache.kafka.connect.data.Schema schema, CloudEventsConverter.CESchemaBuilder ceSchemaBuilder, Function<String,String> nameMapper, boolean alwaysOptional)

    • convertToCeExtensionSchema

      private org.apache.kafka.connect.data.Schema convertToCeExtensionSchema(org.apache.kafka.connect.data.Schema schema, boolean alwaysOptional)
      Converts the given source attribute schema into a corresponding CE extension schema. The types supported there are limited, e.g. int64 can only be represented as string.

    • convertToCeExtensionSchema

      private org.apache.kafka.connect.data.Schema convertToCeExtensionSchema(org.apache.kafka.connect.data.Schema schema)

    • defineSchema

      private static CloudEventsConverter.CESchemaBuilder defineSchema()

    • withValue

      private static CloudEventsConverter.CEValueBuilder withValue(org.apache.kafka.connect.data.Schema schema)

    • adjustExtensionName

      @VisibleForTesting static String adjustExtensionName(String original)
      Adjust the name of CloudEvents attributes for Debezium events, following CloudEvents attribute naming convention as follows:
      • prefixed with EXTENSION_NAME_PREFIX
      • CloudEvents attribute names MUST consist of lower-case letters ('a' to 'z') or digits ('0' to '9') from the ASCII character set, so any other characters are removed
      Parameters:
      original - the original field name
      Returns:
      the valid extension attribute name

    • txExtensionName

      private static String txExtensionName(String name)

    • isValidExtensionNameCharacter

      private static boolean isValidExtensionNameCharacter(char c)