Package io.debezium.converters

Class CloudEventsConverter

  • All Implemented Interfaces:
    org.apache.kafka.connect.storage.Converter
    public class CloudEventsConverter
extends Object
implements org.apache.kafka.connect.storage.Converter
    Implementation of Converter that express schemas and objects with CloudEvents specification. The serialization format can be Json or Avro.

    The serialization format of CloudEvents is configured with cloudevents.serializer.type option.

    The serialization format of the data attribute in CloudEvents is configured with cloudevents.data.serializer.type option.

    Configuration options of the underlying converters can be passed through using the json and avro prefixes, respectively.

    There are two modes for transferring CloudEvents as Kafka messages: structured and binary. In the structured content mode, event metadata attributes and event data are placed into the Kafka message value section using an event format. In the binary content mode, the value of the event data is placed into the Kafka message's value section as-is, with the content-type header value declaring its media type; all other event attributes are mapped to the Kafka message's header section.

    Since Kafka converters has not support headers yet, right now CloudEvents converter use structured mode as the default.

    • Field Detail

      • CONFLUENT_AVRO_CONVERTER_CLASS

        private static final String CONFLUENT_AVRO_CONVERTER_CLASS
        Instantiated reflectively to avoid hard dependency to Avro converter.
        See Also:
        Constant Field Values

      • CONFLUENT_SCHEMA_REGISTRY_URL_CONFIG

        private static final String CONFLUENT_SCHEMA_REGISTRY_URL_CONFIG
        See Also:
        Constant Field Values

      • APICURIO_AVRO_CONVERTER_CLASS

        private static String APICURIO_AVRO_CONVERTER_CLASS

      • DATA_SCHEMA_SUFFIX

        private static final String DATA_SCHEMA_SUFFIX
        Suffix appended to schema names of data schema in case of Avro/Avro, to keep data schema and CE schema apart from each other
        See Also:
        Constant Field Values

      • LOGGER

        private static final org.slf4j.Logger LOGGER

      • CONVERT_TO_CONNECT_METHOD

        private static Method CONVERT_TO_CONNECT_METHOD

      • jsonCloudEventsConverter

        private final org.apache.kafka.connect.json.JsonConverter jsonCloudEventsConverter

      • jsonDataConverter

        private final org.apache.kafka.connect.json.JsonConverter jsonDataConverter

      • enableJsonSchemas

        private boolean enableJsonSchemas

      • jsonDeserializer

        private final org.apache.kafka.connect.json.JsonDeserializer jsonDeserializer

      • avroConverter

        private org.apache.kafka.connect.storage.Converter avroConverter

      • schemaRegistryUrls

        private List<String> schemaRegistryUrls

    • Constructor Detail

      • CloudEventsConverter

        public CloudEventsConverter()

      • CloudEventsConverter

        public CloudEventsConverter​(org.apache.kafka.connect.storage.Converter avroConverter)

    • Method Detail

      • configure

        public void configure​(Map<String,​?> configs,
                      boolean isKey)
        Specified by:
        configure in interface org.apache.kafka.connect.storage.Converter

      • fromConnectData

        public byte[] fromConnectData​(String topic,
                              org.apache.kafka.connect.data.Schema schema,
                              Object value)
        Specified by:
        fromConnectData in interface org.apache.kafka.connect.storage.Converter

      • lookupCloudEventsProvider

        private static CloudEventsProvider lookupCloudEventsProvider​(org.apache.kafka.connect.data.Struct record)
        Lookup the CloudEventsProvider implementation for the source connector.

      • convertToCloudEventsFormatWithDataAsAvro

        private org.apache.kafka.connect.data.SchemaAndValue convertToCloudEventsFormatWithDataAsAvro​(String topic,
                                                                                              RecordParser parser,
                                                                                              CloudEventsMaker maker)
        Creates a CloudEvents wrapper, converting the "data" to Avro.

      • getSchemaIdFromAvroMessage

        private String getSchemaIdFromAvroMessage​(byte[] serializedData)
        Obtains the schema id from the given Avro record. They are prefixed by one magic byte, followed by an int for the schem id.

      • toConnectData

        public org.apache.kafka.connect.data.SchemaAndValue toConnectData​(String topic,
                                                                  byte[] value)
        Specified by:
        toConnectData in interface org.apache.kafka.connect.storage.Converter

      • reconvertData

        private org.apache.kafka.connect.data.SchemaAndValue reconvertData​(String topic,
                                                                   byte[] serializedData,
                                                                   SerializerType dataType,
                                                                   Boolean enableSchemas)

      • convertToCloudEventsFormat

        private org.apache.kafka.connect.data.SchemaAndValue convertToCloudEventsFormat​(RecordParser parser,
                                                                                CloudEventsMaker maker,
                                                                                org.apache.kafka.connect.data.Schema dataSchemaType,
                                                                                String dataSchema,
                                                                                Object serializedData)

      • convertToCeExtensionSchema

        private org.apache.kafka.connect.data.Schema convertToCeExtensionSchema​(org.apache.kafka.connect.data.Schema schema,
                                                                        boolean alwaysOptional)
        Converts the given source attribute schema into a corresponding CE extension schema. The types supported there are limited, e.g. int64 can only be represented as string.

      • convertToCeExtensionSchema

        private org.apache.kafka.connect.data.Schema convertToCeExtensionSchema​(org.apache.kafka.connect.data.Schema schema)

      • adjustExtensionName

        @VisibleForTesting
static String adjustExtensionName​(String original)
        Adjust the name of CloudEvents attributes for Debezium events, following CloudEvents attribute naming convention as follows:
        • prefixed with EXTENSION_NAME_PREFIX
        • CloudEvents attribute names MUST consist of lower-case letters ('a' to 'z') or digits ('0' to '9') from the ASCII character set, so any other characters are removed
        Parameters:
        original - the original field name
        Returns:
        the valid extension attribute name

      • txExtensionName

        private static String txExtensionName​(String name)

      • isValidExtensionNameCharacter

        private static boolean isValidExtensionNameCharacter​(char c)