io.debezium.converters

Class CloudEventsConverter

java.lang.Object

io.debezium.converters.CloudEventsConverter

All Implemented Interfaces:

org.apache.kafka.connect.storage.Converter

public class CloudEventsConverter
extends Object
implements org.apache.kafka.connect.storage.Converter

Implementation of Converter that express schemas and objects with CloudEvents specification. The serialization format can be Json or Avro.

The serialization format of CloudEvents is configured with cloudevents.serializer.type option.

The serialization format of the data attribute in CloudEvents is configured with cloudevents.data.serializer.type option.

Configuration options of the underlying converters can be passed through using the json and avro prefixes, respectively.

There are two modes for transferring CloudEvents as Kafka messages: structured and binary. In the structured content mode, event metadata attributes and event data are placed into the Kafka message value section using an event format. In the binary content mode, the value of the event data is placed into the Kafka message's value section as-is, with the content-type header value declaring its media type; all other event attributes are mapped to the Kafka message's header section.

Since Kafka converters has not support headers yet, right now CloudEvents converter use structured mode as the default.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	CloudEventsConverter.CESchemaBuilder Builder of a CloudEvents envelope schema.
static interface	CloudEventsConverter.CEValueBuilder Builder of a CloudEvents value.

Field Summary

Fields

Modifier and Type	Field and Description
private static String	AVRO_CONVERTER_CLASS Instantiated reflectively to avoid hard dependency to Avro converter.
private org.apache.kafka.connect.storage.Converter	avroConverter
private SerializerType	ceSerializerType
private static Method	CONVERT_TO_CONNECT_METHOD
private static String	DATA_SCHEMA_SUFFIX Suffix appended to schema names of data schema in case of Avro/Avro, to keep data schema and CE schema apart from each other
private SerializerType	dataSerializerType
private boolean	enableJsonSchemas
private static String	EXTENSION_NAME_PREFIX
private org.apache.kafka.connect.json.JsonConverter	jsonCloudEventsConverter
private org.apache.kafka.connect.json.JsonConverter	jsonDataConverter
private org.apache.kafka.connect.json.JsonDeserializer	jsonDeserializer
private static org.slf4j.Logger	LOGGER
private static String	SCHEMA_REGISTRY_URL_CONFIG
private List<String>	schemaRegistryUrls
private static String	TX_ATTRIBUTE_PREFIX

Constructor Summary

Constructors

Constructor and Description
CloudEventsConverter()
CloudEventsConverter(org.apache.kafka.connect.storage.Converter avroConverter)

Method Summary

Modifier and Type	Method and Description
(package private) static String	adjustExtensionName(String original) Adjust the name of CloudEvents attributes for Debezium events, following CloudEvents attribute naming convention as follows: prefixed with EXTENSION_NAME_PREFIX CloudEvents attribute names MUST consist of lower-case letters ('a' to 'z') or digits ('0' to '9') from the ASCII character set, so any other characters are removed
private void	ceSchemaFromSchema(org.apache.kafka.connect.data.Schema schema, CloudEventsConverter.CESchemaBuilder ceSchemaBuilder, Function<String,String> nameMapper, boolean alwaysOptional)
private void	ceValueFromStruct(org.apache.kafka.connect.data.Struct struct, org.apache.kafka.connect.data.Schema schema, CloudEventsConverter.CEValueBuilder ceValueBuilder, Function<String,String> nameMapper)
void	configure(Map<String,?> configs, boolean isKey)
private org.apache.kafka.connect.data.Schema	convertToCeExtensionSchema(org.apache.kafka.connect.data.Schema schema)
private org.apache.kafka.connect.data.Schema	convertToCeExtensionSchema(org.apache.kafka.connect.data.Schema schema, boolean alwaysOptional) Converts the given source attribute schema into a corresponding CE extension schema.
private org.apache.kafka.connect.data.SchemaAndValue	convertToCloudEventsFormat(RecordParser parser, CloudEventsMaker maker, org.apache.kafka.connect.data.Schema dataSchemaType, String dataSchema, Object serializedData)
private org.apache.kafka.connect.data.SchemaAndValue	convertToCloudEventsFormatWithDataAsAvro(String topic, RecordParser parser, CloudEventsMaker maker) Creates a CloudEvents wrapper, converting the "data" to Avro.
private static CloudEventsConverter.CESchemaBuilder	defineSchema()
byte[]	fromConnectData(String topic, org.apache.kafka.connect.data.Schema schema, Object value)
private String	getSchemaIdFromAvroMessage(byte[] serializedData) Obtains the schema id from the given Avro record.
private static boolean	isValidExtensionNameCharacter(char c)
private org.apache.kafka.connect.data.SchemaAndValue	reconvertData(String topic, byte[] serializedData, SerializerType dataType, Boolean enableSchemas)
org.apache.kafka.connect.data.SchemaAndValue	toConnectData(String topic, byte[] value)
private static String	txExtensionName(String name)
private static CloudEventsConverter.CEValueBuilder	withValue(org.apache.kafka.connect.data.Schema schema)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.kafka.connect.storage.Converter

fromConnectData, toConnectData

Field Detail

EXTENSION_NAME_PREFIX

private static final String EXTENSION_NAME_PREFIX

See Also:

Constant Field Values

TX_ATTRIBUTE_PREFIX

private static final String TX_ATTRIBUTE_PREFIX

See Also:

Constant Field Values

AVRO_CONVERTER_CLASS

private static final String AVRO_CONVERTER_CLASS

Instantiated reflectively to avoid hard dependency to Avro converter.

See Also:

Constant Field Values

SCHEMA_REGISTRY_URL_CONFIG

private static final String SCHEMA_REGISTRY_URL_CONFIG

See Also:

Constant Field Values

DATA_SCHEMA_SUFFIX

private static final String DATA_SCHEMA_SUFFIX

Suffix appended to schema names of data schema in case of Avro/Avro, to keep data schema and CE schema apart from each other

See Also:

Constant Field Values

LOGGER

private static final org.slf4j.Logger LOGGER

CONVERT_TO_CONNECT_METHOD

private static Method CONVERT_TO_CONNECT_METHOD

ceSerializerType

private SerializerType ceSerializerType

dataSerializerType

private SerializerType dataSerializerType

jsonCloudEventsConverter

private final org.apache.kafka.connect.json.JsonConverter jsonCloudEventsConverter

jsonDataConverter

private final org.apache.kafka.connect.json.JsonConverter jsonDataConverter

enableJsonSchemas

private boolean enableJsonSchemas

jsonDeserializer

private final org.apache.kafka.connect.json.JsonDeserializer jsonDeserializer

avroConverter

private org.apache.kafka.connect.storage.Converter avroConverter

schemaRegistryUrls

private List<String> schemaRegistryUrls

Constructor Detail

CloudEventsConverter

public CloudEventsConverter()

CloudEventsConverter

public CloudEventsConverter(org.apache.kafka.connect.storage.Converter avroConverter)

Method Detail

configure

public void configure(Map<String,?> configs,
                      boolean isKey)

Specified by:

configure in interface org.apache.kafka.connect.storage.Converter

fromConnectData

public byte[] fromConnectData(String topic,
                              org.apache.kafka.connect.data.Schema schema,
                              Object value)

Specified by:

fromConnectData in interface org.apache.kafka.connect.storage.Converter

convertToCloudEventsFormatWithDataAsAvro

private org.apache.kafka.connect.data.SchemaAndValue convertToCloudEventsFormatWithDataAsAvro(String topic,
                                                                                              RecordParser parser,
                                                                                              CloudEventsMaker maker)

Creates a CloudEvents wrapper, converting the "data" to Avro.

getSchemaIdFromAvroMessage

private String getSchemaIdFromAvroMessage(byte[] serializedData)

Obtains the schema id from the given Avro record. They are prefixed by one magic byte, followed by an int for the schem id.

toConnectData

public org.apache.kafka.connect.data.SchemaAndValue toConnectData(String topic,
                                                                  byte[] value)

Specified by:

toConnectData in interface org.apache.kafka.connect.storage.Converter

reconvertData

private org.apache.kafka.connect.data.SchemaAndValue reconvertData(String topic,
                                                                   byte[] serializedData,
                                                                   SerializerType dataType,
                                                                   Boolean enableSchemas)

convertToCloudEventsFormat

private org.apache.kafka.connect.data.SchemaAndValue convertToCloudEventsFormat(RecordParser parser,
                                                                                CloudEventsMaker maker,
                                                                                org.apache.kafka.connect.data.Schema dataSchemaType,
                                                                                String dataSchema,
                                                                                Object serializedData)

ceValueFromStruct

private void ceValueFromStruct(org.apache.kafka.connect.data.Struct struct,
                               org.apache.kafka.connect.data.Schema schema,
                               CloudEventsConverter.CEValueBuilder ceValueBuilder,
                               Function<String,String> nameMapper)

ceSchemaFromSchema

private void ceSchemaFromSchema(org.apache.kafka.connect.data.Schema schema,
                                CloudEventsConverter.CESchemaBuilder ceSchemaBuilder,
                                Function<String,String> nameMapper,
                                boolean alwaysOptional)

convertToCeExtensionSchema

private org.apache.kafka.connect.data.Schema convertToCeExtensionSchema(org.apache.kafka.connect.data.Schema schema,
                                                                        boolean alwaysOptional)

Converts the given source attribute schema into a corresponding CE extension schema. The types supported there are limited, e.g. int64 can only be represented as string.

convertToCeExtensionSchema

private org.apache.kafka.connect.data.Schema convertToCeExtensionSchema(org.apache.kafka.connect.data.Schema schema)

defineSchema

private static CloudEventsConverter.CESchemaBuilder defineSchema()

withValue

private static CloudEventsConverter.CEValueBuilder withValue(org.apache.kafka.connect.data.Schema schema)

adjustExtensionName

@VisibleForTesting
static String adjustExtensionName(String original)

Adjust the name of CloudEvents attributes for Debezium events, following CloudEvents attribute naming convention as follows:

• prefixed with EXTENSION_NAME_PREFIX
• CloudEvents attribute names MUST consist of lower-case letters ('a' to 'z') or digits ('0' to '9') from the ASCII character set, so any other characters are removed

Parameters:

original - the original field name

Returns:

the valid extension attribute name

txExtensionName

private static String txExtensionName(String name)

isValidExtensionNameCharacter

private static boolean isValidExtensionNameCharacter(char c)