io.debezium.transforms

Class ByLogicalTableRouter<R extends org.apache.kafka.connect.connector.ConnectRecord<R>>

java.lang.Object

io.debezium.transforms.ByLogicalTableRouter<R>

Type Parameters:

R - the subtype of ConnectRecord on which this transformation will operate

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.kafka.common.Configurable, org.apache.kafka.connect.transforms.Transformation<R>

public class ByLogicalTableRouter<R extends org.apache.kafka.connect.connector.ConnectRecord<R>>
extends Object
implements org.apache.kafka.connect.transforms.Transformation<R>

A logical table consists of one or more physical tables with the same schema. A common use case is sharding -- the two physical tables `db_shard1.my_table` and `db_shard2.my_table` together form one logical table. This Transformation allows us to change a record's topic name and send change events from multiple physical tables to one topic. For instance, we might choose to send the two tables from the above example to the topic `db_shard.my_table`. The config options TOPIC_REGEX and TOPIC_REPLACEMENT are used to change the record's topic. Now that multiple physical tables can share a topic, the event's key may need to be augmented to include fields other than just those for the record's primary/unique key, since these are not guaranteed to be unique across tables. We need some identifier added to the key that distinguishes the different physical tables. The field name specified by the config option KEY_FIELD_NAME is added to the key schema for this purpose. By default, its value will be the old topic name, but if a custom value is desired, the config options KEY_FIELD_REGEX and KEY_FIELD_REPLACEMENT may be used to change it. For instance, in our above example, we might choose to make the identifier `db_shard1` and `db_shard2` respectively.

Author:

David Leibovic, Mario Mueller

Field Summary

Fields

Modifier and Type	Field and Description
private org.apache.kafka.common.cache.Cache<org.apache.kafka.connect.data.Schema,org.apache.kafka.connect.data.Schema>	envelopeSchemaUpdateCache
private static Field	KEY_FIELD_NAME
private static Field	KEY_FIELD_REGEX
private static Field	KEY_FIELD_REPLACEMENT
private String	keyFieldName
private Pattern	keyFieldRegex
private String	keyFieldReplacement
private org.apache.kafka.common.cache.Cache<String,String>	keyRegexReplaceCache
private org.apache.kafka.common.cache.Cache<org.apache.kafka.connect.data.Schema,org.apache.kafka.connect.data.Schema>	keySchemaUpdateCache
private static org.slf4j.Logger	logger
private SchemaNameAdjuster	schemaNameAdjuster
private static Field	TOPIC_REGEX
private static Field	TOPIC_REPLACEMENT
private Pattern	topicRegex
private org.apache.kafka.common.cache.Cache<String,String>	topicRegexReplaceCache
private String	topicReplacement

Constructor Summary

Constructors

Constructor and Description
ByLogicalTableRouter()

Method Summary

Modifier and Type	Method and Description
R	apply(R record)
void	close()
org.apache.kafka.common.config.ConfigDef	config()
void	configure(Map<String,?> props)
private org.apache.kafka.connect.data.SchemaBuilder	copySchemaExcludingName(org.apache.kafka.connect.data.Schema source, org.apache.kafka.connect.data.SchemaBuilder builder)
private org.apache.kafka.connect.data.SchemaBuilder	copySchemaExcludingName(org.apache.kafka.connect.data.Schema source, org.apache.kafka.connect.data.SchemaBuilder builder, boolean copyFields)
private String	determineNewTopic(String oldTopic) Determine the new topic name.
private org.apache.kafka.connect.data.Struct	updateEnvelope(org.apache.kafka.connect.data.Schema newEnvelopeSchema, org.apache.kafka.connect.data.Struct oldEnvelope)
private org.apache.kafka.connect.data.Schema	updateEnvelopeSchema(org.apache.kafka.connect.data.Schema oldEnvelopeSchema, String newTopicName)
private org.apache.kafka.connect.data.Struct	updateKey(org.apache.kafka.connect.data.Schema newKeySchema, org.apache.kafka.connect.data.Struct oldKey, String oldTopic)
private org.apache.kafka.connect.data.Schema	updateKeySchema(org.apache.kafka.connect.data.Schema oldKeySchema, String newTopicName)
private org.apache.kafka.connect.data.Struct	updateValue(org.apache.kafka.connect.data.Schema newValueSchema, org.apache.kafka.connect.data.Struct oldValue)
private static int	validateKeyFieldReplacement(Configuration config, Field field, Field.ValidationOutput problems) If KEY_FIELD_REGEX has a value that is really a regex, then the KEY_FIELD_REPLACEMENT must be a non-empty value.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TOPIC_REGEX

private static final Field TOPIC_REGEX

TOPIC_REPLACEMENT

private static final Field TOPIC_REPLACEMENT

KEY_FIELD_REGEX

private static final Field KEY_FIELD_REGEX

KEY_FIELD_NAME

private static final Field KEY_FIELD_NAME

KEY_FIELD_REPLACEMENT

private static final Field KEY_FIELD_REPLACEMENT

logger

private static final org.slf4j.Logger logger

schemaNameAdjuster

private final SchemaNameAdjuster schemaNameAdjuster

topicRegex

private Pattern topicRegex

topicReplacement

private String topicReplacement

keyFieldRegex

private Pattern keyFieldRegex

keyFieldReplacement

private String keyFieldReplacement

keyFieldName

private String keyFieldName

keySchemaUpdateCache

private final org.apache.kafka.common.cache.Cache<org.apache.kafka.connect.data.Schema,org.apache.kafka.connect.data.Schema> keySchemaUpdateCache

envelopeSchemaUpdateCache

private final org.apache.kafka.common.cache.Cache<org.apache.kafka.connect.data.Schema,org.apache.kafka.connect.data.Schema> envelopeSchemaUpdateCache

keyRegexReplaceCache

private final org.apache.kafka.common.cache.Cache<String,String> keyRegexReplaceCache

topicRegexReplaceCache

private final org.apache.kafka.common.cache.Cache<String,String> topicRegexReplaceCache

Constructor Detail

ByLogicalTableRouter

public ByLogicalTableRouter()

Method Detail

validateKeyFieldReplacement

private static int validateKeyFieldReplacement(Configuration config,
                                               Field field,
                                               Field.ValidationOutput problems)

If KEY_FIELD_REGEX has a value that is really a regex, then the KEY_FIELD_REPLACEMENT must be a non-empty value.

configure

public void configure(Map<String,?> props)

Specified by:

configure in interface org.apache.kafka.common.Configurable

apply

public R apply(R record)

Specified by:

apply in interface org.apache.kafka.connect.transforms.Transformation<R extends org.apache.kafka.connect.connector.ConnectRecord<R>>

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Specified by:

close in interface org.apache.kafka.connect.transforms.Transformation<R extends org.apache.kafka.connect.connector.ConnectRecord<R>>

config

public org.apache.kafka.common.config.ConfigDef config()

Specified by:

config in interface org.apache.kafka.connect.transforms.Transformation<R extends org.apache.kafka.connect.connector.ConnectRecord<R>>

determineNewTopic

private String determineNewTopic(String oldTopic)

Determine the new topic name.

Parameters:

oldTopic - the name of the old topic

Returns:

return the new topic name, if the regex applies. Otherwise, return null.

updateKeySchema

private org.apache.kafka.connect.data.Schema updateKeySchema(org.apache.kafka.connect.data.Schema oldKeySchema,
                                                             String newTopicName)

updateKey

private org.apache.kafka.connect.data.Struct updateKey(org.apache.kafka.connect.data.Schema newKeySchema,
                                                       org.apache.kafka.connect.data.Struct oldKey,
                                                       String oldTopic)

updateEnvelopeSchema

private org.apache.kafka.connect.data.Schema updateEnvelopeSchema(org.apache.kafka.connect.data.Schema oldEnvelopeSchema,
                                                                  String newTopicName)

updateEnvelope

private org.apache.kafka.connect.data.Struct updateEnvelope(org.apache.kafka.connect.data.Schema newEnvelopeSchema,
                                                            org.apache.kafka.connect.data.Struct oldEnvelope)

updateValue

private org.apache.kafka.connect.data.Struct updateValue(org.apache.kafka.connect.data.Schema newValueSchema,
                                                         org.apache.kafka.connect.data.Struct oldValue)

copySchemaExcludingName

private org.apache.kafka.connect.data.SchemaBuilder copySchemaExcludingName(org.apache.kafka.connect.data.Schema source,
                                                                            org.apache.kafka.connect.data.SchemaBuilder builder)

copySchemaExcludingName

private org.apache.kafka.connect.data.SchemaBuilder copySchemaExcludingName(org.apache.kafka.connect.data.Schema source,
                                                                            org.apache.kafka.connect.data.SchemaBuilder builder,
                                                                            boolean copyFields)