Package io.debezium.connector.binlog.jdbc

Class BinlogValueConverters

java.lang.Object
io.debezium.jdbc.JdbcValueConverters
io.debezium.connector.binlog.jdbc.BinlogValueConverters
All Implemented Interfaces:
ValueConverterProvider
@Immutable public abstract class BinlogValueConverters extends JdbcValueConverters
Binlog-based connector-specific customizations for converting JDBC values obtained from the binlog client library.

This class uses UTC for the default time zone when converting values without timezone details to values that require timezones. This is because MariaDB TIMESTAMP values are always stored in UTC, unlike types like DATETIME, and aare replicated as such. Meanwhile, the Binlog Client will deserialize these as Timestamp which have no timezone; therefore, are presumed to be UTC.

If a column is Types.TIMESTAMP_WITH_TIMEZONE, the converters will need to convert the value from a Timestamp to an OffsetDateTime using the default time zone, which is always UTC.
Author:
Randall Hauch, Chris Cranford
See Also:
  • AbstractRowsEventDataDeserializer

  • Field Details

    • LOGGER

      private static final org.slf4j.Logger LOGGER

    • INVALID_VALUE_LOGGER

      private static final org.slf4j.Logger INVALID_VALUE_LOGGER

    • TIME_FIELD_PATTERN

      private static final Pattern TIME_FIELD_PATTERN
      Used to parse values of TIME columns. Format: 000:00:00.000000.

    • DATE_FIELD_PATTERN

      private static final Pattern DATE_FIELD_PATTERN
      Used to parse values of DATE columns. Format: 000-00-00.

    • TIMESTAMP_FIELD_PATTERN

      private static final Pattern TIMESTAMP_FIELD_PATTERN
      Used to parse values of TIMESTAMP columns. Format: 000-00-00 00:00:00.000.

    • eventConvertingFailureHandlingMode

      private final CommonConnectorConfig.EventConvertingFailureHandlingMode eventConvertingFailureHandlingMode

  • Constructor Details

  • Method Details

    • schemaBuilder

      public org.apache.kafka.connect.data.SchemaBuilder schemaBuilder(Column column)
      Specified by:
      schemaBuilder in interface ValueConverterProvider
      Overrides:
      schemaBuilder in class JdbcValueConverters

    • converter

      public ValueConverter converter(Column column, org.apache.kafka.connect.data.Field fieldDefn)
      Specified by:
      converter in interface ValueConverterProvider
      Overrides:
      converter in class JdbcValueConverters

    • byteOrderOfBitType

      protected ByteOrder byteOrderOfBitType()
      Overrides:
      byteOrderOfBitType in class JdbcValueConverters

    • convertSmallInt

      protected Object convertSmallInt(Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      Overrides:
      convertSmallInt in class JdbcValueConverters

    • convertInteger

      protected Object convertInteger(Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      Overrides:
      convertInteger in class JdbcValueConverters

    • convertBigInt

      protected Object convertBigInt(Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      Overrides:
      convertBigInt in class JdbcValueConverters

    • convertFloat

      protected Object convertFloat(Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      FLOAT(p) values are reported as FLOAT and DOUBLE.

      A value with precision 0 to 23 results in a 4-byte, single-precision FLOAT column. A value with precision 24 to 53 results in an 8-byte, double-precision DOUBLE column.

      MySQL 8.0.17 deprecated the non-standard FLOAT(M,D) and DOUBLE(M,D) syntax, and should expect support for it to be removed in a future release.

      todo: check the above deprecation with regard to MariaDB
      Overrides:
      convertFloat in class JdbcValueConverters

    • normalizeBinaryData

      protected byte[] normalizeBinaryData(Column column, byte[] data)
      Overrides:
      normalizeBinaryData in class JdbcValueConverters

    • handleUnknownData

      protected Object handleUnknownData(Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      Overrides:
      handleUnknownData in class JdbcValueConverters

    • convertString

      protected Object convertString(Column column, org.apache.kafka.connect.data.Field fieldDefn, Charset columnCharset, Object data)
      Convert the String or byte[] value to a string value used in a SourceRecord.
      Parameters:
      column - the column in which the value appears
      fieldDefn - the field definition for the SourceRecord's Schema; never null
      columnCharset - the Java character set in which column byte[] values are encoded; may not be null
      data - the data; may be null
      Returns:
      the converted value, or null if the conversion could not be made and the column allows nulls
      Throws:
      IllegalArgumentException - if the value could not be converted but the column does not allow nulls

    • convertJson

      protected Object convertJson(Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      Convert the String byte[] value to a string value used in a SourceRecord.
      Parameters:
      column - the column in which the value appears
      fieldDefn - the field definition for the SourceRecord's Schema; never null
      data - the data; may be null
      Returns:
      the converted value, or null if the conversion could not be made and the column allows nulls
      Throws:
      IllegalArgumentException - if the value could not be converted but the column does not allow nulls

    • convertYearToInt

      protected Object convertYearToInt(Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      Converts a value object for a YEAR, which appear in the binlog as an integer though returns from the the JDBC driver as either a short or a Date.
      Parameters:
      column - the column definition describing the data value; never null
      fieldDefn - the field definition; never null
      data - the data object to be converted into a year literal integer value; never null
      Returns:
      the converted value, or null if the conversion could not be made and the column allows nulls
      Throws:
      IllegalArgumentException - if the value could not be converted but the column does not allow nulls

    • convertEnumToString

      protected Object convertEnumToString(List<String> options, Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      Converts a value object for an ENUM, which is represented in the binlog events as an integer value containing the index of the enum option. The JDBC driver returns a string containing the option, so this method calculates the same.
      Parameters:
      options - the characters that appear in the same order as defined in the column; may not be null
      column - the column definition describing the data value; never null
      fieldDefn - the field definition; never null
      data - the data object to be converted into an ENUM literal String value
      Returns:
      the converted value, or null if the conversion could not be made and the column allows nulls
      Throws:
      IllegalArgumentException - if the value could not be converted but the column does not allow nulls

    • convertSetToString

      protected Object convertSetToString(List<String> options, Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      Converts a value object for a SET, which is represented in the binlog events contain a long number in which every bit corresponds to a different option. The JDBC driver returns a string containing the comma-separated options, so this method calculates the same.
      Parameters:
      options - the characters that appear in the same order as defined in the column; may not be null
      column - the column definition describing the data value; never null
      fieldDefn - the field definition; never null
      data - the data object to be converted into an SET literal String value; never null
      Returns:
      the converted value, or null if the conversion could not be made and the column allows nulls
      Throws:
      IllegalArgumentException - if the value could not be converted but the column does not allow nulls

    • charsetFor

      protected Charset charsetFor(Column column)
      Return the Charset instance with the database-specific character set name used by the given column.
      Parameters:
      column - the column in which the character set is used; never null
      Returns:
      the Java Charset, or null if there is no mapping

    • matches

      protected boolean matches(String upperCaseTypeName, String upperCaseMatch)
      Determine if the uppercase form of a column's type exactly matches or begins with the specified prefix. Note that this logic works when the column's type contains the type name followed by parentheses.
      Parameters:
      upperCaseTypeName - the upper case form of the column's type name
      upperCaseMatch - the upper case form of the expected type or prefix of the type; may not be null
      Returns:
      true if the type matches the specified type, or false otherwise

    • isGeometryCollection

      protected boolean isGeometryCollection(String upperCaseTypeName)
      Determine if the uppercase form of a column's type is geometry collection independent of JDBC driver or server version.
      Parameters:
      upperCaseTypeName - the upper case form of the column's type name
      Returns:
      true if the type is geometry collection

    • extractEnumAndSetOptionsAsString

      protected String extractEnumAndSetOptionsAsString(Column column)

    • convertSetValue

      protected String convertSetValue(Column column, long indexes, List<String> options)

    • convertPoint

      protected Object convertPoint(Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      Convert a value representing a POINT byte[] value to a Point value used in a SourceRecord.
      Parameters:
      column - the column in which the value appears
      fieldDefn - the field definition for the SourceRecord's Schema; never null
      data - the data; may be null
      Returns:
      the converted value, or null if the conversion could not be made and the column allows nulls
      Throws:
      IllegalArgumentException - if the value could not be converted but the column does not allow nulls

    • convertGeometry

      protected Object convertGeometry(Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      Convert a value representing a GEOMETRY byte[] value to a Geometry value used in a SourceRecord.
      Parameters:
      column - the column in which the value appears
      fieldDefn - the field definition for the SourceRecord's Schema; never null
      data - the data; may be null
      Returns:
      the converted value, or null if the conversion could not be made and the column allows nulls
      Throws:
      IllegalArgumentException - if the value could not be converted but the column does not allow nulls

    • convertUnsignedTinyint

      protected Object convertUnsignedTinyint(Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      Convert a value representing a Unsigned TINYINT value to the correct Unsigned TINYINT representation.
      Parameters:
      column - the column in which the value appears
      fieldDefn - the field definition for the SourceRecord's Schema; never null
      data - the data; may be null
      Returns:
      the converted value, or null if the conversion could not be made and the column allows nulls
      Throws:
      IllegalArgumentException - if the value could not be converted but the column does not allow nulls

    • convertUnsignedSmallint

      protected Object convertUnsignedSmallint(Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      Convert a value representing a Unsigned SMALLINT value to the correct Unsigned SMALLINT representation.
      Parameters:
      column - the column in which the value appears
      fieldDefn - the field definition for the SourceRecord's Schema; never null
      data - the data; may be null
      Returns:
      the converted value, or null if the conversion could not be made and the column allows nulls
      Throws:
      IllegalArgumentException - if the value could not be converted but the column does not allow nulls

    • convertUnsignedMediumint

      protected Object convertUnsignedMediumint(Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      Convert a value representing a Unsigned MEDIUMINT value to the correct Unsigned SMALLINT representation.
      Parameters:
      column - the column in which the value appears
      fieldDefn - the field definition for the SourceRecord's Schema; never null
      data - the data; may be null
      Returns:
      the converted value, or null if the conversion could not be made and the column allows nulls
      Throws:
      IllegalArgumentException - if the value could not be converted but the column does not allow nulls

    • convertUnsignedInt

      protected Object convertUnsignedInt(Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      Convert a value representing a Unsigned INT value to the correct Unsigned INT representation.
      Parameters:
      column - the column in which the value appears
      fieldDefn - the field definition for the SourceRecord's Schema; never null
      data - the data; may be null
      Returns:
      the converted value, or null if the conversion could not be made and the column allows nulls
      Throws:
      IllegalArgumentException - if the value could not be converted but the column does not allow nulls

    • convertUnsignedBigint

      protected Object convertUnsignedBigint(Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      Convert a value representing a Unsigned BIGINT value to the correct Unsigned INT representation.
      Parameters:
      column - the column in which the value appears
      fieldDefn - the field definition for the SourceRecord's Schema; never null
      data - the data; may be null
      Returns:
      the converted value, or null if the conversion could not be made and the column allows nulls
      Throws:
      IllegalArgumentException - if the value could not be converted but the column does not allow nulls

    • convertDurationToMicroseconds

      protected Object convertDurationToMicroseconds(Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      Converts a value object for an expected type of Duration to Long values that represents the time in microseconds.

      Per the JDBC specification, databases should return Time instances, but that's not working because it can only handle Daytime 00:00:00-23:59:59. We use Duration instead that can handle the range of -838:59:59.000000 to 838:59:59.000000 of a TIME type and transfer data as signed INT64 which reflects the DB value converted to microseconds.
      Parameters:
      column - the column definition describing the data value; never null
      fieldDefn - the field definition; never null
      data - the data object to be converted into a Duration type; never null
      Returns:
      the converted value, or null if the conversion could not be made and the column allows nulls
      Throws:
      IllegalArgumentException - if the value could not be converted but the column does not allow nulls

    • convertTimestampToLocalDateTime

      protected Object convertTimestampToLocalDateTime(Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)

    • extractEnumAndSetOptions

      protected abstract List<String> extractEnumAndSetOptions(Column column)

    • getJavaEncodingForCharSet

      protected abstract String getJavaEncodingForCharSet(String charSetName)

    • adjustTemporal

      public static Temporal adjustTemporal(Temporal temporal)
      A utility method that adjusts ambiguous 2-digit year values of DATETIME, DATE, and TIMESTAMP types using these database-specific rules:
      • Year values in the range 00-69 are converted to 2000-2069.
      • Year values in the range 70-99 are converted to 1970-1999.
      Parameters:
      temporal - the temporal instance to adjust; may not be null
      Returns:
      the possibly adjusted temporal instance; never null

    • stringToDuration

      public static Duration stringToDuration(String timeString)
      Converts a TIME_FIELD_PATTERN time field pattern string to a Duration.
      Parameters:
      timeString - the time field string value; should not be null
      Returns:
      the parsed duration

    • stringToLocalDate

      public static LocalDate stringToLocalDate(String dateString, Column column, Table table)
      Converts a DATE_FIELD_PATTERN date string field pattern to a LocalDate value.
      Parameters:
      dateString - date string to be parsed; may be null or empty
      column - the relational column, should not be null
      table - the relational table, should not be null
      Returns:
      the parsed local date or null if the date's year, month, or day are zero

    • containsZeroValuesInDatePart

      public static boolean containsZeroValuesInDatePart(String timestampString, Column column, Table table)
      Checks if the TIMESTAMP_FIELD_PATTERN timestamp field pattern contains 0 in its year, month, or day parts of the parsed value.
      Parameters:
      timestampString - timestamp string to be parsed, may be null or empty
      column - the relational column, should not be null
      table - the relational table, should not be null
      Returns:
      true if the timestamp's year, month, or day are zero; false otherwise