Package io.debezium.storage.s3.history

Class S3SchemaHistory

java.lang.Object

io.debezium.relational.history.AbstractSchemaHistory

io.debezium.relational.history.AbstractFileBasedSchemaHistory

io.debezium.storage.s3.history.S3SchemaHistory

All Implemented Interfaces:

SchemaHistory

@NotThreadSafe
public class S3SchemaHistory
extends AbstractFileBasedSchemaHistory

A SchemaHistory implementation that records schema changes as normal SourceRecords on the specified topic, and that recovers the history by establishing a Kafka Consumer re-processing all messages on that topic. This implementation provides caching HistoryRecord on the main memory in the case of recovering records. Since S3 does not support Append operation on the object level. AbstractFileBasedSchemaHistory.start() fetches history log from S3 and store the HistoryRecord on the main memory. Also AbstractFileBasedSchemaHistory.storeRecord(HistoryRecord) creates new history object everytime invokes on S3

Author:

hossein.torabi

Field Summary

Fields

Modifier and Type	Field	Description
static final Field	ACCESS_KEY_ID
static final String	ACCESS_KEY_ID_CONFIG
static final Field.Set	ALL_FIELDS
private String	bucket
static final Field	BUCKET
static final String	BUCKET_CONFIG
private software.amazon.awssdk.services.s3.S3Client	client
private software.amazon.awssdk.auth.credentials.AwsCredentialsProvider	credentialsProvider
private URI	endpoint
static final Field	ENDPOINT
static final String	ENDPOINT_CONFIG
private static final org.slf4j.Logger	LOGGER
static final String	OBJECT_CONTENT_TYPE
static final Field	OBJECT_NAME
static final String	OBJECT_NAME_CONFIG
private String	objectName
private software.amazon.awssdk.regions.Region	region
static final Field	REGION
static final String	REGION_CONFIG
static final Field	SECRET_ACCESS_KEY
static final String	SECRET_ACCESS_KEY_CONFIG

Fields inherited from class io.debezium.relational.history.AbstractFileBasedSchemaHistory

documentReader, documentWriter, lock, records, running

Fields inherited from class io.debezium.relational.history.AbstractSchemaHistory

config, logger

Fields inherited from interface io.debezium.relational.history.SchemaHistory

CONFIGURATION_FIELD_PREFIX_STRING, DDL_FILTER, INTERNAL_CONNECTOR_CLASS, INTERNAL_CONNECTOR_ID, INTERNAL_PREFER_DDL, NAME, SKIP_UNPARSEABLE_DDL_STATEMENTS, STORE_ONLY_CAPTURED_DATABASES_DDL, STORE_ONLY_CAPTURED_TABLES_DDL

Constructor Summary

Constructors

Constructor	Description
S3SchemaHistory()

Method Summary

Modifier and Type	Method	Description
void	configure(Configuration config, HistoryRecordComparator comparator, SchemaHistoryListener listener, boolean useCatalogBeforeSchema)
protected void	doPreStart()
protected void	doPreStoreRecord(HistoryRecord record)
protected void	doStart()
void	doStop()
protected void	doStoreRecord(HistoryRecord record)
void	initializeStorage()
boolean	storageExists()
String	toString()

Methods inherited from class io.debezium.relational.history.AbstractFileBasedSchemaHistory

exists, fromHistoryRecord, getRecords, recoverRecords, start, stop, storeRecord, toHistoryRecord

Methods inherited from class io.debezium.relational.history.AbstractSchemaHistory

record, record, recover

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface io.debezium.relational.history.SchemaHistory

recover, recover

Field Details

LOGGER

private static final org.slf4j.Logger LOGGER

ACCESS_KEY_ID_CONFIG

public static final String ACCESS_KEY_ID_CONFIG

See Also:

• Constant Field Values

SECRET_ACCESS_KEY_CONFIG

public static final String SECRET_ACCESS_KEY_CONFIG

See Also:

• Constant Field Values

REGION_CONFIG

public static final String REGION_CONFIG

See Also:

• Constant Field Values

BUCKET_CONFIG

public static final String BUCKET_CONFIG

See Also:

• Constant Field Values

OBJECT_NAME_CONFIG

public static final String OBJECT_NAME_CONFIG

See Also:

• Constant Field Values

ENDPOINT_CONFIG

public static final String ENDPOINT_CONFIG

See Also:

• Constant Field Values

OBJECT_CONTENT_TYPE

public static final String OBJECT_CONTENT_TYPE

See Also:

• Constant Field Values

ACCESS_KEY_ID

public static final Field ACCESS_KEY_ID

SECRET_ACCESS_KEY

public static final Field SECRET_ACCESS_KEY

REGION

public static final Field REGION

BUCKET

public static final Field BUCKET

OBJECT_NAME

public static final Field OBJECT_NAME

ENDPOINT

public static final Field ENDPOINT

ALL_FIELDS

public static final Field.Set ALL_FIELDS

bucket

private String bucket

objectName

private String objectName

region

private software.amazon.awssdk.regions.Region region

endpoint

private URI endpoint

credentialsProvider

private software.amazon.awssdk.auth.credentials.AwsCredentialsProvider credentialsProvider

client

private volatile software.amazon.awssdk.services.s3.S3Client client

Constructor Details

S3SchemaHistory

public S3SchemaHistory()

Method Details

configure

public void configure(Configuration config,
 HistoryRecordComparator comparator,
 SchemaHistoryListener listener,
 boolean useCatalogBeforeSchema)

Specified by:

configure in interface SchemaHistory

Overrides:

configure in class AbstractSchemaHistory

doPreStart

protected void doPreStart()

Overrides:

doPreStart in class AbstractFileBasedSchemaHistory

doStart

protected void doStart()

Overrides:

doStart in class AbstractFileBasedSchemaHistory

doStop

public void doStop()

Overrides:

doStop in class AbstractFileBasedSchemaHistory

doPreStoreRecord

protected void doPreStoreRecord(HistoryRecord record)

Overrides:

doPreStoreRecord in class AbstractFileBasedSchemaHistory

doStoreRecord

protected void doStoreRecord(HistoryRecord record)

Overrides:

doStoreRecord in class AbstractFileBasedSchemaHistory

storageExists

public boolean storageExists()

initializeStorage

public void initializeStorage()

Specified by:

initializeStorage in interface SchemaHistory

Overrides:

initializeStorage in class AbstractSchemaHistory

toString

public String toString()

Overrides:

toString in class Object