Package io.debezium.storage.s3.history

Class S3SchemaHistory

java.lang.Object

io.debezium.relational.history.AbstractSchemaHistory

io.debezium.storage.s3.history.S3SchemaHistory

All Implemented Interfaces:

SchemaHistory

@NotThreadSafe
public class S3SchemaHistory
extends AbstractSchemaHistory

A SchemaHistory implementation that records schema changes as normal SourceRecords on the specified topic, and that recovers the history by establishing a Kafka Consumer re-processing all messages on that topic. This implementation provides caching HistoryRecord on the main memory in the case of recovering records. Since S3 does not support Append operation on the object level. start() fetches history log from S3 and store the HistoryRecord on the main memory. Also storeRecord(HistoryRecord) creates new history object everytime invokes on S3

Author:

hossein.torabi

Field Summary

Fields

Modifier and Type	Field	Description
static final Field	ACCESS_KEY_ID
static final String	ACCESS_KEY_ID_CONFIG
static final Field.Set	ALL_FIELDS
private String	bucket
static final Field	BUCKET
static final String	BUCKET_CONFIG
private software.amazon.awssdk.services.s3.S3Client	client
private software.amazon.awssdk.auth.credentials.AwsCredentialsProvider	credentialsProvider
private final DocumentWriter	documentWriter
private URI	endpoint
static final Field	ENDPOINT
static final String	ENDPOINT_CONFIG
private final FunctionalReadWriteLock	lock
private static final org.slf4j.Logger	LOGGER
static final String	OBJECT_CONTENT_TYPE
static final Field	OBJECT_NAME
static final String	OBJECT_NAME_CONFIG
private String	objectName
private final DocumentReader	reader
private List<HistoryRecord>	records
private software.amazon.awssdk.regions.Region	region
static final Field	REGION
static final String	REGION_CONFIG
private final AtomicBoolean	running
static final Field	SECRET_ACCESS_KEY
static final String	SECRET_ACCESS_KEY_CONFIG

Fields inherited from class io.debezium.relational.history.AbstractSchemaHistory

config, INTERNAL_CONNECTOR_CLASS, INTERNAL_CONNECTOR_ID, INTERNAL_PREFER_DDL, logger

Fields inherited from interface io.debezium.relational.history.SchemaHistory

CONFIGURATION_FIELD_PREFIX_STRING, DDL_FILTER, NAME, SKIP_UNPARSEABLE_DDL_STATEMENTS, STORE_ONLY_CAPTURED_TABLES_DDL

Constructor Summary

Constructors

Constructor	Description
S3SchemaHistory()

Method Summary

Modifier and Type	Method	Description
void	configure(Configuration config, HistoryRecordComparator comparator, SchemaHistoryListener listener, boolean useCatalogBeforeSchema)
boolean	exists()
void	initializeStorage()
protected void	recoverRecords(Consumer<HistoryRecord> records)
void	start()
void	stop()
boolean	storageExists()
protected void	storeRecord(HistoryRecord record)
String	toString()

Methods inherited from class io.debezium.relational.history.AbstractSchemaHistory

record, record, recover, skipUnparseableDdlStatements, storeOnlyCapturedTables

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface io.debezium.relational.history.SchemaHistory

recover, recover

Field Details

LOGGER

private static final org.slf4j.Logger LOGGER

ACCESS_KEY_ID_CONFIG

public static final String ACCESS_KEY_ID_CONFIG

See Also:

• Constant Field Values

SECRET_ACCESS_KEY_CONFIG

public static final String SECRET_ACCESS_KEY_CONFIG

See Also:

• Constant Field Values

REGION_CONFIG

public static final String REGION_CONFIG

See Also:

• Constant Field Values

BUCKET_CONFIG

public static final String BUCKET_CONFIG

See Also:

• Constant Field Values

OBJECT_NAME_CONFIG

public static final String OBJECT_NAME_CONFIG

See Also:

• Constant Field Values

ENDPOINT_CONFIG

public static final String ENDPOINT_CONFIG

See Also:

• Constant Field Values

OBJECT_CONTENT_TYPE

public static final String OBJECT_CONTENT_TYPE

See Also:

• Constant Field Values

ACCESS_KEY_ID

public static final Field ACCESS_KEY_ID

SECRET_ACCESS_KEY

public static final Field SECRET_ACCESS_KEY

REGION

public static final Field REGION

BUCKET

public static final Field BUCKET

OBJECT_NAME

public static final Field OBJECT_NAME

ENDPOINT

public static final Field ENDPOINT

ALL_FIELDS

public static final Field.Set ALL_FIELDS

running

private final AtomicBoolean running

lock

private final FunctionalReadWriteLock lock

documentWriter

private final DocumentWriter documentWriter

reader

private final DocumentReader reader

bucket

private String bucket

objectName

private String objectName

region

private software.amazon.awssdk.regions.Region region

endpoint

private URI endpoint

credentialsProvider

private software.amazon.awssdk.auth.credentials.AwsCredentialsProvider credentialsProvider

client

private volatile software.amazon.awssdk.services.s3.S3Client client

records

private volatile List<HistoryRecord> records

Constructor Details

S3SchemaHistory

public S3SchemaHistory()

Method Details

configure

public void configure(Configuration config,
 HistoryRecordComparator comparator,
 SchemaHistoryListener listener,
 boolean useCatalogBeforeSchema)

Specified by:

configure in interface SchemaHistory

Overrides:

configure in class AbstractSchemaHistory

start

public void start()

Specified by:

start in interface SchemaHistory

Overrides:

start in class AbstractSchemaHistory

stop

public void stop()

Specified by:

stop in interface SchemaHistory

Overrides:

stop in class AbstractSchemaHistory

storeRecord

protected void storeRecord(HistoryRecord record)
                    throws SchemaHistoryException

Specified by:

storeRecord in class AbstractSchemaHistory

Throws:

SchemaHistoryException

recoverRecords

protected void recoverRecords(Consumer<HistoryRecord> records)

Specified by:

recoverRecords in class AbstractSchemaHistory

exists

public boolean exists()

storageExists

public boolean storageExists()

initializeStorage

public void initializeStorage()

Specified by:

initializeStorage in interface SchemaHistory

Overrides:

initializeStorage in class AbstractSchemaHistory

toString

public String toString()

Overrides:

toString in class Object