A Kinesis sink which can be configured by
KinesisSinkConfig.
KinesisSinkConfig accepts
1. awsEndpoint: kinesis end-point url can be found at : https://docs.aws.amazon.com/general/latest/gr/rande.html
2. awsRegion: appropriate aws region eg: us-west-1, us-west-2
3. awsKinesisStreamName: kinesis stream name
4. awsCredentialPluginName: Fully-Qualified class name of implementation of AwsCredentialProviderPlugin.
- It is a factory class which creates an AWSCredentialsProvider that will be used by KinesisProducer
- If it is empty then KinesisSink creates default AWSCredentialsProvider
which accepts json-map of credentials in awsCredentialPluginParam
eg: awsCredentialPluginParam = {"accessKey":"my-access-key","secretKey":"my-secret-key"}
5. awsCredentialPluginParam: json-parameters to initialize AwsCredentialProviderPlugin
6. messageFormat: enum:["ONLY_RAW_PAYLOAD","FULL_MESSAGE_IN_JSON","FULL_MESSAGE_IN_FB"]
a. ONLY_RAW_PAYLOAD: publishes raw payload to stream
b. FULL_MESSAGE_IN_JSON: publish full message (encryptionCtx + properties + payload) in json format
json-schema:
{"type":"object","properties":{"encryptionCtx":{"type":"object","properties":{"metadata":{"type":"object","additionalProperties":{"type":"string"}},"uncompressedMessageSize":{"type":"integer"},"keysMetadataMap":{"type":"object","additionalProperties":{"type":"object","additionalProperties":{"type":"string"}}},"keysMapBase64":{"type":"object","additionalProperties":{"type":"string"}},"encParamBase64":{"type":"string"},"compressionType":{"type":"string","enum":["NONE","LZ4","ZLIB"]},"batchSize":{"type":"integer"},"algorithm":{"type":"string"}}},"payloadBase64":{"type":"string"},"properties":{"type":"object","additionalProperties":{"type":"string"}}}}
Example:
{"payloadBase64":"cGF5bG9hZA==","properties":{"prop1":"value"},"encryptionCtx":{"keysMapBase64":{"key1":"dGVzdDE=","key2":"dGVzdDI="},"keysMetadataMap":{"key1":{"ckms":"cmks-1","version":"v1"},"key2":{"ckms":"cmks-2","version":"v2"}},"metadata":{"ckms":"cmks-1","version":"v1"},"encParamBase64":"cGFyYW0=","algorithm":"algo","compressionType":"LZ4","uncompressedMessageSize":10,"batchSize":10}}
c. FULL_MESSAGE_IN_FB: publish full message (encryptionCtx + properties + payload) in flatbuffer format
d. FULL_MESSAGE_IN_JSON_EXPAND_VALUE: publish full message (topic + key + value + properties + event time) in JSON format using the schema to expand the value in JSON.