Package org.apache.pulsar.io.kinesis

Class KinesisSink

  • All Implemented Interfaces:
    java.lang.AutoCloseable, org.apache.pulsar.io.core.Sink<byte[]>
    public class KinesisSink
extends org.apache.pulsar.io.aws.AbstractAwsConnector
implements org.apache.pulsar.io.core.Sink<byte[]>
    A Kinesis sink which can be configured by KinesisSinkConfig. 
     KinesisSinkConfig accepts
 1. awsEndpoint: kinesis end-point url can be found at : https://docs.aws.amazon.com/general/latest/gr/rande.html
 2. awsRegion: appropriate aws region eg: us-west-1, us-west-2
 3. awsKinesisStreamName: kinesis stream name
 4. awsCredentialPluginName: Fully-Qualified class name of implementation of AwsCredentialProviderPlugin.
    - It is a factory class which creates an AWSCredentialsProvider that will be used by KinesisProducer
    - If it is empty then KinesisSink creates default AWSCredentialsProvider
      which accepts json-map of credentials in awsCredentialPluginParam
      eg: awsCredentialPluginParam = {"accessKey":"my-access-key","secretKey":"my-secret-key"}
 5. awsCredentialPluginParam: json-parameters to initialize AwsCredentialProviderPlugin
 6. messageFormat: enum:["ONLY_RAW_PAYLOAD","FULL_MESSAGE_IN_JSON"]
   a. ONLY_RAW_PAYLOAD:     publishes raw payload to stream
   b. FULL_MESSAGE_IN_JSON: publish full message (encryptionCtx + properties + payload) in json format
   json-schema:
   {"type":"object","properties":{"encryptionCtx":{"type":"object","properties":{"metadata":{"type":"object","additionalProperties":{"type":"string"}},"uncompressedMessageSize":{"type":"integer"},"keysMetadataMap":{"type":"object","additionalProperties":{"type":"object","additionalProperties":{"type":"string"}}},"keysMapBase64":{"type":"object","additionalProperties":{"type":"string"}},"encParamBase64":{"type":"string"},"compressionType":{"type":"string","enum":["NONE","LZ4","ZLIB"]},"batchSize":{"type":"integer"},"algorithm":{"type":"string"}}},"payloadBase64":{"type":"string"},"properties":{"type":"object","additionalProperties":{"type":"string"}}}}
   Example:
   {"payloadBase64":"cGF5bG9hZA==","properties":{"prop1":"value"},"encryptionCtx":{"keysMapBase64":{"key1":"dGVzdDE=","key2":"dGVzdDI="},"keysMetadataMap":{"key1":{"ckms":"cmks-1","version":"v1"},"key2":{"ckms":"cmks-2","version":"v2"}},"metadata":{"ckms":"cmks-1","version":"v1"},"encParamBase64":"cGFyYW0=","algorithm":"algo","compressionType":"LZ4","uncompressedMessageSize":10,"batchSize":10}}

    • Constructor Summary

      Constructors 
      Constructor Description
      KinesisSink()  

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void close()  
      static java.nio.ByteBuffer createKinesisMessage​(KinesisSinkConfig.MessageFormat msgFormat, org.apache.pulsar.functions.api.Record<byte[]> record)  
      void open​(java.util.Map<java.lang.String,​java.lang.Object> config, org.apache.pulsar.io.core.SinkContext sinkContext)  
      void write​(org.apache.pulsar.functions.api.Record<byte[]> record)  

      • Methods inherited from class org.apache.pulsar.io.aws.AbstractAwsConnector

        createCredentialProvider, createCredentialProviderWithPlugin, defaultCredentialProvider

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Field Detail

      • METRICS_TOTAL_INCOMING

        public static final java.lang.String METRICS_TOTAL_INCOMING
        See Also:
        Constant Field Values

      • METRICS_TOTAL_INCOMING_BYTES

        public static final java.lang.String METRICS_TOTAL_INCOMING_BYTES
        See Also:
        Constant Field Values

      • METRICS_TOTAL_SUCCESS

        public static final java.lang.String METRICS_TOTAL_SUCCESS
        See Also:
        Constant Field Values

      • METRICS_TOTAL_FAILURE

        public static final java.lang.String METRICS_TOTAL_FAILURE
        See Also:
        Constant Field Values

    • Constructor Detail

      • KinesisSink

        public KinesisSink()

    • Method Detail

      • write

        public void write​(org.apache.pulsar.functions.api.Record<byte[]> record)
           throws java.lang.Exception
        Specified by:
        write in interface org.apache.pulsar.io.core.Sink<byte[]>
        Throws:
        java.lang.Exception

      • close

        public void close()
           throws java.io.IOException
        Specified by:
        close in interface java.lang.AutoCloseable
        Throws:
        java.io.IOException

      • open

        public void open​(java.util.Map<java.lang.String,​java.lang.Object> config,
                 org.apache.pulsar.io.core.SinkContext sinkContext)
          throws java.lang.Exception
        Specified by:
        open in interface org.apache.pulsar.io.core.Sink<byte[]>
        Throws:
        java.lang.Exception

      • createKinesisMessage

        public static java.nio.ByteBuffer createKinesisMessage​(KinesisSinkConfig.MessageFormat msgFormat,
                                                       org.apache.pulsar.functions.api.Record<byte[]> record)