Package org.apache.pulsar.io.core

Class PushSource<T>

java.lang.Object
org.apache.pulsar.io.core.PushSource<T>
All Implemented Interfaces:
AutoCloseable, Source<T>
@Public @Stable public abstract class PushSource<T> extends Object implements Source<T>
Pulsar's Push Source interface. PushSource read data from external sources (database changes, twitter firehose, etc) and publish to a Pulsar topic. The reason its called Push is because PushSources get passed a consumer that they invoke whenever they have data to be published to Pulsar. The lifecycle of a PushSource is to open it passing any config needed by it to initialize(like open network connection, authenticate, etc). A consumer is then to it which is invoked by the source whenever there is data to be published. Once all data has been read, one can use close at the end of the session to do any cleanup

  • Constructor Details

    • PushSource

      public PushSource()

  • Method Details

    • read

      public Record<T> read() throws Exception
      Description copied from interface: Source
      Reads the next message from source. If source does not have any new messages, this call should block.
      Specified by:
      read in interface Source<T>
      Returns:
      next message from source. The return result should never be null
      Throws:
      Exception

    • open

      public abstract void open(Map<String,Object> config, SourceContext sourceContext) throws Exception
      Open connector with configuration.
      Specified by:
      open in interface Source<T>
      Parameters:
      config - initialization config
      sourceContext - environment where the source connector is running
      Throws:
      Exception - IO type exceptions when opening a connector

    • consume

      public void consume(Record<T> record)
      Attach a consumer function to this Source. This is invoked by the implementation to pass messages whenever there is data to be pushed to Pulsar.
      Parameters:
      record - next message from source which should be sent to a Pulsar topic

    • getQueueLength

      public int getQueueLength()
      Get length of the queue that records are push onto. Users can override this method to customize the queue length
      Returns:
      queue length