Package org.apache.pulsar.transaction.coordinator.impl

Class TxnLogBufferedWriter<T>

java.lang.Object
org.apache.pulsar.transaction.coordinator.impl.TxnLogBufferedWriter<T>
All Implemented Interfaces:
Closeable, AutoCloseable, org.apache.bookkeeper.mledger.AsyncCallbacks.AddEntryCallback
public class TxnLogBufferedWriter<T> extends Object implements org.apache.bookkeeper.mledger.AsyncCallbacks.AddEntryCallback, Closeable
See PIP-160: https://github.com/apache/pulsar/issues/15516. Buffer requests and flush to Managed Ledger. Transaction Log Store And Pending Ack Store will no longer write to Managed Ledger directly, Change to using this class to write Ledger data. Caches “write requests” for a certain number or a certain size of request data and then writes them to the Managed Ledger in one go. After Managed Ledger has written complete, responds to each request-caller. In this process, Managed Ledger doesn't care how many records(or what to be written) in the Entry, it just treats them as a single block of data. The first write-request by transaction components will take a long time to receive a response, because we must wait for subsequent requests to accumulate enough data to actually start writing to the Managed Ledger. To control the maximum latency, we will mark the first request time for each batch, and additional timing triggers writes. You can enable or disabled the batch feature, will use Managed Ledger directly and without batching when disabled.

  • Field Details

    • BATCHED_ENTRY_DATA_PREFIX_MAGIC_NUMBER

      public static final short BATCHED_ENTRY_DATA_PREFIX_MAGIC_NUMBER
      See Also:

    • BATCHED_ENTRY_DATA_PREFIX_MAGIC_NUMBER_LEN

      public static final int BATCHED_ENTRY_DATA_PREFIX_MAGIC_NUMBER_LEN
      See Also:

    • BATCHED_ENTRY_DATA_PREFIX_VERSION

      public static final short BATCHED_ENTRY_DATA_PREFIX_VERSION
      See Also:

    • BATCHED_ENTRY_DATA_PREFIX_VERSION_LEN

      public static final short BATCHED_ENTRY_DATA_PREFIX_VERSION_LEN
      See Also:

  • Constructor Details

    • TxnLogBufferedWriter

      public TxnLogBufferedWriter(org.apache.bookkeeper.mledger.ManagedLedger managedLedger, org.apache.bookkeeper.common.util.OrderedExecutor orderedExecutor, io.netty.util.Timer timer, TxnLogBufferedWriter.DataSerializer<T> dataSerializer, int batchedWriteMaxRecords, int batchedWriteMaxSize, int batchedWriteMaxDelayInMillis, boolean batchEnabled)
      Constructor.
      Parameters:
      dataSerializer - The serializer for the object which called by asyncAddData(T, org.apache.pulsar.transaction.coordinator.impl.TxnLogBufferedWriter.AddDataCallback, java.lang.Object).
      batchedWriteMaxRecords - Caches “write requests” for a certain number, if reach this threshold, will trig Bookie writes.
      batchedWriteMaxSize - Caches “write requests” for a certain size of request data, if reach this threshold, will trig Bookie writes.
      batchedWriteMaxDelayInMillis - Maximum delay for writing to bookie for the earliest request in the batch.
      batchEnabled - Enable or disabled the batch feature, will use Managed Ledger directly and without batching when disabled.
      timer - Used for periodic flush.

  • Method Details

    • asyncAddData

      public void asyncAddData(T data, TxnLogBufferedWriter.AddDataCallback callback, Object ctx)
      Append a new entry to the end of a managed ledger. All writes will be performed in the same thread. Callbacks are executed in strict write order，but after close(), callbacks that fail by state check will execute earlier, and successful callbacks will not be affected.
      Parameters:
      data - data entry to be persisted.
      callback - Will call TxnLogBufferedWriter.AddDataCallback.addComplete(Position, Object) when add complete. Will call TxnLogBufferedWriter.AddDataCallback.addFailed(ManagedLedgerException, Object) when add failure.
      Throws:
      org.apache.bookkeeper.mledger.ManagedLedgerException

    • trigFlush

      public void trigFlush(boolean force, boolean byScheduleThreads)
      Trigger write to bookie once, If the conditions are not met, nothing will be done.

    • addComplete

      public void addComplete(org.apache.bookkeeper.mledger.Position position, io.netty.buffer.ByteBuf entryData, Object ctx)
      see AsyncCallbacks.AddEntryCallback.addComplete(Position, ByteBuf, Object).
      Specified by:
      addComplete in interface org.apache.bookkeeper.mledger.AsyncCallbacks.AddEntryCallback

    • addFailed

      public void addFailed(org.apache.bookkeeper.mledger.ManagedLedgerException exception, Object ctx)
      see AsyncCallbacks.AddEntryCallback.addFailed(ManagedLedgerException, Object).
      Specified by:
      addFailed in interface org.apache.bookkeeper.mledger.AsyncCallbacks.AddEntryCallback

    • close

      public void close()
      Cancel pending tasks and release resources.
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable