org.apache.pulsar.functions.worker

Class FunctionMetaDataManager

java.lang.Object

org.apache.pulsar.functions.worker.FunctionMetaDataManager

All Implemented Interfaces:

AutoCloseable

public class FunctionMetaDataManager
extends Object
implements AutoCloseable

FunctionMetaDataManager maintains a global state of all function metadata. It is the system of record for the worker for function metadata. FunctionMetaDataManager operates in either the leader mode or worker mode. By default, when you initialize and start manager, it starts in the worker mode. In the worker mode, the FunctionMetaDataTailer tails the function metadata topic and updates the in-memory metadata cache. When the worker becomes a leader, it calls the acquireLeadaership thru which the FunctionMetaData Manager switches to a leader mode. In the leader mode the manager first captures an exclusive producer on the the metadata topic. Then it drains the MetaDataTailer to ensure that it has caught up to the last record. After this point, the worker can update the in-memory state of function metadata by calling processUpdate/processDeregister methods. If a worker loses its leadership, it calls giveupLeaderShip at which time the manager closes its exclusive producer and starts its tailer again.

Constructor Summary

Constructors

Constructor and Description
FunctionMetaDataManager(WorkerConfig workerConfig, SchedulerManager schedulerManager, org.apache.pulsar.client.api.PulsarClient pulsarClient, ErrorNotifier errorNotifier)

Method Summary

Modifier and Type	Method and Description
org.apache.pulsar.client.api.Producer<byte[]>	acquireExclusiveWrite(Supplier<Boolean> isLeader) Acquires a exclusive producer.
void	acquireLeadership(org.apache.pulsar.client.api.Producer<byte[]> exclusiveProducer) Called by the leader service when this worker becomes the leader.
void	close()
boolean	containsFunction(String tenant, String namespace, String functionName) Check if the function exists.
List<org.apache.pulsar.functions.proto.Function.FunctionMetaData>	getAllFunctionMetaData() Get a list of all the meta for every function.
org.apache.pulsar.functions.proto.Function.FunctionMetaData	getFunctionMetaData(String tenant, String namespace, String functionName) Get the function metadata for a function.
void	giveupLeadership() called by the leader service when we lose leadership.
void	initialize() Initializes the FunctionMetaDataManager.
Collection<org.apache.pulsar.functions.proto.Function.FunctionMetaData>	listFunctions(String tenant, String namespace) List all the functions in a namespace.
void	processMetaDataTopicMessage(org.apache.pulsar.client.api.Message<byte[]> message) This is called by the MetaData tailer.
void	start()
void	updateFunctionOnLeader(org.apache.pulsar.functions.proto.Function.FunctionMetaData functionMetaData, boolean delete) Called by the worker when we are in the leader mode.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FunctionMetaDataManager

public FunctionMetaDataManager(WorkerConfig workerConfig,
                               SchedulerManager schedulerManager,
                               org.apache.pulsar.client.api.PulsarClient pulsarClient,
                               ErrorNotifier errorNotifier)
                        throws org.apache.pulsar.client.api.PulsarClientException

Throws:

org.apache.pulsar.client.api.PulsarClientException

Method Detail

initialize

public void initialize()

Initializes the FunctionMetaDataManager. We create a new reader

start

public void start()

close

public void close()
           throws Exception

Specified by:

close in interface AutoCloseable

Throws:

Exception

getFunctionMetaData

public org.apache.pulsar.functions.proto.Function.FunctionMetaData getFunctionMetaData(String tenant,
                                                                                       String namespace,
                                                                                       String functionName)

Get the function metadata for a function.

Parameters:

tenant - the tenant the function belongs to

namespace - the namespace the function belongs to

functionName - the function name

Returns:

FunctionMetaData that contains the function metadata

getAllFunctionMetaData

public List<org.apache.pulsar.functions.proto.Function.FunctionMetaData> getAllFunctionMetaData()

Get a list of all the meta for every function.

Returns:

list of function metadata

listFunctions

public Collection<org.apache.pulsar.functions.proto.Function.FunctionMetaData> listFunctions(String tenant,
                                                                                             String namespace)

List all the functions in a namespace.

Parameters:

tenant - the tenant the namespace belongs to

namespace - the namespace

Returns:

a list of function names

containsFunction

public boolean containsFunction(String tenant,
                                String namespace,
                                String functionName)

Check if the function exists.

Parameters:

tenant - tenant that the function belongs to

namespace - namespace that the function belongs to

functionName - name of function

Returns:

true if function exists and false if it does not

updateFunctionOnLeader

public void updateFunctionOnLeader(org.apache.pulsar.functions.proto.Function.FunctionMetaData functionMetaData,
                                   boolean delete)
                            throws IllegalStateException,
                                   IllegalArgumentException

Called by the worker when we are in the leader mode. In this state, we update our in-memory data structures and then write to the metadata topic.

Parameters:

functionMetaData - The function metadata in question

delete - Is this a delete operation

Throws:

IllegalStateException - if we are not the leader

IllegalArgumentException - if the request is out of date.

acquireExclusiveWrite

public org.apache.pulsar.client.api.Producer<byte[]> acquireExclusiveWrite(Supplier<Boolean> isLeader)
                                                                    throws WorkerUtils.NotLeaderAnymore

Acquires a exclusive producer. This method cannot return null. It can only return a valid exclusive producer or throw NotLeaderAnymore exception.

Parameters:

isLeader - if the worker is still the leader

Returns:

A valid exclusive producer

Throws:

WorkerUtils.NotLeaderAnymore - if the worker is no longer the leader.

acquireLeadership

public void acquireLeadership(org.apache.pulsar.client.api.Producer<byte[]> exclusiveProducer)

Called by the leader service when this worker becomes the leader. We first get exclusive producer on the metadata topic. Next we drain the tailer to ensure that we have caught up to metadata topic. After which we close the tailer. Note that this method cannot be syncrhonized because the tailer might still be processing messages

giveupLeadership

public void giveupLeadership()

called by the leader service when we lose leadership. We close the exclusive producer and start the tailer.

processMetaDataTopicMessage

public void processMetaDataTopicMessage(org.apache.pulsar.client.api.Message<byte[]> message)
                                 throws IOException

This is called by the MetaData tailer. It updates the in-memory cache. It eats up any exception thrown by processUpdate/processDeregister since that's just part of the state machine

Parameters:

message - The message read from metadata topic that needs to be processed

Throws:

IOException