dev.langchain4j.model.embedding

Class BertTokenizer

java.lang.Object

dev.langchain4j.model.embedding.BertTokenizer

All Implemented Interfaces:

dev.langchain4j.model.Tokenizer

Deprecated.

Use HuggingFaceTokenizer instead.

@Deprecated
public class BertTokenizer
extends Object
implements dev.langchain4j.model.Tokenizer

Constructor Summary

Constructors

Constructor and Description
BertTokenizer() Deprecated.
BertTokenizer(URL vocabularyFile) Deprecated.

Method Summary

Modifier and Type	Method and Description
int	estimateTokenCountInMessage(dev.langchain4j.data.message.ChatMessage message) Deprecated.
int	estimateTokenCountInMessages(Iterable<dev.langchain4j.data.message.ChatMessage> messages) Deprecated.
int	estimateTokenCountInText(String text) Deprecated.
int	estimateTokenCountInToolExecutionRequests(Iterable<dev.langchain4j.agent.tool.ToolExecutionRequest> toolExecutionRequests) Deprecated.
int	estimateTokenCountInToolSpecifications(Iterable<dev.langchain4j.agent.tool.ToolSpecification> toolSpecifications) Deprecated.
long	tokenId(String token) Deprecated.
List<String>	tokenize(String text) Deprecated.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface dev.langchain4j.model.Tokenizer

estimateTokenCountInForcefulToolExecutionRequest, estimateTokenCountInForcefulToolSpecification, estimateTokenCountInTools, estimateTokenCountInTools

Constructor Detail

BertTokenizer

public BertTokenizer()

Deprecated.

BertTokenizer

public BertTokenizer(URL vocabularyFile)

Deprecated.

Method Detail

estimateTokenCountInText

public int estimateTokenCountInText(String text)

Deprecated.

Specified by:

estimateTokenCountInText in interface dev.langchain4j.model.Tokenizer

estimateTokenCountInMessage

public int estimateTokenCountInMessage(dev.langchain4j.data.message.ChatMessage message)

Deprecated.

Specified by:

estimateTokenCountInMessage in interface dev.langchain4j.model.Tokenizer

estimateTokenCountInMessages

public int estimateTokenCountInMessages(Iterable<dev.langchain4j.data.message.ChatMessage> messages)

Deprecated.

Specified by:

estimateTokenCountInMessages in interface dev.langchain4j.model.Tokenizer

estimateTokenCountInToolSpecifications

public int estimateTokenCountInToolSpecifications(Iterable<dev.langchain4j.agent.tool.ToolSpecification> toolSpecifications)

Deprecated.

Specified by:

estimateTokenCountInToolSpecifications in interface dev.langchain4j.model.Tokenizer

estimateTokenCountInToolExecutionRequests

public int estimateTokenCountInToolExecutionRequests(Iterable<dev.langchain4j.agent.tool.ToolExecutionRequest> toolExecutionRequests)

Deprecated.

Specified by:

estimateTokenCountInToolExecutionRequests in interface dev.langchain4j.model.Tokenizer

tokenize

public List<String> tokenize(String text)

Deprecated.

tokenId

public long tokenId(String token)

Deprecated.