Package dev.ai4j

Class Tokenizer

java.lang.Object

dev.ai4j.Tokenizer

public class Tokenizer
extends Object

Constructor Summary

Constructors

Constructor	Description
Tokenizer()

Method Summary

Modifier and Type	Method	Description
int	countTokens(String text)
String	decode(List<Integer> tokens)
List<Integer>	encode(String text)
List<Integer>	encode(String text, int maxTokensToEncode)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

Tokenizer

public Tokenizer()

Method Details

encode

public List<Integer> encode(String text)

encode

public List<Integer> encode(String text,
 int maxTokensToEncode)

decode

public String decode(List<Integer> tokens)

countTokens

public int countTokens(String text)