public class CommonToken extends Object implements WritableToken, Serializable
| Modifier and Type | Field and Description |
|---|---|
protected int |
channel
This is the backing field for
getChannel() and
setChannel(int). |
protected int |
charPositionInLine
This is the backing field for
getCharPositionInLine() and
setCharPositionInLine(int). |
protected static Tuple2<TokenSource,CharStream> |
EMPTY_SOURCE
|
protected int |
index
This is the backing field for
getTokenIndex() and
setTokenIndex(int). |
protected int |
line
This is the backing field for
getLine() and setLine(int). |
protected Tuple2<? extends TokenSource,CharStream> |
source
This is the backing field for
getTokenSource() and
getInputStream(). |
protected int |
start
This is the backing field for
getStartIndex() and
setStartIndex(int). |
protected int |
stop
This is the backing field for
getStopIndex() and
setStopIndex(int). |
protected String |
text
This is the backing field for
getText() when the token text is
explicitly set in the constructor or via setText(java.lang.String). |
protected int |
type
This is the backing field for
getType() and setType(int). |
DEFAULT_CHANNEL, EOF, EPSILON, HIDDEN_CHANNEL, INVALID_TYPE, MIN_USER_CHANNEL_VALUE, MIN_USER_TOKEN_TYPE| Constructor and Description |
|---|
CommonToken(int type)
Constructs a new
CommonToken with the specified token type. |
CommonToken(int type,
String text)
Constructs a new
CommonToken with the specified token type and
text. |
CommonToken(Token oldToken)
Constructs a new
CommonToken as a copy of another Token. |
CommonToken(Tuple2<? extends TokenSource,CharStream> source,
int type,
int channel,
int start,
int stop) |
| Modifier and Type | Method and Description |
|---|---|
int |
getChannel()
Return the channel this token.
|
int |
getCharPositionInLine()
The index of the first character of this token relative to the
beginning of the line at which it occurs, 0..n-1
|
CharStream |
getInputStream()
Gets the
CharStream from which this token was derived. |
int |
getLine()
The line number on which the 1st character of this token was matched,
line=1..n
|
int |
getStartIndex()
The starting character index of the token
This method is optional; return -1 if not implemented.
|
int |
getStopIndex()
The last character index of the token.
|
String |
getText()
Get the text of the token.
|
int |
getTokenIndex()
An index from 0..n-1 of the token object in the input stream.
|
TokenSource |
getTokenSource()
Gets the
TokenSource which created this token. |
int |
getType()
Get the token type of the token
|
void |
setChannel(int channel) |
void |
setCharPositionInLine(int charPositionInLine) |
void |
setLine(int line) |
void |
setStartIndex(int start) |
void |
setStopIndex(int stop) |
void |
setText(String text)
Explicitly set the text for this token.
|
void |
setTokenIndex(int index) |
void |
setType(int type) |
String |
toString() |
String |
toString(Recognizer<?,?> r) |
protected static final Tuple2<TokenSource,CharStream> EMPTY_SOURCE
protected int type
getType() and setType(int).protected int line
getLine() and setLine(int).protected int charPositionInLine
getCharPositionInLine() and
setCharPositionInLine(int).protected int channel
getChannel() and
setChannel(int).@NotNull protected Tuple2<? extends TokenSource,CharStream> source
getTokenSource() and
getInputStream().
These properties share a field to reduce the memory footprint of
CommonToken. Tokens created by a CommonTokenFactory from
the same source and input stream share a reference to the same
Tuple2 containing these values.
protected String text
getText() when the token text is
explicitly set in the constructor or via setText(java.lang.String).getText()protected int index
getTokenIndex() and
setTokenIndex(int).protected int start
getStartIndex() and
setStartIndex(int).protected int stop
getStopIndex() and
setStopIndex(int).public CommonToken(int type)
CommonToken with the specified token type.type - The token type.public CommonToken(@NotNull
Tuple2<? extends TokenSource,CharStream> source,
int type,
int channel,
int start,
int stop)
public CommonToken(int type,
String text)
CommonToken with the specified token type and
text.type - The token type.text - The text of the token.public CommonToken(@NotNull
Token oldToken)
CommonToken as a copy of another Token.
If oldToken is also a CommonToken instance, the newly
constructed token will share a reference to the text field and
the Tuple2 stored in source. Otherwise, text will
be assigned the result of calling getText(), and source
will be constructed from the result of Token.getTokenSource() and
Token.getInputStream().
oldToken - The token to copy.public int getType()
Tokenpublic void setLine(int line)
setLine in interface WritableTokenpublic void setText(String text)
null, then getText() will return this value rather than
extracting the text from the input.setText in interface WritableTokentext - The explicit text of the token, or null if the text
should be obtained from the input along with the start and stop indexes
of the token.public int getLine()
Tokenpublic int getCharPositionInLine()
TokengetCharPositionInLine in interface Tokenpublic void setCharPositionInLine(int charPositionInLine)
setCharPositionInLine in interface WritableTokenpublic int getChannel()
TokengetChannel in interface Tokenpublic void setChannel(int channel)
setChannel in interface WritableTokenpublic void setType(int type)
setType in interface WritableTokenpublic int getStartIndex()
TokengetStartIndex in interface Tokenpublic void setStartIndex(int start)
public int getStopIndex()
TokengetStopIndex in interface Tokenpublic void setStopIndex(int stop)
public int getTokenIndex()
TokengetTokenIndex in interface Tokenpublic void setTokenIndex(int index)
setTokenIndex in interface WritableTokenpublic TokenSource getTokenSource()
TokenTokenSource which created this token.getTokenSource in interface Tokenpublic CharStream getInputStream()
TokenCharStream from which this token was derived.getInputStream in interface Tokenpublic String toString(@Nullable Recognizer<?,?> r)
Copyright © 1992–2024 Daniel Sun. All rights reserved.