TimeLexerSpeed (ANTLR 4 Performance Tests 4.13.1.4 API)

java.lang.Object
- org.antlr.v4.test.runtime.java.api.perf.TimeLexerSpeed

```
public class TimeLexerSpeed
extends Object
```
Test how fast we can lex Java and some unicode graphemes using old and new unicode stream mechanism. It also tests load time for unicode code points beyond 0xFFFF. Sample output on Linux with Intel Xeon E5-2600 @ 2.20 GHz (us == microseconds, 1/1000 of a millisecond): Java VM args: Warming up Java compiler.... load_legacy_java_utf8 average time 273us size 132266b over 3500 loads of 29038 symbols from Parser.java load_legacy_java_utf8 average time 299us size 128386b over 3500 loads of 13379 symbols from udhr_hin.txt load_new_utf8 average time 535us size 284788b over 3500 loads of 29038 symbols from Parser.java load_new_utf8 average time 439us size 153150b over 3500 loads of 13379 symbols from udhr_hin.txt lex_legacy_java_utf8 average time 624us over 2000 runs of 29038 symbols lex_legacy_java_utf8 average time 1530us over 2000 runs of 29038 symbols DFA cleared lex_new_java_utf8 average time 672us over 2000 runs of 29038 symbols lex_new_java_utf8 average time 1671us over 2000 runs of 29038 symbols DFA cleared lex_legacy_grapheme_utf8 average time 11942us over 400 runs of 6614 symbols from udhr_kor.txt lex_legacy_grapheme_utf8 average time 12075us over 400 runs of 6614 symbols from udhr_kor.txt DFA cleared lex_legacy_grapheme_utf8 average time 10040us over 400 runs of 13379 symbols from udhr_hin.txt lex_legacy_grapheme_utf8 average time 10221us over 400 runs of 13379 symbols from udhr_hin.txt DFA cleared Sample output on OS X with 4 GHz Intel Core i7 (us == microseconds, 1/1000 of a millisecond): Java VM args: -Xms2G -Xmx2G Warming up Java compiler.... load_legacy_java_ascii_file average time 53us size 58384b over 3500 loads of 29038 symbols from Parser.java load_legacy_java_ascii_file average time 27us size 15568b over 3500 loads of 7625 symbols from RuleContext.java load_legacy_java_ascii average time 53us size 65584b over 3500 loads of 29038 symbols from Parser.java load_legacy_java_ascii average time 13us size 32816b over 3500 loads of 7625 symbols from RuleContext.java load_legacy_java_utf8 average time 54us size 65584b over 3500 loads of 29038 symbols from Parser.java load_legacy_java_utf8 average time 118us size 32816b over 3500 loads of 13379 symbols from udhr_hin.txt load_new_utf8 average time 232us size 131232b over 3500 loads of 29038 symbols from Parser.java load_new_utf8 average time 69us size 32928b over 3500 loads of 7625 symbols from RuleContext.java load_new_utf8 average time 210us size 65696b over 3500 loads of 13379 symbols from udhr_hin.txt lex_legacy_java_utf8 average time 342us over 2000 runs of 29038 symbols lex_legacy_java_utf8 average time 890us over 2000 runs of 29038 symbols DFA cleared lex_new_java_utf8 average time 439us over 2000 runs of 29038 symbols lex_new_java_utf8 average time 969us over 2000 runs of 29038 symbols DFA cleared lex_legacy_grapheme_utf8 average time 3971us over 400 runs of 6614 symbols from udhr_kor.txt lex_legacy_grapheme_utf8 average time 4084us over 400 runs of 6614 symbols from udhr_kor.txt DFA cleared lex_legacy_grapheme_utf8 average time 7542us over 400 runs of 13379 symbols from udhr_hin.txt lex_legacy_grapheme_utf8 average time 7666us over 400 runs of 13379 symbols from udhr_hin.txt DFA cleared lex_new_grapheme_utf8 average time 4034us over 400 runs of 6614 symbols from udhr_kor.txt lex_new_grapheme_utf8 average time 4173us over 400 runs of 6614 symbols from udhr_kor.txt DFA cleared lex_new_grapheme_utf8 average time 7680us over 400 runs of 13379 symbols from udhr_hin.txt lex_new_grapheme_utf8 average time 7946us over 400 runs of 13379 symbols from udhr_hin.txt DFA cleared lex_new_grapheme_utf8 average time 70us over 400 runs of 85 symbols from emoji.txt lex_new_grapheme_utf8 average time 82us over 400 runs of 85 symbols from emoji.txt DFA cleared I dump footprint now too (this is 64-bit HotSpot VM): Parser.java (29038 char): org.antlr.v4.runtime.ANTLRFileStream@6b8e0782d footprint: COUNT AVG SUM DESCRIPTION 2 29164 58328 [C 1 24 24 java.lang.String 1 32 32 org.antlr.v4.runtime.ANTLRFileStream 4 58384 (total) RuleContext.java (7625 char): org.antlr.v4.runtime.ANTLRFileStream@76fb7505d footprint: COUNT AVG SUM DESCRIPTION 2 7756 15512 [C 1 24 24 java.lang.String 1 32 32 org.antlr.v4.runtime.ANTLRFileStream 4 15568 (total) Parser.java (29038 char): org.antlr.v4.runtime.ANTLRInputStream@1fc1cb1d footprint: COUNT AVG SUM DESCRIPTION 1 65552 65552 [C 1 32 32 org.antlr.v4.runtime.ANTLRInputStream 2 65584 (total) RuleContext.java (7625 char): org.antlr.v4.runtime.ANTLRInputStream@2c6aa25dd footprint: COUNT AVG SUM DESCRIPTION 1 32784 32784 [C 1 32 32 org.antlr.v4.runtime.ANTLRInputStream 2 32816 (total) Parser.java (29038 char): org.antlr.v4.runtime.ANTLRInputStream@3d08db0bd footprint: COUNT AVG SUM DESCRIPTION 1 65552 65552 [C 1 32 32 org.antlr.v4.runtime.ANTLRInputStream 2 65584 (total) udhr_hin.txt (13379 char): org.antlr.v4.runtime.ANTLRInputStream@486dc6f3d footprint: COUNT AVG SUM DESCRIPTION 1 32784 32784 [C 1 32 32 org.antlr.v4.runtime.ANTLRInputStream 2 32816 (total) Parser.java (29038 char): org.antlr.v4.runtime.CodePointCharStream@798fe5a1d footprint: COUNT AVG SUM DESCRIPTION 1 40 40 [C 1 131088 131088 [I 1 24 24 java.lang.String 1 48 48 java.nio.HeapIntBuffer 1 32 32 org.antlr.v4.runtime.CodePointCharStream 5 131232 (total) RuleContext.java (7625 char): org.antlr.v4.runtime.CodePointCharStream@29cf5a20d footprint: COUNT AVG SUM DESCRIPTION 1 40 40 [C 1 32784 32784 [I 1 24 24 java.lang.String 1 48 48 java.nio.HeapIntBuffer 1 32 32 org.antlr.v4.runtime.CodePointCharStream 5 32928 (total) udhr_hin.txt (13379 char): org.antlr.v4.runtime.CodePointCharStream@1adb8a22d footprint: COUNT AVG SUM DESCRIPTION 1 40 40 [C 1 65552 65552 [I 1 24 24 java.lang.String 1 48 48 java.nio.HeapIntBuffer 1 32 32 org.antlr.v4.runtime.CodePointCharStream 5 65696 (total) The "DFA cleared" indicates that the lexer was returned to initial conditions before the tokenizing of each file. As the ALL(*) lexer encounters new input, it records how it tokenized the chars. The next time it sees that input, it will more quickly recognize the token. Lexing times have the top 20% stripped off before doing the average to account for issues with the garbage collection and compilation pauses; other OS tasks could also pop in randomly. Load times are too fast to measure with a microsecond clock using an SSD so the average load time is computed as the overall time to load n times divided by n (rather then summing up the individual times).

Since:

4.7

Field Summary

Fields
Modifier and Type	Field and Description
`boolean`	`output`
`static String`	`Parser_java_file`
`static String`	`PerfDir`
`static String`	`RuleContext_java_file`
`List<String>`	`streamFootprints`

Constructor Summary

Constructors
Constructor and Description

TimeLexerSpeed()

Constructors
Constructor and Description
`TimeLexerSpeed()`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`double`	`avg(long[] values)`
`static String`	`basename(File file)`
`static String`	`basename(String fullyQualifiedFileName)`
`void`	`compilerWarmUp(int n)`
`static long`	`getResourceSize(ClassLoader loader, String resourceName)`
`void`	`lex_legacy_grapheme_utf8(String fileName, int n, boolean clearLexerDFACache)`
`void`	`lex_legacy_java_utf8(int n, boolean clearLexerDFACache)`
`void`	`lex_new_grapheme_utf8(String fileName, int n, boolean clearLexerDFACache)`
`void`	`lex_new_java_utf8(int n, boolean clearLexerDFACache)`
`void`	`load_legacy_java_ascii_file(String resourceName, int n)`
`void`	`load_legacy_java_ascii(String resourceName, int n)`
`void`	`load_legacy_java_utf8(String resourceName, int n)`
`void`	`load_new_utf8(String resourceName, int n)`
`static void`	`main(String[] args)`
`double`	`std(double mean, List<Long> values)`
`double`	`tokenize(org.antlr.v4.runtime.Lexer lexer, int n, boolean clearLexerDFACache)`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

Parser_java_file

public static final String Parser_java_file

See Also:: Constant Field Values

RuleContext_java_file

public static final String RuleContext_java_file

See Also:: Constant Field Values

PerfDir
```
public static final String PerfDir
```
See Also:

Constant Field Values

output
```
public boolean output
```

streamFootprints
```
public List<String> streamFootprints
```

Constructor Detail
- TimeLexerSpeed
```
public TimeLexerSpeed()
```

Method Detail

main

public static void main(String[] args)
                 throws Exception

Throws:: Exception

compilerWarmUp

public void compilerWarmUp(int n)
                    throws Exception

Throws:: Exception

load_legacy_java_ascii_file

public void load_legacy_java_ascii_file(String resourceName,
                                        int n)
                                 throws Exception

Throws:: Exception

load_legacy_java_ascii

public void load_legacy_java_ascii(String resourceName,
                                   int n)
                            throws Exception

Throws:: Exception

load_legacy_java_utf8

public void load_legacy_java_utf8(String resourceName,
                                  int n)
                           throws Exception

Throws:: Exception

load_new_utf8

public void load_new_utf8(String resourceName,
                          int n)
                   throws Exception

Throws:: Exception

lex_legacy_java_utf8

public void lex_legacy_java_utf8(int n,
                                 boolean clearLexerDFACache)
                          throws Exception

Throws:: Exception

lex_new_java_utf8

public void lex_new_java_utf8(int n,
                              boolean clearLexerDFACache)
                       throws Exception

Throws:: Exception

lex_legacy_grapheme_utf8

public void lex_legacy_grapheme_utf8(String fileName,
                                     int n,
                                     boolean clearLexerDFACache)
                              throws Exception

Throws:: Exception

lex_new_grapheme_utf8

public void lex_new_grapheme_utf8(String fileName,
                                  int n,
                                  boolean clearLexerDFACache)
                           throws Exception

Throws:: Exception

tokenize

public double tokenize(org.antlr.v4.runtime.Lexer lexer,
                       int n,
                       boolean clearLexerDFACache)

avg
```
public double avg(long[] values)
```

std

public double std(double mean,
                  List<Long> values)

basename

public static String basename(String fullyQualifiedFileName)

basename

public static String basename(File file)

getResourceSize

public static final long getResourceSize(ClassLoader loader,
                                         String resourceName)
                                  throws IOException

Throws:: IOException

Class TimeLexerSpeed

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

Parser_java_file

RuleContext_java_file

PerfDir

output

streamFootprints

Constructor Detail

TimeLexerSpeed

Method Detail

main

compilerWarmUp

load_legacy_java_ascii_file

load_legacy_java_ascii

load_legacy_java_utf8

load_new_utf8

lex_legacy_java_utf8

lex_new_java_utf8

lex_legacy_grapheme_utf8

lex_new_grapheme_utf8

tokenize

avg

std

basename

basename

getResourceSize