Java Reference ยท Text, Formatting and Regex
The StringTokenizer class
Reference for java.util.StringTokenizer: splitting a string by delimiter characters, returnDelims, countTokens, and why String.split or Scanner is preferred.
java.util.StringTokenizer splits a string into tokens at any of a set of delimiter characters. It dates from Java 1.0 and is kept only for compatibility. Its own documentation marks it as a legacy class and recommends String.split or java.util.regex instead. It is still found in old code and is occasionally chosen in speed-sensitive input loops because it does no regex work.
The delimiters are a set of single characters, not a pattern and not a string: "; ," means "semicolon, space or comma", each on its own. Runs of delimiters are skipped, so a tokenizer never produces empty tokens.
StringTokenizer st = new StringTokenizer("a,b;c", ",;"); while (st.hasMoreTokens()) { System.out.println(st.nextToken()); }
Try it
Press Run (or Ctrl + Enter) to see the result here.
Program input ยท what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.
Press Run (or Ctrl + Enter) to see the result here.
Program input ยท what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.
Syntax
public class StringTokenizer implements Enumeration<Object> public StringTokenizer(String str) public StringTokenizer(String str, String delim) public StringTokenizer(String str, String delim, boolean returnDelims)
| Name | Description |
|---|---|
str |
The text to tokenize. |
delim |
A set of delimiter characters. Default " \t\n\r\f". |
returnDelims |
If true, each delimiter character is also returned as a one-character token. Default false. |
| Method | Description |
|---|---|
hasMoreTokens(), nextToken() |
Iterate. nextToken() throws NoSuchElementException at the end. |
nextToken(String delim) |
Returns the next token after switching to a new delimiter set. |
countTokens() |
The number of remaining nextToken() calls. Does not consume anything. |
hasMoreElements(), nextElement() |
Enumeration methods; same behavior but typed as Object. |
Throws: NullPointerException from the constructor if str is null.
Usage notes
- No empty tokens.
"a,,b,"gives two tokens, whilesplit(",")givesa, an empty string andb. That can be what you want or a lost field; decide on purpose. - Delimiters are characters. There is no way to split on a multi-character separator such as
"::"or on a pattern; useString.split,Pattern.splitorScanner.useDelimiter. - Not an
Iterator, so it does not work in an enhancedfor. Loop withhasMoreTokens(). - No regex cost, which made it popular in programming-contest input code. Today
BufferedReaderplussplitis also fast enough for most cases. - Not thread-safe. The tokenizer holds a position in the string.
- Stays available. It is not deprecated with
@Deprecated, but new code should usesplit,Scanneror streams:Arrays.stream(text.split("\\s+")).
Related
fmt-scanner,string-split,regex-split,regex-pattern- Lessons: User input, String methods