Skip to content
Tutorials Center
My learning

Topics

Java Reference ยท Text, Formatting and Regex

The StringTokenizer class

Reference for java.util.StringTokenizer: splitting a string by delimiter characters, returnDelims, countTokens, and why String.split or Scanner is preferred.

java.util.StringTokenizer splits a string into tokens at any of a set of delimiter characters. It dates from Java 1.0 and is kept only for compatibility. Its own documentation marks it as a legacy class and recommends String.split or java.util.regex instead. It is still found in old code and is occasionally chosen in speed-sensitive input loops because it does no regex work.

The delimiters are a set of single characters, not a pattern and not a string: "; ," means "semicolon, space or comma", each on its own. Runs of delimiters are skipped, so a tokenizer never produces empty tokens.

Java
StringTokenizer st = new StringTokenizer("a,b;c", ",;");
while (st.hasMoreTokens()) {
    System.out.println(st.nextToken());
}

Try it

Java Tokens and delimiters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Java StringTokenizer vs split

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Syntax

Java
public class StringTokenizer implements Enumeration<Object>
public StringTokenizer(String str)
public StringTokenizer(String str, String delim)
public StringTokenizer(String str, String delim, boolean returnDelims)
Name Description
str The text to tokenize.
delim A set of delimiter characters. Default " \t\n\r\f".
returnDelims If true, each delimiter character is also returned as a one-character token. Default false.
Method Description
hasMoreTokens(), nextToken() Iterate. nextToken() throws NoSuchElementException at the end.
nextToken(String delim) Returns the next token after switching to a new delimiter set.
countTokens() The number of remaining nextToken() calls. Does not consume anything.
hasMoreElements(), nextElement() Enumeration methods; same behavior but typed as Object.

Throws: NullPointerException from the constructor if str is null.

Usage notes

  • No empty tokens. "a,,b," gives two tokens, while split(",") gives a, an empty string and b. That can be what you want or a lost field; decide on purpose.
  • Delimiters are characters. There is no way to split on a multi-character separator such as "::" or on a pattern; use String.split, Pattern.split or Scanner.useDelimiter.
  • Not an Iterator, so it does not work in an enhanced for. Loop with hasMoreTokens().
  • No regex cost, which made it popular in programming-contest input code. Today BufferedReader plus split is also fast enough for most cases.
  • Not thread-safe. The tokenizer holds a position in the string.
  • Stays available. It is not deprecated with @Deprecated, but new code should use split, Scanner or streams: Arrays.stream(text.split("\\s+")).