edu.cmu.minorthird.text
Interface Tokenizer

All Known Implementing Classes:

CompoundTokenizer, FilterTokenizer, RegexTokenizer, SpanTypeTokenizer, SplitTokenizer

public interface Tokenizer

Method Summary

TextToken[]	splitIntoTokens(Document document) Tokenize a document.
java.lang.String[]	splitIntoTokens(java.lang.String string) Tokenize a string.

Method Detail

splitIntoTokens

java.lang.String[] splitIntoTokens(java.lang.String string)

Tokenize a string.

splitIntoTokens

TextToken[] splitIntoTokens(Document document)

Tokenize a document.