|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
public interface Tokenizer
Chops a string or text file into Token instances.
Method Summary | |
---|---|
java.lang.String |
getErrorDescription()
If hasErrors returns true, returns a description of the error encountered. |
Token |
getNextToken()
Returns the next token. |
boolean |
hasErrors()
Returns true if there were errors while reading tokens. |
boolean |
hasMoreTokens()
Returns true if there are more tokens, false otherwise. |
boolean |
isBreak()
Determines if the current token should start a new sentence. |
void |
setInputReader(java.io.Reader reader)
Sets the input reader. |
void |
setInputText(java.lang.String textToTokenize)
Sets the text to be tokenized by this tokenizer. |
void |
setPostpunctuationSymbols(java.lang.String symbols)
Sets the postpunctuation symbols of this Tokenizer to the given symbols. |
void |
setPrepunctuationSymbols(java.lang.String symbols)
Sets the prepunctuation symbols of this Tokenizer to the given symbols. |
void |
setSingleCharSymbols(java.lang.String symbols)
Sets the single character symbols of this Tokenizer to the given symbols. |
void |
setWhitespaceSymbols(java.lang.String symbols)
Sets the whitespace symbols of this Tokenizer to the given symbols. |
Method Detail |
---|
void setInputText(java.lang.String textToTokenize)
textToTokenize
- the text to tokenizevoid setInputReader(java.io.Reader reader)
reader
- the input sourceToken getNextToken()
boolean hasMoreTokens()
boolean hasErrors()
java.lang.String getErrorDescription()
void setWhitespaceSymbols(java.lang.String symbols)
symbols
- the whitespace symbolsvoid setSingleCharSymbols(java.lang.String symbols)
symbols
- the single character symbolsvoid setPrepunctuationSymbols(java.lang.String symbols)
symbols
- the prepunctuation symbolsvoid setPostpunctuationSymbols(java.lang.String symbols)
symbols
- the postpunctuation symbolsboolean isBreak()
|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |