com.github.pmerienne.trident.ml.preprocessing

Class TwitterTokenizer

java.lang.Object

com.github.pmerienne.trident.ml.preprocessing.EnglishTokenizer

com.github.pmerienne.trident.ml.preprocessing.TwitterTokenizer

All Implemented Interfaces:

TextTokenizer, Serializable

public class TwitterTokenizer
extends EnglishTokenizer

See Also:

Serialized Form

Field Summary

Fields inherited from class com.github.pmerienne.trident.ml.preprocessing.EnglishTokenizer

stemExclusionsSet, stopWords

Constructor Summary

Constructors

Constructor and Description
TwitterTokenizer()
TwitterTokenizer(int minNGram, int maxNGram)

Method Summary

Methods

Modifier and Type	Method and Description
protected String	preprocess(String tweet)
List<String>	tokenize(String text)

Methods inherited from class com.github.pmerienne.trident.ml.preprocessing.EnglishTokenizer

createTokenStream, getMaxNGram, getMinNGram, getStemExclusionsSet, getStopWords, isnGram, setMaxNGram, setMinNGram, setnGram, setNGram, setStemExclusionsSet, setStopWords

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TwitterTokenizer

public TwitterTokenizer()

TwitterTokenizer

public TwitterTokenizer(int minNGram,
                int maxNGram)

Method Detail

tokenize

public List<String> tokenize(String text)

Specified by:

tokenize in interface TextTokenizer

Overrides:

tokenize in class EnglishTokenizer

preprocess

protected String preprocess(String tweet)