com.github.pmerienne.trident.ml.preprocessing

Class EnglishTokenizer

java.lang.Object

com.github.pmerienne.trident.ml.preprocessing.EnglishTokenizer

All Implemented Interfaces:

TextTokenizer, Serializable

Direct Known Subclasses:

TwitterTokenizer

public class EnglishTokenizer
extends Object
implements Serializable, TextTokenizer

Abstraction of lucene tokenizer. This file contains custom re-implementation of EnglishAnalyzer, PorterStemFilter and PorterStemmer to fix Issue LUCENE-3335 causing sigsegv!

Author:

pmerienne

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected Set<String>	stemExclusionsSet
protected List<String>	stopWords

Constructor Summary

Constructors

Constructor and Description
EnglishTokenizer()
EnglishTokenizer(int minNGram, int maxNGram)
EnglishTokenizer(List<String> stopWords)
EnglishTokenizer(List<String> stopWords, Set<String> stemExclusionsSet)

Method Summary

Methods

Modifier and Type	Method and Description
protected org.apache.lucene.analysis.TokenStream	createTokenStream(String text)
int	getMaxNGram()
int	getMinNGram()
Set<String>	getStemExclusionsSet()
List<String>	getStopWords()
boolean	isnGram()
void	setMaxNGram(int maxNGram)
void	setMinNGram(int minNGram)
void	setnGram(boolean nGram)
void	setNGram(int minNGram, int maxNGram)
void	setStemExclusionsSet(Set<String> stemExclusionsSet)
void	setStopWords(List<String> stopWords)
List<String>	tokenize(String text)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

stopWords

protected List<String> stopWords

stemExclusionsSet

protected Set<String> stemExclusionsSet

Constructor Detail

EnglishTokenizer

public EnglishTokenizer()

EnglishTokenizer

public EnglishTokenizer(int minNGram,
                int maxNGram)

EnglishTokenizer

public EnglishTokenizer(List<String> stopWords)

EnglishTokenizer

public EnglishTokenizer(List<String> stopWords,
                Set<String> stemExclusionsSet)

Method Detail

tokenize

public List<String> tokenize(String text)

Specified by:

tokenize in interface TextTokenizer

createTokenStream

protected org.apache.lucene.analysis.TokenStream createTokenStream(String text)

setNGram

public void setNGram(int minNGram,
            int maxNGram)

isnGram

public boolean isnGram()

setnGram

public void setnGram(boolean nGram)

getMinNGram

public int getMinNGram()

setMinNGram

public void setMinNGram(int minNGram)

getMaxNGram

public int getMaxNGram()

setMaxNGram

public void setMaxNGram(int maxNGram)

getStopWords

public List<String> getStopWords()

setStopWords

public void setStopWords(List<String> stopWords)

setStemExclusionsSet

public void setStemExclusionsSet(Set<String> stemExclusionsSet)

getStemExclusionsSet

public Set<String> getStemExclusionsSet()