Class TokenTypeUtils

java.lang.Object
org.carrot2.language.TokenTypeUtils

public final class TokenTypeUtils extends Object
Utility methods for working with Tokenizer attributes.
  • Method Details

    • maskType

      public static int maskType(int rawType)
      Mask the given raw token type and leave just the token type bits.
    • isDocumentSeparator

      public static boolean isDocumentSeparator(int type)
      Returns true if the given type has Tokenizer.TF_SEPARATOR_DOCUMENT set.
    • isFieldSeparator

      public static boolean isFieldSeparator(int type)
      Returns true if the given type has Tokenizer.TF_SEPARATOR_FIELD set.
    • isTerminator

      public static boolean isTerminator(int type)
      Returns true if the given type has Tokenizer.TF_TERMINATOR set.
    • isCommon

      public static boolean isCommon(int flag)
      Return true if Tokenizer.TF_COMMON_WORD is set.
    • isInQuery

      public static boolean isInQuery(int flag)
      Return true if Tokenizer.TF_QUERY_WORD is set.