ElasticSearch教程——自定義分詞器（轉學習使用）

一、分詞器

Elasticsearch中，內置了很多分詞器（analyzers），例如standard（標準分詞器）、english（英文分詞）和chinese（中文分詞），默認是standard.

standard tokenizer：以單詞邊界進行切分
standard token filter：什么都不做
lowercase token filter：將所有字母轉換為小寫
stop token filer（默認被禁用）：移除停用詞，比如a the it等等

?

二、修改分詞器設置

啟用english，停用詞token filter

PUT /my_index
{"settings": {"analysis": {"analyzer": {"es_std":{"type":"standard","stopwords":"_english_"}}}}
}

三、標準分詞測試代碼

GET /my_index/_analyze
{"analyzer": "standard","text":"a dog is in the house"
}

結果：

{"tokens": [{"token": "a","start_offset": 0,"end_offset": 1,"type": "<ALPHANUM>","position": 0},{"token": "dog","start_offset": 2,"end_offset": 5,"type": "<ALPHANUM>","position": 1},{"token": "is","start_offset": 6,"end_offset": 8,"type": "<ALPHANUM>","position": 2},{"token": "in","start_offset": 9,"end_offset": 11,"type": "<ALPHANUM>","position": 3},{"token": "the","start_offset": 12,"end_offset": 15,"type": "<ALPHANUM>","position": 4},{"token": "house","start_offset": 16,"end_offset": 21,"type": "<ALPHANUM>","position": 5}]
}

四、設置的英文分詞測試代碼

GET /my_index/_analyze
{"analyzer": "es_std","text":"a dog is in the house"}

結果：

{"tokens": [{"token": "dog","start_offset": 2,"end_offset": 5,"type": "<ALPHANUM>","position": 1},{"token": "house","start_offset": 16,"end_offset": 21,"type": "<ALPHANUM>","position": 5}]
}

五、自定義分詞器

PUT /my_index
{"settings": {"analysis": {"char_filter": {"&_to_and": {"type": "mapping","mappings": ["&=> and"]}},"filter": {"my_stopwords": {"type": "stop","stopwords": ["the", "a"]}},"analyzer": {"my_analyzer": {"type": "custom","char_filter": ["html_strip", "&_to_and"],"tokenizer": "standard","filter": ["lowercase", "my_stopwords"]}}}}
}

測試：

GET /my_index/_analyze
{"text": "tom&jerry are a friend in the house, <a>, HAHA!!","analyzer": "my_analyzer"
}

結果：

{"tokens": [{"token": "tomandjerry","start_offset": 0,"end_offset": 9,"type": "<ALPHANUM>","position": 0},{"token": "are","start_offset": 10,"end_offset": 13,"type": "<ALPHANUM>","position": 1},{"token": "friend","start_offset": 16,"end_offset": 22,"type": "<ALPHANUM>","position": 3},{"token": "in","start_offset": 23,"end_offset": 25,"type": "<ALPHANUM>","position": 4},{"token": "house","start_offset": 30,"end_offset": 35,"type": "<ALPHANUM>","position": 6},{"token": "haha","start_offset": 42,"end_offset": 46,"type": "<ALPHANUM>","position": 7}]
}

六、type中的使用

PUT /my_index/_mapping/my_type
{"properties": {"content": {"type": "text","analyzer": "my_analyzer"}}
}

?