Compound words - Elasticsearch 权威指南中文版

返回介绍

Compound words

发布于 2019-07-04 字数 5153 浏览 774 评论 0

[[ngrams-compound-words]]
=== Ngrams for Compound Words

Finally, let’s take a look at how n-grams can be used to search languages with
compound words. (((“languages”, “using many compound words, indexing of”)))(((“n-grams”, “using with compound words”)))(((“partial matching”, “using n-grams for compound words”)))(((“German”, “compound words in”))) German is famous for combining several small words into one
massive compound word in order to capture precise or complex meanings. For
example:

Aussprachewörterbuch::
Pronunciation dictionary

Militärgeschichte::
Military history

Weißkopfseeadler::
White-headed sea eagle, or bald eagle

Weltgesundheitsorganisation::
World Health Organization

Rindfleischetikettierungsüberwachungsaufgabenübertragungsgesetz::
The law concerning the delegation of duties for the supervision of cattle
marking and the labeling of beef

Somebody searching for Wörterbuch'' (dictionary) would probably expect to seeAussprachewörtebuch” in the results list. Similarly, a search for
Adler'' (eagle) should includeWeißkopfseeadler.”

One approach to indexing languages like this is to break compound words into
their constituent parts using the http://bit.ly/1ygdjjC[compound word token filter].
However, the quality of the results depends on how good your compound-word
dictionary is.

Another approach is just to break all words into n-grams and to search for any
matching fragments–the more fragments that match, the more relevant the
document.

Given that an n-gram is a moving window on a word, an n-gram of any length
will cover all of the word. We want to choose a length that is long enough
to be meaningful, but not so long that we produce far too many unique terms.
A trigram (length 3) is (((“trigrams”)))probably a good starting point:

[source,js]

PUT /my_index { “settings”: { “analysis”: { “filter”: { “trigrams_filter”: { “type”: “ngram”, “min_gram”: 3, “max_gram”: 3 } }, “analyzer”: { “trigrams”: { “type”: “custom”, “tokenizer”: “standard”, “filter”: [ “lowercase”, “trigrams_filter” ] } } } }, “mappings”: { “my_type”: { “properties”: { “text”: { “type”: “string”, “analyzer”: “trigrams” } } } } }

// SENSE: 130_Partial_Matching/40_Compound_words.json

The `text` field uses the `trigrams` analyzer to index its contents as
n-grams of length 3.

Testing the trigrams analyzer with the analyze API

[source,js]

GET /my_index/_analyze?analyzer=trigrams Weißkopfseeadler

// SENSE: 130_Partial_Matching/40_Compound_words.json

returns these terms:

wei, eiß, ißk, ßko, kop, opf, pfs, fse, see, eea,ead, adl, dle, ler

We can index our example compound words to test this approach:

[source,js]

POST /my_index/my_type/_bulk { “index”: { “_id”: 1 }} { “text”: “Aussprachewörterbuch” } { “index”: { “_id”: 2 }} { “text”: “Militärgeschichte” } { “index”: { “_id”: 3 }} { “text”: “Weißkopfseeadler” } { “index”: { “_id”: 4 }} { “text”: “Weltgesundheitsorganisation” } { “index”: { “_id”: 5 }} { “text”: “Rindfleischetikettierungsüberwachungsaufgabenübertragungsgesetz” }

// SENSE: 130_Partial_Matching/40_Compound_words.json

A search for `Adler'' (eagle) becomes a query for the three termsadl,dle, andler`:

[source,js]

GET /my_index/my_type/_search { “query”: { “match”: { “text”: “Adler” } } }

// SENSE: 130_Partial_Matching/40_Compound_words.json

which correctly matches “Weißkopfsee-adler”:

[source,js]

{ “hits”: [ { “_id”: “3”, “_score”: 3.3191128, “_source”: { “text”: “Weißkopfseeadler” } } ] }

// SENSE: 130_Partial_Matching/40_Compound_words.json

A similar query for Gesundheit'' (health) correctly matches Welt-gesundheit-sorganisation,” but it also matches
Militär-__ges__-chichte'' and Rindfleischetikettierungsüberwachungsaufgabenübertragungs-ges-etz,”
both of which also contain the trigram ges.

Judicious use of the minimum_should_match parameter can remove these
spurious results by requiring that a minimum number of trigrams must be
present for a document to be considered a match:

[source,js]

GET /my_index/my_type/_search { “query”: { “match”: { “text”: { “query”: “Gesundheit”, “minimum_should_match”: “80%” } } } }

// SENSE: 130_Partial_Matching/40_Compound_words.json

This is a bit of a shotgun approach to full-text search and can result in a
large inverted index, but it is an effective generic way of indexing languages
that use many compound words or that don’t use whitespace between words,
such as Thai.

This technique is used to increase recall—the number of relevant
documents that a search returns. It is usually used in combination with
other techniques, such as shingles (see <>) to improve precision and
the relevance score of each document.

上一篇:Search as you type

下一篇:Relevance

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

目前还没有任何评论,快来抢沙发吧!