One language per doc - Elasticsearch 权威指南中文版


One language per doc

发布于 2019-07-04 字数 3527 浏览 788 评论 0

=== One Language per Document

A single predominant language per document (((“languages”, “one language per document”)))(((“indices”, “documents in different languages”)))requires a relatively simple setup.
Documents from different languages can be stored in separate indices—blogs-en,
blogs-fr, and so forth—that use the same type and the same fields for each index,
just with different analyzers:


PUT /blogs-en
“mappings”: {
“post”: {
“properties”: {
“title”: {
“type”: “string”,
“fields”: {
“stemmed”: {
“type”: “string”,
“analyzer”: “english”

PUT /blogs-fr { “mappings”: { “post”: { “properties”: { “title”: { “type”: “string”, “fields”: { “stemmed”: { “type”: “string”, “analyzer”: “french” } }}}}}}

Both `blogs-en` and `blogs-fr` have a type called `post` that contains
the field `title`.
The `title.stemmed` subfield uses a language-specific analyzer.

This approach is clean and flexible. New languages are easy to add–just
create a new index–and because each language is completely separate, we
don’t suffer from the term-frequency and stemming problems described in

The documents of a single language can be queried independently, or queries
can target multiple languages by querying multiple indices. We can even
specify a preference(((“indices_boost parameter”, “specifying preference for a specific language”))) for particular languages with the indices_boost parameter:


GET /blogs-*/post/_search { “query”: { “multi_match”: { “query”: “deja vu”, “fields”: [ “title”, “title.stemmed” ] “type”: “most_fields” } }, “indices_boost”: { “blogs-en”: 3, “blogs-fr”: 2 } }

This search is performed on any index beginning with `blogs-`.
The `title.stemmed` fields are queried using the analyzer
specified in each index.
Perhaps the user’s `accept-language` headers showed a preference for
English, and then French, so we boost results from each index accordingly.
Any other languages will have a neutral boost of `1`.

==== Foreign Words

Of course, these documents may contain words or sentences in other languages,
and these words are unlikely to be stemmed correctly. With
predominant-language documents, this is not usually a major problem. The user will
often search for the exact words–for instance, of a quotation from another
language–rather than for inflections of a word. Recall can be improved
by using techniques explained in <>.

Perhaps some words like place names should be queryable in the predominant
language and in the original language, such as Munich and München. These
words are effectively synonyms, which we discuss in <>.

.Don’t Use Types for Languages

You may be tempted to use a separate type for each language,(((“types”, “not using for languages”)))(((“languages”, “not using types for”))) instead of a
separate index. For best results, you should avoid using types for this
purpose. As explained in <>, fields from different types but with
the same field name are indexed into the same inverted index. This means
that the term frequencies from each type (and thus each language) are mixed

To ensure that the term frequencies of one language don’t pollute those of
another, either use a separate index for each language, or a separate field,
as explained in the next section.

上一篇:Language pitfalls

下一篇:One language per field


需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。