In https://github.com/guardian/giant/pull/780/changes#diff-7313b78f0596187098ad28c728d42da778992c1d403d133d5ebbfd45b39deaa2 Phil introduced a new field ‘translation’ inside the languageData block.
The languageData block looks like this:
...
languageData: {
text: {
detectedLanguageCode: "fr",
translation: "good day",
},
ocr: {
detectedLanguageCode: {
english: "fr",
french: "fr",
},
translation: {
english: "good day",
french: "good day",
}
}There's some detail on this in the original language detection PR here #725
We use an explicit mapping for elasticsearch, which means that each time we add new fields we have to update the mapping. On startup, giant does some of this automatically. For example, if a new language is added to Languages.scala then a mapping for that language will be added to all multi language fields.
The current mapping for the languageData block is shown below. There are two key mistakes with it:
- the emailBody, emailSubject and text fields don't have
term_vector set, so we can't do highlighting on the contents of those fields
- in the ocr field,
detectedLanguageCode and translation are full blown multi language fields - with different analyzer depending on the language. This is stupid as detectedLanguageCode doesn't need indexing at all, and definitely not with different analyzers as it's just a 2 character code in the latin alphabet. It's also stupid for translation because that will always be in english.
The problems with this are:
- we can't do highlighting at all on the text/email fields
- ocr indexing is overly complex and wasteful
"languageData": {
"properties": {
"emailBody": {
"properties": {
"detectedLanguageCode": {
"type": "text"
},
"translation": {
"type": "text"
}
}
},
"emailSubject": {
"properties": {
"detectedLanguageCode": {
"type": "text"
},
"translation": {
"type": "text"
}
}
},
"ocr": {
"properties": {
"detectedLanguageCode": {
"properties": {
"arabic": {
"type": "text",
"fields": {
"exact": {
"type": "text",
"term_vector": "with_positions_offsets",
"analyzer": "standard"
}
},
"term_vector": "with_positions_offsets",
"analyzer": "arabic"
},
"english": {
"type": "text",
"fields": {
"exact": {
"type": "text",
"term_vector": "with_positions_offsets",
"analyzer": "standard"
}
},
"term_vector": "with_positions_offsets",
"analyzer": "english"
},
"french": {
"type": "text",
"fields": {
"exact": {
"type": "text",
"term_vector": "with_positions_offsets",
"analyzer": "standard"
}
},
"term_vector": "with_positions_offsets",
"analyzer": "french"
},
"german": {
"type": "text",
"fields": {
"exact": {
"type": "text",
"term_vector": "with_positions_offsets",
"analyzer": "standard"
}
},
"term_vector": "with_positions_offsets",
"analyzer": "german"
},
"persian": {
"type": "text",
"fields": {
"exact": {
"type": "text",
"term_vector": "with_positions_offsets",
"analyzer": "standard"
}
},
"term_vector": "with_positions_offsets",
"analyzer": "persian"
},
"portuguese": {
"type": "text",
"fields": {
"exact": {
"type": "text",
"term_vector": "with_positions_offsets",
"analyzer": "standard"
}
},
"term_vector": "with_positions_offsets",
"analyzer": "portuguese"
},
"russian": {
"type": "text",
"fields": {
"exact": {
"type": "text",
"term_vector": "with_positions_offsets",
"analyzer": "standard"
}
},
"term_vector": "with_positions_offsets",
"analyzer": "russian"
},
"spanish": {
"type": "text",
"fields": {
"exact": {
"type": "text",
"term_vector": "with_positions_offsets",
"analyzer": "standard"
}
},
"term_vector": "with_positions_offsets",
"analyzer": "spanish"
}
}
},
"translation": {
"properties": {
"arabic": {
"type": "text",
"fields": {
"exact": {
"type": "text",
"term_vector": "with_positions_offsets",
"analyzer": "standard"
}
},
"term_vector": "with_positions_offsets",
"analyzer": "arabic"
},
"english": {
"type": "text",
"fields": {
"exact": {
"type": "text",
"term_vector": "with_positions_offsets",
"analyzer": "standard"
}
},
"term_vector": "with_positions_offsets",
"analyzer": "english"
},
"french": {
"type": "text",
"fields": {
"exact": {
"type": "text",
"term_vector": "with_positions_offsets",
"analyzer": "standard"
}
},
"term_vector": "with_positions_offsets",
"analyzer": "french"
},
"german": {
"type": "text",
"fields": {
"exact": {
"type": "text",
"term_vector": "with_positions_offsets",
"analyzer": "standard"
}
},
"term_vector": "with_positions_offsets",
"analyzer": "german"
},
"persian": {
"type": "text",
"fields": {
"exact": {
"type": "text",
"term_vector": "with_positions_offsets",
"analyzer": "standard"
}
},
"term_vector": "with_positions_offsets",
"analyzer": "persian"
},
"portuguese": {
"type": "text",
"fields": {
"exact": {
"type": "text",
"term_vector": "with_positions_offsets",
"analyzer": "standard"
}
},
"term_vector": "with_positions_offsets",
"analyzer": "portuguese"
},
"russian": {
"type": "text",
"fields": {
"exact": {
"type": "text",
"term_vector": "with_positions_offsets",
"analyzer": "standard"
}
},
"term_vector": "with_positions_offsets",
"analyzer": "russian"
},
"spanish": {
"type": "text",
"fields": {
"exact": {
"type": "text",
"term_vector": "with_positions_offsets",
"analyzer": "standard"
}
},
"term_vector": "with_positions_offsets",
"analyzer": "spanish"
}
}
}
}
},
"text": {
"properties": {
"detectedLanguageCode": {
"type": "text"
},
"translation": {
"type": "text"
}
}
}
}
},How to fix this
In #785 I attempt to fix this by updating ElasticsearchSyntax.scala here https://github.com/guardian/giant/pull/785/changes#diff-c0dcd79765470ddbdccc49538f2ed970f248e64c675c1c5a4b55216dba6667fb so that languageData.text gets a proper index that can be used for highlights. However, I ran into problems when trying to apply the mapping update on playground - you can't do this:
curl -X PUT "http://localhost:19200/pfi/_mapping" \
-H 'Content-Type: application/json' \
-d '{
"properties": {
"languageData": {
"properties": {
"text": {
"properties": {
"translation": {
"type": "text",
"fields": {
"exact": {
"type": "text",
"analyzer": "standard",
"term_vector": "with_positions_offsets"
}
},
"term_vector": "with_positions_offsets",
"analyzer": "english"
}
}
}
}
}
}
}'So what to do instead? We can either make a new index and swap over to it (sounds hard) or maybe just abandon languageData.text.translation and introduce languageData.text.translation2 - that's what I did here 9581885 - and I think that would have worked except that it would introduce a translation2 field to languageData.ocr as well, using the current stupid multi language setup.
Proposed fix
I think we should change languageData so that none of the fields are multilingual - they should all just be plain english text fields. OCR should either:
- To keep things simple, we could stop supporting the situation where a document has been OCRd in multiple languages, and instead just pick whichever OCR field has more words in it or something, and then just have a single
languageData.ocr.translation field.
- Or, we can keep the current structure of
languageData.ocr.translation but make it a simple objectField with a dynamic mapping, so we aren't generating the language mapping for every language in languages.scala **Phil this this might be simplest **
For either solution, we would still need to introduce a translation2 field or something to deal with the fact that we can't change the mapping of an existing index.
Sorry to dump this right before going away. I think it's a day or two of work to sort out
In https://github.com/guardian/giant/pull/780/changes#diff-7313b78f0596187098ad28c728d42da778992c1d403d133d5ebbfd45b39deaa2 Phil introduced a new field ‘translation’ inside the languageData block.
The languageData block looks like this:
There's some detail on this in the original language detection PR here #725
We use an explicit mapping for elasticsearch, which means that each time we add new fields we have to update the mapping. On startup, giant does some of this automatically. For example, if a new language is added to Languages.scala then a mapping for that language will be added to all multi language fields.
The current mapping for the languageData block is shown below. There are two key mistakes with it:
term_vectorset, so we can't do highlighting on the contents of those fieldsdetectedLanguageCodeandtranslationare full blown multi language fields - with different analyzer depending on the language. This is stupid as detectedLanguageCode doesn't need indexing at all, and definitely not with different analyzers as it's just a 2 character code in the latin alphabet. It's also stupid fortranslationbecause that will always be in english.The problems with this are:
How to fix this
In #785 I attempt to fix this by updating ElasticsearchSyntax.scala here https://github.com/guardian/giant/pull/785/changes#diff-c0dcd79765470ddbdccc49538f2ed970f248e64c675c1c5a4b55216dba6667fb so that
languageData.textgets a proper index that can be used for highlights. However, I ran into problems when trying to apply the mapping update on playground - you can't do this:So what to do instead? We can either make a new index and swap over to it (sounds hard) or maybe just abandon languageData.text.translation and introduce languageData.text.translation2 - that's what I did here 9581885 - and I think that would have worked except that it would introduce a translation2 field to languageData.ocr as well, using the current stupid multi language setup.
Proposed fix
I think we should change languageData so that none of the fields are multilingual - they should all just be plain english text fields. OCR should either:
languageData.ocr.translationfield.languageData.ocr.translationbut make it a simple objectField with a dynamic mapping, so we aren't generating the language mapping for every language in languages.scala **Phil this this might be simplest **For either solution, we would still need to introduce a
translation2field or something to deal with the fact that we can't change the mapping of an existing index.Sorry to dump this right before going away. I think it's a day or two of work to sort out