You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: Pythia.Udp.Plugin/README.md
+76-1Lines changed: 76 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,7 +2,13 @@
2
2
3
3
Integrating UDPipe in Pythia requires two components: a text filter and a token filter. The text filter is responsible for submitting the document’s text to the UDPipe service and collecting the POS tags returned by it. The token filter is responsible for matching the tokens in the document with the tokens returned by UDPipe, and storing the relevant POS data in the index.
4
4
5
-
## Text Filter
5
+
## Text Filters
6
+
7
+
### XML Tag Filler Text Filter
8
+
9
+
This generic filter is used to blank-fill text corresponding to specified XML elements. For instance a TEI `choice` element typically contains a child `abbr` and a child `expan` and we do NOT want to index the `expan` content because it is an editorial addition outside of the text. So before submitting the text to UDPipe we blank fill the whole `expan` element with its content.
10
+
11
+
### UDPipe Text Filter
6
12
7
13
The UDPipe text filter belongs to the family of text filters components, i.e. it's a filter applied once to the whole document, at the beginning of the analysis process. The purpose of this filter is not changing the document’s text, but only submitting it to the UDPipe service, in order to get back POS tags for its content. This submission usually does not happen at once, but rather in chunks, so that POST requests to the UDPipe service are granted a smaller body. In this case, the text filter is designed in such a way to avoid splitting a sentence into different chunks (unless it happens to be longer than the maximum allowed chunk size, as configured in the profile).
8
14
@@ -13,3 +19,72 @@ In the end this filter collects all the POS tags for the document's content. So,
13
19
Later, after the text has been tokenized, the UDP token filter comes into play. Its task is matching the token being filtered with the token (if any) defined by UDPipe, extract all the POS data from it, and store into the target index the subset of them specified by the analysis configuration.
14
20
15
21
Token matching happens in a rather mechanical way: as the filter has the character-based offset of the token being processed and its length, it scans the POS data got by the UDP text filter and matches the first UDPipe token overlapping it. This is made possible by the fact that the text filter requested the POS data together with the offsets and extent of each token (passed via the CONLLU `Misc` field). So, whatever the original format of the document and the differences in tokenization, in most cases this produces the expected result.
22
+
23
+
A corner case in this filter is represented by **multiword tokens** like Italian "della" = "di" + "la".
24
+
For instance, consider this Italian sentence: "Questo è della casa.". The POS tagger analyzes this as follows:
25
+
26
+
| id | form | lemma | UPos | XPos | Feats | Head | DepRel | Deps | Misc |
Here `della` is the multiword token (tokens 3-4) and its children (3 and 4) follow it.
37
+
38
+
In Pythia we need to get POS data for the single `della` token. Filters applied to this token know only about it; `di` and `la` are 'artifacts' from the POS tagger, while here we need to represent all the data on top of the original text, where only `della` exists.
39
+
40
+
The filter has been implemented to:
41
+
42
+
1. detect the multiword token (`della`);
43
+
2. collect its "children" tokens (`di` and `la`);
44
+
3. lookup the configuration provided in filter options to inject specific POS data into the token being processed depending on its children.
45
+
46
+
In most cases, we need to POS-tag the form as it appears on the text (like `della`) rather than its analysis (`di` + `la`), because we are here focusing on a text-based search which must reflect the document's text. Compare for instance the Morph-It list of Italian inflected forms, where POS tags are designed to fit this specific language, and forms like `della` are at the same level as words like `di` or `la`. In this case, the POS tag is `ART-F:s`, meaning article, feminine, singular.
47
+
48
+
In the same way, here for each typology of such composite forms we must decide which subset of POS data from its components should make their way into the POS tag of the surface form `della`:
Here we will typically let the morphologically and syntactically richer component prevail, so that we treat the whole form `della` just like a `DET.RD`, ignoring its composite origin. Anyway, the decision will vary accoring to the typology of composition in the various forms presenting multi-word tokens. So, we need a generalized and configurable strategy to deal with them.
54
+
55
+
To provide a generic, reusable configuration to build a new lemma and tag by variously collecting data from the children tokens, you use the `Multiwords` option in the filter configuration object (see unit tests for an example). In the case of `della`, the configuration tells the filter to match tokens `ADP.E` followed by `DET.RO`, and provide the token's value as the lemma (which is the default behavior), plus UPOS, XPOS and features from the second token (`la`). The corresponding JSON configuration would be:
56
+
57
+
```json
58
+
{
59
+
"Id": "token-filter.udp",
60
+
"Options": {
61
+
"Props": 43,
62
+
"Language": "",
63
+
"Multiwords": [
64
+
{
65
+
"MinCount": 2,
66
+
"MaxCount": 2,
67
+
"Tokens": [
68
+
{
69
+
"Upos": "ADP",
70
+
"Xpos": "E"
71
+
},
72
+
{
73
+
"Upos": "DET",
74
+
"Xpos": "RD"
75
+
}
76
+
],
77
+
"Target": {
78
+
"Upos": "DET",
79
+
"Xpos": "RD",
80
+
"Feats": {
81
+
"*": "2"
82
+
}
83
+
}
84
+
}
85
+
]
86
+
}
87
+
},
88
+
```
89
+
90
+
This matches any 2-tokens multiword token having `ADP.E` for its first token and `DET.RD` as its second one. We might also add `Feats` to the filters, but that's not required here. The resulting tag for `della` is `DET.RD` and its `Feats` are copied from all the features of the second token. The lemma is just equal to the token's value (`della`), as this is the default when `Lemma` is not specified.
0 commit comments