-
Notifications
You must be signed in to change notification settings - Fork 1
Expand file tree
/
Copy pathdocumentation.html
More file actions
195 lines (173 loc) · 7.43 KB
/
Copy pathdocumentation.html
File metadata and controls
195 lines (173 loc) · 7.43 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0"/>
<title>Important Dates | POLAR @ 2026</title>
<link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/css/bootstrap.min.css" rel="stylesheet"/>
<link rel="stylesheet" href="assets/style.css"/>
</head>
<body>
<nav class="nav nav-tabs justify-content-center shadow-sm bg-white sticky-top">
<a class="nav-link" href="index.html#about">About</a>
<a class="nav-link" href="tasks.html">Tasks</a>
<a class="nav-link" href="participation.html">Participation</a>
<a class="nav-link" href="organizers.html">Organizers</a>
</nav>
<section class="task-documentation-section">
<div class="container" style="margin-top: 2rem; max-width: 960px;">
<!-- Page Title -->
<div class="text--center" style="margin-bottom: 2rem;">
<div style="font-size: 3rem;">📚</div>
<h1>Task Documentation</h1>
<h2 style="font-weight: normal; font-size: 1.25rem; margin-top: 0.5rem;">
SemEval-2026 Shared Task<br>
<em>Attitude Polarization Detection in Multilingual News Commentary (POLAR)</em>
</h2>
</div>
<!-- Overview -->
<div style="margin-bottom: 2rem;">
<h3>Overview</h3>
<p>
This documentation provides all the details needed to understand, implement, and participate in the POLAR shared task. It includes dataset format, annotation scheme, task variants, evaluation metrics, and guidelines for submission.
</p>
</div>
<!-- Task Definition -->
<div style="margin-bottom: 2rem;">
<h3>🧪 Task Definition</h3>
<p>
Participants must develop models to detect and interpret attitude polarization in user comments from news discussions across multiple languages and topics. Given a news comment and context, the system should predict:
</p>
<ul>
<li><strong>Polarization Category</strong> (binary or multi-class depending on subtask)</li>
<li><strong>Token-Level Polarization Cues</strong> (for interpretability)</li>
</ul>
</div>
<!-- Subtasks -->
<div style="margin-bottom: 2rem;">
<h3>🧩 Subtasks</h3>
<ul>
<li><strong>🔹 Subtask A: Polarization Classification</strong><br>
<em>Goal:</em> Classify whether a comment expresses a polarized stance.<br>
<em>Input:</em> News comment + metadata (language, event, topic)<br>
<em>Output:</em> Label – Polarized or Not Polarized
</li>
<li><strong>🔹 Subtask B: Multi-Topic, Multi-Language Classification</strong><br>
<em>Goal:</em> Classify the type of polarization (e.g., political, religious, ethnic)<br>
<em>Output:</em> Multi-class label (e.g., Political, Religious, Ethnic, None)<br>
<em>Languages:</em> English, German, Amharic, Hausa
</li>
<li><strong>🔹 Subtask C: Token-Level Cue Identification</strong><br>
<em>Goal:</em> Highlight which tokens or phrases contribute to perceived polarization<br>
<em>Output:</em> Token-level binary labels (0: neutral, 1: polarizing cue)
</li>
</ul>
</div>
<!-- Dataset -->
<div style="margin-bottom: 2rem;">
<h3>🗃️ Dataset</h3>
<ul>
<li><strong>Languages:</strong> English, German, Amharic, Hausa</li>
<li><strong>Domains:</strong> News comments on real-world events (conflicts, elections, crises)</li>
<li><strong>Annotations:</strong> Human-labeled for both comment-level and token-level polarization</li>
<li><strong>Sources:</strong> Multilingual news portals, social commentary archives</li>
<li><strong>Structure:</strong></li>
</ul>
</div>
<!-- Annotation Guidelines -->
<div style="margin-bottom: 2rem;">
<h3>🏷️ Annotation Guidelines</h3>
<p>
Polarized comments show strong alignment, antagonism, or emotional extremes toward social groups, ideologies, or policies. Cues include slurs, framing, emphasis, exaggeration, selective blame, etc.
Native annotators provided culturally grounded judgments for each language.
</p>
<p>
A full annotation guideline document (PDF) will be made available on the Resources page.
</p>
</div>
<!-- Evaluation -->
<div style="margin-bottom: 2rem;">
<h3>🧮 Evaluation</h3>
<ul>
<li><strong>Subtask A and B:</strong><br>
Accuracy, F1 Score (macro & weighted), Class-wise precision & recall</li>
<li><strong>Subtask C:</strong><br>
Token-level F1 Score, Span-level Match Score (Jaccard), Optional human judgment for interpretability
</li>
</ul>
</div>
<!-- Baseline Models -->
<div style="margin-bottom: 2rem;">
<h3>💾 Baseline Models</h3>
<ul>
<li>Multilingual BERT / XLM-R</li>
<li>AfroXLM-R for low-resource languages</li>
<li>Rule-based polarity patterns (for comparison)</li>
</ul>
<p>
Baseline code and training scripts will be published on our GitHub repository.
</p>
</div>
<!-- Submission Format -->
<div style="margin-bottom: 2rem;">
<h3>🚀 Submission Format</h3>
<p>
<strong>Prediction file:</strong> JSONL format<br>
<strong>Required fields:</strong> id, label, and optionally cue_spans<br>
Each team may submit up to 3 runs per subtask.
</p>
<p>Submissions accepted via CodaLab during evaluation phase</p>
</div>
<!-- Timeline -->
<div style="margin-bottom: 2rem;">
<h3>📅 Timeline</h3>
<table style="width: 100%; border-collapse: collapse;">
<thead>
<tr>
<th style="text-align: left; border-bottom: 1px solid #ccc;">Phase</th>
<th style="text-align: left; border-bottom: 1px solid #ccc;">Date</th>
</tr>
</thead>
<tbody>
<tr><td>Task Release</td><td>Sept 2025</td></tr>
<tr><td>Trial Data Release</td><td>Sept 2025</td></tr>
<tr><td>Training Data Release</td><td>Oct 2025</td></tr>
<tr><td>Evaluation Phase</td><td>Jan 2026</td></tr>
<tr><td>Leaderboard + Feedback</td><td>Feb 2026</td></tr>
<tr><td>Paper Submission Due</td><td>March 2026</td></tr>
</tbody>
</table>
</div>
<!-- Resources -->
<div style="margin-bottom: 2rem;">
<h3>📎 Resources</h3>
<ul>
<li><a href="#">Annotation Guidelines (PDF)</a></li>
<li><a href="#">Baseline Code (GitHub)</a></li>
<li><a href="#">Frequently Asked Questions (FAQ)</a></li>
<li><a href="#">Related Research Papers</a></li>
</ul>
</div>
<!-- Contact -->
<div style="margin-bottom: 2rem;">
<h3>📬 Contact</h3>
<p>
For any questions, reach out to us at:<br>
📧 <a href="mailto:polar-task@semeval.org">polar-task@semeval.org</a><br>
Or open an issue on <a href="#">GitHub Discussions</a>
</p>
</div>
<hr>
</div>
</section>
<footer class="footer-section">
<div class="footer-container text-center">
<img src="assets/logo-small.png" alt="POLAR Task Logo" style="width: 80px; height: auto; margin-bottom: 10px;" />
<p>© 2026 POLAR Shared Task. All rights reserved.</p>
<p>Made with ❤️ by the POLAR Team</p>
</div>
</footer>
<!-- Bootstrap JS -->
<script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/js/bootstrap.bundle.min.js"></script>
</body>
</html>