Internationalization support

docs/architecture.md

@@ -124,6 +124,8 @@ You can drop into a python shell to leverage this as well by running `dojo flask
 
 The docker socket of the docker-in-docker daemon is mapped into the CTFd container, allowing CTFd to start up user challenge containers.
 
+Dojo content can be translated into other languages; see [internationalization](i18n.md).
+
 ## Challenge containers
 
 When a user launches a challenge, CTFd starts a docker container that will run alongside the infrastructure containers, and:

docs/i18n.md

@@ -0,0 +1,179 @@
+# Internationalization
+
+Dojo content — the names and descriptions of dojos, modules, challenges, and module
+resources — can be shipped in more than one language.
+A visitor picks a language from the switcher in the navigation bar, and every piece of
+content that has a translation for it is rendered in that language.
+Anything without a translation falls back to the source text, so a partially translated
+dojo is always coherent.
+
+## Supported languages
+
+The languages offered by the switcher are listed in `dojo_plugin/i18n.py`:
+
+```python
+LANGUAGES = {
+    "en": "English",
+    "ko": "한국어",
+    "zh-CN": "简体中文",
+    "zh-TW": "繁體中文",
+    "it": "Italiano",
+}
+```
+
+A dojo may carry translations for tags that are not in this list; they are simply not
+selectable until the platform adds them.
+Language tags are BCP-47-ish (`ko`, `ko-KR`, `zh-Hans-CN`), and a region-qualified tag
+falls back to its base language before falling back to the source text.
+
+Write translations under the tags in `LANGUAGES` — a dojo that files its Chinese under
+`zh-Hans` is not found by a visitor who selected `zh-CN`.
+`LANGUAGE_ALIASES`, alongside `LANGUAGES`, maps the tags browsers actually send onto the
+ones offered, so `zh`, `zh-Hans-CN`, and `zh-SG` all select `zh-CN`, and `zh-Hant`,
+`zh-HK`, and `zh-MO` all select `zh-TW`.
+It steers the choice of language; it does not affect how a translation is looked up once
+the language is chosen.
+
+## How a language is chosen
+
+For each request, in order:
+
+1. a `?lang=` query parameter
+2. the `dojo_language` cookie, set by the switcher
+3. the browser's `Accept-Language` header
+4. `en`
+
+The choice is per-browser, not per-account: it lives in a cookie and nothing about it is
+written to the database, so signing in on another machine starts from `Accept-Language`
+again.
+
+## Adding translations to a dojo
+
+### The `i18n/` tree
+
+The recommended layout is a parallel tree that mirrors the dojo's own structure, one
+directory per language:
+
+```
+repo-root/
+  dojo.yml
+  DESCRIPTION.md
+  hello/
+    module.yml
+    DESCRIPTION.md
+    hello/
+      DESCRIPTION.md
+  i18n/
+    ko/
+      dojo.yml                  <- translated dojo name
+      DESCRIPTION.md            <- translated dojo description
+      hello/
+        module.yml              <- translated module name + resource names/content
+        DESCRIPTION.md          <- translated module description
+        hello/
+          challenge.yml         <- translated challenge name (optional)
+          DESCRIPTION.md        <- translated challenge description
+```
+
+Every file is optional. Only what exists gets translated.
+
+`i18n/<lang>/dojo.yml` and `i18n/<lang>/<module>/<challenge>/challenge.yml` accept `name`
+and `description`.
+`i18n/<lang>/<module>/module.yml` accepts `name`, `description`, and a `resources` list.
+
+### Translating resources
+
+A module's resources are an ordered, heterogeneous list, so each entry in the translated
+`resources` list carries exactly one locator saying which resource it applies to:
+
+| Locator | Matches | Use for |
+| --- | --- | --- |
+| `id` | the resource's challenge id | challenges |
+| `source` | the resource's source (English) `name` | lectures, markdown resources |
+| `index` | the resource's position in the list | headers, which have neither |
+
+The remaining keys (`name`, `content`, `description`) are the translated values:
+
+```yaml
+# i18n/ko/hello/module.yml
+name: 안녕, 해커
+resources:
+- id: hello
+  name: 명령어 입문
+- source: The Command Line
+  name: 커맨드 라인
+- source: Other Tutorials
+  name: 다른 튜토리얼
+  content: |
+    - [방대한 bash 튜토리얼](https://bash.cyberciti.biz/guide/Main_Page).
+- index: 4
+  content: 고급 섹션
+```
+
+A locator that matches no resource — or more than one — is an error, reported when the
+dojo is created or updated.
+This is deliberate: a translation that silently stops being applied, or lands on the wrong
+resource, is worse than one that tells you it has drifted from the content it translates.
+Consolidated dojos often repeat a resource name (several modules each contributing a
+"Resources" block); use `index` for those.
+
+`source` matches a resource's `name`, so it cannot address a header — headers have only
+`content`, and in a consolidated dojo a header's content is frequently identical to a
+nearby markdown resource's name. Address headers by `index`.
+
+### Inline `translations:`
+
+Translations can also be written directly into the dojo's own yml files, at any level that
+has a `name`, `description`, or `content`:
+
+```yaml
+id: linux-luminarium
+name: Linux Luminarium
+translations:
+  ko:
+    name: 리눅스 루미나리움
+    description: |
+      리눅스 루미나리움에 오신 것을 환영합니다!
+```
+
+An inline `translations:` block wins over the `i18n/` tree, the same way a `description:`
+in `dojo.yml` wins over `DESCRIPTION.md`.
+This form is mostly useful for dojos created from a spec rather than a repository.
+
+## Imported dojos, modules, and challenges
+
+A dojo that imports content inherits that content's translations. The inheritance is
+per-language and per-field, so a consolidated dojo can rename a challenge in its own
+language without losing the translated description it inherits:
+
+```yaml
+- type: challenge
+  id: hello-hello
+  name: Intro to Commands
+  translations:
+    ko:
+      name: 명령어 입문        # wins over the source's translated name
+  import:                       # the source's translated description is still inherited
+    dojo: linux-luminarium
+    module: hello
+    challenge: hello
+```
+
+Note the consequence: an imported challenge has no `description` of its own, so its
+description — and its translation — can only be changed in the dojo it is imported from.
+
+## Implementation notes
+
+Translations are stored in each model's existing JSONB `data` column under a
+`translations` key, so this feature adds no table and no column, and needs no migration.
+The key is written only for content that actually has a translation, so an untranslated
+dojo's rows are exactly what a build without i18n would write.
+`LocalizedMixin` in `dojo_plugin/models/__init__.py` resolves them:
+`dojo.localized_name`, `module.localized_description`, and `resource.localized_content`
+each return the translation for the request's language, or the source value.
+Templates and the JSON API use these properties throughout; the underlying `name` and
+`description` columns always hold the source text, which is what admin views and dojo
+imports operate on.
+
+`Dojos.languages` records which languages the dojo repo ships, derived from the
+subdirectories of `i18n/`.

dojo_plugin/__init__.py

@@ -23,6 +23,7 @@
 from .models import Dojos, DojoChallenges, Belts, Emojis
 from .config import DOJO_HOST, bootstrap
 from .utils import unserialize_user_flag, render_markdown
+from .i18n import init_language
 from .utils.dojo import get_current_dojo_challenge
 from .utils.awards import update_awards
 from .utils.feed import publish_challenge_solve
@@ -35,6 +36,7 @@
 from .pages.users import users
 from .pages.settings import settings_override
 from .pages.discord import discord
+from .pages.language import language
 from .pages.course import course
 from .pages.belts import belts
 from .pages.research import research
@@ -227,6 +229,7 @@ def publish_stat_events_after_request(response):
     app.register_blueprint(workspace)
     app.register_blueprint(sensai)
     app.register_blueprint(discord)
+    app.register_blueprint(language)
     app.register_blueprint(users)
     app.register_blueprint(course)
     app.register_blueprint(belts)
@@ -237,6 +240,8 @@ def publish_stat_events_after_request(response):
 
     app.jinja_env.filters["markdown"] = render_markdown
 
+    init_language(app)
+
     register_admin_plugin_menu_bar("Dojos", "/admin/dojos")
 
     before_request_funcs = app.before_request_funcs[None]

dojo_plugin/api/v1/dojos.py

@@ -47,8 +47,8 @@ def get(self):
         dojos = [
             dict(id=dojo.reference_id,
                  hex_id=dojo.hex_dojo_id,
-                 name=dojo.name,
-                 description=dojo.description,
+                 name=dojo.localized_name,
+                 description=dojo.localized_description,
                  type=dojo.type,
                  official=dojo.official,
                  award=dojo.award,
@@ -179,13 +179,13 @@ def get(self, dojo):
         is_dojo_admin = dojo.is_admin()
         modules = [
             dict(id=module.id,
-                 name=module.name,
-                 description=module.description,
+                 name=module.localized_name,
+                 description=module.localized_description,
                  resources=[
                     dict(id=f"resource-{resource.resource_index}",
-                         name=resource.name,
+                         name=resource.localized_name,
                          type=resource.type,
-                         content=getattr(resource, 'content', None) if resource.type == "markdown" else None,
+                         content=resource.localized_content if resource.type == "markdown" else None,
                          video=getattr(resource, 'video', None) if resource.type == "lecture" else None,
                          playlist=getattr(resource, 'playlist', None) if resource.type == "lecture" else None,
                          slides=getattr(resource, 'slides', None) if resource.type == "lecture" else None,
@@ -195,24 +195,24 @@ def get(self, dojo):
                  ],
                  challenges=[
                     dict(id=challenge.id,
-                         name=challenge.name,
+                         name=challenge.localized_name,
                          required=challenge.required,
-                         description=challenge.description)
+                         description=challenge.localized_description)
                     for challenge in (module.visible_challenges() if not is_dojo_admin
                                       else module.challenges)
                  ],
                  unified_items=[
                      dict(
                          item_type=item.item_type,
                          id=f"resource-{item.resource_index}" if item.item_type == 'resource' else getattr(item, 'id', None),
-                         name=item.name,
+                         name=item.localized_name,
                          type=getattr(item, 'type', None),
-                         content=getattr(item, 'content', None) if hasattr(item, 'type') and item.type in ["markdown", "header"] else None,
+                         content=item.localized_content if hasattr(item, 'type') and item.type in ["markdown", "header"] else None,
                          video=getattr(item, 'video', None) if hasattr(item, 'type') and item.type == "lecture" else None,
                          playlist=getattr(item, 'playlist', None) if hasattr(item, 'type') and item.type == "lecture" else None,
                          slides=getattr(item, 'slides', None) if hasattr(item, 'type') and item.type == "lecture" else None,
                          expandable=getattr(item, 'expandable', True) if hasattr(item, 'type') else None,
-                         description=getattr(item, 'description', None),
+                         description=item.localized_description,
                          required=getattr(item, 'required', None) if item.item_type == 'challenge' else None
                      ) for item in (module.unified_items if is_dojo_admin else module.visible_items)
                  ])
@@ -430,7 +430,7 @@ def get(self, dojo, module, challenge_id):
 
         return {
             "success": True,
-            "description": render_markdown(dojo_challenge.description)
+            "description": render_markdown(dojo_challenge.localized_description)
         }
 
 

dojo_plugin/api/v1/search.py

@@ -4,16 +4,27 @@
 from CTFd.models import db
 from CTFd.utils.user import get_current_user, is_admin
 
+from ...i18n import DEFAULT_LANGUAGE, current_language
 from ...models import Dojos, DojoAdmins, DojoModules, DojoChallenges
 
 search_namespace = Namespace("search", description="Search across dojos, modules, and challenges")
 
+
+def matches(model, language, like_query, *fields):
+    conditions = [getattr(model, field).ilike(like_query, escape="\\") for field in fields]
+    if language != DEFAULT_LANGUAGE:
+        conditions += [model.data["translations"][language][field].astext.ilike(like_query, escape="\\")
+                       for field in fields]
+    return or_(*conditions)
+
+
 @search_namespace.route("")
 class Search(Resource):
     def get(self):
         query = request.args.get("q", "").strip()
 
         user = get_current_user()
+        language = current_language()
 
         if not query or len(query) < 2:
             return {"success": False, "error": "Query too short."}, 400
@@ -22,18 +33,16 @@ def get(self):
         escaped = query.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
         like_query = f"%{escaped}%"
 
-        def ilike(*columns):
-            return or_(*(column.ilike(like_query, escape="\\") for column in columns))
-
-        dojos = Dojos.viewable(user=user).filter(ilike(Dojos.name, Dojos.description))
+        dojos = Dojos.viewable(user=user).filter(
+            matches(Dojos, language, like_query, "name", "description"))
         modules = (DojoModules.query
                    .join(Dojos.viewable(user=user))
-                   .filter(ilike(DojoModules.name, DojoModules.description)))
+                   .filter(matches(DojoModules, language, like_query, "name", "description")))
         challenges = (DojoChallenges.query
                       .join(Dojos.viewable(user=user))
                       .join(DojoModules, and_(DojoModules.dojo_id == DojoChallenges.dojo_id,
                                               DojoModules.module_index == DojoChallenges.module_index))
-                      .filter(ilike(DojoChallenges.name, DojoChallenges.description)))
+                      .filter(matches(DojoChallenges, language, like_query, "name", "description")))
 
         if not is_admin():
             admin_dojo_ids = (db.session.query(DojoAdmins.dojo_id)
@@ -57,42 +66,42 @@ def ilike(*columns):
                 "dojos": [
                     {
                         "id": dojo.reference_id,
-                        "name": dojo.name,
+                        "name": dojo.localized_name,
                         "link": f"/{dojo.reference_id}",
-                        "description": dojo.description,
+                        "description": dojo.localized_description,
                     }
                     for dojo in dojos
                 ],
                 "modules": [
                     {
                         "id": module.id,
-                        "name": module.name,
+                        "name": module.localized_name,
                         "dojo": {
                             "id": module.dojo.reference_id,
-                            "name": module.dojo.name,
+                            "name": module.dojo.localized_name,
                             "link": f"/{module.dojo.reference_id}"
                         },
                         "link": f"/{module.dojo.reference_id}/{module.id}",
-                        "description": module.description,
+                        "description": module.localized_description,
                     }
                     for module in modules
                 ],
                 "challenges": [
                     {
                         "id": challenge.id,
-                        "name": challenge.name,
+                        "name": challenge.localized_name,
                         "module": {
                             "id": challenge.module.id,
-                            "name": challenge.module.name,
+                            "name": challenge.module.localized_name,
                             "link": f"/{challenge.module.dojo.reference_id}/{challenge.module.id}"
                         },
                         "dojo": {
                             "id": challenge.module.dojo.reference_id,
-                            "name": challenge.module.dojo.name,
+                            "name": challenge.module.dojo.localized_name,
                             "link": f"/{challenge.module.dojo.reference_id}"
                         },
                         "link": f"/{challenge.module.dojo.reference_id}/{challenge.module.id}/{challenge.id}",
-                        "description": challenge.description,
+                        "description": challenge.localized_description,
                     }
                     for challenge in challenges
                 ]