Skip to content

Commit 6d5a124

Browse files
tats-uMihaZupan
andauthored
Add CJK-friendly Emphasis Extension (#921)
* Add CJK-friendly Emphasis Extension * Add auto-generated test file * Add name for configuration * Remove useless default value assignments Co-authored-by: Miha Zupan <mihazupan.zupan1@gmail.com> * Make `CheckOpenCloseDelimiterCjkFriendly` internal only * Remove `CjkFriendlyEmphasisExtension` class * Add some comments including links * Add direct tests on `CharHelper.CheckOpenCloseDelimiterCjkFriendly` * Fix generated tests * Add `#if NET` * Skip Rune-dependent tests in .NET Standard tests * Add missing XML Documentation Comments * Fix URL * Change test condition * Add test in .NET Framework 4.8.1 * Add netstandard2.0 to SpecFileGen * Add fallback for netstandard2.0 * Fix * Revert "Fix" This reverts commit 42e998b. * Revert "Add fallback for netstandard2.0" This reverts commit 7400a7b. * Revert "Add netstandard2.0 to SpecFileGen" This reverts commit f9aa8e1. * Revert "Add test in .NET Framework 4.8.1" This reverts commit d8d6d51. * Fix missing indent --------- Co-authored-by: Miha Zupan <mihazupan.zupan1@gmail.com>
1 parent 7959e3b commit 6d5a124

12 files changed

Lines changed: 474 additions & 6 deletions

File tree

.github/workflows/test-netstandard.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,10 +35,10 @@ jobs:
3535

3636
- name: Test Debug
3737
run: |
38-
dotnet build src/Markdig.Tests/Markdig.Tests.csproj -c Debug --no-restore
38+
dotnet build src/Markdig.Tests/Markdig.Tests.csproj -c Debug --no-restore -p:MarkdigNoRuneTests=true
3939
dotnet test src/Markdig.Tests/Markdig.Tests.csproj -c Debug --no-build
4040
4141
- name: Test Release
4242
run: |
43-
dotnet build src/Markdig.Tests/Markdig.Tests.csproj -c Release --no-restore
43+
dotnet build src/Markdig.Tests/Markdig.Tests.csproj -c Release --no-restore -p:MarkdigNoRuneTests=true
4444
dotnet test src/Markdig.Tests/Markdig.Tests.csproj -c Release --no-build

readme.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,7 @@ You can **try Markdig online** and compare it to other implementations on [babel
5151
- [**Diagrams**](src/Markdig.Tests/Specs/DiagramsSpecs.md) extension whenever a fenced code block contains a special keyword, it will be converted to a div block with the content as-is (currently, supports [`mermaid`](https://mermaid.js.org) and [`nomnoml`](https://github.com/skanaar/nomnoml) diagrams)
5252
- [**YAML Front Matter**](src/Markdig.Tests/Specs/YamlSpecs.md) to parse without evaluating the front matter and to discard it from the HTML output (typically used for previewing without the front matter in MarkdownEditor)
5353
- [**JIRA links**](src/Markdig.Tests/Specs/JiraLinks.md) to automatically generate links for JIRA project references (Thanks to @clarkd: https://github.com/clarkd/MarkdigJiraLinker)
54+
- [**CJK-friendly Emphasis**](src/Markdig.Tests/Specs/CJKFriendlyEmphasis.md) to mitigate a CommonMark specification issue in CJK languages (Thanks to @tats-u: https://github.com/tats-u/markdown-cjk-friendly)
5455
- Starting with Markdig version `0.20.0+`, Markdig is compatible only with `NETStandard 2.0`, `NETStandard 2.1`, `NETCoreApp 2.1` and `NETCoreApp 3.1`.
5556

5657
If you are looking for support for an old .NET Framework 3.5 or 4.0, you can download Markdig `0.18.3`.

src/Markdig.Tests/Markdig.Tests.csproj

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,10 @@
1212
<NoWarn>$(NoWarn);NETSDK1138</NoWarn>
1313
</PropertyGroup>
1414

15+
<PropertyGroup Condition="'$(MarkdigNoRuneTests)' == 'true'">
16+
<DefineConstants>$(DefineConstants);MARKDIG_NO_RUNE_TESTS</DefineConstants>
17+
</PropertyGroup>
18+
1519
<ItemGroup>
1620
<PackageReference Include="Microsoft.NET.Test.Sdk" />
1721
<PackageReference Include="NUnit" />
Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
2+
// --------------------------------
3+
// CJK-friendly Emphasis
4+
// --------------------------------
5+
6+
using System;
7+
using NUnit.Framework;
8+
9+
namespace Markdig.Tests.Specs.CJKFriendlyEmphasis
10+
{
11+
[TestFixture]
12+
public class TestCJKFriendlyEmphasisExtension
13+
{
14+
// ## CJK-friendly Emphasis Extension
15+
//
16+
// See https://github.com/tats-u/markdown-cjk-friendly/blob/main/specification.md for details about the spec of this extension.
17+
//
18+
// This extension drastically mitigates [the long-standing issue (specification flaw)](https://github.com/commonmark/commonmark-spec/issues/650) in CommonMark that emphasis in CJK languages is often not parsed as expected.
19+
//
20+
// The plain CommonMark cannot recognize even the following emphasis in CJK languages:
21+
[Test]
22+
public void CJKFriendlyEmphasisExtension_Example001()
23+
{
24+
// Example 1
25+
// Section: CJK-friendly Emphasis Extension
26+
//
27+
// The following Markdown:
28+
// **この文を強調できますか(Can I emphasize this sentence)?**残念ながらこの文のせいでできません(Unfortunately not possible due to this sentence)。
29+
//
30+
// Should be rendered as:
31+
// <p><strong>この文を強調できますか(Can I emphasize this sentence)?</strong>残念ながらこの文のせいでできません(Unfortunately not possible due to this sentence)。</p>
32+
33+
TestParser.TestSpec("**この文を強調できますか(Can I emphasize this sentence)?**残念ながらこの文のせいでできません(Unfortunately not possible due to this sentence)。", "<p><strong>この文を強調できますか(Can I emphasize this sentence)?</strong>残念ながらこの文のせいでできません(Unfortunately not possible due to this sentence)。</p>", "cjk-friendly-emphasis", context: "Example 1\nSection CJK-friendly Emphasis Extension\n");
34+
}
35+
36+
// ````````````````````````````````` example
37+
// 我可以强调**这个`code`**吗(Can I emphasize **this `code`**)?
38+
// .
39+
// <p>我可以强调<code>这个`code`</code>吗(Can I emphasize <strong>this <code>code</code></strong>)?</p>
40+
// `````````````````````````````````
41+
[Test]
42+
public void CJKFriendlyEmphasisExtension_Example002()
43+
{
44+
// Example 2
45+
// Section: CJK-friendly Emphasis Extension
46+
//
47+
// The following Markdown:
48+
// **이 용어(This term)**를 강조해 주세요. (Please emphasize **this term**.)
49+
//
50+
// Should be rendered as:
51+
// <p><strong>이 용어(This term)</strong>를 강조해 주세요. (Please emphasize <strong>this term</strong>.)</p>
52+
53+
TestParser.TestSpec("**이 용어(This term)**를 강조해 주세요. (Please emphasize **this term**.)", "<p><strong>이 용어(This term)</strong>를 강조해 주세요. (Please emphasize <strong>this term</strong>.)</p>", "cjk-friendly-emphasis", context: "Example 2\nSection CJK-friendly Emphasis Extension\n");
54+
}
55+
// You can compare the results with and without this extension: https://tats-u.github.io/markdown-cjk-friendly/?sc8=KirjgZPjga7mlofjgpLlvLfoqr_jgafjgY3jgb7jgZnjgYvvvIhDYW4gSSBlbXBoYXNpemUgdGhpcyBzZW50ZW5jZe-8ie-8nyoq5q6L5b-144Gq44GM44KJ44GT44Gu5paH44Gu44Gb44GE44Gn44Gn44GN44G-44Gb44KT77yIVW5mb3J0dW5hdGVseSBub3QgcG9zc2libGUgZHVlIHRvIHRoaXMgc2VudGVuY2XvvInjgIIKCuaIkeWPr-S7peW8uuiwgyoq6L-Z5LiqYGNvZGVgKirlkJfvvIhDYW4gSSBlbXBoYXNpemUgKip0aGlzIGBjb2RlYCoq77yJ77yfCgoqKuydtCDsmqnslrQoVGhpcyB0ZXJtKSoq66W8IOqwleyhsO2VtCDso7zshLjsmpQuIChQbGVhc2UgZW1waGFzaXplICoqdGhpcyB0ZXJtKiouKQo&gfm=1&engine=markdown-it
56+
//
57+
// You will find how poor the plain CommonMark is for CJK languages.
58+
//
59+
// To use this extension, configure the pipeline as follows:
60+
//
61+
// ```csharp
62+
// var pipeline = new MarkdownPipelineBuilder()
63+
// .UseCJKFriendlyEmphasis() // Add this
64+
// .Build();
65+
// ```
66+
}
67+
}
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
## CJK-friendly Emphasis Extension
2+
3+
See https://github.com/tats-u/markdown-cjk-friendly/blob/main/specification.md for details about the spec of this extension.
4+
5+
This extension drastically mitigates [the long-standing issue (specification flaw)](https://github.com/commonmark/commonmark-spec/issues/650) in CommonMark that emphasis in CJK languages is often not parsed as expected.
6+
7+
The plain CommonMark cannot recognize even the following emphasis in CJK languages:
8+
9+
```````````````````````````````` example
10+
**この文を強調できますか(Can I emphasize this sentence)?**残念ながらこの文のせいでできません(Unfortunately not possible due to this sentence)。
11+
.
12+
<p><strong>この文を強調できますか(Can I emphasize this sentence)?</strong>残念ながらこの文のせいでできません(Unfortunately not possible due to this sentence)。</p>
13+
````````````````````````````````
14+
15+
````````````````````````````````` example
16+
我可以强调**这个`code`**吗(Can I emphasize **this `code`**)?
17+
.
18+
<p>我可以强调<code>这个`code`</code>吗(Can I emphasize <strong>this <code>code</code></strong>)?</p>
19+
`````````````````````````````````
20+
21+
```````````````````````````````` example
22+
**이 용어(This term)**를 강조해 주세요. (Please emphasize **this term**.)
23+
.
24+
<p><strong>이 용어(This term)</strong>를 강조해 주세요. (Please emphasize <strong>this term</strong>.)</p>
25+
````````````````````````````````
26+
27+
You can compare the results with and without this extension: https://tats-u.github.io/markdown-cjk-friendly/?sc8=KirjgZPjga7mlofjgpLlvLfoqr_jgafjgY3jgb7jgZnjgYvvvIhDYW4gSSBlbXBoYXNpemUgdGhpcyBzZW50ZW5jZe-8ie-8nyoq5q6L5b-144Gq44GM44KJ44GT44Gu5paH44Gu44Gb44GE44Gn44Gn44GN44G-44Gb44KT77yIVW5mb3J0dW5hdGVseSBub3QgcG9zc2libGUgZHVlIHRvIHRoaXMgc2VudGVuY2XvvInjgIIKCuaIkeWPr-S7peW8uuiwgyoq6L-Z5LiqYGNvZGVgKirlkJfvvIhDYW4gSSBlbXBoYXNpemUgKip0aGlzIGBjb2RlYCoq77yJ77yfCgoqKuydtCDsmqnslrQoVGhpcyB0ZXJtKSoq66W8IOqwleyhsO2VtCDso7zshLjsmpQuIChQbGVhc2UgZW1waGFzaXplICoqdGhpcyB0ZXJtKiouKQo&gfm=1&engine=markdown-it
28+
29+
You will find how poor the plain CommonMark is for CJK languages.
30+
31+
To use this extension, configure the pipeline as follows:
32+
33+
```csharp
34+
var pipeline = new MarkdownPipelineBuilder()
35+
.UseCJKFriendlyEmphasis() // Add this
36+
.Build();
37+
```

src/Markdig.Tests/Specs/readme.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,5 +32,6 @@ You will find from the following links the supported extensions in markdig and t
3232
- [**Diagrams**](DiagramsSpecs.md)
3333
- [**YAML frontmatter**](YamlSpecs.md)
3434
- [**JIRA links**](JiraLinks.md)
35+
- [**CJK-friendly Emphasis**](CJKFriendlyEmphasis.md)
3536

3637
> Notice that the links above are not yet the final documentation but are "specification" files used for testing the correctness of markdig for each extension

0 commit comments

Comments
 (0)