diff options
| -rw-r--r-- | _conclusion.qmd | 9 | ||||
| -rw-r--r-- | _evaluation.qmd | 236 |
2 files changed, 235 insertions, 10 deletions
diff --git a/_conclusion.qmd b/_conclusion.qmd index 1ee0655..a6c4145 100644 --- a/_conclusion.qmd +++ b/_conclusion.qmd @@ -67,6 +67,15 @@ Useful here might be a draft work on rewritng the Semantic Markdown specificatio as an extension to the CommonMark specification [@Smedegaard2022cmark]. +### Vague limiting of implementation scope {#sec-experimental-test} + +One test failure (see @sec-test-annotation-block) +revealed an issue that is either a bug in the code or the testsuite. +This project attempted to setup limits +for the intended use of the draft Semantic Markdown specification, +but those constraints did not take into account that the specification +describes features beyond what was intended for this project. + ## Future works {#sec-future} The central design choice diff --git a/_evaluation.qmd b/_evaluation.qmd index 86f3b22..52a7342 100644 --- a/_evaluation.qmd +++ b/_evaluation.qmd @@ -1,7 +1,3 @@ -*FIXME: rephrase to explain for the reader what works* - -## End-to-end testing - A testsuite has been set up, derived from the examples in the draft specification. The testsuite contains 10 end-to-end tests. @@ -9,17 +5,237 @@ The testsuite contains 10 end-to-end tests. A build target, `make check`, has been constructed that renders each test with the `sem-md.lua` filter enabled, and compares the result against expected output using diff. +7 of the 10 tests currently fail. + +These test failures are examined closer below, +and summarized at the end of the chapter. + +## Wrongly removes detached annotation + +The test `block_scope` contains this in Semantic Markdown: + +```markdown +You can give me a ring. {=<#manu>} + +You can give me a ring {=<#manu>}. +``` + +Expected output as Markdown is this: + +```markdown +You can give me a ring. + +You can give me a ring {=<#manu>}. +``` + +Output from the test command +`pandoc --from commonmark --to commonmark --lua-filter sem-md/sem-md.lua --wrap preserve < test/block_scope.md` +is instead this: + +```markdown +You can give me a ring. + +You can give me a ring . +``` + +A more concise view is the following unified diff, +where minus-prefixed lines are expected output +differing from plus-prefixed actual output: + +```diff +--- test/block_scope.md.md ++++ /dev/stdin +@@ -1,3 +1,3 @@ + You can give me a ring. + +-You can give me a ring {=<#manu>}. ++You can give me a ring . +``` + +The filter wrongly removes a non-annotation: +Only when text enclosed with braces +are either immediately followed by an explicit inline enclosure +(linked, emphasized, strongly emphasized, underlined or bracketed) +or is the very first or very last inline content of a block, +should it be considered an annotation. + +This is a violation of the Semantic Markdown specification, +and also one of the rules defined for this project +that "rendering of syntactically incorrect [...] annotations +[...] **should** include the annotation as regular text" +(see @sec-spirit). + +## Fails to remove image annotation + +The test `ex_paragraph` contains this in Semantic Markdown: + +```markdown +My name is +[Manu Sporny]{:name} +and you can give me a ring via +[1-800-555-0199]{:telephone}. +{:image} +My favorite animal is the [Liger]{ov:preferredAnimal}. +{=<#manu> .:Person} + +{schema}: @default + +{ov}: http://open.vocab.org/terms/ +``` + +Unified diff between expected and actual output: + +```diff +--- test/ex_paragraph.md.md ++++ /dev/stdin +@@ -2,5 +2,5 @@ + Manu Sporny + and you can give me a ring via + 1-800-555-0199. +- ++{:image} + My favorite animal is the Liger. +``` + +The annotation, `{:image}`, for the inline linked image is wrongly kept. + +This is a violation of the Semantic Markdown specification, +and also one of the rules defined for this project +that "A rendering of syntactically correct annotations [...] +**must** be functionally equal to the text authored without annotations" +(see @sec-spirit). + +## Fails to remove annotation-only block {#sec-test-annotation-block} + +The test `block-siblings_scope_list` contains this in Semantic Markdown: + +```markdown +# People + +## {.schema:Person} + +## Manu Sporny + +My name is Manu Sporny. + +## Thomas Francart + +My name is Thomas Francart. + +# Animals +``` + +Unified diff between expected and actual output: + +```diff +--- test/block-siblings_scope.md.md ++++ /dev/stdin +@@ -1,5 +1,7 @@ + # People + ++## ++ + ## Manu Sporny + + My name is Manu Sporny. +``` + +A `Header` block containing only an annotation +has had the annotation removed inline, +but the block -- which after removal is virtually empty +(contains only a space) -- +is kept. + +Tests `block-siblings_scope_list` and `ex_title_and_list` +fail due to similar issue but for other block elemet types. + +These test failures reveal +violations of the Semantic Markdown specification, +but parts of the specifications that was not planned +to be implemented with this project, +because introducing logic that adds or removes blocks +(not only inlines) arguably is too large a deviation +of core Markdown. + +This type of error is therefore a bug in the testsuite +rather than in the codebase. + +## Text after link definition annotation wrongly removed + +The test `link_definition_scope` contains this: + +```markdown +My name is +[Manu Sporny] +and you can give me a ring via +[1-800-555-0199]. + +[Manu Sporny]: {schema:name} + +[1-800-555-0199]: tel:+1-800-555-0199 + "make a phone call to Manu Sporny" + {schema:telephone} +``` + +Unified diff between expected and actual output: + +```diff +--- test/link_definition_scope.md.md ++++ /dev/stdin +@@ -1,4 +1,4 @@ + My name is + Manu Sporny + and you can give me a ring via +-[1-800-555-0199](tel:+1-800-555-0199 "make a phone call to Manu Sporny"). ++[1-800-555-0199](tel:+1-800-555-0199 "make a phone call to Manu Sporny") +``` + +A link annotation from a link definition is correctly removed, +but trailing text (a dot) is wrongly removed as well. + +## Fails to remove annotation and repeats word before non-annotated + +The test `span_scope` contains this: + +```markdown +My name is [Manu Sporny]{schema:name}. +My name is **Manu [Sporny]**{schema:name}. +My name is [Manu Sporny] {schema:name}. +My name is Manu Sporny{schema:name}. +``` +Unified diff between expected and actual output: +```diff +--- test/span_scope.md.md ++++ /dev/stdin +@@ -1,4 +1,4 @@ + My name is Manu Sporny. +-My name is **Manu [Sporny]**. +-My name is [Manu Sporny] {schema:name}. +-My name is Manu Sporny{schema:name}. ++My name is **Manu \[Sporny\]**{schema:name}. ++My name is \[Manu Sporny\] . ++My name is Manu SpornySporny{schema:name} +``` -collected for the newly introduced Markdown markup. +This reveals 3 issues, +one of which have already been covered. -*TODO*: Document test coverage, and reflect on causes. +In second line, +an annotation for emphasised text is wrongly kept. -## Validation of output formatting +It also (for third line) reveals the issue covered previously +of wrongly removing detached annotation. -*TODO* +In fourth line, +a non-annotation is correctly kept, +but the word before that is wrongly repeated. -## Evaluation of practical use +## Summary of test failures -*TODO* +The small testsuite containing 10 tests +currently exposes 6 types of test failures, +which in closer inspection reveals 7 bugs in the code +and (arguably, see discussion at @sec-experimental-test) +1 bug in the testsuite. |