more evaluation

2 files changed, 235 insertions, 10 deletions

diff --git a/_conclusion.qmd b/_conclusion.qmd
index 1ee0655..a6c4145 100644
--- a/_conclusion.qmd
+++ b/_conclusion.qmd
@@ -67,6 +67,15 @@ Useful here might be a draft work on rewritng the Semantic Markdown specificatio
 as an extension to the CommonMark specification
 [@Smedegaard2022cmark].
 
+### Vague limiting of implementation scope {#sec-experimental-test}
+
+One test failure (see @sec-test-annotation-block)
+revealed an issue that is either a bug in the code or the testsuite.
+This project attempted to setup limits
+for the intended use of the draft Semantic Markdown specification,
+but those constraints did not take into account that the specification
+describes features beyond what was intended for this project.
+
 ## Future works {#sec-future}
 
 The central design choice
diff --git a/_evaluation.qmd b/_evaluation.qmd
index 86f3b22..52a7342 100644
--- a/_evaluation.qmd
+++ b/_evaluation.qmd
@@ -1,7 +1,3 @@
-*FIXME: rephrase to explain for the reader what works*
-
-## End-to-end testing
-
 A testsuite has been set up,
 derived from the examples in the draft specification.
 The testsuite contains 10 end-to-end tests.
@@ -9,17 +5,237 @@ The testsuite contains 10 end-to-end tests.
 A build target, `make check`, has been constructed
 that renders each test with the `sem-md.lua` filter enabled,
 and compares the result against expected output using diff.
+7 of the 10 tests currently fail.
+
+These test failures are examined closer below,
+and summarized at the end of the chapter.
+
+## Wrongly removes detached annotation
+
+The test `block_scope` contains this in Semantic Markdown:
+
+```markdown
+You can give me a ring. {=<#manu>}
+
+You can give me a ring {=<#manu>}.
+```
+
+Expected output as Markdown is this:
+
+```markdown
+You can give me a ring.
+
+You can give me a ring {=<#manu>}.
+```
+
+Output from the test command
+`pandoc --from commonmark --to commonmark --lua-filter sem-md/sem-md.lua --wrap preserve < test/block_scope.md`
+is instead this:
+
+```markdown
+You can give me a ring.
+
+You can give me a ring .
+```
+
+A more concise view is the following unified diff,
+where minus-prefixed lines are expected output
+differing from plus-prefixed actual output:
+
+```diff
+--- test/block_scope.md.md
++++ /dev/stdin
+@@ -1,3 +1,3 @@
+ You can give me a ring.
+ 
+-You can give me a ring {=<#manu>}.
++You can give me a ring .
+```
+
+The filter wrongly removes a non-annotation:
+Only when text enclosed with braces
+are either immediately followed by an explicit inline enclosure
+(linked, emphasized, strongly emphasized, underlined or bracketed)
+or is the very first or very last inline content of a block,
+should it be considered an annotation.
+
+This is a violation of the Semantic Markdown specification,
+and also one of the rules defined for this project
+that "rendering of syntactically incorrect [...] annotations
+[...] **should** include the annotation as regular text"
+(see @sec-spirit).
+
+## Fails to remove image annotation
+
+The test `ex_paragraph` contains this in Semantic Markdown:
+
+```markdown
+My name is
+[Manu Sporny]{:name}
+and you can give me a ring via
+[1-800-555-0199]{:telephone}.
+![](http://manu.sporny.org/images/manu.png){:image}
+My favorite animal is the [Liger]{ov:preferredAnimal}.
+{=<#manu> .:Person}
+
+{schema}: @default
+
+{ov}: http://open.vocab.org/terms/
+```
+
+Unified diff between expected and actual output:
+
+```diff
+--- test/ex_paragraph.md.md
++++ /dev/stdin
+@@ -2,5 +2,5 @@
+ Manu Sporny
+ and you can give me a ring via
+ 1-800-555-0199.
+-![](http://manu.sporny.org/images/manu.png)
++![](http://manu.sporny.org/images/manu.png){:image}
+ My favorite animal is the Liger.
+```
+
+The annotation, `{:image}`, for the inline linked image is wrongly kept.
+
+This is a violation of the Semantic Markdown specification,
+and also one of the rules defined for this project
+that "A rendering of syntactically correct annotations [...]
+**must** be functionally equal to the text authored without annotations"
+(see @sec-spirit).
+
+## Fails to remove annotation-only block {#sec-test-annotation-block}
+
+The test `block-siblings_scope_list` contains this in Semantic Markdown:
+
+```markdown
+# People
+
+## {.schema:Person}
+
+## Manu Sporny
+
+My name is Manu Sporny.
+
+## Thomas Francart
+
+My name is Thomas Francart.
+
+# Animals
+```
+
+Unified diff between expected and actual output:
+
+```diff
+--- test/block-siblings_scope.md.md
++++ /dev/stdin
+@@ -1,5 +1,7 @@
+ # People
+ 
++## 
++
+ ## Manu Sporny
+ 
+ My name is Manu Sporny.
+```
+
+A `Header` block containing only an annotation
+has had the annotation removed inline,
+but the block -- which after removal is virtually empty
+(contains only a space) --
+is kept.
+
+Tests `block-siblings_scope_list` and `ex_title_and_list`
+fail due to similar issue but for other block elemet types.
+
+These test failures reveal
+violations of the Semantic Markdown specification,
+but parts of the specifications that was not planned
+to be implemented with this project,
+because introducing logic that adds or removes blocks
+(not only inlines) arguably is too large a deviation
+of core Markdown.
+
+This type of error is therefore a bug in the testsuite
+rather than in the codebase.
+
+## Text after link definition annotation wrongly removed
+
+The test `link_definition_scope` contains this:
+
+```markdown
+My name is
+[Manu Sporny]
+and you can give me a ring via
+[1-800-555-0199].
+
+[Manu Sporny]: {schema:name}
+
+[1-800-555-0199]: tel:+1-800-555-0199
+  "make a phone call to Manu Sporny"
+  {schema:telephone}
+```
+
+Unified diff between expected and actual output:
+
+```diff
+--- test/link_definition_scope.md.md
++++ /dev/stdin
+@@ -1,4 +1,4 @@
+ My name is
+ Manu Sporny
+ and you can give me a ring via
+-[1-800-555-0199](tel:+1-800-555-0199 "make a phone call to Manu Sporny").
++[1-800-555-0199](tel:+1-800-555-0199 "make a phone call to Manu Sporny")
+```
+
+A link annotation from a link definition is correctly removed,
+but trailing text (a dot) is wrongly removed as well.
+
+## Fails to remove annotation and repeats word before non-annotated
+
+The test `span_scope` contains this:
+
+```markdown
+My name is [Manu Sporny]{schema:name}.  
+My name is **Manu [Sporny]**{schema:name}.  
+My name is [Manu Sporny] {schema:name}.  
+My name is Manu Sporny{schema:name}.
+```
 
+Unified diff between expected and actual output:
 
+```diff
+--- test/span_scope.md.md
++++ /dev/stdin
+@@ -1,4 +1,4 @@
+ My name is Manu Sporny.  
+-My name is **Manu [Sporny]**.  
+-My name is [Manu Sporny] {schema:name}.  
+-My name is Manu Sporny{schema:name}.
++My name is **Manu \[Sporny\]**{schema:name}.  
++My name is \[Manu Sporny\] .  
++My name is Manu SpornySporny{schema:name}
+```
 
-collected for the newly introduced Markdown markup.
+This reveals 3 issues,
+one of which have already been covered.
 
-*TODO*: Document test coverage, and reflect on causes.
+In second line,
+an annotation for emphasised text is wrongly kept.
 
-## Validation of output formatting
+It also (for third line) reveals the issue covered previously
+of wrongly removing detached annotation.
 
-*TODO*
+In fourth line,
+a non-annotation is correctly kept,
+but the word before that is wrongly repeated.
 
-## Evaluation of practical use
+## Summary of test failures
 
-*TODO*
+The small testsuite containing 10 tests
+currently exposes 6 types of test failures,
+which in closer inspection reveals 7 bugs in the code
+and (arguably, see discussion at @sec-experimental-test)
+1 bug in the testsuite.