17 files changed, 509 insertions, 32 deletions
diff --git a/doc/bugs/firefox_doesn__39__t_want_to_load_updated_pages_at_ikiwiki.info.mdwn b/doc/bugs/firefox_doesn__39__t_want_to_load_updated_pages_at_ikiwiki.info.mdwn
index 46e77a265..558eb90c8 100644
--- a/doc/bugs/firefox_doesn__39__t_want_to_load_updated_pages_at_ikiwiki.info.mdwn
+++ b/doc/bugs/firefox_doesn__39__t_want_to_load_updated_pages_at_ikiwiki.info.mdwn
@@ -5,3 +5,10 @@ Only explicitly pressing "reload" helps.
 Is it a bug? I haven't been noticing such problems usually on other sites. --Ivan Z.
 
 This remains to be true now, with Epiphany 2.26.3 (Mozilla/5.0 (X11; U; Linux i686; en; rv:1.9.1.4pre) Gecko/20080528 Epiphany/2.22 Firefox/3.5). --Ivan Z.
+
+> In the most recent ikiwiki release, I added a Cache-Control hack
+> explicitly to work around firefox's broken over-caching.
+> 
+> (When I tested epiphany and chromium, neither had firefox's problem.)
+>
+> [[!tag done]]
diff --git a/doc/bugs/some_but_not_all_meta_fields_are_stored_escaped.mdwn b/doc/bugs/some_but_not_all_meta_fields_are_stored_escaped.mdwn
new file mode 100644
index 000000000..8e1ca42e0
--- /dev/null
+++ b/doc/bugs/some_but_not_all_meta_fields_are_stored_escaped.mdwn
@@ -0,0 +1,39 @@
+[[!template id=gitbranch branch=smcv/unescaped-meta author="[[Simon_McVittie|smcv]]"]]
+[[!tag patch]]
+(Warning: this branch has not been tested thoroughly.)
+
+While discussing the [[plugins/meta]] plugin on IRC, Joey pointed out that
+it stores most meta fields unescaped, but 'title', 'guid' and 'description'
+are special-cased and stored escaped (with numeric XML/HTML entities). This
+is to avoid emitting markup in the `<title>` of a HTML page, or in an RSS/Atom
+feed, neither of which are subject to the [[plugins/htmlscrubber]].
+
+However, having the meta fields "partially escaped" like this is somewhat
+error-prone. Joey suggested that perhaps everything should be stored
+unescaped, and the escaping should be done on output; this branch
+implements that.
+
+Points of extra subtlety:
+
+* The title given to the [[plugins/search]] plugin was previously HTML;
+  now it's plain text, potentially containing markup characters. I suspect
+  that that's what Xapian wants anyway (which is why I didn't change it),
+  but I could be wrong...
+
+* Page descriptions in the HTML `<head>` were previously double-escaped:
+  the description was stored escaped with numeric entities, then that was
+  output with a second layer of escaping! In this branch, I just emit
+  the page description escaped once, as was presumably the intention.
+
+* It's safe to apply this change to a wiki and neglect to rebuild it
+  (assuming I implemented it correctly!), but until the wiki is rebuilt,
+  titles, descriptions and GUIDs for unchanged pages will appear
+  double-escaped on any page that inlines them in `quick=yes` mode, and
+  is rebuilt for some other reason. The failure mode is too much escaping
+  rather than too little, so it shouldn't be a security problem.
+
+* Reverting this change, if applied, is more dangerous; until the wiki is
+  rebuilt, any titles, descriptions and GUIDs on unchanged pages that
+  contained markup could appear unescaped on any page that inlines them
+  in `quick=yes` mode, and is rebuilt for some other reason. The failure
+  mode here would be too little escaping, i.e. cross-site scripting.
diff --git a/doc/bugs/tagged__40____41___matching_wikilinks.mdwn b/doc/bugs/tagged__40____41___matching_wikilinks.mdwn
index e7e4af7c3..a211654f1 100644
--- a/doc/bugs/tagged__40____41___matching_wikilinks.mdwn
+++ b/doc/bugs/tagged__40____41___matching_wikilinks.mdwn
@@ -28,6 +28,8 @@ rationale on this, or what am I doing wrong, and how to achieve what I want?
 >> is valid. [[todo/matching_different_kinds_of_links]] is probably
 >> how it will eventually be solved. --[[Joey]] 
 
+>>> [[Done]]: `tagged` no longer matches other wikilinks. --[[smcv]]
+
 > And this is an illustration why a clean work-around (without changing the software) is not possible: while thinking about [[todo/matching_different_kinds_of_links]], I thought one could work around the problem by simply explicitly including the kind of the relation into the link target (like the tagbase in tags), and by having a separate page without the "tagbase" to link to when one wants simply to refer to the tag without tagging. But this won't work: one has to at least once refer to the real tag page if one wants to talk about it, and this reference will count as tagging (unwanted). --Ivan Z.
 
 > But well, perhaps there is a workaround without introducing different kinds of links. One could modify the [[tag plugin|plugins/tag]] so that it adds 2 links to a page: for tagging -- `tagbase/TAG`, and for navigation -- `tagdescription/TAG` (displayed at the bottom). Then the `tagdescription/TAG` page would hold whatever list one wishes (with `tagged(TAG)` in the pagespec), and whenever one wants to merely refer to the tag, one should link to `tagdescription/TAG`--this link won't count as tagging. So, `tagbase/TAG` would become completely auxiliary (internal) link targets for ikiwiki, the users would edit or link to only `tagdescription/TAG`. --Ivan Z.
diff --git a/doc/forum/an_alternative_approach_to_structured_data.mdwn b/doc/forum/an_alternative_approach_to_structured_data.mdwn
index 045bfd7aa..6e6af8adb 100644
--- a/doc/forum/an_alternative_approach_to_structured_data.mdwn
+++ b/doc/forum/an_alternative_approach_to_structured_data.mdwn
@@ -18,6 +18,12 @@ I think it could be really powerful and useful, especially if it becomes part of
 
 > It looks like an interesting idea.  I don't have time right now to look at it in depth, but it looks interesting. -- [[Will]]
 
+> I agree such a separation makes some sense. But note that the discussion on [[todo/structured_page_data]]
+> talks about associating data types with fields for a good reason: It's hard to later develop a good UI for
+> querying or modifying a page's data if all the data has an implicit type "string". --[[Joey]]
+
+>> I'm not sure that having an implicit type of "string" is really such a bad thing.  After all, Perl itself manages with just string and number, and easily converts from one to the other.  Strong typing is generally used to (a) restrict what can be done with the data and/or (b) restrict how the data is input.  The latter could be done with some sort of validated form, but that, too, could be decoupled from looking up and returning the value of a field. --[[KathrynAndersen]]
+
 ## Second Pass
 
 I have written additional plugins which integrate with the [[plugins/contrib/field]] plugin to both set and get structured page data.
diff --git a/doc/git.mdwn b/doc/git.mdwn
index 379285a49..55967e543 100644
--- a/doc/git.mdwn
+++ b/doc/git.mdwn
@@ -26,7 +26,8 @@ be browsed, subscribed to etc on its
 You are of course free to set up your own ikiwiki git repository with your
 own [[patches|patch]]. If you list it here, the `gitremotes` script will
 automatically add it to git remotes. Your repo will automatically be pulled
-into [[Joey]]'s working tree. This is recommended. :-)
+into [[Joey]]'s working repository where he can see your branches and
+think about merging them. This is recommended. :-)
 
 <!-- Machine-parsed format: * wikilink <git:url> -->
 
@@ -60,6 +61,7 @@ into [[Joey]]'s working tree. This is recommended. :-)
 * [[davrieb|David_Riebenbauer]] `git://git.liegesta.at/git/ikiwiki`
   ([browse](http://git.liegesta.at/?p=ikiwiki.git;a=summary))
 * [[GustafThorslund]] `http://gustaf.thorslund.org/src/ikiwiki.git`
+* [[peteg]] `git://git.hcoop.net/git/peteg/ikiwiki.git`
 
 ## branches
 
diff --git a/doc/ikiwiki/directive/meta.mdwn b/doc/ikiwiki/directive/meta.mdwn
index 557441c0b..5a3919dea 100644
--- a/doc/ikiwiki/directive/meta.mdwn
+++ b/doc/ikiwiki/directive/meta.mdwn
@@ -23,6 +23,13 @@ Supported fields:
   be set to a true value in the template; this can be used to format things
   differently in this case.
 
+  An optional `sortas` parameter will be used preferentially when
+  [[ikiwiki/pagespec/sorting]] by `meta(title)`:
+
+	\[[!meta title="The Beatles" sortas="Beatles, The"]]
+
+	\[[!meta title="David Bowie" sortas="Bowie, David"]]
+
 * license
 
   Specifies a license for the page, for example, "GPL". Can contain
@@ -37,6 +44,11 @@ Supported fields:
 
   Specifies the author of a page.
 
+  An optional `sortas` parameter will be used preferentially when
+  [[ikiwiki/pagespec/sorting]] by `meta(author)`:
+
+	\[[!meta author="Joey Hess" sortas="Hess, Joey"]]
+
 * authorurl
 
   Specifies an url for the author of a page.
diff --git a/doc/ikiwiki/pagespec/sorting.mdwn b/doc/ikiwiki/pagespec/sorting.mdwn
index 697818a2a..37995f1e8 100644
--- a/doc/ikiwiki/pagespec/sorting.mdwn
+++ b/doc/ikiwiki/pagespec/sorting.mdwn
@@ -5,9 +5,20 @@ orders can be specified.
 
 * `age` - List pages from the most recently created to the oldest.
 * `mtime` - List pages with the most recently modified first.
-* `title` - Order by title.
-* `title_natural` - Only available if [[!cpan Sort::Naturally]] is
-  installed. Orders by title, but numbers in the title are treated
+* `title` - Order by title (page name).
+[[!if test="enabled(sortnaturally)" then="""
+* `title_natural` - Orders by title, but numbers in the title are treated
   as such, ("1 2 9 10 20" instead of "1 10 2 20 9")
+"""]]
+[[!if test="enabled(meta)" then="""
+* `meta(title)` - Order according to the `\[[!meta title="foo" sortas="bar"]]`
+  or `\[[!meta title="foo"]]` [[ikiwiki/directive]], or the page name if no
+  full title was set. `meta(author)`, `meta(date)`, `meta(updated)`, etc.
+  also work.
+"""]]
+
+In addition, you can combine several sort orders and/or reverse the order of
+sorting, with a string like `age -title` (which would sort by age, then by
+title in reverse order if two pages have the same age).
 
 [[!meta robots="noindex, follow"]]
diff --git a/doc/plugins/contrib/field/discussion.mdwn b/doc/plugins/contrib/field/discussion.mdwn
index c2b75a76d..103e061e5 100644
--- a/doc/plugins/contrib/field/discussion.mdwn
+++ b/doc/plugins/contrib/field/discussion.mdwn
@@ -21,11 +21,67 @@ behaviour, an auxiliary plugin would be easy.)
 >> (like `map`). Is your plan that `meta` should register itself by
 >> default, and `map` and friends should be adapted to
 >> work based on `getfield()` instead of `$pagestate{foo}{meta}`, then?
->>
+
+>>> Based on `field_get_value()`, yes.  That would be my ideal.  Do you think I should implement that as an ikiwiki branch? --[[KathrynAndersen]]
+
+>>>> This doesn't solve cases where certain fields are treated specially; for
+>>>> instance, putting a `\[[!meta permalink]]` on a page is not the same as
+>>>> putting it in `ymlfront` (in the latter case you won't get your
+>>>> `<link>` header), and putting `\[[!meta date]]` is not the same as putting
+>>>> `date` in `ymlfront` (in the latter case, `%pagectime` won't be changed).
+>>>>
+>>>> One way to resolve that would be to have `ymlfront`, or similar, be a
+>>>> front-end for `meta` rather than for `field`, and call
+>>>> `IkiWiki::Plugin::meta::preprocess` (or a refactored-out function that's
+>>>> similar).
+>>>>
+>>>> There are also some cross-site scripting issues (see below)... --[[smcv]]
+
 >> (On the site I mentioned, I'm using an unmodified version of `field`,
 >> and currently working around the collision by tagging books' pages
 >> with `bookauthor` instead of `author` in the YAML.) --s
 
+>> Revisiting this after more thought, the problem here is similar to the
+>> possibility that a wiki user adds a `meta` shortcut
+>> to [[shortcuts]], or conversely, that a plugin adds a `cpan` directive
+>> that conflicts with the `cpan` shortcut that pages already use. (In the
+>> case of shortcuts, this is resolved by having plugin-defined directives
+>> always win.) For plugin-defined meta keywords this is the plugin
+>> author's/wiki admin's problem - just don't enable conflicting plugins! -
+>> but it gets scary when you start introducing things like `ymlfront`, which
+>> allow arbitrary, wiki-user-defined fields, even ones that subvert
+>> other plugins' assumptions.
+>>
+>> The `pagetemplate` hook is particularly alarming because page templates are
+>> evaluated in many contexts, not all of which are subject to the
+>> htmlscrubber or escaping; because the output from `field` isn't filtered,
+>> prefixed or delimited, when combined with an arbitrary-key-setting plugin
+>> like `ymlfront` it can interfere with other plugins' expectations
+>> and potentially cause cross-site scripting exploits. For instance, `inline`
+>> has a `pagetemplate` hook which defines the `FEEDLINKS` template variable
+>> to be a blob of HTML to put in the `<head>` of the page. As a result, this
+>> YAML would be bad:
+>>
+>>     ---
+>>     FEEDLINKS: <script>alert('code injection detected')</script>
+>>     ---
+>>
+>> (It might require a different case combination due to implementation
+>> details, I'm not sure.)
+>>
+>> It's difficult for `field` to do anything about this, because it doesn't
+>> know whether a field is meant to be plain text, HTML, a URL, or something
+>> else.
+>>
+>> If `field`'s `pagetemplate` hook did something more limiting - like
+>> only emitting template variables starting with `field_`, or from some
+>> finite set, or something - then this would cease to be a problem, I think?
+>>
+>> `ftemplate` and `getfield` don't have this problem, as far as I can see,
+>> because their output is in contexts where the user could equally well have
+>> written raw HTML directly; the user can cause themselves confusion, but
+>> can't cause harmful output. --[[smcv]]
+
 From a coding style point of view, the `$CamelCase` variable names aren't
 IkiWiki style, and the `match_foo` functions look as though they could benefit
 from being thin wrappers around a common `&IkiWiki::Plugin::field::match`
@@ -75,8 +131,8 @@ I think it should just be part of `field` rather than a separate plugin.
 			error("sort=field requires a parameter");
 		}
 
-		my $left = IkiWiki::Plugin::field::field_get_value($_[2], $_[0]);
-		my $right = IkiWiki::Plugin::field::field_get_value($_[2], $_[1]);
+		my $left = IkiWiki::Plugin::field::field_get_value($_[0], $a);
+		my $right = IkiWiki::Plugin::field::field_get_value($_[0], $b);
 
 		$left = "" unless defined $left;
 		$right = "" unless defined $right;
@@ -85,10 +141,186 @@ I think it should just be part of `field` rather than a separate plugin.
 
 	1;
 
--------
+----
+
+Disclaimer: I've only looked at this plugin and ymlfront, not other related
+stuff yet. (I quite like ymlfront, so I looked at this as its dependency. :)
+I also don't want to annoy you with a lot of design discussion 
+if your main goal was to write a plugin that did exactly what you wanted.
+
+My first question is: Why we need another plugin storing metadata
+about the page, when we already have the meta plugin? Much of the
+complication around the field plugin has to do with it accessing info
+belonging to the meta plugin, and generalizing that to be able to access
+info stored by other plugins too. (But I don't see any other plugins that
+currently store such info). Then too, it raises points of confusion like
+smcv's discuission of field author vs meta author above. --[[Joey]] 
+
+> The point is exactly in the generalization, to provide a uniform interface for accessing structured data, no matter what the source of it, whether that be the meta plugin or some other plugin.
+
+> There were a few reasons for this:
+
+>1. In converting my site over from PmWiki, I needed something that was equivalent to PmWiki's Page-Text-Variables (which is how PmWiki implements structured data).
+>2. I also wanted an equivalent of PmWiki's Page-Variables, which, rather than being simple variables, are the return-value of a function.  This gives one a lot of power, because one can do calculations, derive one thing from another.  Heck, just being able to have a "basename" variable is useful.
+>3. I noticed that in the discussion about structured data, it was mired down in disagreements about what form the structured data should take; I wanted to overcome that hurdle by decoupling the form from the content.
+>4. I actually use this to solve (1), because, while I do use ymlfront, initially my pages were in PmWiki format (I wrote (another) unreleased plugin which parses PmWiki format) including PmWiki's Page-Text-Variables for structured data.  So I needed something that could deal with multiple formats.
+
+> So, yes, it does cater to mostly my personal needs, but I think it is more generally useful, also.
+> --[[KathrynAndersen]]
+
+>> Is it fair to say, then, that `field`'s purpose is to take other
+>> plugins' arbitrary per-page data, and present it as a single
+>> merged/flattened string => string map per page? From the plugins
+>> here, things you then use that merged map for include:
+>>
+>> * sorting - stolen by [[todo/allow_plugins_to_add_sorting_methods]]
+>> * substitution into pages with Perl-like syntax - `getfield`
+>> * substitution into wiki-defined templates - the `pagetemplate`
+>>   hook
+>> * substitution into user-defined templates - `ftemplate`
+>>
+>> As I mentioned above, the flattening can cause collisions (and in the
+>> `pagetemplate` case, even security problems).
+>>
+>> I wonder whether conflating Page Text Variables with Page Variables
+>> causes `field` to be more general than it needs to be?
+>> To define a Page Variable (function-like field), you need to write
+>> a plugin containing that Perl function; if we assume that `field`
+>> or something resembling it gets merged into ikiwiki, then it's
+>> reasonable to expect third-party plugins to integrate with whatever
+>> scaffolding there is for these (either in an enabled-by-default
+>> plugin that most people are expected to leave enabled, like `meta`
+>> now, or in the core), and it doesn't seem onerous to expect each
+>> plugin that wants to participate in this mechanism to have code to
+>> do so. While it's still contrib, `field` could just have a special case
+>> for the meta plugin, rather than the converse?
+>>
+>> If Page Text Variables are limited to being simple strings as you
+>> suggest over in [[forum/an_alternative_approach_to_structured_data]],
+>> then they're functionally similar to `meta` fields, so one way to
+>> get their functionality would be to extend `meta` so that
+>>
+>>     \[[!meta badger="mushroom"]]
+>>
+>> (for an unrecognised keyword `badger`) would store
+>> `$pagestate{$page}{meta}{badger} = "mushroom"`? Getting this to
+>> appear in templates might be problematic, because a naive
+>> `pagetemplate` hook would have the same problem that `field` combined
+>> with `ymlfront` currently does.
+>>
+>> One disadvantage that would appear if the function-like and
+>> meta-like fields weren't in the same namespace would be that it
+>> wouldn't be possible to switch a field from being meta-like to being
+>> function-like without changing any wiki content that referenced it.
+>>
+>> Perhaps meta-like fields should just *be* `meta` (with the above
+>> enhancement), as a trivial case of function-like fields? That would
+>> turn `ymlfront` into an alternative syntax for `meta`, I think?
+>> That, in turn, would hopefully solve the special-fields problem,
+>> by just delegating it to meta. I've been glad of the ability to define
+>> new ad-hoc fields with this plugin without having to write an extra plugin
+>> to do so (listing books with a `bookauthor` and sorting them by
+>> `"field(bookauthor) title"`), but that'd be just as easy if `meta`
+>> accepted ad-hoc fields?
+>>
+>> --[[smcv]]
+
+>>> Your point above about cross-site scripting is a valid one, and something I
+>>> hadn't thought of (oops).
+
+>>> I still want to be able to populate pagetemplate templates with field, because I
+>>> use it for a number of things, such as setting which CSS files to use for a
+>>> given page, and, as I said, for titles.  But apart from the titles, I
+>>> realize I've been setting them in places other than the page data itself.
+>>> (Another unreleased plugin, `concon`, uses Config::Context to be able to
+>>> set variables on a per-site, per-directory and a per-page basis).
+
+>>> The first possible solution is what you suggested above: for field to only
+>>> set values in pagetemplate which are prefixed with *field_*.  I don't think
+>>> this is quite satisfactory, since that would still mean that people could
+>>> put un-scrubbed values into a pagetemplate, albeit they would be values
+>>> named field_foo, etc. --[[KathrynAndersen]]
+
+>>>> They can already do similar; `PERMALINK` is pre-sanitized to
+>>>> ensure that it's a "safe" URL, but if an extremely confused wiki admin was
+>>>> to put `COPYRIGHT` in their RSS/Atom feed's `<link>`, a malicious user
+>>>> could put an unsafe (e.g. Javascript) URL in there (`COPYRIGHT` *is*
+>>>> HTML-scrubbed, but "javascript:alert('pwned!')" is just text as far as a
+>>>> HTML sanitizer is concerned, so it passes straight through). The solution
+>>>> is to not use variables in situations where that variable would be
+>>>> inappropriate. Because `field` is so generic, the definition of what's
+>>>> appropriate is difficult. --[[smcv]]
+
+>>> An alternative solution would be to classify field registration as "secure"
+>>> and "insecure".  Sources such as ymlfront would be insecure, sources such
+>>> as concon (or the $config hash) would be secure, since they can't be edited
+>>> as pages.  Then, when doing pagetemplate substitution (but not ftemplate
+>>> substitution) the insecure sources could be HTML-escaped.
+>>> --[[KathrynAndersen]]
+
+>>>> Whether you trust the supplier of data seems orthogonal to whether its value
+>>>> is (meant to be) interpreted as plain text, HTML, a URL or what?
+>>>>
+>>>> Even in cases where you trust the supplier, you need to escape things
+>>>> suitably for the context, not for security but for correctness. The
+>>>> definition of the value, and the context it's being used in, changes the
+>>>> processing you need to do. An incomplete list:
+>>>>
+>>>> * HTML used as HTML needs to be html-scrubbed if and only if untrusted
+>>>> * URLs used as URLs need to be put through `safeurl()` if and only if
+>>>>   untrusted
+>>>> * HTML used as plain text needs tags removed regardless
+>>>> * URLs used as plain text are safe
+>>>> * URLs or plain text used in HTML need HTML-escaping (and URLs also need
+>>>>   `safeurl()` if untrusted)
+>>>> * HTML or plain text used in URLs need URL-escaping (and the resulting
+>>>>   URL might need sanitizing too?)
+>>>>
+>>>> I can't immediately think of other data types we'd be interested in beyond
+>>>> text, HTML and URL, but I'm sure there are plenty.
+
+>>>>> But isn't this a problem with anything that uses pagetemplates?  Or is
+>>>>> the point that, with plugins other than `field`, they all know,
+>>>>> beforehand, the names of all the fields that they are dealing with, and
+>>>>> thus the writer of the plugin knows which treatment each particular field
+>>>>> needs?  For example, that `meta` knows that `title` needs to be
+>>>>> HTML-escaped, and that `baseurl` doesn't.  In that case, yes, I see the problem.
+>>>>> It's a tricky one.  It isn't as if there's only ever going to be a fixed set of fields that need different treatment, either.  Because the site admin is free to add whatever fields they like to the page template (if they aren't using the default one, that is.  I'm not using the default one myself).
+>>>>> Mind you, for trusted sources, since the person writing the page template and the person providing the variable are the same, they themselves would know whether the value will be treated as HTML, plain text, or a URL, and thus could do the needed escaping themselves when writing down the value.
+
+>>>>> Looking at the content of the default `page.tmpl` let's see what variables fall into which categories:
+
+>>>>> * **Used as URL:** BASEURL, EDITURL, PARENTLINKS->URL, RECENTCHANGESURL, HISTORYURL, GETSOURCEURL, PREFSURL, OTHERLANGUAGES->URL, ADDCOMMENTURL, BACKLINKS->URL, MORE_BACKLINKS->URL
+>>>>> * **Used as part of a URL:** FAVICON, LOCAL_CSS
+>>>>> * **Needs to be HTML-escaped:** TITLE
+>>>>> * **Used as-is (as HTML):** FEEDLINKS, RELVCS, META, PERCENTTRANSLATED, SEARCHFORM, COMMENTSLINK, DISCUSSIONLINK, OTHERLANGUAGES->PERCENT, SIDEBAR, CONTENT, COMMENTS, TAGS->LINK, COPYRIGHT, LICENSE, MTIME, EXTRAFOOTER
+
+>>>>> This looks as if only TITLE needs HTML-escaping all the time, and that the URLS all end with "URL" in their name.  Unfortunately the FAVICON and LOCAL_CSS which are part of URLS don't have "URL" in their name, though that's fair enough, since they aren't full URLs.
+
+>>>>>  --K.A.
+
+>>>> One reasonable option would be to declare that `field` takes text-valued
+>>>> fields, in which case either consumers need to escape
+>>>> it with `<TMPL_VAR FIELD_FOO ESCAPE=HTML>`, and not interpret it as a URL
+>>>> without first checking `safeurl`), or the pagetemplate hook needs to
+>>>> pre-escape.
+
+>>>>> Since HTML::Template does have the ability to do ESCAPE=HTML/URL/JS, why not take advantage of that? Some things, like TITLE, probably should have ESCAPE=HTML all the time; that would solve the "to escape or not to escape" problem that `meta` has with titles. After all, when one *sorts* by title, one doesn't really want HTML-escaping in it; only when one uses it in a template. -- K.A.
+
+>>>> Another reasonable option would be to declare that `field` takes raw HTML,
+>>>> in which case consumers need to only use it in contexts that will be
+>>>> HTML-scrubbed (but it becomes unsuitable for using as text - problematic
+>>>> for text-based things like sorting or URLs, and not ideal for searching).
+>>>>
+>>>> You could even let each consumer choose how it's going to use the field,
+>>>> by having the `foo` field generate `TEXT_FOO` and `HTML_FOO` variables?
+>>>> --[[smcv]]
+
+>>>>> Something similar is already done in `template` and `ftemplate` with the `raw_` prefix, which determines whether the variable should have `htmlize` run over it first before the value is applied to the template.  Of course, that isn't scrubbing or escaping, because with those templates, the scrubbing is done afterwards as part of the normal processing.
+
+>>> Another problem, as you point out, is special-case fields, such as a number of
+>>> those defined by `meta`, which have side-effects associated with them, more
+>>> than just providing a value to pagetemplate.  Perhaps `meta` should deal with
+>>> the side-effects, but use `field` as an interface to get the values of those special fields.
 
-Bug report: `field` has an unnecessary `use YAML::Any`, presumably from before
-you separated out `ymlfront`. Trivial patch available from
-field-etc branch in git://git.pseudorandom.co.uk/git/smcv/ikiwiki.git (gitweb:
-<http://git.pseudorandom.co.uk/smcv/ikiwiki.git?a=shortlog;h=refs/heads/field-etc>)
---[[smcv]]
+>>> --[[KathrynAndersen]]
diff --git a/doc/plugins/contrib/pod/discussion.mdwn b/doc/plugins/contrib/pod/discussion.mdwn
index 76e858680..9187b1350 100644
--- a/doc/plugins/contrib/pod/discussion.mdwn
+++ b/doc/plugins/contrib/pod/discussion.mdwn
@@ -8,3 +8,7 @@ supports, or later support latex, that could be problimatic since that
 could maybe be used to include files or run code. --[[Joey]]
 
 > I don't know, either; the documentation for [[!cpan Pod:Xhtml]] is silent on this subject. --[[KathrynAndersen]]
+
+>> I'm afraid the only approach is to audit the existing code in the perl
+>> module(s), and then hope nothing is added to them later that opens a
+>> security hole. --[[Joey]] 
diff --git a/doc/plugins/contrib/ymlfront/discussion.mdwn b/doc/plugins/contrib/ymlfront/discussion.mdwn
index f437624b6..3ad02af29 100644
--- a/doc/plugins/contrib/ymlfront/discussion.mdwn
+++ b/doc/plugins/contrib/ymlfront/discussion.mdwn
@@ -2,3 +2,10 @@ My field-etc branch in git://git.pseudorandom.co.uk/git/smcv/ikiwiki.git (gitweb
 <http://git.pseudorandom.co.uk/smcv/ikiwiki.git?a=shortlog;h=refs/heads/field-etc>)
 has some fixes for compatibility with old YAML modules, mostly done by imitating
 Joey's code in IkiWiki::Setup::Yaml. Please consider merging :-) --[[smcv]]
+
+> I would if I could *find* it.  I checked out the "field-etc" branch, but I can't find the plugins in question under IkiWiki/Plugin; am I looking in the wrong place, or what?
+> --[[KathrynAndersen]]
+
+>> Sorry, I accidentally removed `field-etc` by pushing with `--mirror` from a
+>> different checkout. I've put it back; it's a branch from your `ikiplugins.git`,
+>> so yes, the code should be in `IkiWiki/Plugin`. --[[smcv]]
diff --git a/doc/plugins/sortnaturally.mdwn b/doc/plugins/sortnaturally.mdwn
new file mode 100644
index 000000000..a16381946
--- /dev/null
+++ b/doc/plugins/sortnaturally.mdwn
@@ -0,0 +1,6 @@
+[[!template id=plugin name=sortnaturally core=1 author="[[chrysn]], [[smcv]]"]]
+[[!tag type/meta]]
+
+This plugin provides the `title_natural` [[ikiwiki/pagespec/sorting]]
+order, which uses [[!cpan Sort::Naturally]] to sort numbered pages in a
+more natural order.
diff --git a/doc/plugins/write.mdwn b/doc/plugins/write.mdwn
index 96a2aa16d..707622956 100644
--- a/doc/plugins/write.mdwn
+++ b/doc/plugins/write.mdwn
@@ -633,6 +633,22 @@ reference. Do not modify this hash directly; call `add_link()`.
 
 	$links{"foo"} = ["bar", "baz"];
 
+### `%typedlinks`
+
+The `%typedlinks` hash records links of specific types. Do not modify this
+hash directly; call `add_link()`. The keys are page names, and the values
+are hash references. In each page's hash reference, the keys are link types
+defined by plugins, and the values are hash references with link targets
+as keys, and 1 as a dummy value, something like this:
+
+	$typedlinks{"foo"} = {
+		tag => { short_word => 1, metasyntactic_variable => 1 },
+		next_page => { bar => 1 },
+	};
+
+Ordinary [[WikiLinks|ikiwiki/WikiLink]] appear in `%links`, but not in
+`%typedlinks`.
+
 ### `%pagesources`
 
 The `%pagesources` has can be used to look up the source filename
@@ -718,7 +734,10 @@ Additional named parameters can be specified:
 * `filter` is a reference to a function, that is called and passed a page,
   and returns true if the page should be filtered out of the list.
 * `sort` specifies a sort order for the list. See
-  [[ikiwiki/PageSpec/sorting]] for the avilable sort methods.
+  [[ikiwiki/PageSpec/sorting]] for the avilable sort methods. Note that
+  if a sort method is specified that depends on the
+  page content (such as 'meta(foo)'), the deptype needs to be set to
+  a content dependency.
 * `reverse` if true, sorts in reverse.
 * `num` if nonzero, specifies the maximum number of matching pages that
   will be returned.
@@ -939,11 +958,14 @@ Optionally, a third parameter can be passed, to specify the preferred
 filename of the page. For example, `targetpage("foo", "rss", "feed")`
 will yield something like `foo/feed.rss`.
 
-### `add_link($$)`
+### `add_link($$;$)`
 
 This adds a link to `%links`, ensuring that duplicate links are not
 added. Pass it the page that contains the link, and the link text.
 
+An optional third parameter sets the link type. If not specified,
+it is an ordinary [[ikiwiki/WikiLink]].
+
 ## Miscellaneous
 
 ### Internal use pages
@@ -1110,6 +1132,24 @@ For example, "backlink(foo)" is influenced by the contents of page foo;
 they match; "created_before(foo)" is influenced by the metadata of foo;
 while "glob(*)" is not influenced by the contents of any page.
 
+### Sorting plugins
+
+Similarly, it's possible to write plugins that add new functions as
+[[ikiwiki/pagespec/sorting]] methods. To achieve this, add a function to
+the IkiWiki::SortSpec package named `cmp_foo`, which will be used when sorting
+by `foo` or `foo(...)` is requested.
+
+The names of pages to be compared are in the global variables `$a` and `$b`
+in the IkiWiki::SortSpec package. The function should return the same thing
+as Perl's `cmp` and `<=>` operators: negative if `$a` is less than `$b`,
+positive if `$a` is greater, or zero if they are considered equal. It may
+also raise an error using `error`, for instance if it needs a parameter but
+one isn't provided.
+
+The function will also be passed one or more parameters. The first is
+`undef` if invoked as `foo`, or the parameter `"bar"` if invoked as `foo(bar)`;
+it may also be passed additional, named parameters.
+
 ### Setup plugins
 
 The ikiwiki setup file is loaded using a pluggable mechanism. If you look
diff --git a/doc/todo/allow_plugins_to_add_sorting_methods.mdwn b/doc/todo/allow_plugins_to_add_sorting_methods.mdwn
index 0aca74be2..b523cd19f 100644
--- a/doc/todo/allow_plugins_to_add_sorting_methods.mdwn
+++ b/doc/todo/allow_plugins_to_add_sorting_methods.mdwn
@@ -10,6 +10,9 @@ title over the page name, but for compatibility, I'm not going to (I do wonder
 whether it would be worth making sort=name an alias for the current sort=title,
 and changing the meaning of sort=title in 4.0, though).
 
+> What compatability concerns, exactly, are there that prevent making that
+> change now? --[[Joey]] 
+
 *[sort-hooks branch now withdrawn in favour of sort-package --s]*
 
 I briefly tried to turn *all* the current sort types into hook functions, and
@@ -20,7 +23,7 @@ That earlier version of the branch is also available for comparison:
 
 >> I wonder if IkiWiki would benefit from the concept of a "sortspec", like a [[ikiwiki/PageSpec]] but dedicated to sorting lists of pages rather than defining lists of pages?  Rather than defining a sort-hook, define a SortSpec class, and enable people to add their own sort methods as functions defined inside that class, similarly to the way they can add their own pagespec definitions. --[[KathrynAndersen]]
 
->>> [[!template id=gitbranch branch=smcv/sort-package author="[[Simon_McVittie|smcv]]"]]
+>>> [[!template id=gitbranch branch=smcv/ready/sort-package author="[[Simon_McVittie|smcv]]"]]
 >>> I'd be inclined to think that's overkill, but it wasn't very hard to
 >>> implement, and in a way is more elegant. I set it up so sort mechanisms
 >>> share the `IkiWiki::PageSpec` package, but with a `cmp_` prefix. Gitweb:
@@ -40,7 +43,7 @@ That earlier version of the branch is also available for comparison:
 
 >>>>>> `SortSpec` --[[Joey]] 
 
->>>>>>> Done. --s
+>>>>>>> [[Done]]. --s
 
 >>>> I would be inclined to drop the `check_` stuff. --[[Joey]] 
 
@@ -207,7 +210,26 @@ Unfortunatly, I think that c is closest to the new implementation.
 >     }
 >
 > which would mean that the comparison used `$IkiWiki::SortSpec::a`.
->
+> --s
+
+>> I've now done this. On a wiki with many [[plugins/contrib/album]]s
+>> (a full rebuild takes half an hour!), I tested a refresh after
+>> `touch tags/*.mdwn` (my tag pages contain inlines of the form
+>> `tagged(foo)` sorted by date, so they exercise sorting).
+>> I also tried removing sorting from `pagespec_match_list`
+>> altogether, as an upper bound for how fast we can possibly make it.
+>>
+>> * `master` at branch point: 63.72user 0.29system
+>> * `master` at branch point: 63.91user 0.37system
+>> * my branch, with `@_`: 65.28user 0.29system
+>> * my branch, with `@_`: 65.21user 0.28system
+>> * my branch, with `$a`: 64.09user 0.28system
+>> * my branch, with `$a`: 63.83user 0.36system
+>> * not sorted at all: 58.99user 0.29system
+>> * not sorted at all: 58.92user 0.29system
+>>
+>> --s
+
 > I do notice that `pagespec_match_list` performs the sort before the
 > filter by pagespec. Is this a deliberate design choice, or
 > coincidence? I can see that when `limit` is used, this could be
@@ -218,7 +240,15 @@ Unfortunatly, I think that c is closest to the new implementation.
 
 >> Yes, it was deliberate, pagespec matching can be expensive enough that
 >> needing to sort a lot of pages seems likely to be less work. (I don't
->> remember what benchmarking was done though.) --[[Joey]] 
+>> remember what benchmarking was done though.) --[[Joey]]
+
+>>> We discussed this on IRC and Joey pointed out that this also affects
+>>> dependency calculation, so I'm not going to get into this now... --s
+
+Joey pointed out on IRC that the `titlesort` feature duplicates all the
+meta titles. I did that in order to sort by the unescaped version, but
+I've now changed the branch to only store that if it makes a difference.
+--s
 
 ## Documentation from sort-package branch
 
@@ -262,13 +292,13 @@ Similarly, it's possible to write plugins that add new functions as
 the IkiWiki::SortSpec package named `cmp_foo`, which will be used when sorting
 by `foo` or `foo(...)` is requested.
 
-The function will be passed three or more parameters. The first two are
-page names, and the third is `undef` if invoked as `foo`, or the parameter
-`"bar"` if invoked as `foo(bar)`. It may also be passed additional, named
-parameters.
+The names of pages to be compared are in the global variables `$a` and `$b`
+in the IkiWiki::SortSpec package. The function should return the same thing
+as Perl's `cmp` and `<=>` operators: negative if `$a` is less than `$b`,
+positive if `$a` is greater, or zero if they are considered equal. It may
+also raise an error using `error`, for instance if it needs a parameter but
+one isn't provided.
 
-It should return the same thing as Perl's `cmp` and `<=>` operators: negative
-if the first argument is less than the second, positive if the first argument
-is greater, or zero if they are considered equal. It may also raise an
-error using `error`, for instance if it needs a parameter but one isn't
-provided.
+The function will also be passed one or more parameters. The first is
+`undef` if invoked as `foo`, or the parameter `"bar"` if invoked as `foo(bar)`;
+it may also be passed additional, named parameters.
diff --git a/doc/todo/allow_site-wide_meta_definitions.mdwn b/doc/todo/allow_site-wide_meta_definitions.mdwn
index 7129a44ac..82670250e 100644
--- a/doc/todo/allow_site-wide_meta_definitions.mdwn
+++ b/doc/todo/allow_site-wide_meta_definitions.mdwn
@@ -217,6 +217,16 @@ definitions essentially.
 
 >>> For this to work with websetup and --dumpsetup, it needs to define the
 >>> `meta_*` settings in the getsetup function.
+>>>> 
+>>>> I think this will be problematic with the current implementation of this
+>>>> patch. The datatype here is an array of hash references, with each hash
+>>>> having a variable (and arbitrary) number of key/value pairs.  I can't
+>>>> think of an intuitive way of implementing a way of editing such a
+>>>> datatype in the web interface, let alone registering the option in
+>>>> getsetup.
+>>>> 
+>>>> Perhaps a limited set of defined meta values could be exposed via
+>>>> websetup (the obvious ones: author, copyright, license, etc.) -- [[Jon]]
 >>>
 >>> I also have some concerns about both these patches, since both throw
 >>> a lot of redundant data at meta, which then stores it in a very redundant
@@ -232,6 +242,5 @@ definitions essentially.
 >>> meta special-case the site-wide settings, not store them in these
 >>> per-page data structures, and just make them be used if no per-page
 >>> metadata of the given type is present. --[[Joey]]
-
->>>> Thanks for the review - these are all valid points. I'll get working
->>>> on a revised patch. -- [[Jon]]
+>>>> 
+>>>> that should be easy enough to do. I will work on a patch. -- [[Jon]]
diff --git a/doc/todo/matching_different_kinds_of_links.mdwn b/doc/todo/matching_different_kinds_of_links.mdwn
index c4383c0b7..da3ea49f6 100644
--- a/doc/todo/matching_different_kinds_of_links.mdwn
+++ b/doc/todo/matching_different_kinds_of_links.mdwn
@@ -53,7 +53,7 @@ I don't have any opinion on this syntax (whether it's good or not)...--Ivan Z.
 
 -------
 
->> [[!template id=gitbranch author="[[Simon_McVittie|smcv]]" branch=smcv/link-types]]
+>> [[!template id=gitbranch author="[[Simon_McVittie|smcv]]" branch=smcv/ready/link-types]]
 >> [[!tag patch]]
 
 ## Documentation for smcv's branch
@@ -176,3 +176,21 @@ though... --[[smcv]]
 
 >> Removed in a newer version of the branch. I re-introduced it as a
 >> plugin in `smcv/typedlink`, but I don't think we really need it. --s
+
+----
+
+I am ready to merge this, but I noticed one problem -- since `match_tagged`
+now only matches pages with the tag linktype, a wiki will need to be
+rebuilt on upgrade in order to get the linktype of existing tags in it
+recorded. So there needs to be a NEWS item about this and
+the postinst modified to force the rebuild.
+
+> Done, although you'll need to plug in an appropriate version number when
+> you release it. Is there a distinctive reminder string you grep for
+> during releases? I've used `UNRELEASED` for now. --[[smcv]]
+
+Also, the ready branch adds `typedlink()` to [[ikiwiki/pagespec]],
+but you removed that feature as documented above.
+--[[Joey]]
+
+> [[Done]]. --s
diff --git a/doc/todo/optional_underlaydir_prefix.mdwn b/doc/todo/optional_underlaydir_prefix.mdwn
new file mode 100644
index 000000000..06900a904
--- /dev/null
+++ b/doc/todo/optional_underlaydir_prefix.mdwn
@@ -0,0 +1,46 @@
+For security reasons, symlinks are disabled in IkiWiki.  That's fair enough, but that means that some problems, which one could otherwise solve by using a symlink, cannot be solved. The specfic problem in this case is that all underlays are placed at the root of the wiki, when it could be more convenient to place some underlays in specific sub-directories.
+
+Use-case 1 (to keep things tidy):
+
+Currently IkiWiki has some javascript files in `underlays/javascript`; that directory is given as one of the underlay directories.  Thus, all the javascript files appear in the root of the generated site.  But it would be tidier if one could say "put the contents of *this* underlaydir under the `js` directory".
+
+> Of course, this could be accomplished, if we wanted to, by moving the
+> files to `underlays/javascript/js`. --[[Joey]] 
+
+Use-case 2 (a read-only external dir):
+
+Suppose I want to include a subset of `/usr/local/share/docs` on my wiki, say the docs about `foo`.  But I want them to be under the `docs/foo` sub-directory on the generated site.  Currently I can't do that.  If I give `/usr/local/share/docs/foo` as an underlaydir, then the contents of that will be in the root of the site, rather than under `docs/foo`.  And if I give `/usr/local/share/docs` as an underlaydir, then the contents of the `foo` dir will be under `foo`, but it will also include every other thing in `/usr/local/share/docs`.
+
+Since we can't use symlinks in an underlay dir to link to these directories, then perhaps one could give a specific underlay dir a specific prefix, which defines the sub-directory that the underlay should appear in.
+
+I'm not sure how this would be implemented, but I guess it could be configured something like this:
+
+    prefixed_underlay => {
+         'js' => '/usr/local/share/ikiwiki/javascript',
+         'docs/foo' => '/usr/local/share/docs/foo',
+    }
+
+> So, let me review why symlinks are an issue. For normal, non-underlay
+> pages, users who do not have filesystem access to the server may have 
+> commit access, and so could commit eg, a symlink to `/etc/passwd` (or
+> to `/` !). The guards are there to prevent ikiwiki either exposing the
+> symlink target's contents, or potentially overwriting it.
+> 
+> Is this a concern for underlays? Most of the time, certianly not;
+> the underlay tends to be something only the site admin controls.
+> Not all the security checks that are done on the srcdir are done
+> on the underlays, either. Most checks done on files in the underlay
+> are only done because the same code handles srcdir files. The one
+> exception is the test that skips processing symlinks in the underlay dir.
+> (But note that the underlay directory can be a symlinkt to elsewhere
+> which the srcdir, by default, cannot.)
+> 
+> So, one way to approach this is to make ikiwiki follow directory symlinks
+> inside the underlay directory. Just a matter of passing `follow => 1` to
+> find. (This would still not allow individual files to be symlinks, because
+> `readfile` does not allow reading symlinks. But I don't see much need
+> for that.) --[[Joey]]
+
+>> If you think that enabling symlinks in underlay directories wouldn't be a security issue, then I'm all for it!  That would be much simpler to implement, I'm sure. --[[KathrynAndersen]]
+
+[[!taglink wishlist]]
diff --git a/doc/users/KathrynAndersen/discussion.mdwn b/doc/users/KathrynAndersen/discussion.mdwn
index 288ea8c51..4f2790c39 100644
--- a/doc/users/KathrynAndersen/discussion.mdwn
+++ b/doc/users/KathrynAndersen/discussion.mdwn
@@ -9,6 +9,12 @@ that as a patch to the existing map plugin.) --[[Joey]]
 
 > I think pmap is probably better as a separate plugin, because it has additional dependencies (HTML::LinkList) which people might not want to have to install.
 
+>> One approach commonly used in ikiwiki is to make such optional features
+>> be enabled by a switch somewhere, and 'eval q{use Foo}` so the module
+>> does not have to be loaded unless the feature is used. --[[Joey]]
+
+>>> Unfortunately, HTML::LinkList isn't an optional feature for pmap; that's what it uses to create the HTML for the map. --[[KathrynAndersen]]
+
 > The "includepage" plugin I'm not sure whether it is worth releasing or not; it's basically a cut-down version of "inline", because the inline plugin is so complicated and has so many options, I felt more at ease to have something simpler.
 
 > --[[KathrynAndersen]]