* ikiwiki can now download and aggregate feeds with its new aggregate

  plugin, so it's possible to implement a Planet using ikiwiki!
* --setup --refresh no longer rebuilds wrappers. Use --setup --refresh
  --wrappers to do that.
* Add %IkiWiki::forcerebuild to provide a way for plugins like aggregate
  to update pages that haven't changed on disk.

10 files changed, 81 insertions, 33 deletions

diff --git a/doc/features.mdwn b/doc/features.mdwn
index a43cd1c9a..0a235d708 100644
--- a/doc/features.mdwn
+++ b/doc/features.mdwn
@@ -52,6 +52,10 @@ Some of ikiwiki's features:
   Ikiwiki's own [[TODO]], [[news]], and [[plugins]] pages are good examples
   of some of the flexible ways that this can be used.
 
+  Ikiwiki can also [[plugin/aggregate]] external blogs, feeding them into
+  the wiki. This can be used to create a Planet type site that aggregates
+  interesting feeds.
+
 * [[tags]]
 
   You can tag pages and use these tags in various ways. Tags will show
diff --git a/doc/install.mdwn b/doc/install.mdwn
index d23c88e41..eb5b91e67 100644
--- a/doc/install.mdwn
+++ b/doc/install.mdwn
@@ -4,7 +4,7 @@ Ikiwiki requires [[MarkDown]] be installed, and also uses the following
 perl modules if available: `CGI::Session` `CGI::FormBuilder` (version
 3.02.02 or newer) `HTML::Template` `Mail::SendMail` `Time::Duration`
 `Date::Parse` (libtimedate-perl), `HTML::Scrubber`, `RPC::XML`,
-`XML::Simple`
+`XML::Simple`, `XML::Feed`, `HTML::Parser`
 
 If you want to install from the tarball, you should make sure that the
 required perl modules are installed, then run:
diff --git a/doc/plugins.mdwn b/doc/plugins.mdwn
index 0596fc068..1c04d09f5 100644
--- a/doc/plugins.mdwn
+++ b/doc/plugins.mdwn
@@ -1,7 +1,7 @@
 Most of ikiwiki's [[features]] are implemented as plugins. Beyond the
 [[type/core]] features, there are plugins to [[type/format]] text,
-use [[type/tags]], show [[type/meta]] information, or just have
-[[type/fun]].
+use [[type/tags]], show [[type/meta]] information, do other [[type/useful]]
+stuff, or just have [[type/fun]].
 
 There's documentation if you want to [[write]] your own plugins, or you can
 install and use plugins contributed by others. 
diff --git a/doc/plugins/aggregate.mdwn b/doc/plugins/aggregate.mdwn
new file mode 100644
index 000000000..690904f73
--- /dev/null
+++ b/doc/plugins/aggregate.mdwn
@@ -0,0 +1,53 @@
+This plugin allows content from other blogs to be aggregated into the wiki.
+Aggregate a blog as follows:
+
+	\[[aggregate name="example blog" feedurl="http://example.com/index.rss" url="http://example.com/" updateinterval="15" dir="example"]
+
+That example aggregates posts from the expecified RSS feed, updating no
+more frequently than once every 15 minutes, and puts a page per post under
+the example/ directory in the wiki.
+
+You can then use ikiwiki's [[blog]] support to create a blog of one or more
+aggregated feeds.
+
+## setup
+
+Make sure that you have the [[html]] plugin enabled, as the created pages are
+in html format. The [[tag]] plugin is also recommended.
+
+You will need to run ikiwiki periodically from a cron job, passing it the
+--aggregate parameter, to make it check for new posts. Here's an example
+crontab entry:
+
+	*/15 * * * * ikiwiki --setup my.wiki --aggregate --refresh
+
+## usage
+
+Here are descriptions of all the supported parameters to the `aggregate`
+directive:
+
+* `name` - A name for the feed. Each feed must have a unique name.
+  Required.
+* `url` - The url to the web page for the blog that's being aggregated.
+  Required.
+* `dir` - The directory in the wiki where pages should be saved. Required.
+* `feedurl` - The url to the feed. Optional, if it's not specified ikiwiki
+  will look for feeds on the `blogurl`. RSS and atom feeds are supported.
+* `updateinterval` - How often to check for new posts, in minutes. Default
+  is 15 minutes.
+* `expireage` - Expire old items from this blog if they are older than
+  a specified number of days. Default is to never expire on age.
+* `expirecount` - Expire old items from this blog if there are more than
+  the specified number total. Oldest items will be expired first. Default
+  is to never expire on count.
+* `tag` - A tag to tag each post from the blog with. A good tag to use is
+  the name of the blog. Can be repeated multiple times. The [[tag]] plugin
+  must be enabled for this to work.
+
+Note that even if you are using subversion or another revision control
+system, pages created by aggregation will *not* be checked into revision
+control.
+
+This plugin is not enabled by default.
+
+[[tag type/useful]]
diff --git a/doc/plugins/search.mdwn b/doc/plugins/search.mdwn
index d4a6b4efe..78088aed8 100644
--- a/doc/plugins/search.mdwn
+++ b/doc/plugins/search.mdwn
@@ -4,3 +4,5 @@ full text search to ikiwiki, using the [[HyperEstraier]] engine.
 It's possible to configure HyperEstraier via one of ikiwiki's
 [[templates]], but for most users, no configuration should be needed aside
 from enabling the plugin.
+
+[[tag type/useful]]
diff --git a/doc/plugins/type/useful.mdwn b/doc/plugins/type/useful.mdwn
new file mode 100644
index 000000000..92fcf5af1
--- /dev/null
+++ b/doc/plugins/type/useful.mdwn
@@ -0,0 +1 @@
+These plugins perform various miscellaneous useful functions.
diff --git a/doc/plugins/write.mdwn b/doc/plugins/write.mdwn
index 025a242a6..925717777 100644
--- a/doc/plugins/write.mdwn
+++ b/doc/plugins/write.mdwn
@@ -96,7 +96,7 @@ make arbitrary changes. The function is passed named parameters `page` and
 
 ## htmlize
 
-	IkiWiki::hook(type => "htmlize", id => "ext", call => \&filter);
+	IkiWiki::hook(type => "htmlize", id => "ext", call => \&htmlize);
 
 Runs on the raw source of a page and turns it into html. The id parameter
 specifies the filename extension that a file must have to be htmlized using
@@ -135,7 +135,7 @@ content.
 
 ## delete
 
-	IkiWiki::hook(type => "delete", id => "foo", call => \&dele);
+	IkiWiki::hook(type => "delete", id => "foo", call => \&delete);
 
 Each time a page or pages is removed from the wiki, the referenced function
 is called, and passed the names of the source files that were removed.
@@ -190,6 +190,8 @@ use the following hashes, using a page name as the key:
   Many plugins will need to add dependencies to this hash; the best way to do
   it is by using the IkiWiki::add_depends function, which takes as its
   parameters the page name and a [[GlobList]] of dependencies to add.
+* `%IkiWiki::forcerebuild` any pages set as the keys to this hash will be
+  treated as if they're modified and rebuilt.
 
 # A note on generating html links
 
diff --git a/doc/templates.mdwn b/doc/templates.mdwn
index 10f715d9d..e500638f4 100644
--- a/doc/templates.mdwn
+++ b/doc/templates.mdwn
@@ -26,6 +26,8 @@ It ships with some basic templates which can be customised:
   can read the [[HyperEstraier]] docs and configure it using this.
 * `blogpost.tmpl` - Used for a form to add a post to a blog (and a rss link)
 * `rsslink.tmpl` - Used to add a rss link if blogpost.tmpl is not used.
+* `aggregatepost.tmpl` - Used by the [[plugins/aggregate]] plugin to create
+  a page for a post.
 
 If you like, you can add these to further customise it:
 
diff --git a/doc/todo/aggregation.mdwn b/doc/todo/aggregation.mdwn
index 7d765f9e9..53b3133e2 100644
--- a/doc/todo/aggregation.mdwn
+++ b/doc/todo/aggregation.mdwn
@@ -1,24 +1 @@
-Here's a scary idea.. A plugin that can aggregate feeds from other
-locations. Presumably there would need to be a cron job to build the wiki
-periodically, and each time it's built any new items would be turned into
-pages etc. There might also need to be a way to expire old items, unless
-you wanted to keep them forever.
-
-This would allow ikiwiki to work as a kind of a planet, or at least a
-poor-man's news aggregator.
-
-* XML::Feed has a very nice interface, may require valid feeds though.
-* How to store GUIDs? Maybe as meta tags on pages, although that would need
-  caching of such metadata somewhere.
-* How to configure which feeds to pull, how often, and where to put the
-  pulled entries? One way would be command line/config file, but I think
-  better would be to use preprocessor directives in a wiki page, probably
-  the same page that inlines all the pages together.
-* Where to store when a feed was last pulled?
-
-So I need:
-
-* A way to store info from the preprocessor directives about what pages
-  to pull and expiry.
-* A way to store info on last pull time, guids, etc.
-* Switch for a mode that a) pulls b) expires old c) rebuilds wiki (for cron)
+* Still need to support feed expiry.
diff --git a/doc/usage.mdwn b/doc/usage.mdwn
index 691880a96..a6ded5ec2 100644
--- a/doc/usage.mdwn
+++ b/doc/usage.mdwn
@@ -24,8 +24,7 @@ These options control the mode that ikiwiki is operating in.
 * --refresh
 
   Refresh the wiki, updating any changed pages. This is the default
-  behavior so you don't normally need to specify it. If used with -setup also
-  updates any configured wrappers.
+  behavior so you don't normally need to specify it.
 
 * --rebuild
 
@@ -59,11 +58,19 @@ These options control the mode that ikiwiki is operating in.
 
   The default action when --setup is specified is to automatically generate
   wrappers for a wiki based on data in a config file, and rebuild the wiki.
-  If you also pass the --refresh option, ikiwiki will instead just refresh
-  the wiki described in the setup file.
 
   [[ikiwiki.setup]] is an example of such a config file.
 
+* --wrappers
+
+  If used with --setup --refresh, this makes it also update any configured
+  wrappers.
+
+* --aggregate
+
+  If the aggregate plugin is enabled, this makes ikiwiki poll configured
+  feeds and save new posts to the srcdir.
+
 # CONFIG OPTIONS
 
 These options configure the wiki. Note that plugins can add additional