diff options
| -rw-r--r-- | IkiWiki.pm | 2 | ||||
| -rw-r--r-- | doc/todo/aggregate_locking.mdwn | 66 |
2 files changed, 67 insertions, 1 deletions
diff --git a/IkiWiki.pm b/IkiWiki.pm index 050d4e5b2..8172552cb 100644 --- a/IkiWiki.pm +++ b/IkiWiki.pm @@ -884,7 +884,7 @@ sub loadindex () { #{{{ %oldrenderedfiles=%pagectime=(); if (! $config{rebuild}) { %pagesources=%pagemtime=%oldlinks=%links=%depends= - %destsources=%renderedfiles=%pagecase=(); + %destsources=%renderedfiles=%pagecase=%pagestate=(); } open (my $in, "<", "$config{wikistatedir}/index") || return; while (<$in>) { diff --git a/doc/todo/aggregate_locking.mdwn b/doc/todo/aggregate_locking.mdwn new file mode 100644 index 000000000..91df662a7 --- /dev/null +++ b/doc/todo/aggregate_locking.mdwn @@ -0,0 +1,66 @@ +The [[plugin/aggregate]] plugin's locking is a suboptimal. + +There should be no need to lock the wiki while aggregating -- it's annoying +that long aggregate runs can block edits from happening. However, not +locking would present problems. One is, if an aggregate run is happening, +and the feed is removed, it could continue adding pages for that feed. +Those pages would then become orphaned, and stick around, since the feed +that had created them is gone, and thus there's no indication that they +should be removed. + +To fix that, garbage collect any pages that were created by +aggregation once their feed is gone. + +Are there other things that could happen while it's aggregating that it +should check for? + +Well, things like the feed url etc could change, and it +would have to merge in such changes before saving the aggregation state. +New feeds could also be added, feeds could be moved from one source page to +another. + +Merging that feed info seems doable, just re-load the aggregation state +from disk, and set the `message`, `lastupdate`, `numposts`, and `error` +fields to their new values if the feed still exists. + +---- + +Another part of the mess is that it needs to avoid stacking multiple +aggregate processes up if aggregation is very slow. Currently this is done +by taking the lock in nonblocking mode, and not aggregating if it's locked. +This has various problems, for example a page edit at the right time can +prevent aggregation from happening. + +Adding another lock just for aggregation could solve this. Check that lock +(in checkconfig) and exit if another aggregator holds it. + +---- + +The other part of the mess is that it currently does aggregation in +checkconfig, locking the wiki for that, and loading state, and then +dropping the lock, unloading state, and letting the render happen. Which +reloads state. That state reloading is tricky to do just right. + +A simple fix: Move the aggregation to the new 'render' hook. Then state +would be loaded, and there would be no reason to worry about aggregating. + +Or aggregation could be kept in checkconfig, like so: + +* lock wiki +* load aggregation state +* unlock wiki +* get list of feeds needing aggregation +* exit if none +* attempt to take aggregation lock, exit if another aggregation is happening +* fork a child process to do the aggregation + * lock wiki + * load wiki state (needed for aggregation to run) + * unlock wiki + * aggregate + * lock wiki + * reload aggregation state + * merge in aggregation state changes + * unlock wiki +* drop aggregation lock +* force rebuild of sourcepages of feeds that were aggregated +* exit checkconfig and continue with usual refresh process |