Avoid %links accumulating duplicates. (For TOVA)

This is sorta an optimisation, and sorta a bug fix. In one
test case I have available, it can speed a page build up from 3
minutes to 3 seconds.

The root of the problem is that $links{$page} contains arrays of
links, rather than hashes of links. And when a link is found,
it is just pushed onto the array, without checking for dups.

Now, the array is emptied before scanning a page, so there
should not be a lot of opportunity for lots of duplicate links
to pile up in it. But, in some cases, they can, and if there
are hundreds of duplicate links in the array, then scanning it
for matching links, as match_link and some other code does,
becomes much more expensive than it needs to be.

Perhaps the real right fix would be to change the data structure
to a hash. But, the list of links is never accessed like that,
you always want to iterate through it.

I also looked at deduping the list in saveindex, but that does
a lot of unnecessary work, and doesn't completly solve the problem.

So, finally, I decided to add an add_link function that handles deduping,
and make ikiwiki-transition remove the old dup links.

5 files changed, 7 insertions, 7 deletions

diff --git a/IkiWiki/Plugin/camelcase.pm b/IkiWiki/Plugin/camelcase.pm
index 74a8397d7..088447d6b 100644
--- a/IkiWiki/Plugin/camelcase.pm
+++ b/IkiWiki/Plugin/camelcase.pm
@@ -61,7 +61,7 @@ sub scan (@) {
         my $content=$params{content};
 
 	while ($content =~ /$link_regexp/g) {
-		push @{$links{$page}}, linkpage($1) unless ignored($1)
+		add_link($page, linkpage($1)) unless ignored($1)
 	}
 }
 
diff --git a/IkiWiki/Plugin/img.pm b/IkiWiki/Plugin/img.pm
index d295b833b..a697fea19 100644
--- a/IkiWiki/Plugin/img.pm
+++ b/IkiWiki/Plugin/img.pm
@@ -43,7 +43,7 @@ sub preprocess (@) {
 		return '';
 	}
 
-	push @{$links{$params{page}}}, $image;
+	add_link($params{page}, $image);
 	# optimisation: detect scan mode, and avoid generating the image
 	if (! defined wantarray) {
 		return;
diff --git a/IkiWiki/Plugin/link.pm b/IkiWiki/Plugin/link.pm
index b79273f96..4c1add985 100644
--- a/IkiWiki/Plugin/link.pm
+++ b/IkiWiki/Plugin/link.pm
@@ -86,7 +86,7 @@ sub scan (@) {
 	my $content=$params{content};
 
 	while ($content =~ /(?<!\\)$link_regexp/g) {
-		push @{$links{$page}}, linkpage($2);
+		add_link($page, linkpage($2));
 	}
 }
 
diff --git a/IkiWiki/Plugin/meta.pm b/IkiWiki/Plugin/meta.pm
index 4a22fed30..cc5455d64 100644
--- a/IkiWiki/Plugin/meta.pm
+++ b/IkiWiki/Plugin/meta.pm
@@ -110,7 +110,7 @@ sub preprocess (@) {
 	}
 	elsif ($key eq 'link' && ! %params) {
 		# hidden WikiLink
-		push @{$links{$page}}, $value;
+		add_link($page, $value);
 		return "";
 	}
 	elsif ($key eq 'author') {
diff --git a/IkiWiki/Plugin/tag.pm b/IkiWiki/Plugin/tag.pm
index 8fe9c6828..cdcfaf536 100644
--- a/IkiWiki/Plugin/tag.pm
+++ b/IkiWiki/Plugin/tag.pm
@@ -73,7 +73,7 @@ sub preprocess_tag (@) {
 		$tag=linkpage($tag);
 		$tags{$page}{$tag}=1;
 		# hidden WikiLink
-		push @{$links{$page}}, tagpage($tag);
+		add_link($page, tagpage($tag));
 	}
 		
 	return "";
@@ -88,14 +88,14 @@ sub preprocess_taglink (@) {
 		if (/(.*)\|(.*)/) {
 			my $tag=linkpage($2);
 			$tags{$params{page}}{$tag}=1;
-			push @{$links{$params{page}}}, tagpage($tag);
+			add_link($params{page}, tagpage($tag));
 			return taglink($params{page}, $params{destpage}, $tag,
 				linktext => pagetitle($1));
 		}
 		else {
 			my $tag=linkpage($_);
 			$tags{$params{page}}{$tag}=1;
-			push @{$links{$params{page}}}, tagpage($tag);
+			add_link($params{page}, tagpage($tag));
 			return taglink($params{page}, $params{destpage}, $tag);
 		}
 	}