diff options
Diffstat (limited to 'doc/forum/ikiwiki_and_big_files.mdwn')
| -rw-r--r-- | doc/forum/ikiwiki_and_big_files.mdwn | 102 |
1 files changed, 102 insertions, 0 deletions
diff --git a/doc/forum/ikiwiki_and_big_files.mdwn b/doc/forum/ikiwiki_and_big_files.mdwn new file mode 100644 index 000000000..cd41d9fce --- /dev/null +++ b/doc/forum/ikiwiki_and_big_files.mdwn @@ -0,0 +1,102 @@ +My website has 214 hand written html, 1500 of pictures and a few, err sorry, 114 +video files. All this takes around 1.5 GB of disk space at the moment. +Plain html files take 1.7 MB and fit naturally into git. + +But what about the picture and video files? + +Pictures are mostly static and rarely need to be edited after first upload, +wasting a megabyte or two after an edit while having them in git doesn't really matter. +Videos on the other hand are quite large from megabytes to hundreds. Sometimes +I re-encode them from the original source with better codec parameters and just +replace the files under html root so they are accessible from the same URL. +So having a way to delete a 200 MB file and upload a new one with same name and access URL +is what I need. And it appears git has trouble erasing commits from history, or requires +some serious gitfoo and good backups of the original repository. + +So which ikiwiki backend could handle piles of large binary files? Or should I go for a separate +data/binary blob directory next to ikiwiki content? + +Further complication is my intention to keep URL compatibility with old handwritten and ikiwiki +based site. Sigh, tough job but luckily just a hobby. + +[-Mikko](http://mcfrisk.kapsi.fi) + +ps. here's how to calculate space taken by html, picture and video files: + + ~/www$ unset sum; for size in $( for ext in htm html txt xml log; \ + do find . -iname "*$ext" -exec stat -c "%s" \{\} \; ; done | xargs ); \ + do sum=$(( $sum + $size )); done ; echo $sum + 1720696 + ~/www$ unset sum; for size in $( for ext in jpg gif jpeg png; \ + do find . -iname "*$ext" -exec stat -c "%s" \{\} \; ; done | xargs ); \ + do sum=$(( $sum + $size )); done ; echo $sum + 46032184 + ~/www$ unset sum; for size in $( for ext in avi dv mpeg mp4; \ + do find . -iname "*$ext" -exec stat -c "%s" \{\} \; ; done | xargs ); \ + do sum=$(( $sum + $size )); done ; echo $sum + 1351890888 + +> One approach is to use the [[plugins/underlay]] plugin to +> configure a separate underlay directory, and put the large +> files in there. Those files will then be copied to the generated +> wiki, but need not be kept in revision control. (Or could be +> revision controlled in a separate repository -- perhaps one using +> a version control system that handles large files better than git; +> or perhaps one that you periodically blow away the old history to +> in order to save space.) +> +> BTW, the `hardlink` setting is a good thing to enable if you +> have large files, as it saves both disk space and copying time. +> --[[Joey]] + +Can underlay plugin handle the case that source and destination directories +are the same? I'd rather have just one copy of these underlay files on the server. + +> No, but enabling hardlinks accomplishes the same effect. --[[Joey]] + +And did I goof in the setup file since I got this: + + $ ikiwiki -setup blog.setup -rebuild --verbose + Can't use string ("/home/users/mcfrisk/www/blog/med") as an ARRAY ref while + "strict refs" in use at + /home/users/mcfrisk/bin/share/perl/5.10.0/IkiWiki/Plugin/underlay.pm line 41. + $ grep underlay blog.setup + add_plugins => [qw{goodstuff websetup comments blogspam html sidebar underlay}], + underlaydir => '/home/users/mcfrisk/bin/share/ikiwiki/basewiki', + # underlay plugin + # extra underlay directories to add + add_underlays => '/home/users/mcfrisk/www/blog/media', + $ egrep "(srcdir|destdir)" blog.setup + srcdir => '/home/users/mcfrisk/blog', + destdir => '/home/users/mcfrisk/www/blog', + # allow symlinks in the path leading to the srcdir (potentially insecure) + allow_symlinks_before_srcdir => 1, + # directory in srcdir that contains directive descriptions + +-Mikko + +> The plugin seems to present a bad default value in the setup file. +> (Fixed in git.) A correct configuration would be: + + add_underlays => ['/home/users/mcfrisk/www/blog/media'], + +Umm, doesn't quite fix this yet: + + $ ikiwiki -setup blog.setup -v + Can't use an undefined value as an ARRAY reference at /home/users/mcfrisk/bin/share/perl/5.10.0/IkiWiki + /Plugin/underlay.pm line 44. + $ grep underlay blog.setup + add_plugins => [qw{goodstuff websetup comments blogspam html sidebar underlay}], + underlaydir => '/home/users/mcfrisk/bin/share/ikiwiki/basewiki', + # underlay plugin + # extra underlay directories to add + add_underlays => ['/home/users/mcfrisk/www/blog/media'], + $ ikiwiki --version + ikiwiki version 3.20091032 + +-Mikko + +> Yeah, I've fixed that in git, but you can work around it with this: +> --[[Joey]] + + templatedirs => [], |