A mostly static website with Emacs Orgmode

When I first learned to make web pages, we just hand-coded them in HTML and saved them to folders as text files. There they would stay put until someone pointed their browser e.g. to http://christianmoe.com/en/notes/static-site.html, which let the web server know to retrieve the page static-site.html from the en/notes folder.

Today, even the humblest of blogs is typically a dynamic site driven by a Content Management System (CMS) with a lot of moving parts: not just a web server, but a database and a web application with thousands of lines of PHP slaving away behind the scenes to put together pages on the fly. The folders are virtual and the whole content is just one database file.

The best CMSes offer many advantages:

• Content is easily searched, sorted, filtered, listed, combined…
• Virtual folder structures are easily changed, permalinks remain.
• Common elements, like a list of latest blog posts, are easily added across all pages.
• Changes can be automatically logged.
• Migration to/from a different platform is sometimes supported.
• Authenticated users can be assigned roles in a workflow to create content together; certain contect can be restricted to certain users.
• Comments, pingbacks, and even Fediverse integration can be handled automatically.
• There is support, community, and a well-honed codebase from which to learn if you want to tinker for yourself (assuming it’s open source).
• Reasonable security precautions have hopefully been taken by people who know what they’re doing.

But if you just want your own personal website, without breaking news and archives of thousands of pages, you don’t really need all of this. So some people turn to static-site generators instead. These take care of a lot of the same housekeeping for you as CMSes do (adding common page elements, tags, timestamps etc.).

But instead of putting documents together on the fly every time someone wants to read them, static-site generators only put them together for you every time you press “publish.” They also typically allow you to write in a plain-text format, usually some flavor of Markdown, and convert it to HTML, which saves typing a lot of angle brackets.

Static sites are simple, fast, and secure. The server only has to find the page and serve it, not execute a lot of code and database queries.

Also, CMSes tend to expect you to type your thoughts into a browser window. There are better ways to write.

For me, a better way to write is with the Org-mode major mode of the Emacs text editor. If Emacs is practically an operating system, then Org-mode is practically an office package in plain text: calendar, tasks, lists, notes, tags, links, simple database features, totally awesome plain-text tables that double as a powerful spreadsheet, and in recent years, top-notch native support for citations and bibliographies. Unlike your typical office package, it can also mix your text with code. All this, including both the code and its output, can be exported to various formats. And it can be published to the web.

If I used a CMS, I would use Org-mode to draft content anyway, export it to HTML and paste it in, as I used to do in my Wordpress blog on climate. So why not just publish it directly from Org-mode?

Some people use Org-mode in combination with static-site generators such as Jekyll, sometimes with tricks to get the best of both worlds. For this site, I have tried exploring what can be done using Org-mode’s publishing feature out of the box, plus a few added scripts that a hobbyist like me could write on their own.

To publish a website with Org, you write the content as Org files in the same folder structure you want on your website. The source of this page is the file ~/www/cm/en/staticsite.org, which as you see lies in the en subfolder of the cm project in my www folder.

Next, you have to set up one or more publishing projects. You can read all about it in the manual. (If you’re not familiar with Emacs, you should be aware that you can configure it any way you like by writing code in the Emacs-Lisp programming language in a configuration file that typically lives in your home directory and is called .emacs.) Basically, you define the publishing project by listing your requirements in a Lisp expression that you assign to the variable org-publish-project-alist in your Emacs configuration file. It tells Org-mode where to find your project, where to publish it, and what to do with the pages that are part of it.

Here is the general structure of my configuration right now (Listing 1):

(setq org-publish-project-alist
        '(("Landing"
           :base-directory "~/www/cm/"
           ; (...)
           )
          ("Norsk"
           :base-directory "~/www/cm/nb"
           ; (...)
           )
          ("Esperanto"
           :base-directory "~/www/cm/eo"
           ; (...)
           )
          ("English"
           :base-directory "~/www/cm/en"
           ; (...)
           )
          ("Slovensko"
           :base-directory "~/www/cm/sl"
           ; (...)
           )
          ("CSS" ; and .js scripts
           :base-directory "~/www/cm/css"
           :base-extension "css\\|js"
           ; (...)
           )
          ("media" ; images, HTML, PDF, XML (RSS), raw text, webfonts
           :base-directory "~/www/cm/"
           :base-extension "jpg\\|gif\\|png\\|ico\\|svg\\|htm\\|html\\|pdf\\|xml\\|\\txt\\|ttf\\|woff2"
           ; (...)
           :recursive t)
          ("RSS"
           :base-directory "~/www/cm/"
         :preparation-function cm/write-newsfeed
         ; (...)
          ("ChristianMoe.com"
           :components ("Landing" "English" "Norsk" "Esperanto" "Slovensko" "CSS" "media" "RSS"))))

As you see at the bottom of the list, a project (ChristianMoe.com) can have several component subprojects (Landing, Norsk, English etc.). Importantly, for each subproject you can configure a bunch of export options telling Org how to format your exported document. This allows you to set up different rules for what to do with text files and images, or with files in different subfolders. Here is the full content of the Norwegian (“Norsk”) subproject:

("Norsk"
 :base-directory "~/www/cm/nb"
 :publishing-directory "/var/www/html/nb"
 :language "nb"
 :publishing-function org-html-publish-to-html
 :html-doctype "xhtml-strict"
 :recursive t
 :exclude "^\\."
 :section-numbers nil
 :with-toc nil
 :html-preamble cm/html-preamble:nb
 :html-postamble t
 :auto-sitemap t
 ;; :sitemap-function org-publish-sitemap
 :sitemap-sort-folders last
 :sitemap-title "Nettstedskart"
 :sitemap-sans-extension t
 :sitemap-style tree
 :sitemap-file-entry-format "%t" ;"@@html:<b>%t</b><br><i>Lagt ut %d av %a</i><br>" ; Keywords: %k<br>%e@@"
 :makeindex t
 :html-head-include-default-style nil
 :html-head-include-scripts nil
 :html-head "<link rel=\"stylesheet\" href=\"/css/cm.css\" type=\"text/css\"/><script src=\"/css/jquery-1.9.1.js\" type=\"text/javascript\"></script><script src=\"/css/tagindex.js\" type=\"text/javascript\"></script><script src=\"/css/cm.js\" type=\"text/javascript\"></script><link rel=\"icon\" href=\"/img/favicon.png\" />")

You can still also add export options on a per-page basis. For most of these pages I don’t want auto-generated tables of contents, hence I have :with-toc nil in the config, but in this page I have ordered one by adding #+OPTIONS: toc:t to the top of static-site.org.

You can also tell Org-mode what to put in the html HEAD of each document. In the three lines at the end that start with :html-head-, I tell it not to use the default CSS and scripts Org HTML export offers (which are good), but to use mine instead. Again, you can very easily extend or override this on a per-page basis by putting a stylesheet link or a <style> section after #+HTML_HEAD: or #+HTML_HEAD_EXTRA: on top of the page.

You can also configure a preamble to be inserted before the content of each page, and a postamble afterwards. This is how I add the site banner and menu (preamble) and the right or bottom colophon (postamble) to each page. The simple but powerful out-of-the-box way to do this is by defining the variables org-publish-preamble-format and org-html-postamble-format. These are associative lists of HTML strings keyed to language, so you can have different postambles on a per-language basis, and they comes with a number of format strings (%t for title, %d for date, etc.) for templating, so for many purposes, they give you all the flexibility you need for a multilingual site. I do this for the postamble by setting :html-postamble t. As you see, however, to generate the :html-preamble I’m using a homemade function instead, because I want to do things a little differently not only per language but also e.g. depending on whether the page is a blog entry or not, and because it also lets me call other functions for housekeeping.

By default, Org publishing will overlook files not ending in .org. I have used a “CSS” subproject to also publish stylesheets, scripts, and webfonts, recognized by their file extensions, to a specific folder (and yes, I should rename it, since it’s not just CSS). Unlike .org files, obviously, these should not be exported to HTML with the function org-html-publish-to-html. Instead, I set the :publishing-function to org-publish-attachment, which simply transfers the files. Here’s the subproject of org-publish-project-alist:

("CSS" ; and .js scripts, and webfonts
 :base-directory "~/www/cm/css"
 :base-extension "css\\|js\\|ttf\\|woff2"
 :publishing-directory "/var/www/html/css"
 :publishing-function org-publish-attachment)

Media such as images are handled a little differently. Sometimes I’ll want to use a single image folder for cross-site purposes; often I’ll want to keep images in a folder with the page where they’re used. So I configure Org to recursively go through all the folders and to upload all the files it finds with an image extension (.jpg, .gif, .png, .svg) to a corresponding folder on the server, creating the folder if necessary. I do the same for PDFs, hand-crafted HTML docs, XML (for the RSS feed), and raw text documents. Here is the corresponding subproject of org-publish-project-alist:

("media" ; images, HTML, PDF,  XML (RSS), raw text
 :base-directory "~/www/cm/"
 :base-extension "jpg\\|gif\\|png\\|ico\\|svg\\|htm\\|html\\|pdf\\|xml\\|\\txt"
 :publishing-directory "/var/www/html"
 :publishing-function org-publish-attachment
 ;; Upload images and other media files to corresponding
 ;; folders throughout site regardless of sub-project
 :recursive t)

One of my key requirements is a multilingual site. There is more than one way to do this, but using the modularity of org-publish seems like a sensible approach. By giving each language section its own base directory (folder), I can easily configure it as a subproject. Then I can let Org know to use Norwegian localization where available, insert a postamble in the appropriate language, etc., simply by setting this line in the subproject:

:language "nb"

I also have Org generate a sitemap for each language section.

I’m mostly using SIL’s Gentium Plus font, which has excellent Unicode support for Latin, Cyrillic, and Greek script.

Gentium also looks nice without adjustments to the line height, which is a plus for inline math with MathJax.

I’d like to vary this a bit with a hand-writing font, but it’s hard to find a nice and readable one covering all of the above (see Notes: Hand-writing fonts for details).

When exporting to HTML or other formats, Org lets you create custom “filters” to modify the output. I use one to add my name at the start of each <title> element, and add a <page> division just inside the HTML <body> for layout purposes. Another takes care that root-relative references work. True to my perl roots, these mostly consist of throwing regular expressions at the output text until it behaves.

  (defun cm/filter-site-pages (output backend info)
    "When publishing my site, add my name before the title and a
  .page div inside the body."
    (when (and (eq backend 'html)
                 (string-match "/www/cm/" (buffer-file-name)))
        (setq output (replace-regexp-in-string
                      "<title>\\(.*?\\)<br>\\(.*?\\)</title>"
                      "<title>\\1</title>" output)
              output (replace-regexp-in-string
                      "<title>" "<title>C. Moe | " output)
              output (replace-regexp-in-string
                      "<body>" "<body><div id=\"page\">" output)
              output (replace-regexp-in-string
                      "</body>" "</div><!-- page ends--></body>" output)))
    output)

(defun cm/filter-root-links (output backend info)
  "Fix any root-relative references in links."
  (when (and (eq backend 'html)
               (string-match "/www/cm/" (buffer-file-name)))
    (setq testvar output)
    (replace-regexp-in-string "file:///" "/" output)))

(add-to-list 'org-export-filter-final-output-functions
               'cm/filter-site-pages)
(add-to-list 'org-export-filter-link-functions
               'cm/filter-root-links)

Note that while the first function is added to org-export-filter-final-output-functions and so acts on the whole final html output, the second is added to org-export-filter-link-functions and so only acts on links, so I can write about a file:///path without getting it messed up.

Org-mode is great at tagging entries, but doesn’t leverage those tags for publishing searchable content. Org publishing offers a basic sitemap and the possibility of creating a book-style index. Neither is based on tags. It would be fairly easy to add code to the publishing process to link every tag to a dedicated page listing every other page with that tag. However, tags really come into their own when you can combine them, so I’ve looked for a way to do that. Which raises some questions and choices.

I either need to run a script on my server, or turn to Javascript to run one on your computer instead. I’ve gone for the latter. Javascript-based solutions will not work equally for all readers, but jQuery helps.

Tags can be assigned to headings, and with #+FILETAGS they can be assigned to a whole Org file, but the exporter does not automatically turn them into HTML keywords; for that, #+KEYWORDS is used. HTML keywords are comma-separated, so they allow spaces, which Org-mode tags don’t. Keywords and tags could in principle both be used as two different vocabularies. I’ve opted to treat them as one vocabulary, to use #+KEYWORDS in my .org files, and (at this point) to use only keywords for indexing, and to use heading tags only as clickable links, not targets. In the spirit of Org tags, I’ll use single-word keywords as much as I can, but I have left myself the future option of using underscores in tags and transforming them to spaces.

I have written an elisp function (cm/collect-exported-tags) to extract metadata from a file. To ensure that it’s run every time a page is published from this project, I invoke it from the function that puts the page preamble together. I extract the title, date, description, and keywords, and store them in two associative arrays. One, cm/exported-files, holds all the extracted information for each file, indexed by the unique file path. The other, cm/exported-tags, is indexed by tags, and holds lists of paths to all the files so tagged. (I could have kept all of this in one array, but somehow I find this arrangement easier to think about.) When a file is exported, I save the state of these variables as a lisp file, .tagindex, from which I load them back into memory the next time I export a file.

When publishing is done, this data structure is automatically converted to JSON and saved as a .js file. To update the website, this .js file then needs to be exported. On your browser, it is automatically loaded with the web pages. Additional Javascript functions then work in your browser to add the keywords visibly to the page, transform all tag spans into links, and provide a lookup function for the links to activate when a tag is clicked. They also add little touches like links to previous/next blog posts.

Instead on relying on whatever a CMS does for version control, I can just run Git in my source folder. Git would also be one way of publishing to a remote server, assuming one is allowed to set up a Git repository there. Git would also allow collaborative editing, although it would tend to limit your collaborators to technical types who know their way around the command line.