Category Hacking

Literate Lists

I’ve written before about literate programming, and how one of its most attractive features is that you can write code with the primary goal of conveying information to a person, and only secondarily of telling a computer what to do. So there’s a bit in my .bashrc that adds directories to $PATH that isn’t as reader-friendly as I’d like:

for dir in \
    /usr/sbin \
    /opt/sbin \
    /usr/local/sbin \
    /some/very/specific/directory \
    ; do
    PATH="$dir:$PATH"
done

I’d like to be add a comment to each directory entry, explaining why I want it in $PATH, but sh syntax won’t let me: there’s just no way to interleave strings and comments this way. So far, I’ve documented these directories in a comment above the for loop, but that’s not exactly what I’d like to do. In fact, I’d like to do something like:

$PATH components

  • /usr/sbin
  • /usr/local/bin
for dir in \
    {{path-components}} \
    ; do
    PATH="$dir:$PATH"
done

Or even:

$PATH components

DirectoryComments
/usr/sbinsbin directories contain sysadminny stuff, and should go before bin directories.
/usr/local/binLocally-installed utilities take precedence over vendor-installed ones.
for dir in \
    {{path-components}} \
    ; do
    PATH="$dir:$PATH"
done

Spoiler alert: both are possible with org-mode.

Lists

The key is to use Library of Babel code blocks: these allow you to execute org-mode code blocks and use the results elsewhere. Let’s start by writing the code that we want to be able to write:

#+name: path-list
- /usr/bin
- /opt/bin
- /usr/local/bin
- /sbin
- /opt/sbin
- /usr/local/sbin

#+begin_src bash :noweb no-export :tangle list.sh
  for l in \
      <<org-list-to-sh(l=path-list)>> \
      ; do
      PATH="$l:$PATH"
  done
#+end_src

Note the :noweb argument to the bash code block, and the <<org-list-to-sh()>> call in noweb brackets. This is a function we need to write. It’ll (somehow) take an org list as input and convert it into a string that can be inserted in this fragment of bash code.

This function is a Babel code block that we will evaluate, and which will return a string. We can write it in any supported language we like, such as R or Python, but for the sake of simplicity and portability, let’s stick with Emacs lisp.

Next, we’ll want a test rig to actually write the org-list-to-sh function. Let’s start with:

#+name: org-list-to-sh
#+begin_src emacs-lisp :var l='nil
  l
#+end_src

#+name: test-list
- First
- Second
- Third

#+CALL: org-list-to-sh(l=test-list) :results value raw

The begin_src block at the top defines our function. For now, it simply takes one parameter, l, which defaults to nil, and returns l. Then there’s a list, to provide test data, and finally a #+CALL: line, which contains a call to org-list-to-sh and some header arguments, which we’ll get to in a moment.

If you press C-c C-c on the #+CALL line, Emacs will evaluate the call and write the result to a #+RESULTS block underneath. Go ahead and experiment with the Lisp code and any parameters you might be curious about.

The possible values for the :results header are listed under “Results of Evaluation” in the Org-Mode manual. There are a lot of them, but the one we care the most about is value: we’re going to execute code and take its return value, not its printed output. But this is the default, so it can be omitted.

If you tangle this file with C-c C-v C-t, you’ll see the following in list.sh:

for l in \
    ((/usr/bin) (/opt/bin) (/usr/local/bin) (/sbin) (/opt/sbin) (/usr/local/sbin)) \
    ; do
    PATH="$l:$PATH"
done

    It looks as though our org-mode list got turned into a Lisp list. As it turns out, yes, but not really. Let’s change the source of the org-list-to-sh() function to illustrate what’s going on:

    #+name: org-list-to-sh
#+begin_src emacs-lisp :var l='nil :results raw
  (format "aaa %s zzz" l)
#+end_src

    Now, when we tangle list.sh, it contains

        aaa ((/usr/bin) (/opt/bin) (/usr/local/bin) (/sbin) (/opt/sbin) (/usr/local/sbin)) zzz \

    So the return value from org-list-to-sh was turned into a string, and that string was inserted into the tangled file. This is because we chose :results raw in the definition of org-list-to-sh. If you play around with other values, you’ll see why they don’t work: vector wraps the result in extraneous parentheses, scalar adds extraneous quotation marks, and so on.

    Really, what we want is a plain string, generated from Lisp code and inserted in our sh code as-is. So we’ll need to change the org-list-to-sh code to return a string, and use :results raw to insert that string unchanged in the tangled file.

    We saw above that org-list-to-sh sees its parameter as a list of lists of strings, so let’s concatenate those strings, with space between them:

    #+name: org-list-to-sh
#+begin_src emacs-lisp :var l='nil :results raw
  (mapconcat 'identity
	     (mapcar
	      (lambda (elt)
		(car elt)
		)
	      l)
	     " ")
#+end_src

    This yields, in list.sh:

    for l in \
    /usr/bin /opt/bin /usr/local/bin /sbin /opt/sbin /usr/local/sbin \
    ; do
    PATH="$l:$PATH"
done

    which looks pretty nice. It would be nice to break that list of strings across multiple lines, and also quote them (in case there are directories with spaces in them), but I’ll leave that as an exercise for the reader.

    Tables

    That takes care of converting an org-mode list to a sh string. But earlier I said it would be even better to define the $PATH components in an org-mode table, with directories in the first column and comments in the second. This is easy, with what we’ve already done with strings. Let’s add a test table to our org-mode code, and some code to just return its input:

    #+name: echo-input
#+begin_src emacs-lisp :var l='nil :results raw
  l
#+end_src

#+name: test-table
| *Name*   | *Comment*        |
|----------+------------------|
| /bin     | First directory  |
| /sbin    | Second directory |
| /opt/bin | Third directory  |

#+CALL: echo-input(l=test-table) :results value code

#+RESULTS:

    Press C-c C-c on the #+CALL line to evaluate it, and you’ll see the results:

    #+RESULTS:
#+begin_src emacs-lisp
(("/bin" "First directory")
 ("/sbin" "Second directory")
 ("/opt/bin" "Third directory"))
#+end_src

    First of all, note that, just as with lists, the table is converted to a list of lists of strings, where the first string in each list is the name of the directory. So we can just reuse our existing org-list-to-sh code. Secondly, org has helpfully stripped the header line and the horizontal rule underneath it, giving us a clean set of data to work with (this seems a bit fragile, however, so in your own code, be sure to sanitize your inputs). Just convert the list of directories to a table of directories, and you’re done.

    Conclusion

    We’ve seen how to convert org-mode lists and tables to code that can be inserted into a sh (or other language) source file when it’s tangled. This means that when our code includes data best represented by a list or table, we can, in the spirit of literate programming, use org-mode formatting to present that data to the user as a good-looking list or table, rather than just list it as code.

    One final homework assignment: in the list or table that describes the path elements, it would be nice to use org-mode formatting for the directory name itself: =/bin= rather than /bin. Update org-list-to-sh to strip the formatting before converting to sh code.

    A Few More Thoughts on Literate Programming

    A while back, I became intrigued by Donald Knuth’s idea of Literate Programming, and decided to give it a shot. That first attempt was basically just me writing down what I knew as quickly as I learned it, and trying to pass it off as a knowledgeable tutorial. More recently, I tried a second project, a web-app that solves Wordle, and thought I’d write it in the Literate style as well.

    The first time around, I learned the mechanics. The second time, I was able to learn one or two things about the coding itself.

    (For those who don’t remember, in literate programming, you write code intertwined with prose that explains the code, and a post-processor turns the result into a pretty document for humans to read, and ugly code for computers to process.

    1) The thing I liked the most, the part where literate programming really shines, is having the code be grouped not by function or by class, but by topic. I could introduce a <div class="message-box"></div> in the main HTML file, and in the next paragraph introduce the CSS that styles it, and the JavaScript code that manipulates it.

    2) In the same vein, several times I rearranged the source to make the explanations flow better, not discuss variables or functions until I had explained why they’re there and what they do, without it altering the underlying HTML or JavaScript source. In fact, this led to a stylistic quandary:

    3) I defined a few customization variables. You know, the kind that normally go at the top for easy customization:

    var MIN_FOO = 30;
    var MAX_FOO = 1500;
    var LOG_FILE = "/var/log/mylogfile.log";

    Of course, the natural tendency was to put them next to the code that they affect, somewhere in the middle of the source file. Should I have put them at the top of my source instead?

    4) Even smaller: how do you pass command-line option definitions to getopt()? If you have options -a, -b, and -c, each will normally be defined in its own section. So in principle, the literate thing to do would be to write

    getopt("{{option-a}}{{option-b}}{{option-c}}");

    and have a section that defines option-a as “a“. As you can see, though, defining single-letter strings isn’t terribly readable, and literate programming is all about readability.

    5) Speaking of readability, one thing that can come in really handy is the ability to generate a pretty document for human consumption. Knuth’s original tools generated TeX, of course, and it doesn’t get prettier than that.

    I used org-mode, which accepts TeX style math notation, but also allows you to embed images and graphviz graphs. In my case, I needed to calculate the entropy of a variable, so being able to use proper equations, with nicely-formatted sigmas and italicized variables, was very nice. I’ve worked in the past on a number of projects where it would have been useful to embed a diagram with circles and arrows, rather than using words or ASCII art.

    6) I was surprised to find that I had practically no comments in the base code (in the JavaScript, HTML, and CSS that were generated from my org-mode source file). I normally comment a lot. It’s not that I was less verbose. In fact, I was more verbose than usual. It’s just that I was putting all of the explanations about what I was trying to do, and why things were the way they are, in the human-docs part of the source, not the parts destined for computer consumption. Which, I guess, was the point.

    7) Related to this, I think I had fewer bugs than I would normally have gotten in a project of this size. I don’t know why, but I suspect that it was due to some combination of thinking “out loud” (or at least in prose) before pounding out a chunk of code, and of having related bits of code next to each other, and not scattered across multiple files.

    8) I don’t know whether I could tackle a large project in this way. You might say, “Why not? Donald Knuth wrote both TeX and Metafont as literate code, and even published the source in two fat books!” Well, yeah, but he’s Donald Knuth. Also, he was writing before IDEs, or even color-coded code, were available.

    I found org-mode to be the most comfortable tool for me to use for this project. But of course that effectively prevents people who don’t use Emacs (even though they obviously should) from contributing.

    One drawback of org-mode as a literate programming development environment is that you’re pretty much limited to one source file, which obviously doesn’t scale. There are other tools out there, like noweb, but I found those harder to set up, or they forced me to use (La)TeX when I didn’t want to, or the like.

    9) One serious drawback of org-mode is that it makes it nearly impossible to add cross-reference links. If you have a section like

    function myFunc() {
        var thing;
        {{calculate thing}}
        return thing;
    }

    it would be very useful to have {{calculate thing}} be a link that you can click on to go to the definition of that chunk. But this is much harder to do in org-mode than it should be. So is labeling chunks, so that people can chase cross-references even without convenient links. It has a lot of work to be done in that regard.

    Readable Code: Variable Overload

    It’s well known that reading code is a lot harder than writing it. But I recently got some insight as to why that is.

    I was debugging someone else’s sh script. This one seemed harder to read than most. There was a section that involved figuring out a bunch of dates associated with a particular object. The author was careful to show their work, and not just have a bunch of opaque calculations in the code. I won’t quote their code, but imagine something like:

    NOW=$(date +%s)
THING_CREATION_EPOCH=$(<get $THING creation time, in Unix time format>)
THING_AGE_EPOCH=$(( $NOW - $THING_CREATION_EPOCH ))
THING_AGE_DAYS=$(( $THING_AGE_EPOCH / 86400 ))

    Now imagine this for three or four aspects of $THING, like last modification time, which other elements use $THING, things like that. The precise details don’t matter.

    Each variable makes sense. But there are four of them, just for the thing’s age since creation. And if you’re not as intimately familiar with it as someone who just wrote this code, that means you have to keep track of four variables in your head, and that gets very difficult very quickly.

    Part of the problem is that it’s unclear which variables will be needed further down (or even above, in functions that use global variables), so you have to hold on to these variables; you can’t mentally let them fall. Compare this to something like

    <Initialize some stuff>
for i in $LIST_OF_THINGS; do
    ProcessStuff $i
done
<Finalize some stuff>

    Here, you can be reasonably sure that $i won’t be used outside the loop. Once you get past done, you can let it go. Yes, it still exists, and it’s not illegal to use $i in the finalization code, but a well-meaning author won’t do that.

    Which leads to a lesson to be learned from this: limit the number of variables that are used in any given chunk of code. You don’t want to have to remember some variable five pages away. To do this, try to break your code down into independent modules. These can be functions, classes, even just paragraph-sized chunks in some function. Ideally, you want your reader to be able to close the chapter, forget most of the details, and move on to the next bit.

    In the same vein, this illustrates one reason global variables are frowned upon: they can potentially be accessed from anywhere in the code, which means that they never really disappear and can’t be completely ignored.

    Campaign Manager: Serializing Recursive Objects

    There’s an idea for a game that’s been rattling around my brain for a while, so I finally decided to learn enough Unity to birth it.

    To get an idea of what I have in mind, think Sid Meier’s Civilization, but for elections: you play a campaign manager, and your job is to get your candidate elected. This is complicated by the fact that the candidate may be prone to gaffes (think Joe Biden) or unmanageable (think Donald Trump); union leaders, state governors, CEOs, etc. can all promise their support, but they’ll all want something in return.

    The thing I learned this week is the Unity JSON Serializer. It’s used to store prefabs and other data structures in the editor, so I figured it would be the natural tool for storing canned scenarios, as well as for saving games. It has a few quirks, though: for one thing, it can only serialize basic types (int, string, and the like), and a few constructs like struct, and arrays and List<T> of basic types. If you’re used to the introspection of languages like Python and Perl, you’re likely to be disappointed in C#. For another, in order to avoid runaway recursion, it only serializes to a depth of 7.

    To start with, I defined class Area to represent an area such as a country, like the US. An Area is divided in various ways: politically, it’s divided into states. Culturally, it’s divided into regions like the Pacific Northwest or the Bible Belt. Other divisions are possible, depending on what you’re interested in. So I added a Division class:

    namespace Map {
    [Serializable]
    public class Area
    {
	public string id;
	public string name;
	public List<Division> divisions = new List<Division>();
    }

    public class Division
    {
	public string id;
	public string name;
	public List<Area> children;
    }
}

    As you can see, this is recursive: an Area can have multiple Divisions, each of which can contain other Areas with their own Divisions. This allows us to divide the United States into states, which are in turn divided into Congressional Districts, which in turn are divided into precincts.

    Since neither of our classes are elementary types, they can’t be serialized directly. So let’s add struct SerializableArea and struct SerializableDivision to hold the structures that will actually be stored on disk, as opposed to Area and Division which will be used in memory at run time, and use the ISerializationCallbackReceiver interface that will give our classes a hook called when the object is serialized or deserialized.

    Without wishing to repeat what’s in the docs: in order to get around the various limitations of the Serializer, the way to serialize a tree of objects is to serialize an array of objects, and use identifiers to refer to particular objects. Let’s say our in-memory Area tree looks something like:

    • Area United States
      • Division Political
        • Area Alabama
        • Area Alaska
        • …
      • Division Regions
        • Area Bible Belt
        • Area Pacific Northwest
        • …

    (That’s just an outline, of course. Each node has more members than are shown here.) We can serialize this as two arrays: one with all of the Areas, and one with all of the Divisions:

    • Area United States
      • List<SerializableArea> childAreas:
        • SerializableArea Alabama
        • SerializableArea Alaska
        • SerializableArea Bible Belt
        • SerializableArea Pacific Northwest
      • List<SerializableDivision> serialDivisions:
        • SerializableDivision Political
        • SerializableDivision Regions

    We don’t want to recurse, but we do want to be able to rebuild the tree structure when we deserialize the above. So SerializableArea contains, not a list of Divisions, but a list of identifiers that we can find in serialDivisions. Likewise, SerialDivision contains not a list of Areas, but a list of identifiers that we can look up in childAreas.

    Naturally, Area and Division each contain a Serialize() method that recursively serializes it and its children.

    The next question is: so you’ve been asked to serialize an Area. How do you know whether you’re supposed to add it to an existing childAreas list or start your own?

    Answer: if the serialization was invoked through OnBeforeSerialize(), then you’re serializing the top level object and should allocate a list to put the children in. Otherwise, append to an existing list, which should be passed in as a parameter to Serialize().

    If anyone’s interested in what it looks like when all is said and done, here it is:

    namespace Map {

    [Serializable]
    public class Area : ISerializationCallbackReceiver
    {
	public string id;
	public string name;

	public List<Division> divisions = new List<Division>();

	[Serializable]
	public struct SerializableArea
	{
	    public string id;
	    public string name;
	    public List<string> divisions;
	}

	public List<Division.SerializableDivision> serialDivisions;
	public List<SerializableArea> childAreas = new List<SerializableArea>();

	public void OnBeforeSerialize()
	{
	    serialDivisions =
		new List<Division.SerializableDivision>(divisions.Count);
	    childAreas = new List<SerializableArea>();

	    for (int i = 0; i < divisions.Count; i++)
		divisions[i].Serialize(ref serialDivisions, ref childAreas);
	}

	// Look up a Division by name, so we can avoid adding it twice
	// to serialDivisions.
	private int FindDivisionById(string id,
				     ref List<Division.SerializableDivision>dlist)
	{
	    for (int i = 0; i < dlist.Count; i++)
		if (dlist[i].id == id)
		    return i;
	    return -1;
	}

	public void Serialize(ref List<SerializableArea> alist,
			      ref List<Division.SerializableDivision> dlist)
	{
	    SerializableArea sa = new SerializableArea();
	    sa.id = id;
	    sa.name = name;
	    sa.divisions = new List<string>(divisions.Count);

	    alist.Add(sa);

	    for (int i = 0; i < divisions.Count; i++)
	    {
		sa.divisions.Add(divisions[i].name);

		int d = FindDivisionById(divisions[i].id, ref dlist);
		if (d < 0)
		    // Don't add a Division to dlist twice.
		    divisions[i].Serialize(ref dlist, ref alist);
	    }
	}
    }

    public class Division : ISerializationCallbackReceiver
    {
	public string id;
	public string name;
	public List<Area> children;

	[Serializable]
	public struct SerializableDivision
	{
	    public string id;
	    public string name;
	    public List<string> areas;
	}

	public void Serialize(ref List<SerializableDivision> dlist,
			      ref List<Area.SerializableArea> alist)
	{
	    SerializableDivision sd = new SerializableDivision();
	    sd.id = id;
	    sd.name = name;
	    sd.areas = new List<string>(children.Count);

	    dlist.Add(sd);

	    for (int i = 0; i < children.Count; i++)
	    {
		sd.areas.Add(children[i].name);

		int a = FindAreaById(children[i].id, ref alist);
		if (a < 0)
		    // Don't add an Area to alist twice.
		    children[i].Serialize(ref alist, ref dlist);
	    }
	}

	private int FindAreaById(string id,
				 ref List<Area.SerializableArea> alist)
	{
	    for (int i = 0; i < alist.Count; i++)
		if (alist[i].id == id)
		    return i;
	    return -1;
	}
    }
}
    Ansible As Scripting Language

    Ansible is billed as a configuration manager similar to Puppet or cfengine. But it occurred to me recently that it’s really (at least) two things:

    1. A configuration manager.
    2. A scripting language for the machine room.

    Mode 1 is the normal, expected one: here’s a description; now make the machine(s) look like the description. Same as Puppet.

    Mode 2 is, I think, far more difficult to achieve in Puppet than it is in Ansible. This is where you make things happen in a particular order, not just on one machine (you’d use /bin/sh for that), but on multiple hosts.

    For instance, adding a new user might involve:

    1. Generate a random password on localhost.
    2. Add a user on the Active Directory server.
    3. Create and populate a home directory on the home directory server.
    4. Add a stub web page on the web server.

    This is something I’d rather write as an ansible play, than as a Puppet manifest or module.

    Which brings me to my next point: it seems that for Mode 1, you want to think mainly in terms of roles, while for Mode 2, you’ll want to focus on playbooks. A role is designed to encapsulate the notion of “here’s a description of what a machine should look like, and the steps to take, if any, to make it match that description”, while a playbook is naturally organized as “step 1; step 2; step 3; …”.

    These are, of course, just guidelines. And Mode 2 suffers from the fact that YAML is not a good way to express programming concepts.  But I find this to be a useful way of thinking about what I’m doing in Ansible.

    Pseudo-Numeric Identifiers

    Let’s say you’re a programmer, and your application uses Library of Congress Control Numbers for books, e.g., 2001012345, or ZIP codes, like 90210. What data types would you use to represent them? Or maybe something like the Dewey Decimal System, which uses 320 to classify a book as Political Science, 320.5 for Political Theory, and 320.973 for “Political institutions and public administration (United States)”?

    If you said “integer”, “floating point”, or any kind of numeric type, then clearly you weren’t paying attention during the title.

    The correct answer was “string” (or some kind of array of tokens), because although these entities consist of digits, they’re not numbers: they’re identifiers, same as “root” or “Jane Smith”. You can assign them, sort them, group them by common features, but you can’t meaningfully add or multiply them together. If you’re old enough, you may remember the TV series The Prisoner or Get Smart, where characters, most of them secret agents, refer to each other by their code numbers all the time; when agents 86 and 99 team up, they don’t become agent 185 all of a sudden.

    If you keep in mind this distinction between numbers, which represent quantities, and strings that merely look like numbers because they happen to consist entirely of integers, you can save yourself a lot of grief. For instance, when your manager decides to store the phone number 18003569377 as “1-800-FLOWERS”, dashes and all. Or when you need to store a foreign phone number and have to put a plus sign in front of the country code.

    Programming Tip: Open and Close at the Same Time

    One useful programming habit I picked up at some point is: if you open or start something, immediately close it or end it. If you open a bracket, immediately write its closing bracket. If you open a file, immediately write the code to close it.

    These days, development environments take care of the niggling little details like matching parentheses and brackets for you. That’s great, but that’s just syntax. The same principle extends further, and automatic tools can’t guess what it is you want to do.

    There’s a problem in a lot of code called a resource leak. The classic example is memory leaks in C: the code asks for, and gets, a chunk of memory. But if you don’t free the memory when you’re done with it, then your program will get larger and larger — like a coffee table where a new magazine is added every month but none are ever taken away — until eventually the machine runs out of memory.

    These days, languages keep track of memory for you, so it’s easier to avoid memory leaks than it used to be. But the best way I’ve found to manage them is: when you allocate memory (or some other resource), plan to release it when you’re done.

    The same principle applies to any resource: if you read or write a file, you’ll need a file handle. If you never close them, they’ll keep lying around, and you’ll eventually run out. So plan ahead, and free the resource as soon as you’ve alocated it:

    Once you’ve written

    open INFILE, "<", "/path/to/myfile";

    go ahead and immediately write the code to close that file:

    open INFILE, "<", "/path/to/myfile";
close INFILE;

    and only then write the code to do stuff with the file:

    open INFILE, "<", "/path/to/myfile";
while ()
{
	print "hello\n" if /foo/;
}
close INFILE;

    The corollary of this is, if you’ve written the open but aren’t sure where to put the close, then you may want to take a look at the structure of your code, and refactor it.

    This same principle applies in many situations: when you open a connection to a remote web server, database server, etc., immediately write the code to close the connection. If you’re writing HTML, and you’ve written <foo>, immediately write the corresponding </foo>. If you’ve sent off an asynchronous AJAX request, figure out where you’re going to receive the reply. When you throw an exception, decide where you’re going to catch it.

    And only then write the meat of the code, the stuff that goes between the opening and closing code.

    As I said, I originally came across this as a tip for avoiding memory leaks. But I’ve found that doing things this way forces me to be mindful of the structure of my code, and avoid costly surprises down the line.

    If You Use Unix, Use Version Control

    If you’ve used Unix (or Linux. This applies to Linux, and MacOS X, and probably various flavors of Windows as well), you’ve no doubt found yourself editing configuration files with a text editor. This is especially true if you’ve been administering a machine, either professionally or because you got roped into doing it.

    And if you’ve been doing it for more than a day or two, you’ve made a mistake, and wished you could undo it, or at least see what things looked like before you started messing with them.

    This is something that programmers have been dealing with for a long time, so they’ve developed an impressive array of tools to allow you to keep track of how a file has changed over time. Most of them have too much overhead for someone who doesn’t do this stuff full-time. But I’m going to talk about RCS, which is simple enough that you can just start using it.

    Most programmers will tell you that RCS has severe limitations, like only being able to work on one file at a time, rather than a collection of files, that makes it unsuitable for use in all but a few special circumstances. Thankfully, Unix system administration happens to be one of those circumstances!

    What’s version control?

    Basically, it allows you to track changes to a file, over time. You check a file in, meaning that you want to keep track of its changes. Periodically, you check it in again, which is a bit like bookmarking a particular version so you can come back to it later. And you can check a file out, that is, retrieve it from the history archive. Or you can compare the file as it is now to how it looked one, five, or a hundred versions ago.

    Note that RCS doesn’t record any changes unless you tell it to. That means that you should get into the habit of checking in your changes when you’re done messing with a file.

    Starting out

    Let’s create a file:

    # echo "first" > myfile

    Now let’s check it in, to tell RCS that we want to track it:

    # ci -u myfile
    myfile,v <-- myfile
    enter description, terminated with single '.' or end of file:
    NOTE: This is NOT the log message!
    >> This is a test file
    >> .
    initial revision: 1.1
    done

    ci stands for “check in”, and is RCS’s tool for checking files in. The -u option says to unlock it after checking in.

    Locking is a feature of RCS that helps prevent two people from stepping on each other’s toes by editing a file at the same time. We’ll talk more about this later.

    Note that I typed in This is a test file. I could have given a description on multiple lines if I wanted to, but usually you want to keep this short: “DNS config file” or “Login message”, or something similar.

    End the description with a single dot on a line by itself.

    You’ll notice that you now have a file called myfile,v. That’s the history file for myfile.

    Since you probably don’t want ,v files lying around cluttering the place up, know that if there’s a directory called RCS, the RCS utilities will look for ,v history files in that directory. So before we get in too deep, let’s create an RCS directory:

    # mkdir RCS

    Now delete myfile and start from scratch, above.

    Done? Good. By the way, you could also have cheated and just moved the ,v file into the RCS directory. Now you know for next time.

    Making a change

    All right, so now you want to make a change to your file. This happens in three steps:

    1. Check out the file and lock it.
    2. Make the change(s)
    3. Check in your changes and unlock the file.

    Check out the file:

    # co -l myfile
    RCS/myfile,v --> myfile
    revision 1.1 (locked)
    done

    co is RCS’s check-out utility. In this case, it pulls the latest version out of the history archive, if it’s not there already.

    The -l (lower-case ell) flag says to lock the file. This helps to prevent other people from working on the file at the same time as you. It’s still possible for other people to step on your toes, especially if they’re working as root and can overwrite anything, but it makes it a little harder. Just remember that co is almost always followed by -l.

    Now let’s change the file. Edit it with your favorite editor and replace the word “first” with the word “second”.

    If you want to see what has changed between the last version in history and the current version of the file, use rcsdiff:

    # rcsdiff -u myfile
    ===================================================================
    RCS file: RCS/myfile,v
    retrieving revision 1.1
    diff -u -r1.1 myfile
    --- myfile 2016/06/07 20:18:12 1.1
    +++ myfile 2016/06/07 20:32:38
    @@ -1 +1 @@
    -first
    +second

    The -u option makes it print the difference in “unified-diff” format, which I find more readable than the other possibilities. Read the man page for more options.

    In unified-diff format, lines that were deleted are preceded with a minus sign, and lines that were added are preceded by a plus sign. So the above is saying that the line “first” was removed, and “second” was added.

    Finally, let’s check in your change:

    # ci -u myfile
    RCS/myfile,v <-- myfile
    new revision: 1.2; previous revision: 1.1
    enter log message, terminated with single '.' or end of file:
    >> Updated to second version.
    >> .
    done

    Again, we were prompted to list the changes we made to the file (with a dot on a line by itself to mark the end of our text). You’ll want to be concise yet descriptive in this text, because these are notes you’re making for your future self when you want to go back and find out when and why a change was made.

    Viewing a file’s history

    Use the rlog command to see a file’s history:

    # rlog myfile

    RCS file: RCS/myfile,v
    Working file: myfile
    head: 1.2
    branch:
    locks: strict
    access list:
    symbolic names:
    keyword substitution: kv
    total revisions: 2; selected revisions: 2
    description:
    Test file.
    ----------------------------
    revision 1.2
    date: 2016/06/07 20:36:52; author: arensb; state: Exp; lines: +1 -1
    Made a change.
    ----------------------------
    revision 1.1
    date: 2016/06/07 20:18:12; author: arensb; state: Exp;
    Initial revision
    =============================================================================

    In this case, there are two revisions: 1.1, with the log message “Initial revision”, and 1.2, with the log “Made a change.”.

    Undoing a change

    You’ve already see rlog, which shows you a file’s history. And you’ve seen one way to use rcsdiff.

    You can also use either one or two -rrevision-number arguments, to see the difference between specific revisions:

    # rcsdiff -u -r1.1 myfile

    will show you the difference between revision 1.1 and what’s in the file right now, and

    # rcsdiff -u -r1.1 -r1.2 myfile

    will show you the difference between revisions 1.1 and 1.2.

    (Yes, RCS will just increment the second number in the revision number, so after a while you’ll be editing revision 1.2486 of the file. Getting to revision 2.0 is an advanced topic that we won’t cover here.)

    With the tools you already have, the simplest way to revert an unintended change to a file is simply to see what the file used to look like, and copy-paste that into a new revision.

    Once you’re comfortable with that, you can read the manual and read up on things like deleting revisions with rcs -o1.2 myfile.

    Checking in someone else’s changes

    You will inevitably run into cases where someone changes your file without going through RCS. Either it’ll be a coworker managing the same system who didn’t notice the ,v file lying around, or else you’ll forget to check in your changes after making changes.

    Here’s a simple way to see whether someone (possibly you) has made changes without your knowledge:

    # co -l myfile
    RCS/myfile,v --> myfile
    revision 1.2 (locked)
    writable myfile exists; remove it? [ny](n):

    In this case, either you forgot to check in your changes, or else someone made the file writable with chmod, then (possibly) edited it.

    In the former case, see what you did with rcsdiff, check in your changes, then check the file out again to do what you were going to do.

    The latter case requires a bit more work, because you don’t want to lose your coworker’s changes, even though they bypassed version control.

    1. Make a copy of the file
    2. Check out the latest version of the file.
    3. Overwrite that file with your coworker’s version.
    4. Check those changes in.
    5. Check the file out and make your changes.
    6. Have a talk with your coworker about the benefits of using version control..

    You already know, from the above, how to do all of this. But just to recap:

    Move the file aside:

    # mv myfile myfile.new

    Check out the latest version:

    # co -l myfile

    Overwrite it with your coworker’s changes:

    # mv myfile.new myfile

    Check in those changes:

    # ci -u myfile
    RCS/myfile,v <-- myfile
    new revision: 1.3; previous revision: 1.2
    enter log message, terminated with single '.' or end of file:
    >> Checking in Bob's changes:.
    >> Route around Internet damage.
    >> .
    done

    That should be enough to get you started. Play around with this, and I’m sure you’ll find that this is a huge improvement over what you’ve probably been using so far: a not-system of making innumerable copies of files when you remember to, with names like “file.bob”, “file.new”, “file.new2”, “file.newer”, and other names that mean nothing to you a week later.

    There Are Days When I Hate XML

    …and days when I really hate XML.

    In this case, I have an XML document like

    <?xml version="1.0" ?>
<foo xmlns="http://some.org/foo">
  <thing id="first"/>
  <thing id="second"/>
  <thing id="third"/>
</foo>

    and I want to get things out of it with XPath.

    But I couldn’t manage to select things exactly, such as the <foo> element at the top. You’d think “/foo” would do it, but it didn’t.

    Eventually I found out that the problem is the xmlns=”...” attribute. It looks perfectly normal, saying that if you have <thing> without a prefix (like “<ns:thing>”), then it’s in the “http://some.org/foo” namespace.

    However, in XPath, if you specify “ns:thing”, it means “a thing element in whichever namespace the ns prefix corresponds to”. BUT “thing” means “a thing element that’s not in a namespace”.

    So how do you specify an element that’s the empty-string namespace, as above? The obvious way would be to select “:thing”, but that doesn’t work. Too simple, I suppose. Maybe that gets confused with CSS pseudo-selectors or something.

    No, apparently the thing you need to do is to invent a prefix for the standard elements of the file you’re parsing. That is, add “ns” as another prefix that maps onto “http://some.org/foo” and then select “ns:thing”. There are different ways of doing this, depending which library you’re using to make XPath queries, but still, it seems like a giant pain in the ass.

    DD-WRT Config Backups

    So the other day I managed to hose my DD-WRT configuration at home, badly enough that I figured I really ought to back up my config so I don’t wind up trying to reconstruct my config from memory.

    (If you just want to see the script, you can jump ahead.)

    I have nightly backups of my desktop (and so do you, right? Right?!) so I figured the simplest thing is just to copy the router’s config to my desktop, and let it be swept up with the nightly backups. So then it’s just a question of getting the configuration and such to my desktop.

    In DD-WRT, under “Administration → Backup”, you can do manual backups and restores. This is what we want, except that we want this to happen automatically.

    The trivial way to get the config is

    curl -O -u admin:$PASSWORD http://$ROUTER/nvrambak.bin

    or

    wget --http-user=admin --http-password=$PASSWORD http://$ROUTER/nvrambak.bin

    where $ROUTER is your router’s name or IP address, and $PASSWORD is your admin password. But this sends your password over the network in the clear, so it’s not at all secure.

    Instead, let’s invest some time into setting up ssh on the router; specifically, the public key method that’ll allow passwordless login.

    Go follow the instructions there. When you come back, you should have working ssh, and a key file that we’ll use for backups.

    Back? Good. Now you can log in to the router with

    ssh root@$ROUTER

    and entering your password. Or you can log in without a password with

    ssh -i /path/to/key-file root@$ROUTER

    Once on the router, you can save a copy of the configuration with nvram backup /tmp/nvrambak.bin. So let’s combine those last two parts with:

    ssh -i /path/to/key-file root@$ROUTER "nvram backup /tmp/nvrambak.bin"

    And finally, let’s copy that file from the router to the desktop:

    scp -i /path/to/key-file root@$ROUTER:/tmp/nvrambak.bin /path/to/local/nvrambak.bin

    So the simple script becomes:

    #!/bin/sh
SSH=/usr/bin/ssh
SCP=/usr/bin/scp
RSA_ID=/path/to/keyfile_rsa
ROUTER=router name or IP address
LOCALFILE=/path/to/local/nvrambak.bin

$SSH -4 -q ${RSA_ID} root@${ROUTER} "nvram backup /tmp/nbrambak.bin"
$SCP -4 -q ${RSA_ID} root@${ROUTER}:/tmp/nbrambak.bin ${LOCALFILE}

    I’ve put this in the “run this before starting a backup” section of my backup utility. But you can also make it a daily cron job. Either way, you should wind up with backups of your router’s config.