Archives August 2018

Pseudo-Numeric Identifiers

Let’s say you’re a programmer, and your application uses Library of Congress Control Numbers for books, e.g., 2001012345, or ZIP codes, like 90210. What data types would you use to represent them? Or maybe something like the Dewey Decimal System, which uses 320 to classify a book as Political Science, 320.5 for Political Theory, and 320.973 for “Political institutions and public administration (United States)”?

If you said “integer”, “floating point”, or any kind of numeric type, then clearly you weren’t paying attention during the title.

The correct answer was “string” (or some kind of array of tokens), because although these entities consist of digits, they’re not numbers: they’re identifiers, same as “root” or “Jane Smith”. You can assign them, sort them, group them by common features, but you can’t meaningfully add or multiply them together. If you’re old enough, you may remember the TV series The Prisoner or Get Smart, where characters, most of them secret agents, refer to each other by their code numbers all the time; when agents 86 and 99 team up, they don’t become agent 185 all of a sudden.

If you keep in mind this distinction between numbers, which represent quantities, and strings that merely look like numbers because they happen to consist entirely of integers, you can save yourself a lot of grief. For instance, when your manager decides to store the phone number 18003569377 as “1-800-FLOWERS”, dashes and all. Or when you need to store a foreign phone number and have to put a plus sign in front of the country code.

Removing Magic

So this was one of those real-life mysteries.

I like crossword puzzles. And in particular, I like indie crossword puzzles, because they tend to be more inventive and less censored than ones that run in newspapers. So I follow several crossword designers on Twitter.

Yesterday, one of them mentioned that people were having a problem with his latest puzzle. I tried downloading it on my iPad, and yeah, it wouldn’t open in Across Lite. Other people were saying that their computers thought the file was in PostScript format. I dumped the HTTP header with

lynx -head -dump http://url.to/crossword.puz

and found the header

Content-type: application/postscript

which was definitely wrong for a .puz file. What’s more, other .puz files in the same directory were showing up as

Content-type: application/octet-stream

as they should.

I mentioned all this to the designer, which led to us chatting back and forth to see what the problem was. And eventually I had the proverbial aha moment.

.puz files begin with a two-byte checksum. In this particular case, they turned out to be 0x25 and 0x21. Or, in ASCII, “%!“. And as it turns out, PostScript files begin with “%!“, according to Unix’s magic file.

So evidently what happened was: the hosting server didn’t have a default type for files ending in .puz. Not terribly surprising, since that’s not really a widely-used format. So since it didn’t recognize the filename extension, it did the next-best thing and looked at the first few bytes of the file (probably with file or something equivalent) to see if it could make an educated guess. It saw the checksum as “%!” and decided it was a PostScript file.

The obvious fix was to change something about the file: rewrite a clue, add a note, change the copyright statement, anything to change the contents of the file, and thus the checksum.

The more permanent solution was to add a .htaccess file to the puzzle file directory, with

AddType application/octet-stream .puz

assuming that the hosting provider used Apache or something compatible.

This didn’t take immediately; I think the provider cached this metadata for a few hours. But eventually things cleared up.

I’m not sure what the lesson is, here. “Don’t use two-byte checksums at offset 0”, maybe?