sunday, may 13

a mother's day lawn and garden report

coming over the hill on 36 where
you catch that first view of boulder,
clouds move over and in the gray and green of
the bowl of the valley, against the mountains
sweeping away to the north and west
and in seconds thick drops hit the glass
turning to heavy rain by the edge of town

windshield wipers all the way up 28th and onto
the road north, low rumbling as i park and lug my
bag into the house, half-deranged from the day's
driving and a dozen of the sadnesses that
middle adulthood scores over and over again
into surfaces like these

the cat and i are watching the water come down
out the screendoor when the thunder picks up
i run outside and yank a tarp off the woodpile
in back to throw across the garden
just as the hail really gets going
the flowers from the apple tree falling fast
in the rain and ice, my shirt soaking

the tarp is probably futile, but i have memories
of more than one vegetable crop shredded by a
spring storm like this one
and i'm not sure what else i can do

which is both a metaphor and not.

Tuesday, May 1

wtfm

I’m thinking, far from the first time, about how few open source projects meet a certain standard of practical openness:

Can a user unfamiliar with the project start with the published source code and the included documentation, and wind up with a working installation?

The answer is “no way” a lot more often than it should be.

I’m all too aware that the full context it takes to build a lot of software is a pretty hard thing to explain in a README. All the same, if you’re working on a project of any size, maybe you ought to ask yourself some of the following:

  • Have I documented, in full, an up-to-date list of every environmental condition that a user will have to manually obtain in order to build, install, or run this code?

  • Is it clear what environments this code is developed in, and where it’s known to run without issue? Am I being brutally honest about this part?

  • Is it clear what version of everything I used?

  • When was the last time I tested the installation procedures in my documentation in a clean environment? Has it been since the last time I changed dependencies or configuration requirements, no matter how apparently trivial?

  • To what extent am I relying on implicit details of some OS, distribution, virtual machine, container, configuration management tool, package, init system, etc., without communicating those details to the user?

  • If a user is unfamiliar with details of a language, package manager, build system, or other tooling, are they pretty much just SOL? If so, is my software of interest to anyone outside of my specific technical community, narrowly defined? Is there any potential that it will need to be supported, for example, by admins or ops people who don’t share my context, for use by some less technical audience?

  • If I personally returned to developing the software after leaving it untouched for two years, how many problems would I be required to solve from a combination of my own patchy memory, search engine queries, and painstaking software architecture?

  • Are there any required environment variables, config file values, or command-line parameters that are for some reason undocumented?

  • Am I posturing like a link to some automatically extracted API documentation is a useful substitute for a real user manual?

  • Did I just publish a command-line utility with a crappy builtin help system instead of a real man page?

I could go on like this for a while, but I suppose the point is clear enough. If it sounds like I’m advocating a stricter standard than I usually manage to live up to myself, well, that’s probably fair. Still, I think we could all probably do better.

Thursday, April 19

a git log with both tags and dates

Trying to write a human-readable changelog for wrt, I wanted to see git history that included:

  • commit hash
  • a datestamp
  • any tags pointing at a commit
  • subject line

Since tags are used to designate released versions, this should at least give me a rough idea of what changed between releases.

This was not as obvious as I wanted it to be. You can see the tags in gitk’s view of history, but I didn’t know how to extract it from git log. The usual searching led to:

$ git log --oneline --decorate
99a584f (HEAD -> master, tag: v5.0.0-alpha, origin/master, origin/HEAD) set version string to v5.0.0
e919a56 some comment tweaking; latest readme
7b543b5 address a bunch of hotspots from Devel::NYTProf, removing MethodSpit
3fa80ca link_bar(): retain actual <a> tag for current page in linkbar
cf3dd64 remove failing recent_month test
b5fe642 kill an extraneous space char in Markup.pm
c1e17a9 sync README.pod with latest from WRT.pm
7d80dd5 kill recent_month(), month_before(), and feed_print_latest()
143613f get_date_entries_by_depth()
4f9a71b (tag: v4.3.0) switch example config from p1k3.com to example.com urls

Which is pretty close - you can see the tags inline with the rest of the history, but no date. (I’m not sure if I ever knew about --decorate; almost feels like it should be the default.)

I spent some time with git help log and eventually landed on:

$ git log --date=short --pretty='format:%h %ad %d %s'
99a584f 2018-04-19  (HEAD -> master, tag: v5.0.0-alpha, origin/master, origin/HEAD) set version string to v5.0.0
e919a56 2018-04-19  some comment tweaking; latest readme
7b543b5 2018-04-11  address a bunch of hotspots from Devel::NYTProf, removing MethodSpit
3fa80ca 2018-04-08  link_bar(): retain actual <a> tag for current page in linkbar
cf3dd64 2018-04-08  remove failing recent_month test
b5fe642 2018-04-08  kill an extraneous space char in Markup.pm
c1e17a9 2018-04-08  sync README.pod with latest from WRT.pm
7d80dd5 2018-04-08  kill recent_month(), month_before(), and feed_print_latest()
143613f 2018-04-08  get_date_entries_by_depth()
4f9a71b 2018-04-06  (tag: v4.3.0) switch example config from p1k3.com to example.com urls
a5e5f21 2018-04-06  render feed for last n day entries instead of for current month
77d90d2 2018-02-10  (tag: v4.2.2) fix --config option to wrt-display & wrt-render-all

This is better. In the format:

  • %h is hash
  • %ad is author date (shorted by the --date=short option)
  • %d is any ref pointing at that commit, which will include tags (it’s %d by analogy to the --decorate option)
  • %s is subject

Not super obvious, but it could be worse.

The other thing this did for me is re-cement the notion of what references are. If you look in a repo’s .git/refs/ directory, you’ll see something laid out roughly like the following:

$ tree .git/refs
.git/refs
├── heads
│   ├── jsonfeed
│   ├── master
│   └── subcommands
├── remotes
│   ├── origin
│   │   ├── HEAD
│   │   ├── master
│   │   └── subcommands
│   └── p1k3_server
│       ├── master
│       ├── tag_list
│       └── topiclinks
└── tags
    ├── v4.2.0
    ├── v4.2.2
    ├── v4.3.0
    └── v5.0.0-alpha

If you look at the contents of any of these files, you’ll see that they just contain some hashes which point to objects.

$ cat .git/refs/heads/master
99a584fb080e1f295d9522b684ab05f80f60918e

$ cat .git/refs/remotes/origin/master
99a584fb080e1f295d9522b684ab05f80f60918e

$ cat .git/refs/tags/v4.3.0
0ad7dbe5249aff2eab05e8c7f6d8d73c41fc7323

Because git actually has two kinds of tags, you might have some tags that don’t show up in that directory tree. Lightweight tags are, apparently, just commit objects. I’m not totally sure how this works, but you can get a better idea of it with:

$ git tag -l --format="%(objectname:short) %(refname)%09%(objecttype)"
2c789d3 refs/tags/2007-03-10    commit
a1f5c23 refs/tags/2007-06-20    commit
88a445e refs/tags/2007-06-21    commit
17beaef refs/tags/2007-07-07    commit
dac8154 refs/tags/2007-08-08    commit
221d1a3 refs/tags/2007-09-11    commit
54016b2 refs/tags/2007-10-04    commit
5437005 refs/tags/2008-01-11    commit
3c76277 refs/tags/2008-01-28    commit
20551c0 refs/tags/2008-03-15    commit
4bd68fe refs/tags/v4.2.0        tag
84948fa refs/tags/v4.2.2        tag
0ad7dbe refs/tags/v4.3.0        tag
199f704 refs/tags/v5.0.0-alpha  tag

The ones with an object type of “commit” are lightweight tags, the others are annotated.

Useful docs:

I should probably invest more time in understanding the contents of a .git.

Sunday, April 15

watching: the west wing

(Spoilers follow.)

I didn’t watch a whole lot of TV during the years from 2002 to about 2014. I mean, not no TV, just not most of it. Here and there I lived with roommates who had cable, some kind of streaming subscription, or the ambition to pirate stuff. I eventually saw most of The Office (US), Arrested Development, and a couple seasons of Community. A bunch of Daily Show episodes, towards the end of the Bush years. A season or three of Archer. I know there was other stuff, soaked up during family get-togethers and hotel-room stays, but those shows are everything that comes to mind: Not exactly a comprehensive survey of the form.

And then I bought a lightly-used Roku off a friend and got a Netflix streaming subscription, then subsequently moved in with a girlfriend who owns a bunch of DVDs. We’ve watched what feels like a lot of TV shows since then, but aside from Parks & Rec (which I love without reservation), I’m not much more caught up than I was before. Mostly it turns out I’m just more in tune with how my adult self feels about stuff I originally watched in the 1980s and 90s, because we’ve slowly been working our way through most of Star Trek, The X-Files, Babylon 5, and Cheers.

I probably don’t have a whole lot of utility as a TV critic.

The West Wing is a recent addition to the rotation, probably because Netflix put it on the screen enough times and I clicked. It’s a show I watched during at least its first couple of seasons, enough to pick up on the characters and remember a bunch of plot points, but beyond that memory is hazy. I remember my (conservative Republican) parents being into it.

We’re a couple of seasons in now, more or less. This can be a weird and kind of depressing show to experience in the deepening gloom of 2018, in my late 30s, after 15 years or so of watching the American state do the things it does. It was probably a weird thing to watch in its own time too, with Bush in office for most of its run and its whole premise an ever-more-obvious counterfactual.

Some of the plot material is overtly bad. There’s this whole thing where Sam Seaborn sleeps with an escort he doesn’t know is an escort in like the first episode and it keeps coming back up, and though in the end it’s not as badly handled as it could be and I’d guess at the time “sex workers are just people and your judginess is bullshit” was probably intended as a brave moral stance for late turn-of-the-century network TV, it’s also full of moments where I almost decided to stop watching the show as a whole.

The dialog is hooky and clever, but also almost a parody of itself from the very beginning (like the endless walk-and-talk sequences that the show pauses to make fun of itself for before the first season is up) and occasionally veers into these swelling-music-and-earnest-sentiment moments that don’t really earn their implied significance. Some of the stuff I remembered as a really great fuck you to the Bad Guys of the time now reads as a little bit cheap and pandering. (This one sequence, for example, where the President humiliates an obvious stand-in for Laura Schlessinger in what is apparently actually a lecture lifted from an e-mail forward. Not that Dr. Laura herself wasn’t a frankly evil force in the culture just then.)

It’s also a narrative that can’t quite decide whether politics are inherently fucked and no progress is possible, or whether deep down we’re all on the same team here and we can rise above our etc. The left-ish but also kind of pragmatically neoliberal Democrats who make up the Good Guys are constantly shooting themselves in the foot by compromising some belief to perceived political necessity, which is frustrating to watch, especially when the show goes out of its way to demonstrate that they’d do better if they just did what they knew was right, but then reverts to the same pattern an episode later. Things have a tendency to get kind of patronizing and American-exceptionalist whenever a foreign policy plot comes up.

There’s a lot of other stuff I’m not sure about. I mean, I’m not especially sure how I feel about the entire project of American government at this stage of history, which complicates how I feel about these kinds of stories. But whatever. All that said, it’s a good show. The cast are amazing. The writing is frequently cornball and sometimes juvenile but also really entertaining. It’s often hopeful in a way that I have mostly lost track of how to be. It makes you want to work with good people on something important, and that feeling is most of the reason that I’m such a sucker for workplace drama stuff like this.

Monday, April 9

App::WRT v4.3.0: schwartzian transforms, long-term projects

I should have been doing other things, but I spent a couple of hours over the weekend making wrt, the static site generator I use for p1k3.com, a bit more capable.

I decided I wanted to make the feed wrt generates (like this one) contain the most recent n entries instead of just the entries for the most recent month. For example, instead of just rendering a feed with the entries for this April, I wanted it to contain the last 30 days for which I’d written something.

If wrt entries lived in, say, an SQL database of some sort, this would be just a matter of changing a query to get some different ones. Since they’re just flatfiles in a directory tree without a lot of abstractions around them, it was a bit trickier but also more interesting.

Simplified a lot, the wrt repository for this site looks something like this:

/home/brennen/p1k3/
▾ archives/
  ▾ 2018/
    ▸ 1/
    ▸ 2/
    ▸ 3/
    ▾ 4/
      ▸ 5/
      ▸ 8/
      ▸ 9/

The basic idea is that a file 3 deep in the hierarchy of numerical entries—like 2018/4/9 for this entry—represents a day, inside a month, inside a year. If I wanted to put the last 30 entries into the feed, I’d need to flatten this structure out into a sorted list.

I remembered that I was already getting a list of all the entries for the wrt render-all script that renders the whole site at once, so it seemed simple enough to reuse that list, but there was a catch: Doing a simple reversed sort on that list gave me results like these:

...
2016/10/16
2016/10/14
2016/10/12
2016/10
2016/1/5
2016/1/3
2016/1/28
...

…because in a string comparison, 2018/10 follows 2018/1, not 2018/9.

If I’d decided to pad the months with 0s, like 2018/01, a while back, this would have been less of a problem, but it seemed pretty solvable. I just needed to convert the entry paths to a different format and sort by that.

I wound up reading the Wikipedia entry for the Schwartzian transform, and writing something like the following:

sub get_date_entries_by_depth {
  my $self = shift;
  my ($depth) = @_;

  # Match given $depth:
  my @particles;
  for (my $i = 0; $i < $depth; $i++) {
    push @particles, '\d+';
  }
  my $pattern = join '/', @particles;

  # Sort matching entries by sortable_date_from_entry()
  return map  { $_->[0] }
         sort { $a->[1] cmp $b->[1] }
         map  { [$_, sortable_date_from_entry($_)] }
         grep m{^ $pattern $}x, $self->get_all_source_files();
}

sub sortable_date_from_entry {
  my ($entry) = @_;
  my @parts = map { sprintf("%4d", $_) } split '/', $entry;
  return join '', @parts;
}

First, this builds a regular expression to match entries that are at a certain depth in the hierarchy (1 is a year, 2 is a month, 3 is a day).

Then it:

  1. greps the list returned by get_all_source_files() for entries matching the pattern.
  2. maps the matching entries to a list of two-element arrays where the 0th element is the original path to the entry (2018/4/9), and the 1st element is a format returned by sortable_date_from_entry() that will sort correctly using string comparison (201800040009).
  3. Sorts the overall list by comparing the formatted values.
  4. Re-maps the list to the original format stored in the 0th element.

So now, in order to get the list of entries to turn into a feed, I can just call:

my @entries = reverse $self->get_date_entries_by_depth(3);

…and take the first 30 or so.

Once I had the feed done, I decided to apply the same idea to the set of entries on the front page, and once I’d done that I realized that I could also use the same sorted lists to generate next/previous links for any given node in the date tree.

This was an interesting way to kill some time, both because I revisited an algorithm I’d forgotten about, and because every time I hack on a project like this I’m in a dialog with basic decisions I made before I knew how to write software at all. And maybe, by the same token, looking with fresh eyes at norms that I’d take for granted in any more modern context. wrt isn’t a good piece of software by any contemporary standard, and the approach it represents isn’t one I’d use for anything bigger than a trivial shell script at my day job, but there’s a curious durability to it all the same.

Every few years I revisit some facet of this tiny, mundane tool and apply a bit of understanding I lacked when it was first written, and some structure comes a little clearer that lives in the space between my ignorance at 20 and my experience, such as it is, at whatever age I’ve reached.

Everyone should have a few long-term projects, however small and unremarkable.

linkdump

Mobile Giants: Please Don’t Share the Where — Krebs on Security

More about disabling standard I/O buffering

Sysprof - Statistical, system-wide Profiler for Linux

Apple TV Markup Language Reference: About TVML

The Original Hacker's Dictionary

Inside Google, a Debate Rages: Should It Sell Artificial Intelligence to the Military? - Bloomberg — don't be evil lol

The bedtime story

A plea for stability in the SciPy ecosystem

Planet Computers Gemini PDA LTE Compatibility in United States

Don't Get Distracted — "I’m going to tell you about how I took a job building software to kill people."

cowlicks/privacypossum: Privacy Possum makes tracking you less profitable — An anti-tracking-oriented plugin from a one-time Privacy Badger dev. Probably worth a look.

Sound Blaster: How Sound Cards Took Over Computing — "Sound cards like the Creative Sound Blaster were the missing element that computers needed to take on multimedia. Then, they faded from view. Here's why."

“The healthy will of the people” – a lesson from history – James Christie's personal blog

Audio Adversarial Examples — «We have constructed targeted audio adversarial examples on speech-to-text transcription neural networks: given an arbitrary waveform, we can make a small perturbation that when added to the original waveform causes it to transcribe as any phrase we choose.»

Make Some Noise

Why Did Some Countercultural Types Vote For Trump? - Mondo 2000 — These seem like fairly accurate observations, as far as they go. (Could probably use a side of "lots of people are just racist as hell".)

Google AI Blog: Google Duplex: An AI System for Accomplishing Real-World Tasks Over the Phone

The ghost story

Karr, Wallace, Schneiderman, & the Tyranny of Genius - The Atlantic

Building a Kubernetes on Bare-Metal Cluster to Serve Wikipedia - Alexandros Kosiaris - YouTube

The Mangiapocalypse