Saturday, October 24

userlands, 1 of ?

I want to start sketching out this idea I keep having before I lose it again to distraction.

I wrote userland while experiencing, I now think, the end stages of a particular relationship to an existing toolset and intellectual apparatus. Namely, that of the Unix ecosystem, broadly considered.

That is to say: I had reached a certain level of competence, if not exactly mastery, in the most generally useful command-line environment. I had passed through the amateur enthusiast phase into the working professional1 phase, while retaining a sense that the system we had was grounded in sound principles, to the extent that in its broad outlines it was historically inevitable. Ragged around the edges, sure, and complicated by all sorts of counterproductive bullshit, but still under it all a sort of workingperson’s rugged theoretical coherence.

Last fall, Paul Ford wrote this, in some thoughts on (emphasis mine):

Looking at the cabal it turns out the nerds best known to me are realllly a lot of white dudes. And looking at the signup sheet, this community def. is going to be okay on white dudes. We’re nailing it, white-dudewise. But Unix is for everyone. It’s like the Sistine Chapel or Angkor Wat. It’s bigger than the culture that made it. It’s for our servers and our phones and it’s for all the genders and all the races and most of all Unix is for the children.

That last part really resonated with me, like a lot of ideas did. I figured ok, we have this thing (these many interlocking things) that are a deep part of our cultural heritage. If they were accessible to more people, people would have greater recourse in coping with the evils of computing and the power imbalances of the modern network. The tech is here, I might have said, it is just not very equitably distributed.2

I retain a lot of that impulse, but implicit in it was a judgment of the Unix things as also being more or less the right ground for the efforts of the nerd class. Or at least one such ground: A probability cloud of useful abstractions and scaffolding within which sensible people built tooling, and in ignorance of which much absurd wheel-reinventing was constantly enacted.

I have for a long time now been a conservative, of sorts, about actually existing software and innovation in user-facing systems. Harshly critical of the last decade’s inventions and revolutions in UX and system architecture.

Two aphorisms, as they sound in my head, without looking them up:

“Those who do not understand Unix are condemned to reinvent it, poorly.”

“Unix is the worst OS except for all of the others.”

I switched jobs this year. Where I used to focus on a handful of big, application-layer software projects and a pile of business logic, instead I did customer-facing stuff for Adafruit: I wrote little introduction-to-Linux tutorials and documented simple hardware projects and helped support the Raspberry Pi. I read too much of debian-devel. I wrote a lot of shell scripts. I fought with version control. I did everything wrong.

I also listened to my friends talk about intrinsically hard problems of their work: Deploying and configuring and coordinating big systems. Making little systems behave consistently for many users. Asking questions about system state and getting consistent answers.

Here is a very partial list of things I think I know right now:

— The Bourne Shell and its descendants are genuinely terrible programming languages.

— Systems tasks are of sufficient complexity to benefit from a first-class high level language where fundamental abstractions at the level of “string”, “variable”, “loop”, “conditional”, etc. are not debilitating to use.

— Regexps are a total fucking horrorshow. Because of the Turing Tarpit, or something very like it. Because semantic density only contributes to legibility up to a certain point. Because you can always hurt yourself real bad with a chainsaw, even if you know what you are doing.

— The packaging of software for Debian is an unremitting nightmare. Packaging software is just an unremitting nightmare. Also configuration management.

— Configuration management and deployment tools (Puppet / Chef / Ansible / Salt / your bespoke artisanal Python script) have in recent years adopted the language of declarative state and idempotence. This is, in practice, somewhere between a polite fiction and a dirty lie.

— Determinism in configuration of an entire system is a noble goal, it’s just totally fucking impossible for mere mortals to achieve in the current scheme of things.

— Most humans aren’t smart enough to write safe or secure programs in C. Most useful systems-level programs and services are written in C.

— One way to think of a filesystem is as a powerful baseline abstraction for most things in a computer system. Another is as a massive collection of inconsistently represented and needlessly hierarchical state.

— Statefulness is the Fruit of the Tree of Knowledge of Good and Evil.

Torturing the implementor on behalf of the user may be a grim necessity. Then again, like most forms of torture, it may harm its alleged beneficiaries along with its victims.

— Arbitrarily structured text files as baselines for configuration and source code have nearly intractable advantages. You only need to deal with a system where Every Last Thing is in XML, think hard on the Windows Registry, or maintain somebody’s Microsoft Access business logic to understand this much. On the other hand, it renders every system configuration task a nightmare of ad hoc text manipulation.

— Ad hoc text manipulation is powerful. Ad hoc text manipulation is hell.

— The officially-sanctioned desktop environment situation on nearly every Unix derivative is deeply unpleasant, and has been in a state of decreasing quality since the first decade of this century.3

— The terminal as a primary working environment is a jungle of sharp edges, and has become ever more difficult to treat as a refuge in a world where a) people want to communicate in pictures, and b) device manufacturers are doing everything in their power to kill the physical keyboard no matter what you have to say about it.

I’ll stop.

I guess my point is this: Unix sucks. I say this with love, but still, sucks. And not just “the Unix we actually got”, but also the theoretical, idealized, ur-Unix. Its basic ideas, for all of the brilliance and pragmatic down-in-the-dirt utility threaded through them, are not really sufficient to building humane systems.

More broadly, operating systems as they are currently built are bad environments for normal human beings, and nightmare environments for the specialists who work on them directly.

I’m inclined to think this matters, because I’m still a strong partisan of the view that regular users should have ownership and control over their machines. That a large share of computational utility can (and should) be kept local and distributed, if we are to stand any chance of avoiding the worst outcomes of the architecture of centralized control that the internet has made so practical, so necessary-seeming to all the people with the power to make tech decisions.

I also have a selfish desire, as a specialist, for a world in which the working conditions of programmers and sysadmins are less full of broken crap. Because for all that technical gigs are some of the better work you can get these days, as far as money and respect and not breaking your back go, they are also kind of shit in ways that both grind people down and filter out to the rest of society through countless little channels.

I’m eliding so many things here. The history of this kind of criticism. The other could-have-been worlds of OS structure. Ted Nelson and all that. So on and so forth. Maybe I’ll get to those things. I’m not exactly sure.

To be less negative, though, it’d be interesting to spend time thinking about not just what it is that’s wrong with the current dispensation, but what it is that could be fixed and outright replaced in the light of how our existing systems fail—but also how they succeed, or nearly do. Because it seems like right now we’re in the middle of replacing everything (necessary) with stuff that is built in resolute ignorance of a huge tradition of qualified success and tragicomic failure (doomed).

1 Quiet, you.

2 Uh huh.

3 Yes, I also and especially feel this way about Apple.