notes on notes
(It’s safe to skip this part if you’re just looking for tools / tips / ideas.)
Much of my working life has been struggling to understand things built by other people. A lot of the rest has been struggling to understand things built by me. The former has ramped up over the last few years, as I’ve gotten into other people’s codebases and projects well outside my expertise. The last 6 months or so I spent with Adafruit was a lot like starting over from scratch as a programmer. By contrast, working at the Wikimedia Foundation draws on my core skills, but it’s also set in a hilariously complicated and historically ramified ecosystem conditioned by politics I’ve only just begun to piece together and will never fully know.
As a consequence, I’m spending as much time on my notes as on anything else, because it’s a lever for gradually understanding all of this. It’s become part of the way I think about problems and a space for doing nearly all my other work.
In theory, I’ve been writing things down for a while: Letters to family most of my life, mandatory science class logbooks in highschool, the usual embarrassing crap in the way of stories and poems. I’ve been writing this blog since I was 16, and paper journals since some time in college. At one time or another I’ve done archival research, technical writing, and writing-centric software.
Despite that history, I’ve only recently gotten a handle on the practice of taking and referencing notes. I’m still working out habits and tools, but at least some of what I’m doing now seems to work.
One reason this matters to me now is that I’m older and more burned out than I used to be. At
38 39, my memory and focus aren’t what they were at 18. The machinery wears out, especially with enough of drink, smoke, and bad sleep. The sedentary habits of working at a keyboard don’t help much either. Meanwhile there’re extra decades worth of noise bouncing around in my head, while the field I work in has only gotten more ridiculous.
I now wish I’d learned to take notes while I was younger, when I needed it much less. It always had the dire flavor of homework to me in school, and it’s only years later that I can see it as freeing. At any rate, this seems like a good moment to outline some useful things I’ve learned.
I’ve had a
notes directory for quite a while now. I couldn’t tell you when it first appeared, but for the first 5 or 6 years of its life it just contained some ad hoc text files about various topics.
There’s nothing wrong with that approach and I encourage it.
vimwiki diary pages
In early 2017, I wrote about adopting VimWiki and its diary features. Since then, it’s become a central part of my workflow. A hotkey in my window manager opens and closes a window with a [
tmux][tmux] process inside my
~/notes directory. From there, I typically have
vim open to some pages in
~/notes/vimwiki, usually including the diary page for the current day. I often also have a separate tmux buffer open for grepping files or doing other operations, since
~notes contains lots of random stuff outside of the wiki. All of
~/notes is a git repo, committed once a day or so and backed up to a local network drive.
An example brief diary page
= sunday, february 24, 2019 = Still dogsitting in the mountains. Baby Ruth comes home today. - [X] dishes - [X] morning walk [[../louie-dog|louie-dog]] - [ ] pack - [ ] load recycling - [ ] [[../taxes|taxes]] organize every tax document i can lay hands on - [ ] finish writing about [[../notes|notes]] - [ ] read a chapter of CD book
It includes an unambiguous datestamp, a few notes on the situation for context, a handful of TODO items, and links out to pages on specific topics / tags. It also illustrates the two axes I think of the system as built on: Time and subject matter.
Roughly speaking, I want to be able to index the stream of information by either when a thing happened or what it’s about, and jump between those two concerns. In concrete terms, this means liberal use of datestamps and tags.
For the most part here, a day is sufficient temporal granularity, but on days when it’s not (for example when triaging some sort of production outage, or dumping a bunch of mixed info in from chat streams and mail for later processing), I have these bindings in my
" F8 inserts an ISO-8601 datestamp (mnemonic: eight rhymes with date) map <F8> :r !date -I<CR>kJ " ,F8 inserts a date with seconds precision map <Leader><F8> :r !date -Is<CR>kJ " ,td / ,tD do the same (mnemonic: toDay) map <Leader>td :r !date -I<CR>kJ map <Leader>tD :r !date -Is<CR>kJ
…which generate various timestamps.
vimwiki topic / tag pages
The topical entry for
louie-dog looks like this:
[[contacts]], [[animals]] A dog. I watch him: - [[diary:2019-02-21]] - [[diary:2019-02-22]] - [[diary:2019-02-23]]
The first lines of topical pages often contain links that I think of as “tags”. Although VimWiki has a separate concept of tagging and a specific syntax for that, I haven’t used those features much, because I don’t think of there as being much distinction between a link and a tag within wiki systems like this. (This follows ideas I developed while working on WalaWiki.) If a dated diary page links to a topic, it’s effectively tagged with that topic, and vice versa.
Pages often link out to filesystem locations, version control repositories, web URLs, etc.
VimWiki topic pages themselves often map to tags I use in other systems like Pinboard and p1k3, and I’ll probably expand this pattern to include other tooling like command-line history and photo organization.
I name both VimWiki pages and tags like so:
- all lowercase
- dashes instead of spaces
- usually pluralized when a tag denotes an instance of a general category of thing (this is pretty arbitrary, but it works in my head)
- generally limited to the ASCII characters; though I think this may not need to be a hard and fast rule in 2019, it does simplify the life of an American English speaker working with sometimes-crufty CLI utilities
A sample of recently used tags: bicycle, boulder, boulder-county, cars, colorado, filesystem-stuff, icons, images, longmont, mediawiki, money, packaging, placelogging, public-domain, python, real-estate, taxes, urban-planning, via-irc, via-quiddity, warelogging, wikimedia, wikimedia-commons, etc.
Most of these are simple keywords for future searching. Others have specific semantics:
warelogging is for info about software and hardware (often a way for me to flag it for later inspection).
placelogging denotes geographic locations. Tags prefixed with
via- indicate a medium or specific person as the original source of an item. It’s often easier to remember that, for example, Casey told me about an article on IRC than it is to remember the title or publisher of the article.
filesystem-stuff is a collection of interrelated things that I reference when trying to recall documentation, software, or philosophy on the subject, and so forth.
I used to be leery of tagging systems, but I’ve pretty much embraced them at this point and tag with everything that seems relevant when creating new VimWiki pages or bookmarking things, which brings me to…
Pinboard is a straightforward subscription web app for bookmarking stuff. You sign up for an account and install either a couple of bookmarklets or a browser extension, and then when you see something interesting on the web you click a button and (optionally) add a description and some tags. If I have time I try to add a meaningful description, or paste a useful excerpt in the description field, bracketed by « and » (quotation marks not often used in English, which saves me from compulsively worrying about the proper nesting of single and double quote marks).
I’ve had a Pinboard account since 2013, and have tended to use it increasingly over that time. The rolling linkdump here is based in part on my public links. It’s good software that I’m happy to support with dollars, and I tend to enjoy the author’s opinionating and other prose.
Still, this is a locus of several irritating questions for me: Is there a good reason not to have most of my web browsing history just logged locally, with archives? How much am I going to regret my reliance on a piece of closed-source software whose author might lose interest or get hit by a bus?
I periodically export the data, but at some point I might do something more drastic about bookmarking.
blog posts / p1k3
I don’t always do a great job of it, but I try to publish usable notes on problems other people might have.
I use a growing set of tags to further categorize p1k3 entries.
I like to try writing a detailed public explanation for significant changes I make to open source software projects. So, for example, if I do a project for Adafruit that isn’t otherwise described in a tutorial, release a new version of wrt, or write a new utility script for bpb-kit, I’ll try to write a post along the lines of these:
- platform detection with linux on single-board computers
- App::WRT v4.3.0: schwartzian transforms, long-term projects
A preponderance of dull technical notes makes p1k3 drier and more boring than it would otherwise be for any general audience. That’s kind of depressing for someone who wants to share things like poems and photos, but it’s also largely by design. As I keep writing, I’m reluctant to publish personal material into a network environment as culturally and technically fucked up as the present-day internet. As it stands, it’s probably dangerous enough to provide a text corpus for future machine learning models to use in convincingly imitating me. (More about that under security.)
command line history
I keep a lot of command line history, right now by way of two mechanisms.
The first (and simplest) is that I tell
bash to store as much history as possible in the same file:
# from .bashrc export HISTFILE=~/.histfile export HISTCONTROL=ignoredups export HISTSIZE=-1 export HISTFILESIZE=-1 # from .zshrc export HISTFILE=~/.histfile export HISTSIZE=15000 export SAVEHIST=9999999
The second is a utility called
commandlog that I sketched out in October of 2016. As of this writing, the current version hasn’t changed much. It just jams command history into an SQLite database, along with some context like working directory, hostname, and so forth. It’s definitely buggy as hell and does almost nothing. Still, I’d like to build on the idea.
I add rough human-readable comments to history for future searching and forensics work with a no-op script called
comment. Eventually, I’d like to be able to add detailed notes to individual history entries, tag specific commands, and use tags to easily pull together scripts out of related sets of commands (or automatically build menus on a per directory or per project basis).
My theory is that we should be able to treat command history as, at least:
- a simple log (this one is obvious)
- an act of documentation
- an act of cumulative programming
The sheer amount of work that’s done on the command line is simultaneously a tragic waste of repetive motion and an untapped technical resource. I’d like to usefully formalize the habit a lot of people have of hitting Ctrl-R to search through history for the last time they dealt with a problem. This should be a healthy way to accumulate knowledge in one of our primary interfaces rather than a hacky fallback to bad habits.
At any given time, I have a single primary notebook which collects dated entries in sequence. It goes in my bag when leaving the house, traveling, etc.
I look for the following features:
- On the order of 8x5" or a bit bigger.
- Blank pages, ideally pre-numbered.
- Hardcover, thread bound, should lay flat.
- Decent quality paper, sufficient weight to take fountain pen ink without too much bleed-through.
- A back pocket.
- Ribbon bookmark(s).
In 2019, a lot of notebooks are available that fit this rough profile. I used to use Moleskines, but the quality is uneven at best. These days I buy the medium / A5-sized Leuchtturm 1917 books.
When I start a new notebook, I write the starting date on the inside front cover, along with a PO box address that in can be mailed to in case someone else finds it. I add a $20 bill to the back pocket to cover mailing costs (and because having a twenty in reserve for emergencies comes in handy). I leave several pages blank for adding entries to a table of contents (some notebooks have a table already printed for this).
I start a fresh page for every day / entry, and headline the entry at the top of the page with the full date, including day of week and year, like “Saturday, March 3rd, 2019”. Periodically I write the page number, date, and some subject keywords in the table of contents.
A notebook entry might contain anything from grocery lists to a rough draft of a blog post or letter, but in general I use this notebook for personal journaling. This is by contrast to VimWiki pages and the like, which are generally for working notes and organizing practical knowledge about systems.
I don’t copy local notes to anywhere on the network, but this entry still gestures at my feelings about continuing to use paper for a lot of work:
Writing on paper crossed an interesting threshold for me recently: I’d been thinking of it as almost purely an affectation, a thing I persisted in for reasons of aesthetic stubbornness and simple physical attachment to the ritual—all well and good, but scarcely justifiable on any practical grounds beyond “I like it, and writing the way I enjoy writing helps me write at all”.
Lately, though, I feel a deepening appreciation for any technology that’s outside the reach of the network and software. Much like printed paper books live outside of Amazon’s surveillance machinery, stubbornly resist deletion, and can be freely lent out, paper notes are cognitive tools that don’t have Google Analytics and a sea of inscrutable machine-learning slapped on them. I guess that’ll only last until sufficiently high-resolution always-on cameras inescapably cover every angle in every building (or come with whatever augmented-reality system we all have to use in order to keep participating in society) but it’s no small thing for the moment.
I don’t yet have a great system for indexing into these books by date, but I’d like to have them arranged on shelves and labeled.
In addition to the primary notebook / journal, I often use one or two separate “scratch” notebooks for working out ideas, taking random notes during the workday, making sketches, trying out new pens, etc.
For these purposes I’m not as picky about features. I use spiral bound notebooks, cheap freebies, or sketchbooks. I haven’t tended to worry too much about dates, though I usually label pages with a rough subject heading.
I’ve mentioned dates a lot. In general, it’s important that human readable dates be “complete” - including the year and day of week. This way you can tell the temporal context of an entry at a glance, which is particularly useful on paper.
I recently purchased a date stamp and have started using it on more paper items and at the top of notebook pages.
It’s important that machine readable dates be easy to cross reference, sort, and parse.
VimWiki uses ISO 8601 dates for diary entries. That is: zero-padded YYYY-MM-DD. So
~/notes/vimwiki/diary/2019-03-03.wiki is the diary file for March 3, 2019. This is a good habit to follow: It’s easy to sort or manipulate with common tools, fixed-width, and unambiguous.
p1k3 entries are instead stored in a directory hierarchy like
~/p1k3/archives/2019/3/3 contains the entry for March 3, 2019. Months and days aren’t zero-padded, so sorting can get a bit awkard, but I’ve more or less been happy with this scheme for many years. It can be constraining, but often simple rigid constraints like “blog entries are stored by day in a rigid directory hierarchy” are useful scaffolding which can free an author to experiment in other ways.
Apart from excerpts when bookmarking, I often quote from relevant chat traffic, e-mail, and documentation in my electronic notes, sometimes editing for concision or clarity.
Quotes can make grepping a lot easier, and there’s often little reason to re-phrase things that someone else has already put a lot of thought into communicating to me. (Though summarizing and re-stating are good tools for improving your own memory and understanding of a topic, so I do plenty of that as well.)
As with web bookmarking and re-finding URLs to things, never assume you’re going to be able to find some piece of information again in the future just because we live in the age of Google and everything is surveilled to within an inch of its life by our malevolent click-mongering cloud marketeer overlords. If it’s important enough, as a general idea or a set of instructions or a specific technical artifact or whatever, maybe you should think about a local copy.
When working on paper, copying out material longhand is obviously much more labor-intensive, but can be a useful aid to memory.
As an important caveat to this entire practice, don’t quote stuff that would embarrass, incriminate, or expose other people unless you’re specifically trying to create that sort of audit trail. Be mindful that your notes, however locked down, might be a vector of compromise for privileged communications.
TODOs and checklists
In August of 2014, I wrote about my checklist habits:
I keep a lot of notes on paper. For stuff I want to do, I make lists with little square checkboxes next to invidual items. If I get something done, I put a checkbox in. If I move the item to another list, I draw a little circle in the box. If I decide it doesn’t need done any more, I draw a line through the whole list item. At work I keep these under my keyboard on the biggest piece of printer paper I can find. Elsewhere I’ve usually got a notebook going. Every once in a while I go through the last couple sheets of paper and the most recent notebook, and make sure everything gets marked done, not-needed, or moved.
…and described an approach to doing the same inside a
I still use this approach, although more of my TODO items live in VimWiki pages now.
Though this is a document about the utility of writing things down, there’s a tension at work here. The first thing is that if you’re really worried about security, don’t write shit down.
(As a corollary to this, the kind of logging that I do of things like command line history has potential downsides. You need to be aware of the potential for accidentally storing credentials or dangerous metadata in plaintext if you’re doing that kind of thing.)
The last 30 years have shown us conclusively that in the wrong hands—those of corporation, state, programmer, ideologue, troll, stalker, and political agitator—data is very often a toxin, a vulnerability, and a fatal temptation.
If you work in a capacity that exposes you to the personally identifiable information of customers or users of software, your organization might have a policy on the topic. If not, or if your organization’s policy isn’t strict, here’s a substitute policy: Don’t copy that stuff into your notes.
Maybe it goes without saying, but if you are doing anything illegal where you live, you shouldn’t write about it, period. Not on paper, nor in a text file, let alone on the internet. The same thinking applies if you are doing anything legal but still likely to get you killed, beaten, fired, divorced, publicly humiliated, etc.
I try to be realistic about my threat model: While it’s true that American governance appears to be slowly collapsing, for right now I live in a stable region with low crime, I’m extremely un-famous, and I’m of no particular political interest. I’m a straight white male with above-average income in a prosperous part of the central US. This all means that my threat model involves:
- My network attack surface: Targeted or opportunistic attacks on my employers and, to a lesser extent, my public personal infrastructure.
- The opportunistic theft of a device or notebook.
- A break-in at my home (unlikely but always possible).
- Doxing by the shithead internet (again, unlikely but possible).
- The ambient surveillance, malware, and fraud we’re all subject to.
- Routine interactions with border agencies and the TSA.
- Occasional natural disasters.
It specifically does not involve:
- The direct attention of:
- State-level actors / three-letter agencies.
- High-level law enforcement.
- Muckraking journalism, political opposition research, etc.
- An ideological / cultural conflict in which my beliefs or identity are likely to get me jailed or murdered.
- The collapse of civil society.
Given these priors, I keep the following rules in mind with regards to note materials:
- Encrypt all drives, including laptops and mobile devices.
- Automatically lock your screens.
- Don’t write credentials (or account details, if you can help it) on paper, especially paper that’s going to travel. Exceptions exist for keeping passphrases or 2-factor auth backup codes on paper in a safe location.
- Use a password manager with data separate from the body of your notes.
- Don’t trust cloud services.
- Don’t travel with anything you’d rather not have compromised, especially across national borders.