Wednesday, January 27

profiling perl

I wanted to profile a Perl script and get some idea if there were any absurd hotspots I could cut out. I googled “perl profiler” and hit on mention of Devel::NYTProf, which I vaguely remember but I’m not sure I’ve ever used.

Out of laziness, I installed by way of apt instead of a CPAN client (I have no idea which CPAN client you’re supposed to be using these days).

$ sudo apt install libdevel-nytprof-perl

It was a little harder to find a straightforward NOW DO THIS TO RUN IT than I wanted (I’d say I was about three glasses of wine in at this point), but after the install I eventually landed on the following:

$ cd ~/code/projectroot
$ perl -d:NYTProf ./script_to_test.pl
$ nytprofhtml

After that, you can open ~/code/projectroot/nytprof/ in a browser. As an example, my results for a recent test run (rendering this site, specifically) are here. It’s not the least-inscrutable thing ever written, but it didn’t take too long to figure out how to see the interesting stuff. The HTML output is good enough that I didn’t feel any need to fire up KCachegrind or anything, and I dig how it breaks down execution times line-by-line.

Unsurprisingly, this is a Tim Bunce joint. That dude is a monster of practical utility.

tags: topics/perl, topics/technical, topics/wrt

p1k3 / 2016 / 1 / 28

Wednesday, January 27

Recent paid work:

Some notes:

The Fail2ban one is derivative of Justin Ellingwood’s How To Protect SSH with Fail2Ban on Ubuntu 14.04.

I would pretty strongly discourage anyone not super comfortable with the gritty details from upgrading any production servers to PHP 7 as of just yet, though it looks like a really valuable release.

I read stuff by a ton of people while writing the CoreOS one. I think that, at least at the exact moment of publication, it may actually have been the most complete and tested guide to the subject reachable from Google. That’s not because it’s great. It’s just that it would be difficult to overstate how much work the CoreOS documentation and error reporting needs around the security stuff. I’d like to say I might do some of that work, but realistically I’m not a domain expert and it is way down my list.

❉

While I’m at it, a list of tutorials I’ve done for Adafruit:

And code I touched:

None of this is exactly a work for the ages, and some of it was current for about five minutes total after publication. Still, I’m not entirely unhappy with the little penguin comix.

tags: topics/adafruit, topics/digitalocean, topics/php, topics/technical

p1k3 / 2016 / 1 / 27

Monday, January 25

debugging fail2ban

Trying to document a basic fail2ban install on CentOS 7.

There’s an /etc/fail2ban/jail.conf. You’re not supposed to edit this directly. According to official docs, my understanding of what I should do is create a jail.local that looks something like this:

[DEFAULT]
# Ban hosts for one hour:
bantime = 3600

[sshd]
enabled = true

…where those values will override things already defined in jail.conf, which contains a whole bunch of defaults. This doesn’t quite seem to work—fail2ban starts, logs things, and reports that it’s banning IPs, but as far as I can tell, the iptables firewall I expect it to be configuring isn’t touched, and bans are definitely not happening.

Whereas if I do:

$ sudo cp jail.conf jail.local

…then add enabled = true to the [sshd] section, and restart fail2ban:

$ sudo systemctl restart fail2ban

…then iptables -L shows me more or less what I’d expect.

I don’t think my mental model of how these config files fit together is correct. What’s different between the two scenarios?

I wonder if you can get fail2ban-client to dump configuration values, and it turns out that you can:

$ sudo fail2ban-client -d

With this it’s easy enough to see what changes between the two scenarios:

$ sudo fail2ban-client -d > one
# switch from sparse jail.local to full copy of jail.conf
$ sudo fail2ban-client -d > two
$ diff -u one two > jails.diff

It’d be ugly to inline here, but here’s jails.diff. As you can tell, a bunch of stuff referring to firewallcmd-ipset is replaced by stuff referring to iptables-multiport. That makes some intuitive sense—firewalld, not iptables, is standard on CentOS. My question is still ok, so why are the configuration values differing? Does some section of jail.conf not get executed if jail.local is present? Or am I just totally wrong about how this fits together?

To be continued!

❦

Continued: Ok, so on first inspection, the fail2ban configuration stuff appears to be an impenetrable thicket of everything-lives-somewhere-else OO spaghetti. Eventually, however, I grep my way to realizing that /etc/fail2ban/jail.d/00-firewalld.conf looks like so:

[root@centos7-fail2ban fail2ban]# cat jail.d/00-firewalld.conf 
# This file is part of the fail2ban-firewalld package to configure the use of
# the firewalld actions as the default actions.  You can remove this package
# (along with the empty fail2ban meta-package) if you do not use firewalld
[DEFAULT]
banaction = firewallcmd-ipset

And the documented execution order of config files turns out to go:

  1. jail.conf
  2. jail.d/*.conf (in alphabetical order)
  3. jail.local
  4. jail.d/*.local (in alphabetical order).

…which means that unless jail.local overrides it, the jail.d/00-firewalld.conf will set banaction to firewallcmd-ipset. The fail2ban-firewalld package seems to be a hard dependency of fail2ban, so I have a bit more puzzling to do, but in short I was being an idiot.

✨

I land on this version of jail.local:

[DEFAULT]
# Ban hosts for one hour:
bantime = 3600

# Override firewalld banning:
banaction = iptables-multiport

[sshd]
enabled = true

☆

In sysadmin-land, you’re very often looking for one of two moments:

  1. I am being an idiot.
  2. Someone else is wrong.

Always bet on #1, if you have to bet.

tags: topics/technical

p1k3 / 2016 / 1 / 25

Monday, January 18

travis ci and perl modules

I just spent a little time cleaning up the relationship between this site and Display.pm, the Perl module which renders most of it.

Along the way, I noticed that I’d signed up for a Travis CI account at some point, and it was attempting to run tests when I pushed to the Display.pm repo.

The deal with Travis is that you include a .travis.yml file in your project’s root, and it will try to follow instructions to check out your code and run any tests you’ve defined, then squawk at you if they fail. It also provides those little build status badges you see all over GitHub these days. After reading Building a Perl Project, I wound up with this:

language: perl
perl:
  - "5.22"
  - "5.14"

After that, it took me a half dozen tries to figure out why the tests were failing and make them genuinely self-contained, but lo, they pass now.

☙

This was probably a lot easier because I’m already using Module::Build. I’m years and years out of touch with the Perl ecosystem, so I don’t know if Module::Build is still the thing to use, or if I’m supposed to have moved on to something shinier by now. It still seems to work pretty well, at least for my trivial needs.

Something I continue to appreciate about Perl, 15+ years after I first started spending time with it, is that I come back to things like this and usually find that they still work, are still documented, etc. This particular kind of durability seems, if anything, to have decreased dramatically in the tools that dominate now. It’s odd to find it so much more often in projects built on Perl, C, shell—even sometimes PHP!—than in the space of the languages, libraries, and frameworks considered acceptably modern for most projects.

Well, ok, not odd. But certainly striking.

tags: topics/perl, topics/technical, topics/wrt

p1k3 / 2016 / 1 / 18

Saturday, January 16

importing workings-book to the p1k3 tree

For a chunk of last year, I was keeping workings-book as a kind of technical logbook:

I’ve been working on a short book on the basics of the command line. This one is something else altogether. It’s a long-form technical logbook or journal, capturing the outlines of problems and their solutions as I encounter them. It is dry, incomplete, and sometimes wrong. It is unlikely to be useful for the general reader. I’m compiling it because I want a record of what I learn, and I hope that writing documentation will help me do cleaner, more reproducible work. If it helps people with similar problems along the way, so much the better.

Todd recently reminded me of this when he started updating his own version of this idea. It’s something I’d like to pick back up, both for the benefit of my own shaky memory and as a low-key contribution to general knowledge.

Thinking about it, though, the current format (one long HTML document with a table of contents) got cumbersome after more than a handful of entries. Since I already have one long-running blog with a lot of nerd stuff, I might as well just merge workings-book into p1k3, and publish future technical notes here.

I could do this by hand, but would it make sense to automate the process? Maybe. workings-book is laid out like so:

/home/brennen/code/workings-book/
▾ entries/
    0-introduction.md
    2014-12-03.md
    2014-12-05.md
    ...
    copy.sh
    ...

I was about to go into writing a script to copy the entries/yyyy-mm-dd.md files into p1k3 entries (which are stored in files like archives/yyyy/mm/dd), first making sure there’d be no collision with an existing entry. Then I noticed copy.sh, which looks like so:

#!/usr/bin/env bash

cp 2014-12-03.md ~/p1k3/archives/2014/12/3
cp 2014-12-05.md ~/p1k3/archives/2014/12/5
cp 2014-12-07.md ~/p1k3/archives/2014/12/7
cp 2014-12-08.md ~/p1k3/archives/2014/12/8
...

It seems like I must have started in on this process once already. If I had to guess, I’d bet that instead of writing logic for the problem, I just copied a directory listing into Vim and then used blockwise visual select to double the column of filenames, insert ~/p1k3/archives/, chop off the trailing .md, and replace - with /. I’m not sure how I checked for collisions, but it may have been as simple as hitting Ctrl-W f with the cursor on each target filename, or running the script and checking git status on the p1k3 repo.

Here’s an annotated sequence of keystrokes that (more or less) repeats the process:

:r !ls ~/code/workings-book/entries^M
You can read the output of a shell command into the current buffer with `:r !command`
kkd}{jddkdd
Up, up, delete to end of block, top of block, down, delete a line, up, delete a line. (Trimming some uninteresting filenames from the list.)
^V}yA ^[p
You can enter blockwise visual select mode with Ctrl-V and select a column of text with } or {. You can copy this, like any visual selection, with y Jumping to the end of a line and hitting p will then paste the selected lines in a blockwise fashion as well—appending them to existing lines instead of after the current line. The first time you try this, it's sort of magical.
^Vjjjjjjjjjjjjjjjjjjjjjjjjjjlllllllll
Lazily selects just the date-formatted portion of the second filename column with movement keys.
:s/\%V-/\//g^M
I had to look this up again, but you can restrict a substitution command to just the blockwise selection by using something like :s/\%V-/\//g. (By default, substitutions usually just apply to each line in the selection, which is super confusing.)
^Vkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk>
Selects all the lines and indents them one level with >. I could have used regular line-wise selection or just indented the whole block or whatever, but I didn't.
^Vjjjjjjjjjjjjjjjjjjjjjjjjjjjjscp^[
Selects the first space and replaces it with cp.
$hh^Vjjjjjjjjjjjjjjjjjjjjjjjjjj$d
Jumps to the end of the line, blockwise selects all the trailing .md extensions, zaps them. There are cleverer ways to do this, but it most always takes me longer to think up a substitution regex than it does to just delete some columns of characters.

Anyhow, this kind of script-munging is a pattern I’ve leaned on a lot without ever really talking about it much that I can recall. I’d be willing to bet it’s a common practice for a lot of Unix types. It’s somewhere in-between classical scripting and whatever it is that people are doing when they bash data around in a spreadsheet with mouse and keyboard commands.

Part of my nerd brain thinks it’s more “correct” to solve problems like this one in code. It’s not like it would be much code—a shell one-liner would do it, or a few lines of Perl if I wanted something more readable and reusable. Then again, there’s something to be said for levering your way through a problem with the low-level, almost-physical indirections provided by tools like Vim’s visual selection modes. It can spare programmer-brain cycles for more important tasks, and its utility probably says something about how software like Excel has become so essential to people who need to do data manipulation but don’t have general-purpose programming languages or robust databases in their working toolsets.

tags: topics/cli, topics/notes, topics/technical, topics/userland

p1k3 / 2016 / 1 / 16

thursday, january 14

then again, maybe writing is just
about exactly like mowing the lawn
or painting a wall
or splitting wood in the summer
when last winter's air is a fading
memory and next winter's is barely
yet a premonition

maybe all it ever is
whatever it is
is the making of effort
across time

foredoomed, certainly
for of all things that accumulate
only entropy has any lease on the future
but cumulative, anyhow, on the scale of a life
the way brushstrokes and cordwood
are cumulative, on the scale of a season

tags: topics/poem, topics/writing

p1k3 / 2016 / 1 / 14

tuesday, january 12

looking up through sunglasses as i step out of the alley
i catch the sphere of what at first i take for the moon
through trees and clouds low in the sky to the west
realizing it's the sun i look for seconds
longer than you're ever supposed to look at the sun

tags: topics/poem

p1k3 / 2016 / 1 / 12

tuesday, january 5

it's gotten easier with time
than in schoolpapers
and childhood letters
but writing never gets easy
the way that some things
have, like the mechanical
skills that any trade accumulates

like putting one foot in
front of the other until
you eat up the miles

oh i want it
to feel that way
but it never has and
never probably will

writing isn't like
mowing the lawn, or
painting a wall
i wish it was
it's more like
the long fevered arc
of some doomed but
magnetic relationship
like a marriage that swings
between desperate tedium
and brief moments of ecstatic
communion

tags: topics/education, topics/poem, topics/writing

p1k3 / 2016 / 1 / 5

Sunday, January 3

I started a new paper notebook today. There were a few pages left in the old one, but it felt like time. New year, new book, I guess. It’s been since last November that I went through this ritual. Longer than I usually make it in one book, but a lot of my time writing has gone elsewhere: A pile of tutorials, a couple of letters, chat buffers.

p1k3 / 2016 / 1 / 3