Given a file which contains a list of floating point numbers in IEEE 754 single-precision format stored in big endian byte order, how do you view and manipulate this data using command-line tools? This is an actual problem one of my officemates had.
$ od --endian=big -f file
0000000 1.7155696e-07 1.0432226e-08 4.563314e+30 6.162976e-33
I changed the URI scheme of this blog recently from
/posts/YYYY/MM/slug/ to /YYYY/MM/DD/slug/. The
latter looks better and makes the actual day of the post more visible.
But I already had posts using the old scheme and
cool URIs don't change. Luckily, someone wrote a Pelican
plugin called pelican-alias which allows articles to be
tagged with additional URIs to redirect to their canonical location. All I
had to do was add an Alias: /posts/2015/02/... line to the top of each
of the posts I had already written and the plugin would take care of the
rest.
The non-trivial part of automating this is that the URIs include the article's slug, which may have been generated by Pelican from the title, so Pelican has to be involved in generating the correct redirects.
There are two ways I could have automated this process:
Alias: lines into the blog posts.I took the latter approach because it was simpler and involved no new code to maintain.
SSH public key authenication can make SSH much more convenient because you do not need to type a password for every SSH connect. Instead, in common usage, the first time you connect, GNOME Keyring, KWallet, or your desktop environment's equivalent will pop up and offer to keep your decrypted private key securely in memory. Those programs will remember your key until the next time you reboot your computer (or possibly until you log out completely and log back in).
But those are tied to your desktop environment. If you are not at a GUI, either using a computer in text-mode using a console or connecting over SSH, then you do not have access to those programs.
When using multiple terminals on different hosts, it can sometimes be confusing to remember which host you are on. The hostname appears in the command prompt, but it's easy to skim past that if you are not paying attention.
One solution that works pretty well for me is recoloring the prompt based on what host I am on. This is in fact why I researched how to get 256 colors terminals working in the first place: in order to have enough colors to be able to make a good choice for each host I use frequently.
By default, terminals on Linux only use 8 colors (or 16 if setup to
use bright
variants instead of bold text). Everything else on a
modern computer uses 24-bit color, allowing for millions of colors. More
colors in the terminal would allow for better syntax highlighting and
color output of various commands to be more readable.
In practice, while a few terminals support full 24-bit RGB color (at least Konsole does), it is not widespread enough to be used much. On the other hand, most terminals support 256 colors, which is significantly better than just 8.
Yesterday, I described how to setup DANE in order to verify HTTPS keys through DNSSEC. I also noted a very important caveat: no one supports it and both Mozilla and Google are unlikely to ever support it. So we can't expect any security gain from implementing DANE in the real world.
Let's be fair: DNSSEC is no panacea. Browser vendors aren't ignoring it out of spite. They are choosing to not implement it for solid technical and social reasons.
Let's take a step back and look at why we wanted to use DANE in the first place. The problem is that the HTTPS security model is based on certificate authorities. Your browser has a list of certificate authorities that it trusts and to run a HTTPS site, you ask one (or more) of them to sign your server's public key asserting that your server really is the right one for your domain(s). The catch is that any certificate authority can make assertions about any domain and there's a lot of certificate authorities, many of which are suspected to be under the influence of various governments. This means that you are not just relying on the security of the certificate authority that you choose: you are relying on the security of every single certificate authority in the world. In short, the HTTPS security model is broken.
DNS is the system that provides information on hostnames like
the IP address of aweirdimagination.net so your browser
can connect to this website. DNSSEC is an extension which uses
cryptographic signatures in order to verify that information
is actually correct, preventing certain classes of attacks
which could cause you to believe you are connecting to one server while
actually connecting to a computer under the attacker's control. Additionally,
since DNSSEC verifies information obtained through DNS has not been
tampered with, it allows for DNS to be used for certificates,
so servers can be authenticated for encrypted protocols without the need
for relying on certificate authorities.
Unfortunately, DNSSEC support is not widespread in clients or servers.
Particularly, I did not want to run my own DNS server and have to
worry about keeping it updated and being aware of any security
vulnerabilities. I wanted to be able to, for a reasonable price, have
a domain with full DNSSEC support and use it for securely advertising
the https certificate for this website and the ssh server key
for the web server. (Admittedly, I am trusting the DNS host more than
strictly necessary, but realistically, they are also my registrar so
they could simply publish their own keys for my domain if they wanted to
take it over.)
I settled on using easyDNS, since they were the only DNS
hosting provider I could find that offered what I wanted; specifically,
they very recently added support for TLSA and SSHFP records
(for https and ssh keys, respectively). I later found mentions of
RAGE4, which also looks like it should work.
When developing code or creating visual artifacts in non-WYSIWYG systems, it is very useful to constantly be aware of the output of the compiler and the appearance of the artifact you are creating, whether it is a GUI, a chart, a graph, or a paper. The common way of doing this is to have an IDE specialized for the system you are using; for example, LyX provides a WYSIWYG editor for LaTeX. Similarly, there may be plugins for your text editor to support whatever kind of development you are doing. On the other hand, we can use the shell to create a solution independent of the text editor and the availability of plugins for the particular system being developed for.
While shell programing lets you write very concise programs, it turns out that the primary use case of working with files is unfortunately much harder than it seems. That detailed article by David A. Wheeler does a good job of explaining all of the various problems that a naive shell script can run into due to various characters which are allowed in filenames which the shell treats specially in some way.
One surprising one is that filenames beginning with a dash (-) can
be interpreted as options due to the way globbing works in the
shell. Suppose we set up a directory as follows:
$ cat > -n
Some secret text.
$ cat > test
This is a test.
It has multiple lines.
Quick, what will cat * do here?
$ cat *
1 This is a test.
2 It has multiple lines.
Probably not what you wanted. The reason that happens is that the *
is expanded by the shell before being fed to cat, so the command
executed is cat -n test and
gets
interpreted not as a filename but as an option telling -ncat to
number the lines of the output.
The workaround is to use ./* instead of *, so the - will
not actually be the first character and therefore will not get
misinterpreted as an option. But there are many other things that
can go wrong with unexpected filenames and remembering to handle
all of them everywhere is error-prone.
The solution is shellcheck. shellcheck will
warn you about mistakes like the cat * problem and many
other issues you may not be aware of.
As I have many shellscripts around that I wrote before learning about
shellcheck, I wanted to run it on all of the shell scripts (but
not binaries or other language scripts) in my ~/bin directory, so
naturally I wrote a script to do so:
#!/bin/sh
find -exec file {} \; \
| grep -F 'shell script' \
| sed s/:[^:]*$// \
| xargs shellcheck
This uses the file command to identify shell scripts and
then selects out their file names to run shellcheck on all of them
using xargs.
shellcheck is written to support integration into IDEs.
I use Vim to edit shell scripts, so I installed the
syntastic (using Vundle which makes installing
Vim plugins off GitHub very easy). Note to follow the instructions on
the Syntastic page, specifically the recommended settings: without any
settings it won't do anything at all. Once set up, it automatically runs
shellcheck on every save, identifies lines with warnings and shows
a list of warnings that can be double-clicked to jump to the location of
the warning.
If you use the other text editor, then the shellcheck
website recommends the flycheck plugin.
The command-line is an expressive interface which allows powerful
commands to be written concisely. Sometimes you want a longer, less
direct way of implementing a task. For example, merely writing
wc -l is far too straightforward for counting lines in
a file. Surely we can devise a more convoluted way to accomplish that
task.
cat "$file" |
expr $(od -t x1 |
sed 's/ /\n/g' |
grep '^0a$' |
sed -z 's/\n//g' |
wc -c) / 2