The problem#

There's a handy Vim plugin openssl.vim that allows you to easily edit encrypted files with Vim simply by giving the file an extension like .aes. Then Vim will ask for a password upon loading and saving the file in order to decrypt and encrypt it with openssl.

Unfortunately, the plugin was last updated in 2008 and makes some assumptions about openssl's defaults which are no longer valid. The most pressing issue is that the plugin now outputs a warning message when encrypting. By itself, that's worrisome, but, worse, that warning message gets output into the file along with the ciphertext. Needless to say, the resulting file cannot be decrypted without manually removing the warning text.

The solution#

Simply fixing the options the script passes to openssl is a good start, but I also wanted to make sure any files encrypted with the old settings could be decrypted. My updated openssl.vim¹ does both in addition to fixing some other annoyances.

The details#

Checking for prior work#

The first step of building my own solution to any problem is, of course, checking if someone else has already solved it. I did searches on "openssl.vim" and the warning message that openssl outputs:

*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.

I was unable to find much evidence that anyone was using openssl.vim.

The one lead I did find is that vim-scripts.org mirrors their scripts on GitHub, so I found the repository for openssl.vim, and through there the forks by other users on GitHub. There weren't many, but this fork by GitHub user François Charlier had some useful changes that I included in my fork. The most visible change is that his changes fix the issue where openssl.vim didn't work in gvim by using Vim's inputsecret() to read in the password instead of letting openssl generate a prompt, which also added flexibility on how openssl is called in the script which most of my changes relied on.

Decrypting with multiple options#

The obvious fix for the warning above is to simply add -pbkdf2 to the encryption and decryption commands, which I did. But then files encrypted without that option can't be opened.

Thanks to the change mentioned above to store the password in a Vim variable, with a little extra work, we can just re-use the password in multiple calls to openssl and see if any of them succeed. Additionally, since some files will have that warning at the beginning, we check if it's there and remove it if it's found.

As to which options to try decrypting with, we want to try any options any previous version of openssl.vim with any previous version of openssl could have produced. This means trying with and without -pbkdf2 due that flag being added in the new version due to that warning. And also trying with and without -a (Base64 encoding) as different versions of openssl.vim disagree on including it or not for AES encryption. Additionally, the man page for encryption with openssl suggests that the -nosalt option used to be default and says that the current default for -md (Message Digest) is sha-256 although it used to be md5 (which is no longer considered secure).

All combinations of those options would be 16 decryption attempts... but running the decryption is fast, so we just try all of them until one succeeds or all fail and we inform the user they should try a different password.

Capturing warnings from openssl#

While we don't want to write warnings out to the encrypted file, we don't want to ignore them either. Unfortunately, it does not appear to be possible to simultaneously capture standard error and standard output from the same command run in Vim. Luckily, there's a pretty straightforward workaround: run the command twice, capturing one the first time and the other the second time. This is acceptable in this case because the encryption is fast and doesn't have any side-effects, so running it twice is awkward but not actually a problem.

There's also a bit of dance to capture the result of the filter in a variable without modifying any registers:

silent! execute l:expr . " 2>&1 >/dev/null"
" Backup @" register and restore it afterward.
let l:register_tmp = getreg('"', 1, 1)
let l:register_tmp_mode = getregtype('"')
silent! 0,$y
let l:openssl_error = @"
call setreg('"', register_tmp, register_tmp_mode)
unlet l:register_tmp
unlet l:register_tmp_mode

Check on changing password#

Other than encryption settings changing, the more common way to fail to be able to decrypt a file is to not know the password. Typing a password a lot, it's easy to make typos. The "confirm password" prompt helps, but doesn't guarantee the lack of typos. As an extra improvement, I made it so that when encrypting a file, if the password works to decrypt that same file, then it won't require retyping the password. This gives an additional check against accidentally changing the password on a file when saving it.

Other options for encryption#

Vim also has built-in support for encryption. If you want encryption to keep track of passwords securely, consider using a password manager instead.