Given a file which contains a list of floating point numbers in IEEE 754 single-precision format stored in big endian byte order, how do you view and manipulate this data using command-line tools? This is an actual problem one of my officemates had.
$ od --endian=big -f file 0000000 1.7155696e-07 1.0432226e-08 4.563314e+30 6.162976e-33
Binary in shell
The shell has a lot of useful utilities centered around processing
text, often line-by-line. As previously discussed, shell
programs can run into trouble when the text they are dealing with is has
weird characters in it. And there
weird meant characters like
-, space, and newline.
So what happens when we want to process binary data? For the most part, the shell is simply the wrong tool for the job, but sometimes it's nice to be able to use the shell for quick tasks anyway.
There are three programs for printing binary data in a text format:
all with different options and different output formats.1
If you want to edit binary files,
-r option will read in
strings formatted like its output and write the changes back to a file
hexer is a command-line interactive editor for binary
files which is designed for users familiar with
The options for
-f, short for
-t fF, which means to
interpret the file as floats. The
F means single-precision float, as
D for double-precision float.
For handling byte-order,
od also has a
which is necessary because nearly all modern systems are natively
little endian. Or, at least, newer versions do. The version in Ubuntu's
April 2014 release does not have that flag, so it may not be
available on your system.
As a work-around, Python's
struct package offers an
easy way to deal with binary data. As an example, the following script
takes a filename as an argument, interprets its first 4 bytes as a
big-endian float, and prints out that float:
1 2 3 4 5 6 7 8
#!/usr/bin/env python import struct import sys with(open(sys.argv, 'rb')) as input: f = '>f' # big-endian float print(struct.unpack(f, input.read(struct.calcsize(f))))
Out of curiosity of why there are so many tools to do a single task, I checked what package each came from using
$ wajig whichpkg $(which od) INSTALLED MATCHES (x1) ---------------------- coreutils: /usr/bin/od $ wajig whichpkg $(which hexdump) INSTALLED MATCHES (x1) ---------------------- bsdmainutils: /usr/bin/hexdump $ wajig whichpkg $(which xxd) INSTALLED MATCHES (x1) vim-common: /usr/bin/xxd