The next step up, the quadruple precision float, is already part of the standard, but, as far as I know, not implemented anywhere. Intel x86 does have something in between for its internal float format on 80 bits, the so-called extended precision, but it’s not really standard as it is not sanctioned by the IEEE standards, and, generally speaking, and surprisingly enough, not really supported well by the instruction set. It’s sometimes supported by the long double C type. But, anyway, what’s in a floating point number?

Let us take the 32-bits float as an example. There’s one bit assigned to the sign (therefore it’s not a 2s complement representation), 8 bits for the exponent, representing values of -128 to 127 (with a bias of +127), and 23 bits for the mantissa, the “precision bits” (with the most significant one removed, it allows one more low-weight bit to be shoved in): That is, it looks like:

(From Wikipedia)

and the value is given by

where is the sign, the exponent as stored in the float, and are the bits of the mantissa, from left to right in the above picture.

The double uses one sign bit, 11 exponent bits, and 52 (+1 virtual) bits for precision, for a total of 64 bits. The quadruple (which is not a C nor a C++ keyword, unfortunately) uses one sign bit, 15 exponent bits, and 112 (+1 virtual) bits for precision. Its precision is about 34 significant digits. And hypothetical octuple would eat up 256 bits, 32 bytes, probably with something like 24 or 32 bits for the exponent, leaving 224 bits of precision (that’s about significant digits (225 because of the virtual leading bit)). That’s significantly more precise than either float or double, but it would also ask significantly bigger FPUs to deal with these formats!

Generalizing these formats to smaller floats is also straightforward, but we now face serious lack of precision. The half float, on 16 bits, uses one sign bit, 5 exponent bits (with a bias of 15), and 10 (+1 virtual) precision bits.The range is evidently smaller (from to ) but the steps are rather large (all things considered) at . That’s about 3 significant digits. A byte-size floating point number doesn’t seem to make all that much sense.

The half-float format makes sense only if you’re interested in saving storage space, or gaining computation speed, or minimizing hardware, at the expense of precision—you can’t have all three. But could we do better with 16 bits than this? Well, since 16 bits aren’t that much, the first thing that comes to mind is that we could just learn a 64k-entry table filled with higher-precision floats optimized for the precision of the computation at hand. This is in fact a quantization problem where the objective function is not directly the mean square error to the prototypes (as would be scalar quantization optimized over a probability distribution) but the squared error between the quantized computation and the distribution of the “real” result using high-precision arithmetic everywhere. We know have a look-up for each value, but the table may fit in cache, and we could use rather large computations (that do not fit in cache).

*
* *

The idea generalizes to other problems. In instruction encoding, one usually decide how to allocate bits within an instruction word to encode the operations to be performed, the type of the operands (immediate, register, memory reference) and the operands. In simple CPUs, like the Z80 (no, I’m not that old, I know of it because I used to to embedded programming a lot) the instructions are really crude and the opcode fits, always, on one byte.

The instruction set is hierarchical, with groups of prefixes for similar instructions, probably to ease decoding, but that’s not necessary to do it like this. If you’re going to have microcode anyway, the instruction encoding itself is essentially irrelevant: it becomes merely a pointer in the microcode table, and you can assign pretty much every instruction you want to each of your 8-bits (or 16-bits) opcode, without worrying about having a correct prefix structure or enough bits to encode all possible/interesting registers. The look-up happens inside the CPU, at a very deep level in the micro-architecture, and so is not really concerned by the speed discrepancy between the CPU and the memory.

If you want it to be cute, you can enforce a couple of special instructions to have a specific opcode (say 0x00 puts the CPU on HALT (and no, not necessarily, and catch fire), 0x01 for an interrupt, etc.). You could even think of reconfiguring instruction set and microcode on the fly. Maybe a by-process setting?

*
* *

Extending significantly the precision of the usual float formats, say, moving to 128 and 256 bits, poses many problems. While it is undeniable that we will need those eventually for scientific computing, it means some profound changes in FPU architectures. First, we will need wider data buses. That, we already partly have, with GPUs and newer CPUs. A 32-byte long float is smaller than a cache-line, so on that side, that’s not too bad. But what about the hardware that actually computes operations between big floats?

Well, first, I think we all agree on the fact that if we are going forward with these formats, we will not accept painfully slow implementations. This means that the hardware will expand significantly. I would suspect quadratically in the number of input bits in general, while it’s still open for debate how low the circuit complexity can get [1].

*
* *

The idea that we should learn representations rather than designing them by hand is central to data compression and machine-learning (which, in fact, can be unified at a rather deep level). In both cases, we want to extract information from the data with as little a priori knowledge as possible and let the algorithms find a representation (or, more accurately, a re-representation) of the data so that there are exploitable structures for further analysis stages. The criteria for “exploitable” may vary. In data compression, you want a representation for which you can find a short code (i.e., for which the entropy is minimized); in machine-learning it may be that you want to minimize prediction error; finding a short code and formulating an accurate prediction are not exactly the same thing but they are very similar.

[1] Amos R. Omondi — Computer Arithmetic Systems: Algorithms, Architecture, and Implementation — Prentice Hall,1994

In this last installment of the series, I consider an efficient (but approximate) algorithm based on heaps to compute the median.

A heap is an efficient tree-like data structure used to maintain, for example, priority queues. The basic invariant of a max-heap (where we are interested in knowing the largest value; there’s also a min-heap where we want to know the smallest value) is that, unless a leaf, an internal node contain a key that is larger than both its children’s keys. If this invariant is respected through all of the heap, then the root contains the maximum value contained in the heap (and, respectively for a min-heap, the minimum value). A max-heap would look something like this:

The best part is that you can make any array a heap in linear time. But, how does that help us for the median? Well, we could think of a med-heap, where, at every node (that is not a leaf), the invariant is that the node has a key that is the median amongst itself and its two children! A (very) basic med-heap class would look something like:

template <typename T>
void cmpexchg(T & a, T & b) 
 { 
  using std::swap; // ADL safe
  if (a>b) swap(a,b); 
 }

template <typename T>
const T & median3( T & a, T & b, T & c )
 {
  cmpexchg(a,c);
  cmpexchg(a,b);
  cmpexchg(b,c);
  return b;
 }

template <typename T>
class med_heap
 {
 private:

  std::vector<T> & heap;

  int left_child(int i) { return 2*i+1; }
  int right_child(int i) { return 2*i+2; }
 
  void heapify()
   {
    for (int current=heap.size()/2-1;current>-1;current--)
     {
      int left=left_child(current);
      int right=right_child(current);

      // check if it has two children
      // (otherwise give up)
      //
      if ( (left < heap.size()) && 
           (right < heap.size()))
       {
        T & a = heap[current];
        T & b = heap[left];
        T & c = heap[right];

        const T & med = median3(a,b,c);

        if (b==med)
         std::swap(a,b);
        else
         if (c==med)
          std::swap(a,c);
         else
          ; // a is already the median
       }
      else
       ; // has only one child, so
         // already "median"
     }
   }

 public:

  size_t size() const { return heap.size(); }

  // lets you peek at the next
  const T & median() const
   {
    return heap[0];
   }


  med_heap(std::vector<T> & v)
   : heap(v)
   {
    heapify();
   }
 };

Note that the class does not copy the vector, it references an already existing one (this not only avoids computing the time for allocation and copy, it is also fair to select, as it also only uses a reference to a vector).

So here the magic happens in heapify(). Using the addressing described in a previous post, it becomes simple to scan the array from the middle backwards to the beginning and enforce at each step the invariant. This takes “small” linear time, because at each entry, we only need to examine three values.

The problem is that, the median of median is not the median of the whole data, and that there will be imprecision in exchange of speed. That may be a good trade-off, as we will see.

*
* *

OK, the real contenders so far are select and the med-heap. We will compare to the stl::sort algorithm since is a bona fide comparison as 1) simple to use and 2) a likely solution for someone not wanting to reinvent the wheel (or not knowing about selection). The following shows times to find the median in an array of 1000 entries (with 1000 instances of the problem, the same for all three methods), with values on 0…9999:

From the box plots, we see that the stl::sort fares worse (in fact a lot worse) than the two other alternatives. The med-heap is also significantly faster than select, 2-3× faster.

What about accuracy? Looking at the distribution of the errors, we see that 50% of the times, the relative error is within ±5%, and 95% of the time it is less than ±20%. This is distribution seems to be rather indifferent to the range of the values, for example, with a range of 0…255, you get more or less the same results.

*
* *

In essence, therefore, you trade off accuracy (±5% error 50% of the time) for a rather interesting speed-up (2-3×) over an exact algorithm such as select, which is an interesting results on its own. Now, whether or not a med-heap is adequate for your needs is another story. You could argue that for some applications, it is necessary to have the exact algorithm, and you could make the case where, for another application, the ±5% error 50% of the time is unimportant or unnoticeable.

*
* *

Here, we only have a rather sketchy implementation of a med-heap that provides only the fun part as far as knowing the median is concerned, but we could just as easily as with a min- or max-heap, provide the necessary functions to pop the median, insert, and remove, values from the med-heap. In fact, it would be exactly the same code as with a min- or max-heap, but for the median instead of min or max.

*
* *

Full Test Code here.

*
* *

This is the 300th post.

*
* *

The STL function nth_element seems to be doing a good job at select (and a better one than me), but is still much slower than the med-heap:

(Frontières imaginées, 1920×1200)

You can find more wallpapers here

(Sibérie Minimaliste, 1920×1200)

You can find more wallpapers here

In this post, we’ll discuss the simple script I cooked up to do just that, and a bit on how Flac works, and how it compares to MP3.

First, to run the script, you will need cdparanoia and flac, both packages readily available from Debian/Ubuntu repositories (and, for the MP3 version of the script, found at the end of this post, you will need lame, also found in the default repositories).

The first thing to do is to probe the CD for its Table Of Contents (TOC hereafter). Fortunately, cdparanoia is a fine piece of software and it detects the default/loaded CD/DVD drive. It suffice to invoke it with the probe option like this

echo probing...

if cdparanoia -Q 2> cd-rip-out.tmp~
then
...

To find the drive and fetch the TOC, here written to a temporary file. Redirection from stderr is needed because cdparanoia outputs to stderr (for some reason). This temporary file is parsed to show the tracks and get their numbers:

grep -e '^[[:blank:]]*[0-9]*\.' cd-rip-out.tmp~ > cd-rip-toc.tmp~
tracks=$( cut -d. -f 1 cd-rip-toc.tmp~ )

(The exact poutine is cd-paranoia-specific), but in short, it gets the track number and forms a list such as 1 2 3 4. The user is also, later on, able to either chose the default list or specify his own, including cdparanoia-specific syntax such as 12-15 to grab tracks 12 to 15, inclusive, as a single track. We also add a couple of cosmetic details such as promoting track numbers from 1 to 001 (so that they all line up all pretty), ask for the track title, and invoke flac:

    # main grab loop
    for track in ${track_list[@]}
    do
        ((t++))
        # sed hack from Vincent "gnuvince" Foley
        beautiful_track=$(echo $track \
        | sed 's/\([0-9]*\)/000\1/g;s/[0-9]*\([0-9]\{3\}\)/\1/g')

        read -p "tracks $beautiful_track ($t/$t_l) title: " this_track_title 
        this_track_name="$artist -- $album - $beautiful_track - $this_track_title"

        if cdparanoia --never-skip=2 $track -w "$this_track_name.wav"
        then
            # grab successful!

            flac --best \
                "$this_track_name.wav" \
                --tag=artist="$artist" \
                --tag=album="$album" \
                --tag=track="$beautiful_track" \
                --tag=title="$beautiful_track $this_track_title" \
                --tag=year="$year"
        else
            echo An error occurred while grabbing $track
        fi
    done

(The ugly sed hack to convert tracks number from 1 to 001 is due to gnuvince).

Flac provides many options, but it seems that --best does indeed provide very good parameters, choosing the best compression settings regardless of CPU time (and, let us be serious, it’s rather unimportant that it takes “more” CPU time when it takes about 30s to compress an entire CD on a modern CPU—once the sound is ripped from the CD).

*
* *

Flac encodes sound losslessly, that is, if you decompress a .flac file, you get, bit for bit, the .wav you input. MP3, on the other hand, is a conspicuously lossy encoder, that is, it destroys information in order to achieve much higher compression ratios. One essential piece of a modern MP3 encoder is its psycho-acoustic model, an engine that analyzes sound and determines, with quantization enabled, what will be the perceived loss incurring from destroying this or that feature in the sound. Needless to say, the better the model is, the better sound quality can be because the encoder will be able to take the right decisions about what information to destroy—destroy sound information that you should hear the least.

But at very low bit-rates, it’s the quantization/decimation engine that dominates. To meet the very low bit-rates, the encoder will destroy plenty of information, and even if the psycho-acoustic model helps it take the right decisions while coding, the result will still be a sound with lots of artifacts, and it will not sound that good. Effects will be noticeable or even irritating. At high bit-rates, however, it will be the psycho-acoustic model that will dominate. At high bit-rate, there is no need to fiercely quantize and decimate, but the psycho-acoustic model can still force the encoder to remove features in the sound that are deemed inaudible (according to the model engine) despite having enough bits available to code them. In this regime, cranking the bit-rate will result only in marginal file size increase, because what extra bits you throw at the encoding are lost to the model that removes features from the sound.

We have seen this before, haven’t we?

Relative file sizes for --vbr-new

In the graph, we kind of see that some files do not grow as bit-rate increase, others grow only somewhat; at least, quite sub-linearly.

*
* *

With lossless coding, we keep everything, so it is crucial that we have a good prediction model, that is, an engine that can give a good prediction on the next sample from the previous samples, . If the error is (expectedly) small (well, to be exact, if it has a small entropy), then we can devise an efficient code for it and get good compression.

Flac does not use an elaborate sound model. It uses linear prediction to predict values. Linear prediction can be written as:

(or, in vector notation, —it is only a dot-product, after all.) At each time , a new set of values are computed in order to minimize the expected error (there are possible variants) for the next time step. Turns out it’s not too intensive if the window is small-ish (especially for somewhat recent CPUs), because if you need to invert a matrix, it is of a special kind, it is a Toeplitz Matrix. Because of the structure of the matrix, it as degrees of freedom rather than , and the inversion of a Toeplitz matrix (or solving a Toeplitz system; same difference) can be performed in much better than : it can be done in !

Linear Prediction Coding isn’t something new, it’s been in use in various speech coding schemes (such as the GSM Speech Codecs). What makes it attractive is that, well, it seems to work fairly well on speech and sound, and it has efficient algorithms to solve for the parameters of the linear prediction.

*
* *

Once the prediction is done, the difference is coded. In Flac, it seems that this difference is considered to be geometrically distributed, and accordingly encoded using a Golomb Code. It is the exact encoding the residual that makes the recovery of the original signal: indeed, .

*
* *

You can download the full scripts here for Flac and here for MP3 high bit-rate VBR. Feel free to tinker with them; that’s what they’re for.

*
* *

The Flac homepage is on sourceforge… Follow the link to know more about Flac.

(Buy At Amazon.com)

Contrary to what the title may indicate, this book has little to do with actual digital photography techniques for landscapes or natural wonders; it is rather a tourist guide of everything you should see in the major american national parks. After expediting the basics of landscape photography, the guide leads you into a detailed journey into the United States major national parks, giving you all the good hints as to how to reach this-or-that natural wonder, what time of year, or even the time of day will lend itself the best for photography.

While this seems moderately interesting for a foreigner (especially if one doesn’t really want to visit all the state parks), the book is entirely redeemed by Fitzharris’ absolutely superb photography. A must get, especially if you’re interested in landscape photography.

(Buy At Amazon)

This book is quite the step up from Mini Weapons Of Mass Destruction 2: not only the projects presented are a lot more interesting—flame-thrower and all—they are presented in their historical and scientific contexts, explaining the active principles of each device/contraption. While Mini Weapons was something of a kid’s book, The Practical Pyromaniac certainly isn’t. I must say it will appeal quite a lot to every one with a little envie de destruction.

Computing the median of three numbers is a simple as sorting the three numbers (an operation that can be done in constant time, after all, if comparing and swapping are constant time) and picking the middle. However, if the objects compared are “heavy”, comparing and (especially) moving them around may be expensive.

One possibility, is to use comparison, but no swapping, to find the median. Basically, for three numbers , , , we want to know which one of the six following arrangements

,

,

,

,

,

or

holds.

The goal is therefore to devise a testing procedure that determines, in an optimal number of steps, which one of the above six arrangements corresponds to the actual values of , , and . We can proceed by elimination. Testing any two pairs, say , will split the above six arrangements in two groups; one in which holds (with three arrangements) and one in which it does not (also with three arrangements). Each group can be further divided by asking another question, say, , which also creates two sub-groups, and so on, until all groups have been broken down to exactly one arrangement. In C++, that would look pretty much like:

template <typename T>
const T & median3( const T & a,
                   const T & b,
                   const T & c )
 {
   if (a<b) // {a,b,c}, {a,c,b}, {c,a,b}
     if (b<c)
      return b; // {a,b,c}
     else
      // {a,c,b}, {c,a,b}
      if (a<c)
       return c; // {a,c,b}
      else
       return a; // {c,a,b}
    else
     // {b,a,c}, {b,c,a}, {c,b,a}
     if (b<c)  // {b,a,c}, {b,c,a}
      if (a<c)
       return a; // {b,a,c}
      else
       return c; // {b,c,a}
     else
      return b; // {c,b,a}
 }

You understood that the method works (it will ask essentially questions for numbers, and it does not move object around), but also that the resulting if-tree grows rapidly in the number of items to sort. That is, good luck if you have to find the median of 25 numbers.

Moreover, this method is branch-intensive: it will branch randomly, and will be difficult to predict by the branch predictor, therefore (probably) harming performance quite a bit.

If we allow swaps (and consider them inexpensive), we can do something using sorting networks.

A sorting network can be understood as a train-yard sorting station metaphor. You have lanes, and along the lanes, you have a certain number of exchanging junctions onto other lanes. That is, if a train on a lane switches to lane , then the train on lane switches to lane .

The goal is to figure out the most efficient network (with the smallest number of switches) for a given number of lanes. Let us see what sorting networks look like.

For or , there isn’t much to do: with , you just compare the two values and swap them if one is larger than the other (that is, , if you want increasing order). In C++, the basic operation is therefore given by:

template <typename T>
void cmpexchg(T & a, T & b)
 {
  using std::swap; // ADL trick
  if (a>b) swap(a,b);
 }

and the graph representing the network is denoted as:

Where lanes are joined by lines, and circle/dots indicate that the lanes are connected through the line (this is somewhat of an electrical engineering notation, but the large dots, although redundant, make it easier to understand the graph). At , we have a network such as:

(where the color lines locate ‘sequence points’ in the network which determine which compares can be done in parallel) which would yield the code:

template <typename T>
T median3_network( T a,
                   T b,
                   T c )
 {
  cmpexchg(a,c);
  cmpexchg(a,b);
  cmpexchg(b,c);
  return b;
 }

One of the possible networks for is given by:

and finally, for , one of the possible solutions looks like:

…which is already rather more complex than with .

The advantage with a sorting network is that, in theory, a great number of comparison/exchange can be made in parallel (as they are mutually independent) and thus collapse the depth of the network significantly. Assuming that we can do up to comparison in parallel, the depth of such a network is (but with size ).

The disadvantage is that, in general, it is hard (in the co-NP-complete sense) to determine whether or not a given network is optimal for the size of the problem (or, conversely, it is NP-Complete to arrive at an optimal network)!

*
* *

So sorting networks are great if you can have parallel compare/exchange operations, but when we examine the primitive cmpexchg(T & a, T & b), we see they’re not fundamentally more efficient than the if-tree based solution. If one thing, they’re much worst if the cost of swapping is high.

*
* *

Can we use a primitive such as median3 to get the median of a much larger number of values than just three? What is a median3 of three median3?

To Be Continued…

However, if the list is sorted (in ascending order, say) and the tree is not one of the self-balancing varieties, insertion is , because the “tree” created by the successive insertions of sorted key is in fact a degenerate tree, a list. So, what if the list is already sorted and don’t really want to have a self-balancing tree? Well, it turns out that you can build a(n almost perfectly) balanced tree in .

Let us make the simplifying assumption that we can have two types of nodes in the tree: leaves, that contains the actual data, and internal nodes (or just nodes for the remainder of this post) that holds only a key.

The first strategy that comes to mind, using this assumption, is to use a method reminiscent of how Huffman Codes are constructed. That is, we scan the original list from left to right: we take two nodes (if there are at least two left), remove them from the list, and insert back in their place a new node containing the key of the second list item (remember, the list is supposed to be sorted in ascending order), with the two original nodes (or leaves) as children. That is, the operation goes something like this:

(Notice the metaphor: leaves are green, internal nodes brown.) We simply re-scan the list until we have a single remaining node, which will be the root of the tree. The procedure is simple enough, indeed. Let us work out a full example with 5 leaves:

Then we do one pass of merges:

Then another:

…and finally:

What do you notice? The three isn’t all that balanced. In fact, it’s really unbalanced. What went wrong? Well, if for Huffman Codes, this algorithm is optimal, for search trees, it is clearly not. For one thing, Huffman’s method wants to build trees with average path-lengths as close as possible to the source’s entropy; here our goal is to have path-lengths as equal as possible.

Fortunately, we can “repair” the algorithm quite easily, and it’s another code that will come to our aid: Phase-In Codes, which have average code lengths close to (as I show here).

The key observation is that if the length of the list is a power of two, then the list can be successively merged together by the above algorithm and yield a tree that has exactly depth. Therefore, if we somehow modify the list to have exactly items in it, we’ll be able to have our optimal depth. Phase-In Codes provides the mean to do so. If

with the largest and with (thus imposing the uniqueness of the solution, with ), we can merge the first nodes, and it will result in a list of nodes ( of which are now internal nodes). Once we have merged the first nodes, we stop the iteration there and restart from the beginning of the list, which is now of length , and will yield the most-equal depth tree we wanted.

Again, with : , so , we take the two first nodes

and merge them:

and voilà, we proceed with the successive merges:

and then

and we have a tree of almost equal depth, with average depth of (while the actual ) rather than , a considerable improvement, even for this trivial tree.

*
* *

While it is not the customary way of representing a tree in memory, the segregation of leaves and internal nodes may be justified in the context where the leaves and the tree itself live in different memory locations. For example, if the leaves are rather large compared to just the key, they may have to reside in external memory—that is, the disk—while you will still like to keep the index, the tree structure itself, in memory.

One could also argue that this scheme is cache-friendlier as one can use some kind of compact tree storage to maintain the keys and the structure of the tree in a small amount of memory, and with keys that are nearby in the tree being laid-out nearby in memory, thus reducing cache misses quite a lot.

In all cases, we still have reduced the complexity of building the initial tree from to essentially ,

(Buy at Amazon.com)

A quite amusing little books on needlessly complicated hacks, but that can bring quite the ruckus in the office/school. Q-Tip launchers, (paper) ninja stars, rubberband weapons, CD periscopes, … all built from readily available office supplies. In fact, they are all way too complicated, but sooo much fun.