In this last installment of the series, I consider an efficient (but approximate) algorithm based on heaps to compute the median.

A heap is an efficient tree-like data structure used to maintain, for example, priority queues. The basic invariant of a max-heap (where we are interested in knowing the largest value; there’s also a min-heap where we want to know the smallest value) is that, unless a leaf, an internal node contain a key that is larger than both its children’s keys. If this invariant is respected through all of the heap, then the root contains the maximum value contained in the heap (and, respectively for a min-heap, the minimum value). A max-heap would look something like this:

The best part is that you can make any array a heap in linear time. But, how does that help us for the median? Well, we could think of a med-heap, where, at every node (that is not a leaf), the invariant is that the node has a key that is the median amongst itself and its two children! A (very) basic med-heap class would look something like:

template <typename T>
void cmpexchg(T & a, T & b) 
 { 
  using std::swap; // ADL safe
  if (a>b) swap(a,b); 
 }

template <typename T>
const T & median3( T & a, T & b, T & c )
 {
  cmpexchg(a,c);
  cmpexchg(a,b);
  cmpexchg(b,c);
  return b;
 }

template <typename T>
class med_heap
 {
 private:

  std::vector<T> & heap;

  int left_child(int i) { return 2*i+1; }
  int right_child(int i) { return 2*i+2; }
 
  void heapify()
   {
    for (int current=heap.size()/2-1;current>-1;current--)
     {
      int left=left_child(current);
      int right=right_child(current);

      // check if it has two children
      // (otherwise give up)
      //
      if ( (left < heap.size()) && 
           (right < heap.size()))
       {
        T & a = heap[current];
        T & b = heap[left];
        T & c = heap[right];

        const T & med = median3(a,b,c);

        if (b==med)
         std::swap(a,b);
        else
         if (c==med)
          std::swap(a,c);
         else
          ; // a is already the median
       }
      else
       ; // has only one child, so
         // already "median"
     }
   }

 public:

  size_t size() const { return heap.size(); }

  // lets you peek at the next
  const T & median() const
   {
    return heap[0];
   }


  med_heap(std::vector<T> & v)
   : heap(v)
   {
    heapify();
   }
 };

Note that the class does not copy the vector, it references an already existing one (this not only avoids computing the time for allocation and copy, it is also fair to select, as it also only uses a reference to a vector).

So here the magic happens in heapify(). Using the addressing described in a previous post, it becomes simple to scan the array from the middle backwards to the beginning and enforce at each step the invariant. This takes “small” linear time, because at each entry, we only need to examine three values.

The problem is that, the median of median is not the median of the whole data, and that there will be imprecision in exchange of speed. That may be a good trade-off, as we will see.

*
* *

OK, the real contenders so far are select and the med-heap. We will compare to the stl::sort algorithm since is a bona fide comparison as 1) simple to use and 2) a likely solution for someone not wanting to reinvent the wheel (or not knowing about selection). The following shows times to find the median in an array of 1000 entries (with 1000 instances of the problem, the same for all three methods), with values on 0…9999:

From the box plots, we see that the stl::sort fares worse (in fact a lot worse) than the two other alternatives. The med-heap is also significantly faster than select, 2-3× faster.

What about accuracy? Looking at the distribution of the errors, we see that 50% of the times, the relative error is within ±5%, and 95% of the time it is less than ±20%. This is distribution seems to be rather indifferent to the range of the values, for example, with a range of 0…255, you get more or less the same results.

*
* *

In essence, therefore, you trade off accuracy (±5% error 50% of the time) for a rather interesting speed-up (2-3×) over an exact algorithm such as select, which is an interesting results on its own. Now, whether or not a med-heap is adequate for your needs is another story. You could argue that for some applications, it is necessary to have the exact algorithm, and you could make the case where, for another application, the ±5% error 50% of the time is unimportant or unnoticeable.

*
* *

Here, we only have a rather sketchy implementation of a med-heap that provides only the fun part as far as knowing the median is concerned, but we could just as easily as with a min- or max-heap, provide the necessary functions to pop the median, insert, and remove, values from the med-heap. In fact, it would be exactly the same code as with a min- or max-heap, but for the median instead of min or max.

*
* *

Full Test Code here.

*
* *

This is the 300th post.

*
* *

The STL function nth_element seems to be doing a good job at select (and a better one than me), but is still much slower than the med-heap:

(Frontières imaginées, 1920×1200)

You can find more wallpapers here

(Sibérie Minimaliste, 1920×1200)

You can find more wallpapers here

In this post, we’ll discuss the simple script I cooked up to do just that, and a bit on how Flac works, and how it compares to MP3.

First, to run the script, you will need cdparanoia and flac, both packages readily available from Debian/Ubuntu repositories (and, for the MP3 version of the script, found at the end of this post, you will need lame, also found in the default repositories).

The first thing to do is to probe the CD for its Table Of Contents (TOC hereafter). Fortunately, cdparanoia is a fine piece of software and it detects the default/loaded CD/DVD drive. It suffice to invoke it with the probe option like this

echo probing...

if cdparanoia -Q 2> cd-rip-out.tmp~
then
...

To find the drive and fetch the TOC, here written to a temporary file. Redirection from stderr is needed because cdparanoia outputs to stderr (for some reason). This temporary file is parsed to show the tracks and get their numbers:

grep -e '^[[:blank:]]*[0-9]*\.' cd-rip-out.tmp~ > cd-rip-toc.tmp~
tracks=$( cut -d. -f 1 cd-rip-toc.tmp~ )

(The exact poutine is cd-paranoia-specific), but in short, it gets the track number and forms a list such as 1 2 3 4. The user is also, later on, able to either chose the default list or specify his own, including cdparanoia-specific syntax such as 12-15 to grab tracks 12 to 15, inclusive, as a single track. We also add a couple of cosmetic details such as promoting track numbers from 1 to 001 (so that they all line up all pretty), ask for the track title, and invoke flac:

    # main grab loop
    for track in ${track_list[@]}
    do
        ((t++))
        # sed hack from Vincent "gnuvince" Foley
        beautiful_track=$(echo $track \
        | sed 's/\([0-9]*\)/000\1/g;s/[0-9]*\([0-9]\{3\}\)/\1/g')

        read -p "tracks $beautiful_track ($t/$t_l) title: " this_track_title 
        this_track_name="$artist -- $album - $beautiful_track - $this_track_title"

        if cdparanoia --never-skip=2 $track -w "$this_track_name.wav"
        then
            # grab successful!

            flac --best \
                "$this_track_name.wav" \
                --tag=artist="$artist" \
                --tag=album="$album" \
                --tag=track="$beautiful_track" \
                --tag=title="$beautiful_track $this_track_title" \
                --tag=year="$year"
        else
            echo An error occurred while grabbing $track
        fi
    done

(The ugly sed hack to convert tracks number from 1 to 001 is due to gnuvince).

Flac provides many options, but it seems that --best does indeed provide very good parameters, choosing the best compression settings regardless of CPU time (and, let us be serious, it’s rather unimportant that it takes “more” CPU time when it takes about 30s to compress an entire CD on a modern CPU—once the sound is ripped from the CD).

*
* *

Flac encodes sound losslessly, that is, if you decompress a .flac file, you get, bit for bit, the .wav you input. MP3, on the other hand, is a conspicuously lossy encoder, that is, it destroys information in order to achieve much higher compression ratios. One essential piece of a modern MP3 encoder is its psycho-acoustic model, an engine that analyzes sound and determines, with quantization enabled, what will be the perceived loss incurring from destroying this or that feature in the sound. Needless to say, the better the model is, the better sound quality can be because the encoder will be able to take the right decisions about what information to destroy—destroy sound information that you should hear the least.

But at very low bit-rates, it’s the quantization/decimation engine that dominates. To meet the very low bit-rates, the encoder will destroy plenty of information, and even if the psycho-acoustic model helps it take the right decisions while coding, the result will still be a sound with lots of artifacts, and it will not sound that good. Effects will be noticeable or even irritating. At high bit-rates, however, it will be the psycho-acoustic model that will dominate. At high bit-rate, there is no need to fiercely quantize and decimate, but the psycho-acoustic model can still force the encoder to remove features in the sound that are deemed inaudible (according to the model engine) despite having enough bits available to code them. In this regime, cranking the bit-rate will result only in marginal file size increase, because what extra bits you throw at the encoding are lost to the model that removes features from the sound.

We have seen this before, haven’t we?

Relative file sizes for --vbr-new

In the graph, we kind of see that some files do not grow as bit-rate increase, others grow only somewhat; at least, quite sub-linearly.

*
* *

With lossless coding, we keep everything, so it is crucial that we have a good prediction model, that is, an engine that can give a good prediction on the next sample from the previous samples, . If the error is (expectedly) small (well, to be exact, if it has a small entropy), then we can devise an efficient code for it and get good compression.

Flac does not use an elaborate sound model. It uses linear prediction to predict values. Linear prediction can be written as:

(or, in vector notation, —it is only a dot-product, after all.) At each time , a new set of values are computed in order to minimize the expected error (there are possible variants) for the next time step. Turns out it’s not too intensive if the window is small-ish (especially for somewhat recent CPUs), because if you need to invert a matrix, it is of a special kind, it is a Toeplitz Matrix. Because of the structure of the matrix, it as degrees of freedom rather than , and the inversion of a Toeplitz matrix (or solving a Toeplitz system; same difference) can be performed in much better than : it can be done in !

Linear Prediction Coding isn’t something new, it’s been in use in various speech coding schemes (such as the GSM Speech Codecs). What makes it attractive is that, well, it seems to work fairly well on speech and sound, and it has efficient algorithms to solve for the parameters of the linear prediction.

*
* *

Once the prediction is done, the difference is coded. In Flac, it seems that this difference is considered to be geometrically distributed, and accordingly encoded using a Golomb Code. It is the exact encoding the residual that makes the recovery of the original signal: indeed, .

*
* *

You can download the full scripts here for Flac and here for MP3 high bit-rate VBR. Feel free to tinker with them; that’s what they’re for.

*
* *

The Flac homepage is on sourceforge… Follow the link to know more about Flac.

(Buy At Amazon.com)

Contrary to what the title may indicate, this book has little to do with actual digital photography techniques for landscapes or natural wonders; it is rather a tourist guide of everything you should see in the major american national parks. After expediting the basics of landscape photography, the guide leads you into a detailed journey into the United States major national parks, giving you all the good hints as to how to reach this-or-that natural wonder, what time of year, or even the time of day will lend itself the best for photography.

While this seems moderately interesting for a foreigner (especially if one doesn’t really want to visit all the state parks), the book is entirely redeemed by Fitzharris’ absolutely superb photography. A must get, especially if you’re interested in landscape photography.

(Buy At Amazon)

This book is quite the step up from Mini Weapons Of Mass Destruction 2: not only the projects presented are a lot more interesting—flame-thrower and all—they are presented in their historical and scientific contexts, explaining the active principles of each device/contraption. While Mini Weapons was something of a kid’s book, The Practical Pyromaniac certainly isn’t. I must say it will appeal quite a lot to every one with a little envie de destruction.

Computing the median of three numbers is a simple as sorting the three numbers (an operation that can be done in constant time, after all, if comparing and swapping are constant time) and picking the middle. However, if the objects compared are “heavy”, comparing and (especially) moving them around may be expensive.

One possibility, is to use comparison, but no swapping, to find the median. Basically, for three numbers , , , we want to know which one of the six following arrangements

,

,

,

,

,

or

holds.

The goal is therefore to devise a testing procedure that determines, in an optimal number of steps, which one of the above six arrangements corresponds to the actual values of , , and . We can proceed by elimination. Testing any two pairs, say , will split the above six arrangements in two groups; one in which holds (with three arrangements) and one in which it does not (also with three arrangements). Each group can be further divided by asking another question, say, , which also creates two sub-groups, and so on, until all groups have been broken down to exactly one arrangement. In C++, that would look pretty much like:

template <typename T>
const T & median3( const T & a,
                   const T & b,
                   const T & c )
 {
   if (a<b) // {a,b,c}, {a,c,b}, {c,a,b}
     if (b<c)
      return b; // {a,b,c}
     else
      // {a,c,b}, {c,a,b}
      if (a<c)
       return c; // {a,c,b}
      else
       return a; // {c,a,b}
    else
     // {b,a,c}, {b,c,a}, {c,b,a}
     if (b<c)  // {b,a,c}, {b,c,a}
      if (a<c)
       return a; // {b,a,c}
      else
       return c; // {b,c,a}
     else
      return b; // {c,b,a}
 }

You understood that the method works (it will ask essentially questions for numbers, and it does not move object around), but also that the resulting if-tree grows rapidly in the number of items to sort. That is, good luck if you have to find the median of 25 numbers.

Moreover, this method is branch-intensive: it will branch randomly, and will be difficult to predict by the branch predictor, therefore (probably) harming performance quite a bit.

If we allow swaps (and consider them inexpensive), we can do something using sorting networks.

A sorting network can be understood as a train-yard sorting station metaphor. You have lanes, and along the lanes, you have a certain number of exchanging junctions onto other lanes. That is, if a train on a lane switches to lane , then the train on lane switches to lane .

The goal is to figure out the most efficient network (with the smallest number of switches) for a given number of lanes. Let us see what sorting networks look like.

For or , there isn’t much to do: with , you just compare the two values and swap them if one is larger than the other (that is, , if you want increasing order). In C++, the basic operation is therefore given by:

template <typename T>
void cmpexchg(T & a, T & b)
 {
  using std::swap; // ADL trick
  if (a>b) swap(a,b);
 }

and the graph representing the network is denoted as:

Where lanes are joined by lines, and circle/dots indicate that the lanes are connected through the line (this is somewhat of an electrical engineering notation, but the large dots, although redundant, make it easier to understand the graph). At , we have a network such as:

(where the color lines locate ‘sequence points’ in the network which determine which compares can be done in parallel) which would yield the code:

template <typename T>
T median3_network( T a,
                   T b,
                   T c )
 {
  cmpexchg(a,c);
  cmpexchg(a,b);
  cmpexchg(b,c);
  return b;
 }

One of the possible networks for is given by:

and finally, for , one of the possible solutions looks like:

…which is already rather more complex than with .

The advantage with a sorting network is that, in theory, a great number of comparison/exchange can be made in parallel (as they are mutually independent) and thus collapse the depth of the network significantly. Assuming that we can do up to comparison in parallel, the depth of such a network is (but with size ).

The disadvantage is that, in general, it is hard (in the co-NP-complete sense) to determine whether or not a given network is optimal for the size of the problem (or, conversely, it is NP-Complete to arrive at an optimal network)!

*
* *

So sorting networks are great if you can have parallel compare/exchange operations, but when we examine the primitive cmpexchg(T & a, T & b), we see they’re not fundamentally more efficient than the if-tree based solution. If one thing, they’re much worst if the cost of swapping is high.

*
* *

Can we use a primitive such as median3 to get the median of a much larger number of values than just three? What is a median3 of three median3?

To Be Continued…

However, if the list is sorted (in ascending order, say) and the tree is not one of the self-balancing varieties, insertion is , because the “tree” created by the successive insertions of sorted key is in fact a degenerate tree, a list. So, what if the list is already sorted and don’t really want to have a self-balancing tree? Well, it turns out that you can build a(n almost perfectly) balanced tree in .

Let us make the simplifying assumption that we can have two types of nodes in the tree: leaves, that contains the actual data, and internal nodes (or just nodes for the remainder of this post) that holds only a key.

The first strategy that comes to mind, using this assumption, is to use a method reminiscent of how Huffman Codes are constructed. That is, we scan the original list from left to right: we take two nodes (if there are at least two left), remove them from the list, and insert back in their place a new node containing the key of the second list item (remember, the list is supposed to be sorted in ascending order), with the two original nodes (or leaves) as children. That is, the operation goes something like this:

(Notice the metaphor: leaves are green, internal nodes brown.) We simply re-scan the list until we have a single remaining node, which will be the root of the tree. The procedure is simple enough, indeed. Let us work out a full example with 5 leaves:

Then we do one pass of merges:

Then another:

…and finally:

What do you notice? The three isn’t all that balanced. In fact, it’s really unbalanced. What went wrong? Well, if for Huffman Codes, this algorithm is optimal, for search trees, it is clearly not. For one thing, Huffman’s method wants to build trees with average path-lengths as close as possible to the source’s entropy; here our goal is to have path-lengths as equal as possible.

Fortunately, we can “repair” the algorithm quite easily, and it’s another code that will come to our aid: Phase-In Codes, which have average code lengths close to (as I show here).

The key observation is that if the length of the list is a power of two, then the list can be successively merged together by the above algorithm and yield a tree that has exactly depth. Therefore, if we somehow modify the list to have exactly items in it, we’ll be able to have our optimal depth. Phase-In Codes provides the mean to do so. If

with the largest and with (thus imposing the uniqueness of the solution, with ), we can merge the first nodes, and it will result in a list of nodes ( of which are now internal nodes). Once we have merged the first nodes, we stop the iteration there and restart from the beginning of the list, which is now of length , and will yield the most-equal depth tree we wanted.

Again, with : , so , we take the two first nodes

and merge them:

and voilà, we proceed with the successive merges:

and then

and we have a tree of almost equal depth, with average depth of (while the actual ) rather than , a considerable improvement, even for this trivial tree.

*
* *

While it is not the customary way of representing a tree in memory, the segregation of leaves and internal nodes may be justified in the context where the leaves and the tree itself live in different memory locations. For example, if the leaves are rather large compared to just the key, they may have to reside in external memory—that is, the disk—while you will still like to keep the index, the tree structure itself, in memory.

One could also argue that this scheme is cache-friendlier as one can use some kind of compact tree storage to maintain the keys and the structure of the tree in a small amount of memory, and with keys that are nearby in the tree being laid-out nearby in memory, thus reducing cache misses quite a lot.

In all cases, we still have reduced the complexity of building the initial tree from to essentially ,

(Buy at Amazon.com)

A quite amusing little books on needlessly complicated hacks, but that can bring quite the ruckus in the office/school. Q-Tip launchers, (paper) ninja stars, rubberband weapons, CD periscopes, … all built from readily available office supplies. In fact, they are all way too complicated, but sooo much fun.

One way to limit the influence of the outliers for denoising is to use the median. However, computing the median is usually more tricky than computing the average, or mean, and this first post (in a series of three, in the next few weeks), discusses how to compute the median efficiently using the selection algorithm.

*
* *

Before continuing, let us say a word on why the average is sensitive to outliers. There’s of course the intuitive explanation that if you have three values, say, 1,2, and 10, then the average is , which is rather far away of the majority of the samples that are grouped around 1 and 2. A less intuitive, but also rather straightforward reason is that the average sample average is the solution to the least square equation

that is, it is the value that minimizes the square error between itself and all the sample values. If there’s an exceedingly large , it will pull the average towards itself quite a lot.

The median, say , is, on the other hand, the solution to the equation

that is, the value that minimizes absolute value. An overly large will not pull on the solution at all if we use the sample median, which is defined as the middle , if all are sorted. Basically, median (mostly) ignore outliers, which is something we might want for filtering.

*
* *

Since we’re interested in the sample median, we must find a way of finding the the that occupies middle rank. Usually, the median is defined for if is odd, and if is even (or sometime the average of the two sample lying at the center is used, but that’s a minor detail for what we will discuss).

A first idea would be to sort the list, in using a robust sorting algorithm such as Quicksort, or even in linear time if one can use Radix Sort. But Radix Sort is a “big” linear time algorithm because it depends on the size of the machine-specific integers you’re using as a key. Would it even be that sorting completely the list is overkill, because we’re not interested in sorting them all, just in getting the middle value, when (sufficiently) sorted?

So how do we sort an array “just enough” to assess the median value?

Well, it turns out that Quicksort will lend us quite a hand here. Remember, Quicksort sorts quickly by dividing the array in two sub-arrays, with one on the left having all values smaller than the pivot (a value picked within the array) and with the other on the right with values larger than or equal to the pivot. If the pivot is chosen wisely, both sub-arrays should be roughly equal in length. Recursion on each sub-array will do again the same thing, all the way down where arrays are so small as to contain only one element. At that point, when recursion visited all possible sub-arrays, the array is sorted and the algorithm terminates.

So what if we only sorted one side, the side that might contain the median? The idea is to use the Quicksort partition algorithm (and there are more than one, but that’s more or less irrelevant right now) and look at the two resulting sub-array. Is the median position, somewhere in the left or the right sub-array? If it’s in the left, we reapply only on the left sub-array; if it’s in the right, we reapply only on the right sub-array. Again, we will obtain two sub-sub-arrays, and we check whether is in the left sub-sub-array or the right sub-sub-array. We continue splitting until we create a sub-sub-…-sub-array of size one, in position : it will contain the median value!

A minimal C++ implementation would probably look something like this:

////////////////////////////////////////
//
// Implements the generic selection
// algorithm loosely based on the
// QuickSort partition algorithm
// (more or less any of the QuickSort
// partition algorithms will do just
// fine)
//
// (it may be due to Wirth, from his
// book algorithms + data structures
// = programs)
//
// NOTE: array v gets modified in the
// process!
//
template <typename T>
const T & select( std::vector<T> & v,
                  int select_index)
  {
   // check if lo_index <= select_index <= hi_index
   // ...eventually
   //
   int lo_index=0;
   int hi_index=v.size()-1;

   while (hi_index>lo_index)
    {
     // classic mid-point
     // pivot selection (greatly
     // reduces chances of n^2
     // behavior if v is
     // nearly sorted, as it would
     // be after a few call to select.
     //
     const T pivot = v[(lo_index+hi_index)/2];

     int x=lo_index;
     int y=hi_index;

     // basic pivot/partition
     // algorithm
     //
     while (x<=y)
      {
       while (v[x]<pivot) x++;
       while (v[y]>pivot) y--;
       if (x<=y)
        {
         std::swap(v[x],v[y]);
         x++;
         y--;
        }
      }

     // check on which side of
     // the partition lands the
     // select_index, so that we
     // iterate only on the half
     // that contains the select_index
     //
     if (select_index >= x)
      lo_index=x;
     else
      hi_index=x-1;
    }

   // in our application, the actual
   // color isn't very important, unless
   // where at the last stage
   return v[select_index];
  }

You would invoke the above using, say

std::vector<T> v;

//...fill v....

T med = select(v,v.size()/2);

for some type T.

*
* *

The above method has an expected run-time , but, just like Quicksort, it can run as high as , which is quite bad. The choice of the pivot in the middle of the current sub-array reduces this risk quite a lot (and it would take quite an evil array to push it back to behavior). Furthermore, each time select is called on the array,it gets more and more sorted, making the choice of the pivot better and better with time.

Furthermore, it does swap elements multiple times (but only a number of times expected linear in ), and, depending on the size of the array and the partition method (as I said, there are many variants, for example above is probably Hoare’s, but there is also Lomuto’s method, discussed in Cormen et al‘s Introduction to Algorithms), might not be cache-friendly at all.

In the next post, we will examine other sort-like algorithms to obtain the median, maybe at lower cost.

To Be Continued…