For the past several days, I’ve been working my way through Prof. N.G. de Bruijn’s book Asymptotic Methods in Analysis; and I want to share some of the fun I’ve had reading it.

This post is not a review or anything (here’s an image of the back cover with some reviews). Below are just fragments of what I’ve read so far and found fascinating. You could also consider this my attempt at trying to persuade you to get a copy of this beautiful little book! Sadly, new copies appear not to be available at most online bookstores, so it might be hard to get one.

Prof. de Bruijn, his cycles, and average height of trees

This book is by the same Prof. de Bruijn well-known, among other things, for his cycles, which are defined as following: An -ary de Bruijn cycle is a sequence of radix- digits such that every combinations of digits occurs consecutively in the cycle. For example, one possible binary de Bruijn cycle of length is:

Treating this sequence as a cycle, notice that each of the 16 possible 4-bit patterns (0000 … 1111) appear exactly once in it. Knuth describes at least three interesting algorithms for generating such cycles in TAOCP Section 7.2.1.1 (printed in Volume 4, Fascicle 2 and downloadable as pre-fascicle 2a).

There’s surely lots to say about Prof. de Bruijn’s work (including his cycles), but I’ll reserve them for future posts. Take a look, for example, at the index of TAOCP Vol I (perhaps also in other volumes) and you’ll see references to many of his results, sometimes framed as exercises. And if you enjoy asymptotic calculations and analyses of algorithms, you would find it interesting to read his co-authored paper (with Knuth and Rice) where it is shown that the average height of planted plane trees is:

.

As an interesting note, Knuth in his musings on The Joy of Asymptotics (30 May 2000) said:

I’ve dedicated this book [Selected Papers on Analysis of Algorithms] to Prof. de Bruijn … in the Netherlands because he has been my guru … Ever since I got my PhD, I considered him my post-graduate doctoral advisor. Every time I had a problem that I got stuck on, I would write to him, and he would write me back a letter that would help me get unstuck. And that’s been going on for over fourty years now. So I dedicated this book to him. And what I want to talk to you about today is one of the things he has spent most of his life on — asymptotic methods.

By the way, that paper deriving the cool formula for the height of trees is included in these Selected Papers (Ch. 15).

The Book: A Pragmatic Exposition

I had got myself a copy of this book several years ago after first reading about it in the section on asymptotic calculations in TAOCP (Sec 1.2.11.3). But unfortunately, the book suffered the same fate as countless others — it just kept resting on my bookshelf. But a few days ago, I decided to give it a try.

A brief history. As described in its preface, this book grew out of lectures given in 1954–55 and was first published in 1958. The Dover edition I have is a reprint of the third edition from 1970.

Most computer programmers are familiar with the Big-O notation, or the Bachmann-Landau notation, for analyzing space/time-complexity of an algorithm. Among other things, this book talks of the tricks involved in getting precise asymptotic bounds once we have reduced the problem at hand to, say, a sum or an integral.

This is a pragmatic book — its focus is on demonstrating general techniques by working out several examples in detail (sometimes in more than one way) rather than expounding highly generalized theories of analysis. In this context, I found these lines from the preface reassuring:

Usually in mathematics one has to choose between saying more and more about less and less on the one hand, and saying less and less about more and more on the other. In a practical subject like this, however, the proper attitude should be to say as much as possible in a given situation.

From my reading, at certain places it, of necessity, assumes a basic knowledge of analysis and complex variable theory. Other than that, it is a fairly elementary treatment. (Note: as Feynman made a quip in one of his lectures, “Elementary” means that very little is required to know ahead of time in order to understand it, except to have an infinite amount of intelligence.”)

It’s a small book (200 pages) with very few exercises and so can be read in about two weeks, if you read an hour a day. And since there are lots of cliff-hanging moments (well, to the extent a book on analysis can have!) and the reader is left wondering how a tricky sum/integral would be tamed, your might be compelled to finish it even sooner!

de Bruijn has a good sense of humor too. At the start of the first chapter, when trying to answer the question “What is Asymptotics?”, he says:

The safest and not the vaguest definition is the following one: Asymptotics is that part of analysis which considers problems of the type dealt with in this book.

Didn’t see that one coming!

And while the book is tiny, it covers lots of tricks & techniques. So, to give my interested readers a taste, I’ve picked a couple of my favorite from the first chapter. All examples are de Bruijn’s.

Uniform Estimates

Uniformity of estimates is this little thing that keeps popping up every so often when analyzing the asymptotics of some characteristic of an algorithm. But it is rarely explained what it means or why uniformity is crucial to the application at hand. Grown up boys and girls are supposed to know what uniformity is, I suppose.

Before looking at uniformity, let’s recap the definition of the -notation. We say:

.

the -notation implies two implicit constants

Returning to the question of uniformity, suppose is a positive integer and and are arbitrary functions. Then:

, because,

the implicit “constants” and depend on the parameter

On the other hand, in the asymptotic relation (with again being a positive integer):

.

Hence if we write , with , we can choose to be 1. In such a case — when the hidden constants in the -notation do not depend on the parameter — the estimate is called uniform. This requirement is much like that of uniform convergence or uniform continuity in analysis.

So why trouble oneself with uniformity? Here’s one case where uniform estimates prove helpful: In “balancing” the asymptotic terms of a sum evenly to achieve a tight bound.

For instance, if we have somehow shown that with and where is a parameter. If the formula holds uniformly, we can get a tight bound by reasoning as follows: When is small, the first part, , is small, but the second part, , is large; the reverse holds if is large. Hence we wish to find a “balance” that minimizes the sum of the terms. And this happens when both the terms are asymptotically equivalent. So to achieve such a balance, we simply equate the two expressions , giving , and we finally get .

Notice that because of uniformity, the four hidden constants (, , , ) of the two do not vary when we vary the parameter . If the two original estimates had not been uniform; that is, had their hidden constants (implied by the -notation) varied with , we wouldn’t have been able to apply this simplistic “balancing” idea. For instance, while for small , the term might have been small, the hidden constant in might have grown, leading to unpredictable overall behavior.

The good thing about uniform estimates is that they are sometimes more useful; but proving uniformity might require a more careful analysis.

Asymptotic Series and Convergence: Some Curious Facts

An asymptotic series or asymptotic expansion for a function is a formula like:

, and,

.

A large class of examples of asymptotic expansions are convergent power series, like:

.

But there could be asymptotic expansions that are not convergent power series. And with such asymptotic expansions, some curious things can happen:

• The series of the asymptotic expansion need not be convergent.
• If the series does converge, its sum need not be equal to
• It is even possible to have and ’s such that the sum of the series does not have the series as its own asymptotic expansion!

The essential reason for these seemingly strange facts, as de Bruijn explains, is:

… that convergence means something for some fixed whereas the -formulas are not concerned with , but with . Convergence of the series, for all , say, means that for every individual there is a statement about the case . On the other hand, the statement that the series is the asymptotic expansion of means that for every individual there is a statement about the case .

For example, when , the asymptotic expansion of is:

,

So…

The book covers lots of useful stuff. In particular, It talks about Lagrange’s inversion formula (using which we can solve for the “tree function”), it has a especially thorough treatment of the saddle-point method. And much much more.

My experience with Project Euler Problems

As I attempted to solve more and more problems, I found a disproportionately large number of them centered around the theory of numbers. These involved important ideas of number sieves, congruences, continued fractions, the Farey series, solving Pell’s equation, implementing Shanks Tonelli algorithm etc., and had me sifting through Hardy and Wright’s book every so often. But after a while they weren’t so much fun. I did, however, find enough problems of my interest to get me hooked — enumeration and dynamic programming (208, 161, 209, 172, 169, 215, 219), probability (227, 213) and those occasionally unique ones like 197 (concerning the “steady-state” behavior of a certain sequence). Working through these problems made me realize what a treasure PE was. Kudos to Colin “Euler” Hughes and the PE-team for their effort in running this great site!

In addition to having the fun of solving the problems myself, I could study the solutions worked out by other members in the forum. Seeing the elegance, efficiency and analyses of some of their solutions was a rewarding (even if a bit humbling) experience. A case in point is the solution to Robot Walks (208) by sajninredoc and stijn263 (among others), where they reduce the enumeration problem to a single summation. And then there were those APL/J programmers with their cute one-liners!

In this entry, I shall outline my solutions (and their performance characteristics) to the Trominoes Tiling (161) and Number Mind (185) problems. To solve these problems, I used the ZDD techniques I had just studied in Knuth’s pre-fascicle 1B (now in print as Vol 4 Fascicle 1). I had blogged earlier on Knuth’s Fun with ZDDs musing.

Trominoes Tiling (161): Enumerating Exact Covers using ZDDs

Trominoes Tiling (161) is almost the tiling problem of TAOCP 7.1.4 — (130) with the difference that only trominoes are allowed (no monominoes or dominoes) and the grid-size is slightly larger (9x12 instead of 8x8). I reuseed, with the obvious changes, the ZDD routines that I had coded while working out that section. Since Knuth has already explained the ideas involved so beautifully (see the text around 7.1.4 — (129)), I shall only briefly sketch the ZDD construction before giving some performance statistics.

To begin, we first model the tiling problem as an Exact Cover. This involves creating a boolean matrix of 9x12 = 108 columns (corresponding to cells of the board) and 526 rows (corresponding to the ways of placing the L and I trominoes in all possible orientations on a 9x12 grid). We have iff tromino placement occupies cell . The strategy for placement numbering I used is shown below for the simpler 4x4 case (the cells are numbered in the usual row-major order). Since the placement strategy decides the variable-ordering of the ZDD and hence its size (unless, of course, we choose to sift/reorder variables later on), it is important to pick a placement strategy that is not too inefficient.

Tromino Placement Numbering for a 4x4 grid (also used for the rows of the exact cover matrix and ZDD variable ordering)

Exact Cover Matrix (for the simplified 4x4 case)

Having constructed the boolean matrix, to enumerate the tilings, we find the number of ways to select some rows of the matrix such that if we inspect any column, precisely one of the selected rows contains a 1 in that column. Hence the name “exact” cover — we neither want to leave any cell uncovered, nor do we want parts of two or more trominoes to overlap.

Exact covers can be enumerated using Knuth’s Algorithm X — an efficient backtracking technique implemented using an idea Knuth calls Dancing Links (DANCE program, paper at arXiv, Computer Musing video, ASL Implementations for dancing_links and toroid_node_t). Algorithm X not only enumerates the solution, it in fact generates them all!

To enumerate exact covers using ZDDs, using the matrix produced in the step above, we construct the boolean function (Eq. 7.1.4 — (129)):

For various ways to efficiently construct the above ZDD, see Exercise 7.1.4 — 212. The function can itself be implemented using Exercise 207’s “Symmetrizing” operation.

Once we have the ZDD for the boolean function above, the number of solutions, i.e., the number of vectors that make true (which are precisely the vectors representing exact covers) can be readily found using the ZDD-analog of Algorithm 7.1.4C.

Runtime performance of the solution: The resulting ZDD involved 526 variables and had a size of 7893743 zeads. Without invoking any garbage collection, the peak memory usage was ~600MB (where each zead in my implementation was a 20-byte node); time taken was 34s (user) and 85s (elapsed). Given that my memos weren’t very optimized (I had used GCC’s std::map) and neither had I tried doing any variable reordering, the performance seemed reasonable.

A dynamic programming approach (like one I later used for Crack-free Walls (215)), should have been able to give the results in about a second (this was confirmed by posts on the forum). Nevertheless, the ZDD solution was fast enough to keep my conscience clear of any violations of Project Euler’s one-minute-rule.

Aside: The Crack-free Walls (215) problem is kind of like TAOCP Exercise 7.1.4 — 214 (Knuth calls it “faultfree”) and should be amenable to the ZDD attack. There, however, appears to be a danger of hitting space-out since the grid-size 10x32 is somewhat large. I’ve not tried this approach.

Number Mind (185): Using ZDDs to satisfy an ad-hoc set of constraints

Number Mind (185) was the other PE problem that I solved with ZDDs. In this problem we’re to uncover a 16-digit number given a set of “guesses” of the form:

5616185650518293 ;2 correct
3847439647293047 ;1 correct
5855462940810587 ;3 correct
. . .

The guesses along with the “hit-rates” provide partial information about the secret number. Our aim is to find the “secret” number corresponding to the set of guesses.

To solve this problem, we create variables representing the condition that ^th digit () is (). We then have the following constraints:

• Since each position can hold just one digit, for each exactly one of can be true. Constraints of this kind correspond to terms like , where is again our friend, the symmetric boolean function.
• Each of the given guess “hit-rates” must be satisfied. As an example, the third constraint “5855462940810587 (3 correct)” is represented as:
  

Using the symmetrizing operator from Exercise 7.1.4 — 207, both the above kinds of constraints are easily represented. Finally, we compute the AND (or, INTERSECT, if one prefers family-of-subsets point-of-view) of the individual constraints — and we’re left with the final ZDD representing the family of feasible solutions (in our case, the solution in fact turns out to be unique).

Runtime performance of the solution: The ZDD had 160 variables , program execution had a peak memory usage of ~1GB without any garbage collection or reordering (zead-size 20-bytes), size of the largest partial function was 4665450 zeads. The running time was ~16s (both user and elapsed).

Conclusion

Comparing the ZDD solutions to the two PE problems, I think it was the kind of “unstructured” problem like Number Mind where the ZDD technique really shined.

In this entry, I’ll attempt to record the important ideas Knuth presented in his 14^th Annual Christmas Tree Lecture, part of his regular Computer Musings.

While I wasn’t fortunate enough to attend the lecture in person, I did go through the recording that, thankfully, was quite well done. Like all other “musings”, this one included some fascinating anecdotal bits (no pun intended) and Knuth’s good sense of humor was sprinkled throughout; but most noticeable was his infectious enthusiasm for the topics he spoke about. The preface to Volume 4 (printed in Fascicle 0) begins with:

The title of Volume 4 is Combinatorial Algorithms, and when I proposed it, I was strongly inclined to add a subtitle: The Kind of Programming I Like Best.

His excitement for these topics clearly showed in the lecture. It is a must-watch for anyone interested in The Art of Computer Programming (especially Volume 4 on Combinatorial Algorithms), or interested in learning a very useful data structure for combinatorial applications — ZDDs.

ZDDs (Zero-suppressed Binary Decision Diagrams) were introduced by Shin-Ichi Minato in 1993, about 7 years after Randal Bryant introduced his key ideas of the BDD data structure; one of which was that keeping the BDD both reduced and ordered is important in practice. In his “Fun with BDDs” musing (this June), Knuth mentioned that this 1986 paper of Bryant’s took computer science almost by storm; remaining the most-cited paper in the field for many years (see here).

Minato’s idea (of zero-suppression in BDDs) was motivated by his observation that when working with combinatorial objects, very often tests in BDDs have to be made just to ensure a potentially large number of variables are FALSE. So, in ZDDs, we just “skip over” any node whose HI pointer leads to (the FALSE sink) — if having TRUE results in the subfunction rooted at to become FALSE, then we skip testing altogether. That’s the basic difference between BDDs and ZDDs. And while ZDD theory is much like the BDD theory, it differs sufficiently in the sense of requiring a different “mind-set”.

The pre-fascicle 1B that discusses BDDs and ZDDs is available from Knuth’s News page (direct link). Interested readers can download it (but Knuth cautions against printing it out just yet, since he’ll be putting up an updated version in a couple of weeks). That pre-fascicle is a 140-odd page booklet with more than 260 exciting exercises! It will be published as part of fascicle 1 (Knuth said it’ll go to printing around the end of January 2009); fascicle 1 would in turn form part of Volume 4A (along with the already published fascicles 0, 2, 3 and 4).

The Fascicle 1 of Volume 4 of The Art of Computer Programming is now in print. In addition to the material on BDD and ZDD techniques (described in this post), it includes a subsection on bitwise tricks and techniques, including broadword computing (formerly known as “Platologic computing”). In Knuth’s own words, it is “cram-packed with lots of new stuff among nearly 500 exercises and answers to exercises.”

There’s also a bundle of Volume 4, Fascicles 0-4 available at a discounted price.

Get your copies today!

Families of subsets and their ZDD representation

BDD is a data structure for (representing, manipulating, working with) boolean functions. ZDD, on the other hand, is a data-structure for what Knuth calls Families of Sets — a large number of combinatorial problems are based on studying families of sets (which are just Hypergraphs). Families of sets are another way of looking at boolean functions since there’s a natural one-to-one correspondence between the two — the solutions of a boolean function (the set of n-tuples of boolean values such that is TRUE) correspond to family of subsets, with each solution corresponding to a subset where belong to the subset iff [see TAOCP Exercise 7.1.4 -- 203(b)]. But even though the families of sets can be encoded as boolean functions, it is sometimes better to think of them directly as families of sets, rather than in boolean sense. This is the thinking cap we’ll put on when working with ZDDs.

Families of subsets are taken from a well-ordered universe . We use the following rules to represent those using ZDDs [again, see TAOCP Exercise 7.1.4 -- 203]:

• The Empty Family, , is represented by (the FALSE sink node).
• The Unit Family (the set comprising just the empty set), , is represented by (the TRUE sink node).
• If is any other family (besides empty and unit), we know that it is nonempty, and that at least one of its members is in turn nonempty. In that case, let be the least element of universe that supports — that appears in at least one set in . [see answer to TAOCP Exercise 7.1.4 -- 197 for the definition of supports]. Then define sub-families and of with and . These sub-families correspond to those sets that don’t and do contain . Then, inside the computer, the family corresponds to a zead (ZDD-node) labeled with LO and HI branches going to the ZDD representations of and respectively. Notice the recursive definition (see Knuth’s diagram below).

Defining a ZDD recursively (@19m)

Ex. 1: ZDD of a function (@24m)

Throughout the talk, Knuth took up various problems, and worked them out in real-time for everyone to follow, rather than putting up a bunch of slides with everything already worked out. This is one of the beauties of his musing lectures — it’s interactive. While Knuth works his way through the problems, the audience can work with him or with their own pencil and paper. And it’s even more convenient when watching the recorded session over net, where one can pause, work the problem out, and then play to see if they got it right, and go back etc.). Knuth also joked about how he sometimes had to sit through presentations where the speaker doing PowerPoint would whiz through the slides a bit too fast. So, anyway, the most instructive (and most enjoyable) way to understand these examples, I think, would be to just watch the video and work them out yourself (possibly after going through the relevant material in the pre-fascicle).