A frog wants to hop down n steps into a pond. Each hop covers 1 or 2 steps. How many distinct ways can it get down there (say, when n=30)?

In mathematical terms we want a function f(n) for the frog’s jumps. For a function on non-negative integers we can start from small cases, and it is not hard to get f(1)=1, f(2)=2. The problem is that brute-force enumeration gets painful fast.

Imagine you are that frog facing n steps. On the first hop you can take 1 step, leaving n-1 steps with f(n-1) ways down; or you can take 2 steps, leaving n-2 steps with f(n-2) ways down. So the answer is nothing new — it is the Fibonacci sequence:

$$ \begin{aligned} F_{0} &=0 \ F_{1} &=1 \ F_{n} &=F_{n-1}+F_{n-2} \quad(\mathrm{n} \geq 2) \end{aligned} $$

The first step in solving this kind of function-value problem is to find a recurrence. Translate the recurrence into Python:

def fib(n):
    if n==0:
        return 1
    if n==1:
        return 1
    return fib(n-1)+fib(n-2)

%%time
fib(30)

Wall time: 269 ms
832040

The runtime is 284ms — slow enough. Why? Because there is far too much repeated computation. Take f(5) as an example; the calls look like this:

f(5)==>f(4), f(3)
f(4)==>f(3), f(2), f(3)==>f(2), f(1)
f(3)==>f(2), f(1), f(2)==>f(1), f(0), f(2)==>f(1), f(0), f(1)
f(2)==>f(1), f(0), f(1), f(1), f(0), f(1), f(0), f(1)
f(1), f(0), f(1), f(1), f(0), f(1), f(0), f(1)

The natural idea, then, is to cache every intermediate result. Fortunately Python ships that “battery” already.

from functools import lru_cache
@lru_cache()
def fib(n):
    if n==0:
        return 1
    if n==1:
        return 1
    return fib(n-1)+fib(n-2)

%%time
fib(30)
Wall time: 0 ns
832040

So fast that no time could be measured. The basic mechanism of lru_cache in Python is a dict whose key is the call argument and whose value is the result for that argument. It is roughly equivalent to this code:

def fib(n):
    if n in fib.cache:
        return fib.cache[n]
    if n==0:
        ans = 1
    elif n==1:
        ans = 1
    elif:
        ans = fib(n-1)+fib(n-2)
    fib.cache[n] = ans
    return ans
fib.cache = {}

Of course, for this specific problem we can use a more careful caching scheme, or even drop recursion in favor of a loop (equivalent to keeping only two cached values, which cuts space usage a lot — but if we need to compute many different n values over and over, the previous method may be the better fit):

def fib(n):
    a, b = 0, 1
    for i in range(n):
        a, b = b, a+b
    return a

This problem is identical to leetcode 70 . The python3 solution on leetcode is:

from functools import lru_cache
class Solution:
    @lru_cache()
    def climbStairs(self, n: int) -> int:
        if n==0:
            return 1
        if n==1:
            return 1
        return self.climbStairs(n-1)+self.climbStairs(n-2)

Runtime 52 ms, memory 13.2MB.

2. A simplified, practical dynamic programming

The frog gives us a fairly general lesson for problems where a recurrence can be constructed:

1. Find a recurrence relation, write the recursive function, and the problem becomes solving a set of subproblems;
2. To avoid recomputing the same subproblem, use a cache — trade space for time.

General algorithm textbooks and interview write-ups spend a lot of time designing that cache structure. In real engineering we are usually not that sensitive to using a bit more cache space, so writing the recursive function plus a generic cache basically solves the problem. Only when cache space itself becomes a concern do we need to think about a smaller cache tailored to the problem.

To test this approach, let us look at a few more problems, pulled straight from leetcode.

2.1 Maximum Subarray

Given an integer array nums, find the contiguous subarray with the largest sum (the subarray contains at least one element) and return that sum.

Example:

Input: [-2,1,-3,4,-1,2,1,-5,4], Output: 6 Explanation: The contiguous subarray [4,-1,2,1] has the largest sum, 6.

Consider the recurrence for the maximum sum obtainable by a subarray ending at each position in the array. $$ \begin{aligned} f(0)&=nums(0) \ f(k)&=max(f(k-1), 0)+nums(k) \quad(k>0) \end{aligned} $$ From that, the final answer is easy: $$ ans = max_{i=0}^n(f(i)) $$ Translated into python3 on leetcode:

from functools import lru_cache
class Solution:
    def maxSubArray(self, nums: List[int]) -> int:
        self.nums = nums
        return max(self.f(i) for i in range(len(nums)))
    
    @lru_cache()
    def f(self, k):
        if k == 0:
            return self.nums[0]
        else:
            return max(self.f(k-1), 0) + self.nums[k]

Runtime 76 ms, memory 13.7 MB.

2.2 Minimum Path Sum

Given an m x n grid of non-negative integers, find a path from the top-left corner to the bottom-right corner such that the sum of the numbers along the path is minimized.

Example:

Input: [ [1,3,1], [1,5,1], [4,2,1] ] Output: 7 Explanation: The path 1→3→1→1→1 has the smallest sum.

Treating each cell of the matrix as the bottom-right corner and computing the minimum path sum to it gives the recurrence below:

$$ \begin{align} f(0, 0) &= grid(0, 0) \\ f(x, 0) &= f(x-1, 0) + grid(x, 0) \quad(x>0) \\ f(0, y) &= f(0, y-1) + grid(0, y) \quad(y>0) \\ f(x, y) &= \min(f(x-1, y), f(x, y-1)) + grid(x, y) \quad(x>0, y>0) \end{align} $$

Translated into python3 on leetcode:

from functools import lru_cache
class Solution:
    def minPathSum(self, grid: List[List[int]]) -> int:
        self.grid = grid
        return self.f(len(grid)-1, len(grid[0])-1)
    @lru_cache()
    def f(self, x, y):
        if x == 0 and y == 0:
            return self.grid[0][0]
        elif y == 0:
            return self.f(x-1, 0) + self.grid[x][0]
        elif x == 0:
            return self.f(0, y-1) + self.grid[0][y]
        else:
            return min(self.f(x-1,y), self.f(x,y-1)) + self.grid[x][y]

Runtime 1052ms, memory 13.9M.

Given a dataset $D={(\mathbf x_1, y_1), (\mathbf x_2, y_2), …, (\mathbf x_m, y_m)}$ , where $y_i\in {-1, 1} $ and $\mathbf x_i=(x_{i1}, x_{i2}, …x_{id})$ is a d-dimensional vector. Note that some sources prefer ${0, 1}$ for the two classes. Which numbers you use makes no difference in principle, but it does produce small differences in how the math is written.

“Score value” is not a formal term in machine learning algorithms, but it is a useful lens to adopt. We want a scoring function $s(\mathbf x)$ for a sample $\mathbf x$, and its basic property is simple: it must have the same sign as the target value $y$ — positive samples score positive, negative samples score negative — so that the final binary classification function is $sign(s(\mathbf x))$.

So how do we build the scoring function? The intuitive idea is a weighted sum over the sample’s $d$ attributes, which expresses how important each attribute is to the score. A score built this way does not necessarily discriminate at the value 0, so we add a bias term, giving the scoring function $s(\mathbf x)=w^T\mathbf x+b$. Many algorithms derive from this basic scoring function.

The basic criterion for a binary classification problem is the classification error rate. Per sample, the loss is 0 when the classification is correct and 1 when it is wrong; this loss is usually called the 0-1 loss, written $l_{0-1}=\mathbb{I}[sign(s(\mathbf x))\neq y]$.

If we consider positive and negative samples uniformly and let $z=s(\mathbf x)*y$ , then $z$ is always positive when the classification is correct. Call it the absolute score: $z$ measures whether the classification is right. Then $l_{0-1}=\mathbb{I}[sign(z)= -1]$, so $l_{0-1}$ is a piecewise function of $z$. From the perspective of optimization theory, this function has no gradient or subgradient , so we need to find an approximation of it to optimize. The basic criteria for that approximation $\widetilde l$ are: it is always non-negative; and when it is small, $l_{0-1}$ is small.

The score is a linear combination of the attributes $X$; from another angle, the score is also a linear combination of the weights $w$. In general we transform $X$ into a new feature space to get $\phi (\mathbf x)$. The score is then $z=(w^T\mathbf \phi (\mathbf x)+b)y$. The analysis above is essentially unchanged, only the model is more complex. Unless stated otherwise, the following analysis stays in the original space.

1 Linear regression

We require the score to have the same sign as the target. If we set a narrower goal, we can make the score itself ${-1, 1}$; that is the linear regression problem, with the squared error as the loss function. (This admits an intuitive reading, and it can also be derived from maximum likelihood by assuming the sample error is normally distributed; that is not the focus of this post.) Note that $l_{sqr}=(s(\mathbf x)-y)^2=(s(\mathbf x) * y-y^2)^2=(z-1)^2$, so the error function is a quadratic function of the absolute score $z$. As the figure shows, $l_{sqr}$ is always greater than $l_{0-1}$, and when $l_{sqr}$ is relatively small, $l_{0-1}$ is relatively small too, so it can serve as an approximation of $l_{0-1}$.

Linear regression has a closed-form solution $w=(X^TX)^{-1}X^Ty$, so binary classification with linear regression is very simple, but the results are generally not good. Why? The $l_{sqr}$ curve shows that when the absolute score is large, $l_{0-1}$ should be 0; intuitively, a large absolute score is a good thing, indicating high discriminative power, yet $l_{sqr}$ penalizes it more heavily. That is unreasonable. Linear regression can still give a passable $w$, usable as an initial value for other algorithms.

2 Perceptron learning

Perceptron learning can be understood through the following intuition: for the absolute score, when it is positive the classification is correct and we consider the error to be 0; when it is negative, the farther it is from 0 the larger the error, i.e. $l_p=max(0, -z)$. The $l_p$ curve is also an approximation of $l_{0-1}$. It is a piecewise function too, but it admits a subgradient: for $z\geqslant 0$ the gradient is taken as 0, and for $z<0$, $\frac{dl_p}{dw}=\frac {dl_p}{dz}\frac{dz}{dw}=-1*y\mathbf x=-y\mathbf x$. Using stochastic gradient descent, $w\leftarrow w+\eta y\mathbf x$ — that is, we optimize only when the score is wrong, which is exactly what the perceptron learning algorithm does.

3 Support vector machine

The perceptron’s error function can be improved further. We want the absolute score to be sufficiently large, so that we are more confident a sample is positive; since $z$ scales with $w,b$, we may define “sufficiently large” as greater than 1. This gives $l_{hinge}=max(0, 1-z)$, known as the hinge loss. Accounting for structural risk, our optimization objective is

$$ \underset{w,b}{min}=\frac{1}{2}\left |w \right |^2+C\sum_{i=1}^{m}max(0, 1-z) $$

With the slack variable method this becomes the soft-margin support vector machine, and the model is found by quadratic programming.

In terms of score, a sufficiently large score gives the model good robustness on new samples, and that is exactly where the support vector machine excels. If a feature space is introduced, solving the SVM requires computing many inner products; simplifying the inner products in feature space into a kernel function in the original space brings in the kernel method.

4 Logistic regression

On top of the $z$ value, we want it to yield a probability that measures how likely, or how strongly, the classification is correct. That is, map the range of $z$ from $(-\infty , +\infty )$ to $(0,1)$. The sigmoid function is a good choice: it is differentiable and gives exactly probability 0.5 at 0. That is,

$$ sigmoid(z)=\frac {1}{1+exp(-z)} $$

Then, based on this sigmoid function, we emulate the loss function $l_{0-1}(z)$. We require the loss to decrease with respect to $z$ and to stay positive; taking the log of the probability and then negating it gives

$$ l_{log}(z)=-log(sigmoid(z))=log(1+exp(-z)) $$

The figure shows that this is also an approximation of $l_{0-1}(z)$. Taking the log loss as the optimization objective:

$$ \underset{w,b}{min}=\sum_{i=1}^{m}log(1+exp(-z))+ \lambda\left |w \right |^2 $$

This is regularized logistic regression. Since $l_{log}(z)$ is differentiable everywhere, it can be solved with optimization methods such as gradient descent.

5 AdaBoost

The exponential function also satisfies the basic conditions for approximating $l_{0-1}$, i.e. the loss is defined as $l_{exp}(z)=exp(-z)$.

Combining the exponential loss function with an additive model yields the AdaBoost algorithm.

An additive model can be understood like this: suppose the d-dimensional vector $\phi(\mathbf x)$ we obtain in feature space corresponds to d weak classifiers, that is, classifiers a little better than a 50% random guess. The so-called Boosting method effectively combines these weak classifiers to build a strong classifier. An additive model is still essentially a linear model, and still has the scoring function $s(\mathbf x)=w^T\mathbf \phi (\mathbf x)$. (Since a weak classifier can carry its own bias, we take $b=0$.) The only difference is that here $\phi(\mathbf x)$ represents a weak classifier rather than a simple attribute transform (which, in essence, it still is).

AdaBoost is a step-by-step iterative process. Suppose that after $m-1$ rounds of iteration we have the scoring function $s_{m-1}(\mathbf x)=\sum_{i=1}^{m-1}w_i\phi_i(\mathbf x)$, and after the $m$-th round we obtain $w_m, \phi_m(\mathbf x), s_m(\mathbf X)$; then $s_m(\mathbf X)=s_{m-1}(\mathbf X)+w_m\phi_m(\mathbf x)$. AdaBoost’s optimization objective is to make the scoring function obtained at each step minimize the exponential loss, that is,

$$ w_m, \phi_m = arg\underset{w,\phi}{min}\sum_{i=1}^Nexp(-z_{mi}) \quad z_{mi}=y_is_m(\mathbf X_i) $$

Further derivation of this expression is not detailed here.

Summary

Six loss functions so far. All of them are monotonically non-increasing functions of the score, and the last five are surrogate approximations of the 0-1 loss. Whether solving the surrogate yields the solution of the original problem has been studied in depth, under the name of the “consistency” of surrogate losses³.

In addition, if we need to find all positive examples among the samples and rank them, the score must already carry ranking properties while being optimized. The log loss and exponential loss are monotonically decreasing, so the scores computed by models optimized on them can take part in ranking; the other loss functions are not monotonically decreasing, so their scores are not suitable for ranking.

1 Basic principles

The root of software complexity is the complexity of the real world. When software complexity exceeds what a person can hold in their head, development becomes difficult or fails outright. A good software engineering methodology supplies a good abstract model of the real world, managing complexity through a sound hierarchy of abstraction so that a person can grasp the software level by level, from the whole down to the details. Complexity that comes from the real world does not shrink or vanish, but it does get constrained.

High configurability can, to some degree, solve the problem of fast-changing requirements: take the invariant part of the software as the model, and constrain the volatile part through configuration files and the like.

graph TB
    subgraph "Software 1"
        complexity2[Complexity]-->abstract[Common abstraction]
    end
    subgraph "Software"
        complexity1[Complexity]-->complexity3[Complexity]
    end

2 How to build it

To build highly configurable software, you should first establish a general software model for the domain, then apply configuration techniques to that model. Different configurations plus the model finally produce different software instances.

graph TB
    subgraph "Software 2"
        config2[Software configuration 2]-->abstract2[General model]
    end
    subgraph "Software 1"
        config1[Software configuration 1]-->abstract1[General model]
    end

2.1 Modeling

Modeling is a hard problem. It depends on the scenario at design time, requires domain knowledge and design experience, and calls for close collaboration with domain experts. There are a few classic books on this: Domain-Driven Design and Implementing Domain-Driven Design .

2.2 Configuration techniques

2.2.1 Configuration files

Configuration files are the most widely used and most basic configuration technique. They mainly describe the data a program needs at startup and while running, and generally do not describe the program’s runtime logic directly. Common formats include ini, xml and json; a quick comparison:

2.2.2 Scripting languages

Large programs are often built by combining several programming languages: a compiled language implements the relatively fixed parts and the parts where runtime performance is critical, while an interpreted language implements the volatile parts. Linux, for instance, implements many tools in C and then combines them through bash scripts; large games are built on a C++ engine and use Lua scripts to describe the logic of game actions.

2.2.3 Domain-specific languages

A domain-specific language sits between ordinary configuration files and program code: a computer programming language with restricted expressiveness, aimed at one particular domain.

Domain-specific languages divide further into internal DSLs, external DSLs and language workbenches. An internal DSL is a special way of writing code in the host language and hardly counts as configuration; an external DSL designs a new language to do the expressing and still falls within the range of textual configuration. A language workbench mainly offers a visual way of configuring, otherwise known as visual programming.

A DSL sits between configuration files and scripts. It offers high expressiveness within its domain, is easier for domain experts to understand, and speeds up turning expert knowledge into software.

Depending on the model, all three techniques can be applied in highly configurable software development. The table below compares their main characteristics and typical uses.

3 Engineering practice

As the figure shows, highly configurable software development actually changes the traditional development process, and raises the standing of software configuration in the software life cycle. What follows describes the impact of high configurability on each phase of traditional development.

graph TB
    subgraph "Highly configurable software process"
    	requirements2[Requirements analysis]-->construct2[Building the software core]
    	construct2-->config[Software configuration]
        config-->test2[Testing the software core]
        config-->config_test[Testing the configuration]
		test2-->maintain2[Maintaining the software core] 
		config_test-->config_maintain[Maintaining the configuration]
    end
    
    subgraph "General software process"
    	requirements1[Requirements analysis]-->construct1[Software construction]
        construct1-->test1[Software testing]
		test1-->maintain1[Software maintenance] 
    end

3.1 Requirements analysis

At the requirements phase, compared with ordinary development, highly configurable development has to analyze two extra questions:

1. Whether high configurability is necessary. This is decided mainly by the software’s scope of applicability: is the software meant to be generalized, and are the requirements it must meet volatile? The more scenarios it has to fit and the more volatile the requirements, the greater the necessity. If the software serves one basically fixed scenario, there is no need to use configuration.
2. Separate model requirements from configuration requirements. When gathering requirements, gather them for all the software’s application scenarios, analyze the requirement extensions that generalizing may bring, study as many cases as possible, dig down to extract the invariants of the system, abstract the system’s model, and identify the volatile parts as configuration requirements.

3.2 Software construction

Construction in highly configurable development splits into two phases:

1. Building the software core: done by programmers. High configurability places high demands on architecture design. Under the object-oriented paradigm you need to understand and apply OO design principles deeply, make heavy use of the various structural design patterns, and use runtime association techniques such as reflection and dependency injection.
2. Configuring the software: can be done by a programmer, or by a domain expert. Since configuration has become an important part of the software, configuration files should be managed as strictly as source code and put under version control. Manual configuration is prone to error, so provide visual configuration tools wherever possible and restrict users from editing configuration at will.

3.3 Software testing

Corresponding to the two construction phases, testing of highly configurable software also splits into two parts:

1. Testing the software core. Test cases should cover configurations for a variety of scenarios, and those scenarios must include the boundary cases of the model, so that as many core defects as possible are eliminated by testing rather than surfacing after configuration when the software is rolled out.

2. Testing the configuration. Only after configuration testing can the software be applied in a concrete scenario. Configuration is often done by domain experts rather than programmers, so it is best to validate the configuration and to provide testing tools and a written test procedure for the configured state, so that domain experts can complete the testing too.

3.4 Software maintenance

Corresponding to the two construction phases, maintenance of highly configurable software also splits into two parts:

1. Maintaining the software core: general feature upgrades are done by upgrading the core. Changes to core functionality should keep configurations compatible; when the configuration format has to change, provide an upgrade and conversion facility for configurations.

2. Maintaining the configuration: when user requirements and application scenarios change in ways the core anticipated, a domain expert can make the change by editing the configuration. Configuration gradually becomes an important asset of the user.

But software is still within the realm of science and technology, and debugging ought to be a technical skill too. Debugging Rules! is a rare book that talks about debugging as technique. Its nine general principles apply not only to developing and maintaining software and hardware, but even to daily life. Finding this book took some effort; luckily Turing Community still sells the legitimate e-book .

Honestly, many of the author’s examples lean toward hardware, so there’s a gap when reading them, but the nine principles are genuinely practical. In my earlier development and debugging work I probably summed up a few of them without realizing it, yet after reading the book it still felt like being handed a martial arts manual. Now let me walk through the nine principles again to deepen my understanding.

1 Understand the system

This is the most important principle. To analyze a bug you obviously have to understand how the system works, which means learning some fundamentals and carefully reading the documentation for the specific libraries, tools and technologies involved. I only got into the habit of seriously reading official documentation in the last few years. Search engines made us Google-oriented programmers: facing a problem we want it solved in one shot, and the result is usually a pile of detours, all because we begrudge the time to read the signposts. The book particularly insists on reading the manual “word for word” — a blunt blow to the impatient. Starting to debug without understanding the system is the same lineage as writing code without understanding the principles; the common name for it is “programming by coincidence”.

2 Make it fail

This principle is about why and how to reproduce a bug. A bug definitely happens under some particular condition; peel it apart layer by layer until you find that condition and you’re halfway there. From my hardware maintenance experience: it helps enormously if the person who found the bug records the whole process in detail. Many bugs need fairly narrow conditions to appear, hence one of a programmer’s stock phrases, “works on my machine”. For this kind of bug, first try to find a way to simulate it, for instance speeding up the interaction and running stress tests; second, have a detailed and sensible logging mechanism inside the software, so the reproducing conditions can be found from the inside.

3 Don’t think, look

This one stresses the importance of observation: reality beats idle speculation. Language shapes thinking, so the phrase on hitting a bug should be “let me look” rather than “I guess it’s because”. We should try every means to see the details of the bug clearly, so a mature program should carry debugging infrastructure; in my experience, logging at minimum is indispensable. Junior programmers tend to debug entirely with breakpoints, except a breakpoint itself changes the flow of execution (especially with multiple threads). Guessing is still useful, of course: it narrows the range of observation, or at least sets a priority for what to look at first, which is where experience pays off. But no matter what, experience cannot replace observation.

4 Divide and conquer

Binary search turns lookup from linear time into logarithmic time. It isn’t only an algorithm inside programs, it’s a methodology for debugging. Once a system has this awareness, the key question is how to divide the system — back to principle 1. At the very least, when you debug with breakpoints, bisection can locate the bug, and the same principle applies when inserting log statements. The book also mentions bugs influencing each other; I’d say there’s even a bug seesaw. The way out is to let none of them go. When debugging reveals that the code quality is really bad, refactoring is sometimes necessary — you can’t build a tower on loose sand — and after an effective refactor the bug often turns up and gets removed on its own.

5 Change one thing at a time

This is like the control principle in experimental science: examine one variable at a time, compare the normal case against the abnormal case, and never ignore any difference in test conditions. The previous principle says let no bug go, but they still have to be handled one by one, and when refactoring don’t hack away at code you don’t understand. Much code is dangerous because one pull of a hair moves the whole body; I think that’s the biggest risk in fixing bugs — you often remove one bug and add several. Back to principle 1: understand the system. Although understanding other people’s code is easier said than done, which is a reminder to build up good karma when writing code: don’t imagine code is written for the compiler, it’s written for whoever maintains it later.

6 Keep an audit trail

This principle is about why and how to keep records. I think of it as a doctor taking a history: you need to ask the right questions and get concrete, useful answers. My own experience is that once I start debugging a bug I sink in completely and can’t get out; writing things down pulls me out of the mental swamp and lets me keep correcting my plan and method for debugging. With some specialized system software you need to train the users in how to record bug occurrences in effective language; without records, the person debugging has to ask the right questions to jog the user’s memory. Beyond that, a bug you can’t crack at midnight deserves a night’s sleep, letting divergent thinking work on its own; usually it’s solved by morning.

7 Check the plug

This principle is roughly about so-called “stupid mistakes” — like my mother telling me the computer speakers have gone silent, and my first question having to be whether the power is on. Or writing main as mian. Once I set a breakpoint and it just wouldn’t hit; later I found that two blocks of code looked much alike and I had set the breakpoint in the wrong place. Depending on the scenario and your experience, ask yourself whether you’ve made the mistake that is common in this scenario; that often solves a pile of problems on the spot, just as the troubleshooting FAQ in many appliance manuals also tells you to check the plug. You could say a stupid mistake is stupid in difficulty, not in frequency.

8 Get a fresh view

This principle is about asking for help and talking things over. The important part is using another person’s viewpoint to break your own mental set, so it’s best to describe only the symptoms in detail and not your guesses, lest you contaminate their judgment. For a programmer, Google-oriented programming is astonishingly powerful at this point: nine out of ten problems don’t need you to actually ask anyone, just the right search terms. For Chinese programmers, searching in English is a required skill, and of course really understanding what you find matters too — don’t swallow every pill you see. If Google genuinely can’t find it, that proves you’ve climbed a level; then go ask on communities like Stack Overflow.

9 If you didn’t fix it, it ain’t fixed

This principle says don’t count on luck, and don’t play ostrich just because the bug only flashed by once. What has to come will come; the bill isn’t cancelled, only deferred. Real situations can be complicated, of course — software engineering is a compromise among time, cost and quality — but even when you let some bugs go, work out the consequences that may follow and put adequate safeguards in place, which is why many software certification schemes include software safety integrity levels. This also touches on reproducing the bug and generalizing the fix: confirm that you have really found the cause, provide an effective fix, and sweep for all similar problems. Fix one memory leak and you have to fix every piece of code that causes that kind of leak.

Postscript: how to write code that is easy to debug

Seen from the other side, the principles of debugging help with building software itself. Debugging only patches holes, after all; better to write code with fewer bugs and leave yourself more handles for when you do debug. A brief summary of my own experience, which probably needs continual refinement:

1. The program should provide a logging mechanism, emitting debug information in the debug build, and even in release recording the details of what went wrong (an exception thrown in C#, say);
2. Write code fit for humans to read, otherwise whoever maintains it later can’t understand it, let alone debug it. Code Complete says the most about how; besides the various idiomatic ways of writing, I think putting yourself in the reader’s shoes matters a lot;
3. When using libraries or tools, try to understand the principles and read the official documentation carefully. Some libraries explain in detail how to avoid memory leaks and how to handle multithreading; read it clearly before you use it and you plant fewer seeds of disaster — at least you won’t be a headless fly guessing at causes when the debugging comes;
4. Write code that is easy to test, so that problems can be isolated during debugging, or bisected. TDD practice covers this fairly well; I need to work on it myself.