<CharlieDigital/> Programming, Politics, and uhh…pineapples


Lessons from Fred Brooks

Posted by Charles Chen

Brooks is one of my revered writers on the subject of software engineering. The basic lessons in The Mythical Man Month are so obvious and fundamental yet often obscured or forgotten in many of the projects that I've worked on. Certainly, even this classic is "no silver bullet", as Brooks himself would concede, but it offers sage advice for aspiring developers and architects.

In this month's Wired magazine (8.10), he dishes some more wisdom in an interview with Wired's Kevin Kelly.

KK: You say that the Job Control Language you developed for the IBM 360 OS was "the worst computer programming language ever devised by anybody, anywhere." Have you always been so frank with yourself?
FB: You can learn more from failure than success.  In failure you're forced to find out what part did not work.  But in success you can believe everything you did was great, when in fact some parts may not have worked at all. Failure forces you to face reality.

KK: In your experience, what's the best process for design?
FB: Great design does not come from great processes; it comes from great designers.

Both these points resonate with me and I think the last point is particularly salient.  Brooks highlights an example in Steve Jobs:

KK: You're a Mac user.  What have you learned from the design of Apple products?
FB: Edwin Land, inventor of the Polaroid camera, once said that his method of design was to start with a vision of what you want and then, one by one, remove the technical obstacles until you have it.  I think that's what Steve Jobs does.  He starts with a vision rather than a list of features.

Brooks' The Mythical Man Month and  The Design of Design should be on every developer, architect, and IT project manager's reading list.

Filed under: DevLife No Comments

HTML-ifying Text in SharePoint

Posted by Charles Chen

This has probably been done to death, but here's my version of how to "HTML-ify" HTML text in SharePoint calculated fields into rendered HTML using jQuery:

var htmlPattern = /[^<]*<([^ ]+)[^<]+<\/\1>.*/;
var ltPattern = /&lt;/g;
var gtPattern = /&gt;/g;
$(selectors).each(function(index, item){
    var html = $(item).html().replace(ltPattern, "<").replace(gtPattern, ">");

    if(!htmlPattern.test(html)) { return; }


Perhaps the most interesting part of this script is the regular expression used to capture the HTML tag. Unlike other versions I've seen, it easily handles any HTML tag and will match-and-replace even elements that don't start and end with HTML tags (your content can have leading and trailing text or other HTML). This is exceedingly useful if your calculated field contains textual content or if you need to HTML-ify some string that is rendered in say the newsletter list style.

Specify your selectors wisely to minimize the number of elements to scan.

Here it is again in a "best practices" format hooked up to a simple namespace:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
<html xmlns="http://www.w3.org/1999/xhtml">
<title> Sample HTMLIFY </title>
<script type="text/javascript"
<script type="text/javascript">
// Create a simple namespace.
var my = {};

// Define the util class.
function util() {
var htmlPattern = /[^<]*<([^ ]+)[^<]+<\/\1>.*/;
var ltPattern = /&lt;/g;
var gtPattern = /&gt;/g;

return {
// The main function which takes a set of jQuery selectors.
htmlify: function(selectors) {

$(selectors).each(function(index, item){
var html = $(item).html()
.replace(ltPattern, "<")
.replace(gtPattern, ">");

if(!htmlPattern.test(html)) { return; }


// Instantiate the object.
my.util = new util();

// Convert to rendered HTML
$(document).ready(function() {
my.util.htmlify(".text1, .text2");

<div class="text1">This is some HTML text &lt;a href=""&gt;a link&lt;/a&gt;</div>
<div class="text2">This is some HTML text &lt;a href=""&gt;a link&lt;/a&gt;
<span style="background:green">This text is in a span</span></div>
<div class="text2">This is some HTML text &lt;span style="background:red"&gt;some text&lt;/span&gt;</div>

You should be able to copy/paste that and run it.

Filed under: Dev, SharePoint No Comments

Moving to WordPress and WebFaction

Posted by Charles Chen

For some time, I've had hosting with WebFaction for some personal python+django projects I was (well, am still...) working on while my main blog was hosted with ServerIntellect (a great hosting company, by the way).

While WebFaction breaks one of my steadfast rules of hosting by not having a plainly visible phone number (actually, I can't find one anywhere on their site), I've been incredibly pleased with the hosting overall.

I've been running my own Trac installs, SVN servers, Mercurial servers, and now WordPress for < $10/mo.

And now I'm hosting two top level domains as well.  Still < $10/mo.

While I do miss being able to call someone 24-7, their documentation on how to install and configure the various apps is great, their reps are active in the support forums, the plethora of out-of-the-box applications is great (django, Trac, SVN -- just to name a few), and I've gotten email responses to newb-ish Linux questions in about 2-3 minutes.  But overall, it's just a more useful platform for me.

The whole blog move seemed daunting when I approached it, but I really just needed to slice out a small chunk of time and get it done.  All-in-all, I had my content migrated and visible in I'd say less than 20 minutes.

I found these resources helpful in getting it all done:

At the end of the day, it wasn't nearly as dreadful as I feared it would be.  However, it was ultimately worth it to stop paying for two hosting providers and the pains of the aging dasBlog engine.


Meeting Hell

Posted by Charles Chen

From one of my favorite software engineering books, Eric Brechner's I.M. Wright's Hard Code:

None of us is as dumb as all of us

An especially evil form of interruption is the meeting.   A meeting forces you to stop productive work and throw yourself into a frustrating, time-consuming, black hole of wasted life from which you can never recover.  However, there are several actions you can take to minimize the life-draining effect of meetings:

  • Stop going.  There are only a few meetings you must attend: your on-on-ones, your project status meetings, and your staff meetings.  Almost all other meetings are discretionary.  If a meeting feels optional, try skipping it.  If nothing bad happens, don't go again.
  • Make someone else attend.  Try delegating the meeting to someone else.
  • Run effective meetings.  As for meetings that are left, make them as effective as possible.
  • Put all your meetings back to back.  I know this sounds strange, but the idea is to reduce context switches.  First, try to schedule your project and staff meetings early in the week, and then schedule all your one-on-one meetings around them.  Sure, the early part of your week will be hellish, but the middle and the end of your week will have uninterrupted blocks of time.

Sage advice.

Filed under: Rants No Comments

The Math of Mediocrity

Posted by Charles Chen

Professionally, almost nothing aggravates me more than the Math of Mediocrity.  The only thing worse than observing failure based on the Math of Mediocrity is having to actively participate in it.

Steve Jobs’ Parable of the Concept Car is a perfect illustration of how companies fail to execute because they fall for the Math of Mediocrity:

"Here's what you find at a lot of companies," he says, kicking back in a conference room at Apple's gleaming white Silicon Valley headquarters, which looks something like a cross between an Ivy League university and an iPod. "You know how you see a show car, and it's really cool, and then four years later you see the production car, and it sucks? And you go, What happened? They had it! They had it in the palm of their hands! They grabbed defeat from the jaws of victory!

"What happened was, the designers came up with this really great idea. Then they take it to the engineers, and the engineers go, 'Nah, we can't do that. That's impossible.' And so it gets a lot worse. Then they take it to the manufacturing people, and they go, 'We can't build that!' And it gets a lot worse."

When Jobs took up his present position at Apple in 1997, that's the situation he found. He and Jonathan Ive, head of design, came up with the original iMac, a candy-colored computer merged with a cathode-ray tube that, at the time, looked like nothing anybody had seen outside of a Jetsons cartoon. "Sure enough," Jobs recalls, "when we took it to the engineers, they said, 'Oh.' And they came up with 38 reasons. And I said, 'No, no, we're doing this.' And they said, 'Well, why?' And I said, 'Because I'm the CEO, and I think it can be done.' And so they kind of begrudgingly did it. But then it was a big hit."

This doesn’t just happen in automobile manufacturing or engineering, but it also occurs plenty often in software development.  One particular example is what I call the “Communism of Failure”.  In this case, the best solutions, the best ideas, the ones that will benefit the end users the most, the ones that will get the users excited, the ideas and solutions that will help people be more productive or more efficient are…shelved.  Why?  because it can’t be supported by the commoners in tech support.

Certainly, this is a valid concern -  it would be absolutely foolish to believe otherwise; a solution is useless if only the brightest and most exceptional of minds can understand it, deconstruct it, rebuild it, fix it and so on.  But proper software engineering and project management offers ways to mitigate this through process and practice.  Pair programming, strict guidelines and expectations for documentation, well documented common coding styles and techniques, leveraging automation and code generation, encouraging reuse of code assets by thinking in terms of frameworks.  The point is, building an exceptional solution is not exclusive of building a sustainable solution.

The Legalist philosopher Han Fei-Tzu wrote:

If it were necessary to rely on a shaft that had grown perfectly straight, within a hundred generations there would be no arrow. If it were necessary to rely on wood that had grown perfectly round, within a thousand generations there would be no cart wheel. If a naturally straight shaft or naturally round wood cannot be found within a hundred generations, how is it that in all generations carriages are used and birds shot? Because tools are used to straighten and bend. But even if one did not rely on tools and still got a naturally straight shaft or a piece of naturally round wood, a skillful craftsman would not value this. Why? Because it is not just one person that needs to ride and not just one arrow that needs to be shot.

Indeed, a solution catered towards the brightest of minds is equally as bad as a solution catered towards the most common of capabilities.  But through the usage of the right tools and the application of bending and straightening, we can bridge the two.  It's not a compromise, but rather an effort to build a framework that allows excellence to propagate.

As in Jobs’ parable, where solution and enterprise architects fail at the Math of Mediocrity is that they cede to the concerns of the plebeians; they sacrifice excellence for “good enough” so that tech support can support the solution.  On the one hand, Solution A can solve the problem for the end user in 1 click.  On the other hand, Solution B requires the end user to perform more than 20 clicks to complete the same operation but is easier for tech support.  Which is better?  Jobs’ would surely side with Solution A, even if it’s the more technically complex solution because it delivers a better user experience and improves efficiency and productivity.  Amazon loved Solution A so much that they gave it a name and patented it.

The problem arises because of ambiguity in calculating the true cost of Solution A compared to Solution B.  Solution A may require $300/HR consultants and twice as much time to implement.  Solution B may require $150/HR consultants and cost only half as much as Solution A.  These costs are concrete and easy to quantify and grasp.  What seems to escape the calculus is the cost of lost productivity and efficiency by having the end users, who have to use the applications day in and day out, suffer through an inferior solution because of the marginal costs of contracting better developers, working smarter, building from a framework, and hiring more competent tech support.

The same math carries over to the support staff, who are a marginal percentage of the overall workforce for any large organization or for a particular project or initiative.  The question is whether it’s better to hire more competent support staff who can maintain a more complex but better solution or to hire less competent support staff at lower costs.  And again, the question comes back to the issue of productivity and efficiency and comparing the gains made across an organization of tens of thousands of people versus the additional costs associated with a few dozen people.  No question: I improve the user experience which not only improves productivity and efficiency if done right, but also aids in adoption and uptake;  cost is but only one metric of success and perhaps not even the most important one.  In the end, it really doesn't matter how much you saved by approaching the problem using Solution B; if the end result is a clunky, hard to use, inefficient, productivity drain, then the project has failed, regardless of how much money was saved by catering to the mediocre.

A solution architected and designed around a compromise for the average can work, but the problem must be approached differently.  Leverage better developers from the get-go, make documentation a priority, standardize code, leverage patterns, ensure that the right tools and platforms are in place, build frameworks to support the most common scenarios, use pair programming and code reviews to ensure cross pollination of skills and knowledge, make learning and education a primary job conern, find solutions through engineering and process, not through capitulation to the lowest common denominator.

Update: An article in the New York Times by Damon Darlin this weekend caught my attention and adds another layer to this:

Remember those lines? Back when commissars commanded the Soviet Union’s economy like Knute commanding the tides, people would wait for hours in long queues for free bread. Although the bread was free, people paid for it with their time.

To economists, the long lines were a real-life example of the market requirement that payment be made one way or another — in money or in time. (In this country, the long lines would be for an Apple gadget, which is neither cheap nor scarce. But explaining that mystery is for another time.)

Paying with time rather than money seems just as common on the Web...Technology could very well make the Soviet bread line disappear. Do you remember how long it took to do a Google search a dozen years ago, when the service started? Probably not, but Google engineers calculate that their refinements have saved users a billion seconds a day. Using Google to quickly make the calculation, that comes out to about 1,800 lifetimes.

Indeed, the question is whether a business wants to make payment in money or in time (well, for businesses, time is money).  For that reason, enterprise architects should think long and hard about the priorities of the platform or solution they are architecting.  Is it just to do the minimum and keep maintenance effort and costs low?  Or is it to actually streamline and improve the business processes and improve productivity and efficiency?  How can you achieve the latter while sacrificing minimally of the former?

Filed under: DevLife, Rants 2 Comments