Bumble Bee Soft Blog

Old-School Code Visualization
There is some hint as to why I like old video games in this page:

http://benfry.com/distellamap/

Maybe I just don’t want to contend with the sort of complexity of modern programs, or maybe I see in them the seeds of a useful approach that can be extrapolated in these more easily understood situations, but I love these diagrams. Because the images are hard coded as bits, visualizing the bit structures recreates the freaking graphics in the diagram! Because the translation method that the code used to turn data into images was so simple, the visualization was able to achieve a satisfying result via a transformation that didn’t feel like a cheat. That simple translation from bits to visualizability is simply lost in any programs more complicated.

This is maybe further proof that the complexity of software really doesn’t appeal to me past a certain threshold, a threshold that has been long since passed by modern computing. Whether I can make software appeal to me through some clever conceit, or whether I′m in the wrong field remains to be seen.

PS:
http://benfry.com/disarticulate/
This picture awes me in its own way. It is less understandable, but it is so striking, and appears to be on much more complex code. The way that the appearance of frenzied, ragged brushstrokes is created by the rote mathematical operation of a program is hugely appealing. A friend pointed out that some aesthetic choices must have been made in terms of code layout, something which only increased my enjoyment of the image.

Update
So this blog has been lying stagnant for a while now, for a number of reasons. Partially, I just got caught up in the holidays and got out of the habit. Also, lately I feel like its hard to make posts that are interesting in and of themselves, since most of my thinking is tied up in larger conceits, wrapped up in epic theories that simply don’t lend themselves to blog-sized nuggets. I′ll try to get some posts back on here at some point, but I can’t let it get in the way of my “real” research, which is plenty consuming on its own. Time will tell…

Making software is like: designing a video game (abstraction and intuitiveness edition)
I am at a bit of a block lately. My earlier revelation has brought a weight to this blog that I can’t possibly maintain, leading me to far too many self-referential comments like this one. So I am just going to get back to what made me what I am today and just talk out my bum about a metaphor for software.

I’m a real old-schooler when it comes to games. I generally prefer board games to video games, and if I am going to play a video game, I prefer older games to modern ones. If forced to choose between my Playstation 2 and the arcade emulator on my laptop, I’d take the latter every time. In many ways, those things that I dislike about modern video games are those things that I dislike about making software.

Older video games were, by necessity of the limited hardware available to them, pretty simple. Unable to smother you in graphical propaganda, they had to make precision strikes into the core of your brain, triggering the necessary imagination and impulses to create the illusion of an experience beyond button-pressing. While it was largely employed out of necessity and not artistic vision, the abstraction found in these games is very appealing to me.

And older games are simply easier to play. Many such games were originally arcade games, and in such a setting the player needs to be able to walk up and get a satisfying experience almost immediately. Plus, most such games were two dimensional, which I generally find vastly preferable to newer, 3d games. The fact is, 99% of games are played with 2d controllers such as joysticks and mice, or vaguely one-dimensional controllers such as buttons. In a 2d game, up can mean up, and right can be right, and you can accomplish the onscreen actions you desire almost subconsciously. But when using a 2d controller to play a 3d game, real-world actions often must undergo some sort of transformation, relative to the game world, to determine the onscreen result. In driving games, or even first-person games, this transformation is often fairly simple, but in a 3d platformer I find the equation “right = move perpendicular to the direction the camera is pointing” to be too cumbersome to perform hundreds of times a minute while I try to enjoy myself.

This is soft of a recap of my early board game post, but making a document to be used within a software process is sort of like making a game. You are setting up a certain situation in which another person is going to roam around. You can’t control what a programmer using your design is going to do, but you can set the stage in a way that is as optimal as possible. But how intuitive is that experience going to be? When they use your design, are they going to be able to translate their needs into results intuitively, or is it going to be relative to some internal actors, rules and constructs that you have established?

And what about abstraction? I don’t mean abstraction in the “dealing with complexity” sense, but rather the “allowing for imagination” sense (discussion of what the hell I mean by these senses, and whether they are truly different, needs to be put off for now). Are you creating enough structure so that the interpretations fall within an acceptable range, while still allowing them to create the vision of the project that they need to in order to understand things?

If you were creating Space Invaders (note the talk of achieving a zen-like rhythm in the tricks section here), you would want to provide enough information and imagery so that players will get certain things right. You need to be specific enough so that the player understands where the bad guys are, where their bullets are, and what sort of progress you are making in defeating them. But I don’t think you need to show them exactly what the invaders look like down to the last drooling detail. I don’t think you need to create a realistic starfield, a soundly engineered ship or even a third dimension. Even though all of these things would likely exists in an actual battle for the earth against trans-galactic forces, I think that trying to include them is only going to be distracting. Let the player fill in all the details he or she needs to make the game fun, just provide them with enough framework so that you can tap into their basic perception and let their survival instincts and stress receptors do the rest. Or at least, that’s my philosophy.

And I think that it applies to the creation of software artifacts. You need to make sure that your player understands the basic goals, actions and means of achieving them, but you don’t need to breathe down their neck with rendered details. Just get enough to make sure they know how to play the game.

This goes back to my post I mentioned about constraints vs. freedom in board game design. There I made an argument that was sort of mechanical, in terms of a stodgy sense of usability and constraining decision trees. Here, I am looking more at the emotional side of this tradeoff. In a board game, every rule has a very real cost in terms of the players’ ability to keep the rules in mind and the gameplay moving forward. With video games, the computer can handle these rules, and the assumption seems to be that we should then just pile them on as we see fit. After all, the player doesn’t need to know the rules.

I don’t think this is really the case. Maybe the player doesn’t need, truly need, to keep every rule in mind when playing a video game as they do in a board game, they need to know how to do what they want to do. If the nitty gritty rules that handle this are being hidden away, the player better have a damn good metaphor for how they are working so that they can forge their plan. This is especially true given the real-time nature of most games, there’s no time to dig through the rulebook, and I don’t think there’s even really time to think. No, players in a video game don’t need to know the rules; it’s worse. They need to have so thoroughly embraced the rules that they no longer know them in a knowledge sense, they need to know them like instincts, and be guided by them.

Such a powerful understanding is often going to rely heavily on leveraging instincts they already have about the way the world works.

I think that software documentation would, ideally, work the same way. Ideally, making software artifacts wouldn’t be like making board games at all, they’d be like making really good, intuitive, action-packed, old-school video games. A really good requirements document will be just abstract enough (in that vaguely creative sense) and provide a set of rules just intuitive enough, that after reading it a designer simply understands, and can proceed in a way that their instincts tell them is right. Its not that they think to themselves, requirement 25a indicates that I should do things this way, they just have a sense for how the world works, and how to proceed to make it work for them.

Now, I’m off in the land of elves and fairies again, since I think that wholeheartedly embracing such an approach would be exceptionally difficult and prone to ambiguity and misunderstanding. What if the intuition wasn’t quite right, which it likely wouldn’t be? This could be really bad in a situation that demands precise understanding of specific constraints. But supposing that the intuitive guidance you receive was magically infallible, it sounds like a pretty nice way to go. Is working within a given document fraught with hundreds of tiny calculations a second, or is it invisibly shaping your decisions while you, as far as you can tell in the moment, are just “doing what you want”. That latter case, while a largely unattainable ideal, seems like something to strive for.

Software assumptions: will we ever have scheduled 10-year+ development cycles?
Unable to bring myself to create just a regular post, I got thinking about those posts I’ve already made. I’ve been trying to make head or tail of why the same themes keep coming up (besides the fact that I write about things I know and like).

Last post I talked about how hard I find software. I’m not backing down from that making software is really hard, but the reasons I present seem to fall into two camps. I often point out that the creation of a software solution is really hard because of the nature of the product: it is this complicated thing being made for people, who are unpredictable. If you are making an intermediate software artifact its even worse, since you are then making this complicated thing for people who will use it to make an even more complicated thing for other people. And at each intermediate step there is all this uncertainty of the results of the product.

The main angle I take is that the process of creating software makes it really hard. Especially when it comes time to implement code. This process is often either based on guesses at goals or third-hand documents, and must be performed concurrently with dozens of others. Even the best process, plan or design struggles under the weight of such demands.

To some extent these are essential problems with large-scale software development. Despite the immense complexity of software, and the diminishing returns associated with adding people to a project, we demand that software get completed as quickly as possible, so we take any returns we can get. I hate to crib Brooks yet again, but those are the facts of the matter, and we dump hundreds of people on a project anyway. I’m not even saying we shouldn’t, the market is what it is, and you have to get that junk out the door.

I do have this dream of taking a 20 man project with the schedule of a year, and letting two people spend 10 years on it. Assuming they are the right guys, and still have access to a little consulting and QA, I think those two guys would make an amazing piece of software. This is no big revelation, but it’s a thought I always see to return to.

There have certainly been projects that people have spent 10 years on, and have been amazing. The problem is, in the software world, the basic platform, cooperating software, hardware capabilities and public expectations are being pulled out from under you so fast, to speak nothing of the specific requirements themselves, that you’d never end up with something useful on that time scale.

So we have three options. We can stick with the status quo, we can try to make these two guys make software miraculously quickly (unlikely) or we can hope that software will endure a bit longer. And really, it is this last point I am after here: is this something worth hoping for? Is there any chance that a 10-year development cycle will be feasible? I wonder this really regardless of the development approach used, is this a reasonable time scale to ever expect to create successful, well-used software?

I think that in the current state of things, this is only a valid hope for a very specific, possibly mythical, piece of software. It would have to be a program that revolves around something fundamental, and that spans most platform and interaction considerations. I would count, for example, an internet search engine to be a viable candidate. Google has been in development for years, and could reasonably end up being a 10-year project ever under development. That’s a bit of a cheat, since it is fully deployed during its development cycle, but what if it had been sheltered away, with comparable feedback (somehow) from QA as the search engine has received from its actual users. That is, what if on January 1st, 2006, the contemporary version of Google just appeared out of nowhere and dropped on the unexpecting public. Page and Brin created the Google precursor Backrub in January of 1995, so this isn’t such a bad example, actually. I think that once word got out, it would blow people away.

Would this be better than doing it the way they did? No. Would the tool be as well-received? No. Could any amount of internal testing tell them what years of use undoubtedly has? I doubt it. All I’m really saying is that its technically possible that one could start a tool and it would still be useful 10 years later, but I wouldn’t recommend it.

Will this always be the case? Well, with the benefit of the impossibility of truly accurately predicting the future, I can say that no, this won’t necessarily always be the case. I need every ounce of that benefit of the doubt, but I’m willing to say we might see a day when the market pressures, hardware accelerations and shifting views finally mellow out enough to make feasible software masterpieces.

Suppose that hardware has, for whatever reason capped out, or at least its acceleration has slowed enough so that every new year doesn’t bring a tiny revolution. Suppose that someone or something wins the platform war and we all use the same type of machine for more or less everything. If computing were to become as stable a field as architecture or even art (in the sense that a painting started 10 years ago could certainly be worthwhile now), I could see the necessary time scale being expanded. I think that we have seen computers changing and accelerating for so long that we have come to expect that things will always be this way. Perhaps not in our lifetime, but I think that this may eventually change, and with it our priorities about software creation. In such a world, truly ideally realized, not only would a project be useful 10 years after it was started, but would continue to have an impact on the way we use computers for decades thereafter. In such a world, given the informational nature of software, and therefore its easy reproducibility, a single program that does one thing really, really, really well would be invaluable. It would potentially change the way that everyone does something on their computer for years to come. In this unlikely, idealized world, certainly spending 10 years on a piece of software would be a worthwhile endeavor.

It’s a long way off, and perhaps never to arrive, such a vision keeps me from truly accepting the modern time-scale of software engineering as an inevitability.

Revelation - Making software is like: my arch-nemesis
As I was driving to the university this morning, I gave some thought to a possible blog post, as I am wont to do. I was considering a post about old-school video games, abstraction and imagination. As I drove, I got thinking about whether I was repeating myself, not so much with the game-based theme, but with the general message: I don’t like making software.

It made me realize how personal this blog really is. Yes, it contains my opinions. And yes, I largely talk about things that I know (games, music). But something more than that. I started this blog because I thought things about software that weren’t publishable, nor useful, but that I felt needed to be recorded. I had no idea why, but I’m starting to figure it out.

Let me reiterate, I don’t like making software. It is a discipline that demands the utmost order, and yet defies any attempt to create it. We need to create this exacting program, but we need to do it in this swirling vortex of change, uncertainty and human emotion. It’s frustrating, it’s overwhelming, it’s emphemeral and elusive. It is like a moaning army of cryptic, ghosts speaking a dead language and demanding answers, all at once, with unrelenting shrillness. We create software for these computers who are like demanding, unscruitable gods. We are constantly making sacrifices to them, and explaining that sometimes they cause bad things to happen to good people, because that is their way, ever bowing to their wishes.

These rants express emotionally what I will now explain more rationally. I’m not overly well-suited for making software. I’m a divergent thinker with little patience for futzing around with details. I’m very visual, I like working with my hands, and I’m overwhelmed by excessive complexity. I’ve always sort of known this in the back of my mind, it has made me nervous about entering a career as an academic in software engineering if I can′t really do what I purport to know. Should I try to teach a discipline that I hate?

I realized today, that I don’t think I will ever be able to create software well, assuming I make it the way that it is created now. But I don’t hate the idea of software itself. I like generalizing, organizing, making metaphors and rules that are intuitive for people to understand. I like the idea of forming a plan and having a computer carry it out. Is there a way to seperate what I hate from what I love? Is there a way to seperate “software” from “making software” as currently defined?

The fundamental question is:

Is there a way to change the process of making software so that somebody like me can do it?

This is the challenge I have accidentally taken on, and I have only realized it through this blog. The first step was realizing how much I hate software, and what I hate about it. This is implied in most of my “Making software is like…” posts. The second step was starting to size up my enemy, and see whether I can defeat it. This has begun to emerge in my “Software Assumptions” posts.

I will continue in both of these directions, but the fundamental question remains. Can I fundamentally change software creation so that people like me can make it? Even if this requires an absurd amount of hard work, outside-the-boxism, and outright genius, I am willing to try this is something that I am willing to dedicate my career to as long as that possiblity exists.

Or are the things I hate about software simply inherent to its nature. Are its essential difficulties, if they are truly essential, enough to prevent me from ever being happy with it? Is software always going to be slippery, complicated and generally a huge pain in the ass? Is that just something we have to live with, and do the best we can with?

I’m sure the answer lies somewhere in between. Software creation can certainly be improved, and yet it will likely never be truly easy. If I really want to get better at making software, I will need to work at it, but advances in technology and practice will certainly help me along the way.

But where does this meeting point lie? Does it lie so far in the land of “making software is fundamentally, unchangably everything that I hate” that I will be constrantly frustrated when trying to perform it, study it, teach it and talk about it?

I realize that I have taken on a difficult task in studying a subject that I strongly dislike the state of. I’ve probably done so with equal parts bravery and naivete, but either way, I am commited to this battle for at least another few years. This will either result in striking changes that allow the creative, impatient and feckless alike to easily crete software, or the crushing of my spirit as I realize the peak to be unassailible, my foe to be invincible and my battle to be unwinable.

Hints of the outcome will be sprinkled here, as I uncover them.

Making software is like: finding music
I like music, and I like finding new artists, albums and songs to listen to. But how do I find the music that I’ll like from among all of the music out there, given that I have limited time and money for purchasing and listening to music?

I could just buy CDs at random and I hope that I like them, but this seems cost ineffective. I probably would enjoy a very small percentage of them. I could listen to CDs at random, and then buy the ones that I like, but even this would be very time consuming given the thousands of hours’ worth of recordings released each year. I could guide my choices based on the album art, or by record label, or by purchasing music by artists whose previous releases I’ve enjoyed, but these are not often necessarily good indicators of the enjoyment I’ll find.

What I really need is some sort of system that can tell me which CDs to listen to, an oracle for suggesting music. There are 3 basic forms of such systems that spring to mind.

1) Recommendations based on knowledge of your tastes. The old fashioned way, where a friend who knows what you like will recommend something that they think fits into the space of music that you like. They may not know what it is about the suggested album that makes it likely that you will enjoy it, they just have this vague idea of what you like in music and see an overlap. If their tastes are very similar to yours, they may just be able to use their own appreciation of a given album as the basis of this suggestion. The power that guides the recommendation is a holistic understanding of what you need, which is likely to be effective. But it is difficult to achieve, requiring a person who both knows of the album you might like, and knows you well enough to know that you might like it.

2) Recommendations based on a given preference. This is the very essence of Amazon.com’s “People who bought this also bought album X”. The low-tech version is when the guy at the record store sees you buying the Of Montreal album and suggests an old Apples in Stereo album. Once again, the suggesting party doesn’t really understand what it is you like about an album, they just know that people who like it show a tendency for liking another album. If you are in Venn diagram circle A, maybe you’re in A and B, and therefore would like B. Some internet services allow you to enter a great many preferences, and will cross-reference these with others’ lists, looking for likely overlaps. The power that guides the recommendation is the combined preferences of all the people that the service has access to. The system does not really understand what you like, nor what anyone else likes, but allows these preferences to imply similarities in the underlying qualities of the music, and matches them.

3) Recommendations based on underlying qualities. This is the type of recommendation system that I was recently exposed to, and which inspired this post. At pandora.com there is a service into which you enter the name of an artist or song that you like, and it will play you music that you might like. This is all based on something called the music genome project, which endeavors to examine thousands and thousands of songs, identifying certain basic qualities to them, from “varying tempo and time signatures” to “eclectic bass riff” to “east coast rap influence”. It appears that these qualities have been manually applied, with a group of people listening to all the songs and deciding which qualities are good fits. The result is that when you indicate that you like the Beatles, it gets the idea that you might like qualities that are commonly found in their songs, including “major key tonality”, “mellow rock instrumentation” and “a clear focus on studio production”. These qualities are obviously subjectively applied, but they are fairly objectively phrased; there is no mention of “high quality” or “strangeness” or anything that would otherwise be so relative. Then other songs with these qualities are played. The power of the recommendation is the qualities that were sort of hidden away and implied in the other two approaches, but which are explicitly leveraged here. This means that the qualities will more likely be transitioned, but there is no holistic understanding of whether this will really result in music that you like.

So, speaking of “hidden away”, I have been especially coy about the point here. When it comes time to build software, we would like to be able to reuse others’ ideas. These ideas might come in the form of high-level understandings of a problem domain, design patterns or actual reusable code. But once again, there are so many ideas out there, with costs in terms of time and money to find and use them. Our project has certain tastes, that is, it needs certain kinds of ideas that are useful to what is being attempted. We can’t really spell out exactly what it is that we need. We can explain what ideas we already have, or we can try to describe our goals, but there are so many underlying factors as to what idea/pattern/code segment will be useful during a given project that it is very difficult to find useful ones.

I will confess that I am not up-to-the-minute on the current reuse research, and I would appreciate any comments about where I’m wrong, but this is my take on this metaphor. In each case, it doesn’t take much to ruin a good match. A single element of a given song might be enough to turn someone off to it, and there are myriad details that could make a given software idea usable in a certain context. The biggest difference is likely the stakes; if a person doesn’t like a CD its no big deal, whereas a poorly applied software idea could be very costly indeed. But how well do our three basic approaches to music finding apply to software?

1) Recommendations based on knowledge of your tastes (that is, needs). This is the current approach to reuse, as I understand it. Someone is familiar with the software you are developing and the situation around it, and also knows of an idea that seems like a good fit, so they recommend it. In some sense you may need to recommend such an idea to yourself; that is, you must understand your own problem, and identify the connection. This is often something that is much more difficult to keep in your head than your own musical needs.

2) Recommendations based on a given preference. Can we automate this idea suggesting process? I imagine rudimentary systems for this exist, but I haven’t heard of anything very impressive. I think that such a system could be created, but what it lacks is the necessary user data. If we had every document created by software development teams over the last 5 years, we would likely be able to hash out some details of overlapping ideas, concepts, keywords, diagram sections and code segments. It seems like it would be possible to then enter your project, in whatever state it is in, and receive suggestions based on similar projects. Some uniform formatting across projects would, of course, ease things, but I don’t think it would be necessary for results.

I’m making a lot of assumptions about the technology here, and certainly finding projects willing to give up such information is a near impossibility, but I like the idea in its mythical, impractical state nonetheless. I know that Chris Jensen is working on some stuff to data mine open source projects – similar technology might be applied.

3) Recommendations based on underlying qualities. This is the approach that inspired this post, and the one that seems most interesting to me. I was not so much impressed by the recommendations that Pandora.com gave, as I was by the endeavor of deciding on these qualities and then hand-describing thousands and thousands of songs. If we tried to do the same thing to software, what would it look like? What would our qualities be? Would they relate to the domain? Perhaps the architectural style? Could we determine what the emphasis of a given design was? These seem analogous to the instrumentation, style influence and instrument emphasis qualities, respectively, as found on the music site. I wonder if there is some underlying structure of classification of specific designs that can be found. If so, constructing a version for software design would be much easier if we had something more generic to follow.

I think that the benefit of such an endeavor goes beyond my initial point. Yes, once we hashed out the details of such a set of qualities, we could analyze individual designs and try to match them up with each other. This would potentially be useful, and seems in many ways to hold more potential for good matches than approach #2 has for software, and more potential than this approach #3 has for music, even.

But finding these qualities and trying to really decide whether a design has them seems like it would be really fun, for lack of a better word. It would, for me, give me some sense of software aesthetics and qualities beyond the –illities. It is perhaps unrealistic to think that appropriate qualities could be found and applied, but that is what I would have thought about music. Maybe there is hope after all.

The software portion of this post started off about reuse and ended in seeking the nature of software. We know that reuse requires a good understanding of the nature of a given system, and I think the route may be through my, more specific, area of interest: the nature of software in general.

** For those who would like to check out pandora.com, be warned that it is fairly limiting in terms of how many songs you can skip, how long you can use it before registering for free, and then how long you can use it without paying. However, as is often the case, there are some easy workarounds if you are willing to do a little research.

Making Software is Like: Making an Animated Film
In a meeting I was in today, we were brainstorming interesting design fields to inteview practitioners in, seeking a broad understanding of software. One that caught my attention, and which seemed like it would make the subject of an interesting quick post is film creation. In particular, an animated film has a structure to its design decisions that I find very appealing.

In software, one way of considering the design process is that there is some head honcho who decides what kind of product will be built. We then have an architect who makes the highest level design decisions, and then there are module designers who plan things 0ut further as restricted by the architecture, and programmers who implement the truly low-level design decisions, which are the most predictable, and presumably least creative, of all.

At an animation studio, the big-wigs decide what movie to make, writers and artists storyboard it out, a animators make up the key frames based on those storyboards, and tweeners fill in the frames to smoothly animate between those key frames. At each stage, there are more and more people making lower-level design decisions, as restricted by those decisions above them. The lower levels are, arguably, less free and in some sense, creative, than the higher levels.

We see some semblance of this structure in architecture as well, and likely in design fields I haven′t even considered. I wonder how universal this structure is, and in what kind of domains it can be applied. Is it an element of complexity, and what kinds of complexity are well-suited to being broken up in this way? Its something I will have to consider, and will likely follow up on here, shortly.

Making Software is Like: Hunting and Gathering
I am no student of the subject, but in my layman’s concept of the development of early man I see some interesting parallels to software development. Originally, there were hunter-gatherers who relied on opportunities that nature presented them for sustenance. They went out and hoped that they would find the appropriate plants to gather and animals to hunt. There were several ways they could increase their chances of survival.
1) Knowing where to look. The first step in keeping yourself fed in a hunter-gatherer world is finding that which nature provides, and knowing where to look it certainly helpful.
2) Ensuring successful collection. While some approaches might be faster in terms of pounds-gathered-per-minute-of-gathering, gathering plants once you’ve found them is usually easy enough. But once you find that herd of delicious wildebeests, you still need to kill one. Hunting tools and techniques become very helpful in aiding this, often cooperative, activity.
3) Improving preparation. Finally, you have the food back at your camp. The steps that you perform before you eat it (cooking it, removing poisonous parts, etc.) can also greatly influence its usefulness to you.

The problem is that, if nature denies you access to the food in the first place, none of the rest of the steps matter. And in a hunter-gatherer world, this is largely beyond your control. With the development of agriculture and the domestication of animals, we really harness this uncertainty. It is the ultimate in “knowing where to look” since you know that your crops and herds are right where you left them. Furthermore, collection is certainly facilitated when you have control of your prey’s coming’s and goings. You’ve brought predictability to the process.

While I don’t know if this is an anthropologically valid conclusion, I see automation of the food-getting process as the next step. This is truly harnessing the third method of “improving preparation”. Once we have the techniques down as to how we want to prepare the food, we can automate the process and really get that food out there. Modern food-getting-systems add vast improvements in storage and distribution, but this is getting beyond the needs of my metaphor.

So, back to software. There has long been a call to make software production more manageable, more predictable. Some have asked why we can’t have mass-produced software, why making software can’t be more like making an automobile. Or at least more like our food-getting-system. I think that even the most demanding researcher would be willing to accept a system whereby there was some low-level uncertainty on the level of actually picking the coffee beans. Sure, there are some low level details that require human attention, they might say, but after that, lets automate this and get it out there. I can see this being a somewhat reasonable request if we see the “food” of software as code. We want to know where to gather our code, let some developers gather chunks of it, and let some system churn through the rest of the steps.

The problem is, I think the food of software is ideas. Ideas are what allow of to conceive of a basic premise for a program, what allow us to constrain its execution with requirements, what allow us to design it. Even code consists of little ideas about how to implement what is needed. And I just don’t think that ideas can be domesticated, nor lined up in rows to be picked.

Let’s compare primitive idea gathering to primitive food gathering. There are 3 steps, which roughly correspond to the find/get/use improvements I discussed before.
1) We go out into the world looking for ideas. We happen upon one – aha! What a good idea!
2) We have to wrangle it, gather it, get ahold of it, like we might hunting an animal – Ah! I see an interesting application of this idea!
3) Finally, we need to do something interesting with the idea, apply it, hone in on exactly what it is that it represents. We need to prepare it to actually provide us with benefit, as food provides sustenance - Ahhhhh! What a good idea that was!

I will quickly note that there is a nice correspondence to the three stages of design that John Chris Jones proposes: divergence, transformation and convergence. But more pertinently, which of these steps can we improve? I don’t think we can domesticate ideas, it is too hard to control where they come from. Creativity is opportunistic, just as the hunter-gatherers were forced to be. We can’t have an idea farm, and go out and pick ideas when we need them, we have to hunt them out. At best, we can improve our ability to know where to look for ideas, but they won’t be in neat rows behind the barn.

To some extent though, we can improve our techniques for catching ideas once we find them, and making them useful. We can improve our ways of thinking about software and build up better philosophical frameworks, so that when an idea comes our way we know how to think about it. As far as converging on a useful application of an idea, our methods and notations of expression can help us to build software out of the ideas that we have more readily.

But I don’t think we can truly automate these processes, ideas are just too different from one another. We can only try to tease out the similarities and generalizations that span software ideas and find ways to hone our skills in ways that are likely to be useful. In short, I don’t think we can really move beyond the hunter-gatherer level when it comes to software development we can only be the best hunter-gatherers we can be.

We dream of the day when software is produced like an automobile, churned off an assembly line, composed neatly of pre-made parts. I think a software house can instead seek only to be a seller of a mythical, impossibly finicky, rare poisonous pufferfish. This fish does not fare well in captivity, but can be found in certain distant reefs in the Pacific. We will search these areas with our knowledge of what types of coral is likely to attract the fish, hoping to find a few. With specially-designed spearguns, we will catch the fish. Though its eratic swimming makes this process imprecise, we are well trained in the use of our weapons. Finally, when the fish are returned to the warehouse, well-trained chefs expertly cut away the parts of the fish that are poisonous, using a series of cuts that has been found to yield the largest amount usable meat with the least risk of toxicity. But, of course, each fish is slightly different, and requires the skilled judgment of a professional to prepare. At last, the prepared meat is sent around the world, and sold at impossibly high prices. Every once in a while it kills someone, which is bad for business, but seen as intrinsic to the practice.

I think this is all we can hope for in software. We’re going to need wisdom, good tools and better people to find, harness and use ideas. We will not be able to truly automate these steps, given the unpredictable nature of our quarry and the product it produces, but we can hone the necessary skills to keep our families fed.

Making Software is Like: Kobe winning the Game with 0.6 Seconds Left
I was issued a challenge of sorts, to compare software to this game winning shot, which occured in a last-minute win by the Lakers over the Nuggets in their first game of the season.

What strikes me about this comparison is the time pressure, the risk of failure, and the human element. You never have enough time in software development, schedules seem to contract to fit the capabilities of the developers and the technology, and often contract a little more still. The game has a set length, and over its course, there seems to be plenty of time, but sometimes it comes down, after all that, to one shot to determine the winner. Point six seconds seems about right to capture the frenzied action and desperation of the last minute shot that some projects take on in the home stretch. Just enough time to catch and shoot, as they say. Just enough time to size up the situation, act and hope.

And in many cases, this shot is for the game. This shot is the difference between the project being used in whatever capacity it was intended, or being more or less useless. If the usual scary statistics about high project failure rates are to believed, there is a chance that the project might be completely blown, as a result, perhaps of this last-ditch effort’s failings.

If it’s a high risk situation, and time is short, you better get the ball to your best guys. After all, in basketball, as in software, the people make all the difference. Not to retread mythical man month territory, but only one person can take the shot in basketball. They might be assisted by another player, or benefit from a good screen, but the ball ends up in one man’s hands. I will stop short of advocating Brooks’ surgical team, but I will say that the success of this last minute push will often rest in the actions of a select few key individuals who need to get things done.

Though in the game in question, the ball actually went first to Kwame Brown, presumably a move by Lakers coach Phil Jackson to build the much-maligned, number one pick’s confidence. Brown missed the shot, but managed to rebound the ball and get it to one of the game’s top clutch players. I’ll withhold management advice of this level of detail, but is the moral that you take your first shot with some time left, you might have time for another? Seems like a lot to ask given the time pressures involved.

Software Assumptions: do we have to code electronically?
Back in action! The ICSE paper is out there, and hopefully on its way to acceptance. I want to start back in with an idea I raised a few posts ago (a few weeks ago), where I mentioned that I was such a software hippie that I wanted software engineers to sit around in a circle weaving software out of reeds. Ha ha, what hyperbole.

But would this be so bad? If you are going to work with people trying to create something really complex and really abstract, it seems like sitting in a circle and using your hands is a nice way to go. Using one’s hands to create something allows for subtle, intuitive actions to have subtle, intuitive effects. The creator receives tactile feedback, and usually has ready access to feedback in terms of their other senses as well; they can look at the product from different angles, smell it, or even taste it. Compare that to the process of creating software, where our actions are comparably clumsy and unintuitive, and our use of our senses is greatly limited. You would think, given how complicated the product we are trying to create is, we would want all these senses and subtleties at our disposal.

Of course, the big obstacle is machine interpretability, which as I’ve established is essential to programming being programming at all. We are trying to “weave software out of reeds”, all we’re doing is basket weaving, until we make that basket machine readable in some sense. We would need to find a way to have that basket make a computer do something.

Now in several ways, we already participate in tangible programming. Ubicomp folks have developed a variety of tangible interfaces, and using these to program provides some of the advantages of physical object manipulation. But so far, as far as I know, no such interfaces have been developed for programming, which is still stuck in a very textual paradigm, and thus largely keyboard-bound.

We can also consider the use of punch cards, back in the day as a form of tangible programming. Here was the crafting of a physical object that would eventually lead to a program. Given the abstract nature in which that object would be interpreted, most of the advantages of a tangible production product were absent in this process. You can smell your program! Not particularly useful. But again, of course, this medium was not designed with the needs of the programmer in mind, at this point in our development as proto-software-engineers, the machine’s needs were paramount, and we would give it whatever it needed, even if it was really a hassle for the human component.

Now, that’s not so much the case. Computers are powerful enough that we can make them do practically anything; and frankly, the border between the physical object and a digital representation is easier than ever to cross. The most obvious example is a 3d scanner, which usually sweeps a laser around an object, combining the reflections to calculate a digital, 3d model of the object. Sometimes sound or magnetism is used to track the object; many variations exist.

I think we could have a system whereby we create a model out of clay, scan it into a computer, and let that 3d object be automatically translated (compiled) into a computer program. There are some obvious problems with this, but do any of them prevent this from being basically possible? Yes, the program would be imprecise; trying to get it to exactly that specific thing that you want it to would be really tough. But as I mentioned in my last “software assumptions” post, I’m not sure that a program even needs to be deterministic, let alone predictable.

Also, would such a program have a proper design space? That is, the number of possible programs would be necessarily limited by the resolution of the scanner. Some quick research indicates that the resolution of one such scanner is 0.5 mm. So suppose you created a clay program that was a decimeter (about 4 inches) to a side. In this case, there would only be 200 points of resolution in each dimension, meaning only 8 million possible combinations. Assuming that we wanted any given program to be possible, that would only give us about 23 bits. Pretty miserably small, I’ll admit. And that even assumes that we could have any combination of matter/no-matter, which is a gross approximation given the limitations on scanning hollow or overly concave objects under most scanning systems. Resolutions will inevitably improve, color scanning will likely be developed and the programs could certainly be larger, but this method will likely never have the bitspace to work with that textual programs do.

But I don’t think that we necessarily need to be able to create any possible program, in a bitwise sense. I think that this would be a very, very high-level language, perhaps prototyping automated interface components, organic-looking visualizations or AI routines for a specific goal. I could also see this language being used to create algorithms for highly constrained purposes. For example, is it so far fetched for a clay model to represent a sorting algorithm? I think its possible.

Where would such narrow design space and imprecision be tolerable? When would it be worth it? Do we really need to be messing around with these things, scanning them into a computer and making programs? Is this sort of like the semaphore version of Wuthering Heights? Maybe there’s no answer barring further advances in technology, and we should just keep this possibility in mind.

Alternately, such a language might have applications as an educational tool, introducing students to concepts of design in a way that is easily understood and subject to discussions of quality and aesthetics. If an intuitive metaphor for the object created and the program that would result could be created, students could program collectively, manipulating their programs in an intuitive way, examining and commenting on the strengths and weaknesses of a given program. In some ways, this would bring the learning process into line with that found in other, more tangible disciplines such as graphic design and architecture.

But is this really programming? What’s the difference between this and having students just design buildings, since this is so far removed from programming as it is actually likely to be practiced. For one, I think having students design buildings wouldn’t be such a bad thing. But further, I think that such an exercise would teach students a lot about making a computer “do things”, and even the restricted solution space and imprecision would teach important lessons about high-level languages and unpredictability in software. And perhaps by explicitly avoiding the low-level precision of coding, we could teach students a thing or two about the process of actually honing in on a design, and on design methods, thought processes and techniques, while still remaining true, on some level, to software engineering.

Now such ideas are still fairly far off, given, if nothing else, the cost of such scanning technology. But it’s a thought to keep in mind: software creation does not need to involve the creation of information products alone, and I don’t think electronic creation is an assumption we can truly make about making software.