Why not?

Using Apache on Mac OS X to serve files outside ~/Sites

2012-01-14T15:57:00.000-05:00

I'm working on a web project that basically contains just static HTML and Javascript. (well OK, there's also one, small PHP script, but it might be going away in the near future). I tend to keep all my source code in ~/src, but to host it, I also need it to appear in ~/Sites. After some small trial-and-error, I ended up putting the everything (git repo and all) into ~/Sites, and then symlinked it to ~/src. It wasn't pretty, but it worked.

So I just did some reorganization that pretty much invalidated that old structure. In particular, I have moved everything that needs to be deployed into ~/src/project/web. However, I want it to be accessible via http://localhost/me/project. I tried physically moving the project back into ~/src, and then making a symlink to the subdirectory, but that didn't work. Apache would still produce 403s for all the relevant files. So I had to roll up my sleeves and dive into Apache configuration.

Before I go further, I'm compelled to pull out the old soapbox. I have painfully little experience with Apache - I have never had to configure or support it in a production environment, and that makes me happy. From this position of ignorance, I have decided that Apache is a dinosaur that should have died a long time ago. For example, instead of configuring the server from the request's point of view (as has been popularized by Rails' routing logic), it is configured from the filesystem's point of view. The default Mac OS configuration has, buried somewhere in the middle of the file, a directive that disallows the serving of all files under /. Because, I guess, they would be served by default if that directive wasn't present? But still, nobody seems to want to spend the time to produce a replacement web server, and so we struggle on. </rant>

The default Apache install on MacOS 10.7 uses a split apache configuration file. The bulk of the configuration is in /etc/apache2/httpd.conf. However, each user also gets their own /etc/apache2/users/me.conf, which are all imported into the main configuration. And while the main configuration file specifies the FollowSymlinks option, I disconvered that the same is not true in my personal config file. All I had to do was to add the FollowSymlinks option to that configuration file, restart Apache, and everything started working.

So if you have only basic web serving needs, the default config should suffice. If, however, you want/need to spread the files around your disk, you need to mess with the Apache configuration.

Mysterious, Blank User in 10.7 Sharing Dialog

2011-09-05T21:40:00.001-04:00

I wanted to copy some files from my PC to my Mac. When I went to turn on SMB sharing, I came across this:

I was wondering about the identity of this phantom user. It turns out that it is the macports user. He doesn't show up on the login screen. He never showed up under 10.6. Apparently, Apple changed something about the way users are reported to applications.

If it bothers you, you can fix it with dscl.

sudo dscl . -create /Users/macports RealName macports

The first parameter is the machine you want to administer; . is apparently a shortcut for localhost. Then we give the command - we want to create a new key. Then we specify where this key should be created - in this case, the macports user's Directory Services path. Next is the name of the key - RealName is what appears to be used by the sharing dialog. (RealName is also assigned on users you create through System Preferences). Finally, we provide a value for this user's name. Now, we have this:

DES Encryption As Used in VNC Authentication

2010-09-14T00:42:00.000-04:00

A few notes about how DES is used in the VNC Authentication security type (type 2)

DES is used in ECB mode.
The ECB Key is based upon an ASCII password. The key must be 8 bytes long. The password is either truncated to 8 bytes, or else zeros are added to the end to bring it up to 8 bytes. As an additional twist, each byte in flipped. So, if the ASCII password was "pword" [0x 70 77 6F 72 64], the resulting key would be [0x 0E EE F6 4E 26 00 00 00].
The VNC Authentication scheme sends a 16 byte challenge. This challenge should be encrypted with the key that was just described, but DES in ECB mode can only encrypt an 8 byte message. So, the challenge is split into two messages, encrypted separately, and then jammed back together.

Here is some pseudocode (in Erlang) that should explain better than words can.

password_to_key(Password) ->
    Flipped = lists:map(fun flip_byte/1, Password),
    Truncated = truncate(Flipped, 8),
    pad(Truncated, 8, $\0).

encrypt_challenge(Password, Challenge) ->
    Key = password_to_key(Password),
    <<High:8/binary, Low:8/binary>> = Challenge,
    EncHigh = crypto:des_ecb_encrypt(Key, High),
    EncLow = crypto:des_ecb_encrypt(Key, Low),
    <<EncHigh/binary, EncLow/binary>>.

Unpacking a Safari Extension

2010-07-28T22:03:00.001-04:00

So, now that Safari extensions are official (and not just a developer curiosity), I decided to see what people had managed to make over at the extension gallery. It looks like there are some cool ideas out there. I was somewhat interested in the Exposer extension, which sounded a bit like Exposé for Safari. It seems like it kinda works, except that it doesn't always bring up the list of windows, and it's also really slow (it looks like visibleContentsAsDataURL is the culprit, natch, plus I have dozens of tabs open at a time).

Anyway, while I was checking it out, I realized that I had no idea what some of these Safari extensions were doing in the background. Stop and think for a moment; do you really want to run code that some Jimmy wrote in his basement to be able to watch everything that you do in your browser? Maybe I'm just paranoid, but I'd like to know what is really going on.

So, naturally, I tried unpacking an extension. It wasn't particularly hard, but you have to realize that a .safariextz file isn't a ZIP archive. It's a XAR. I know; I opened it up in my hex editor.

Here's how you can unpack one for yourself:

xar -xvf extension.safariextz -C ~/Desktop

Don't worry; there's a directory just inside the safariextz archive. Now to see if there's anything malicious in these extensions. (Exposer looks clean so far.)

What I learned today

2010-06-28T23:33:00.000-04:00

Drybrushing comes first.

Even More Space Marines

2010-04-29T19:48:00.000-04:00

I've spent some more time on my Space Marine squad (it's going on something like 6 months at this point - I just don't get a ton of time to paint). Anyway, the tactical squad is basically done. Notice the highlights.

In addition, I started to work on some figures from Assault on Black Reach, including some terminators. In particular, I spent quite a while trying to get the white helmets to look good. I'm also pretty pleased with the eyes. No Golden Demons here, but not bad for tabletop.

Finally, I had a spare marine sitting around, so I decided to make him up as a Blood Angel. I have two copies of Space Hulk that I want to paint, but I wasn't going to do so until I was ready. I think I'm almost ready. Also, another shot of the terminators' eyes.

Things I Learned While Debugging an SSL Issue

2010-04-15T09:02:00.001-04:00

SSL is sometimes actually TLS. SSL is apparently on the way out, though TLS is only supported in a subset of common browsers. Fortunately, both use the same kind of certificate, so it's mostly transparent.
Java 1.6u17 removed SSL client support for MD2-signed root certificates. Except it sometimes didn't. Some u17 installs worked for me, some failed. 1.6u19 failed every time. If you have a Java client connecting to a SSL server, make sure that the server certificate was generated against a SHA1-signed root certificate.
WireShark will analyze both SSL and TLS. If there's any confusion about what is coming from the server, WireShark can help you figure it out.
The server sends the whole certificate chain to the client. I had thought that this was the case, but I had a hard time finding the documentation that spells it out. In the end, I used WireShark to find out.
Web browsers sometimes lie. When I would ask the web browser for the certificate chain, it would tell me something different from what the server actually sent. The root certificate from the server was signed with SHA1, but the browser would tell me that it was signed with MD2. This occurred in Internet Explorer, Firefox, and Safari. This was also a red herring that caused me to waste a lot of time.
Make sure you are looking at the right server. I had made an assumption about how the Java client software talked to the server, and that assumption was incorrect. In the end, the problematic certificate was on a different server altogether. Go figure.

Deriving the Y Combinator in Erlang - Part 2: Abstraction

2010-01-29T12:53:00.000-05:00

This is the second post in a series on the Y combinator. Part 1

In the last post on the Y combinator, we established that some functional languages (such as Erlang) make it hard to have recursive, anonymous functions. Namely, there is no name that is bound to the "current" anonymous function. Furthermore, we established that we could work around this problem by passing the anonymous function to itself. When we concluded, we had derived this much:


Foo = fun(AlsoFoo, N) ->
    case N of
        1 -> 1;
        _ -> N * AlsoFoo(AlsoFoo, N - 1)
    end
end,

Fact = fun(X) -> Foo(Foo, X) end

This post will focus on extracting the Y combinator from the above code. We will follow a sequence of transformations, each with a specific intent. In the end, we will hopefully have some beautiful code.

Reducing the Number of Arguments

This factorial algorithm started with a function that took a single parameter. This function called itself with successively smaller values, until it reached a base case. However, we muddied the waters by passing the function around as well. We would like to return to having a one-parameter function. We can accomplish this by creating another level of closure:


Foo = fun(AlsoFoo) ->
    fun(N) ->
        case N of
            1 -> 1;
            _ -> N * (AlsoFoo(AlsoFoo))(N - 1)
        end
    end
end,

Fact = fun(X) -> (Foo(Foo))(X) end

Now it is clear that our recursive function is really only calling itself with one parameter.

Simplifying the Recursive Function Call

The place where we make the recursive call is rather ugly. We have to build the function upon which we will recurse before we can actually call it. It would be better if the recursive function was explicitly named. We can do that, too:


Foo = fun(AlsoFoo) ->
    fun(N) ->
        case N of
            1 -> 1;
            _ -> 
                AlsoFooAlsoFoo = fun(Z) -> (AlsoFoo(AlsoFoo))(Z) end,
                N * AlsoFooAlsoFoo(N - 1)
        end
    end
end,

Fact = fun(X) -> (Foo(Foo))(X) end

We can simplify the body further by moving AlsoFooAlsoFoo to a higher scope.


Foo = fun(AlsoFoo) ->
    AlsoFooAlsoFoo = fun(Z) -> (AlsoFoo(AlsoFoo))(Z) end,
    fun(N) ->
        case N of
            1 -> 1;
            _ -> N * AlsoFooAlsoFoo(N - 1)
        end
    end
end,

Fact = fun(X) -> (Foo(Foo))(X) end

As an aside, you might find this step overly complicated. You may ask, why did we not simply define AlsoFooAlsoFoo = AlsoFoo(AlsoFoo). I hope to get into the details in a later post, but for now, just realize that would change the evaluation order in a bad way. Or, try it yourself and find out why it doesn't work.

Extracting the Anonymous Function

Right now, the body of our algorithm is embedded deeply within some necessary plumbing. We would like to extract our algorithm from the center of this. This is quite easy:


Foo = fun(AlsoFoo) ->
    AlsoFooAlsoFoo = fun(Z) -> (AlsoFoo(AlsoFoo))(Z) end,
    FactRec = fun(Self) ->
        fun(N) ->
            case N of
                1 -> 1;
                _ -> N * Self(N - 1)
            end
        end
    end,
    FactRec(AlsoFooAlsoFoo)
end,

Fact = fun(X) -> (Foo(Foo))(X) end

Now we can pull it outside the body of Foo.


FactRec = fun(Self) ->
    fun(N) ->
        case N of
            1 -> 1;
            _ -> N * Self(N - 1)
        end
    end
end,

Foo = fun(AlsoFoo) ->
    AlsoFooAlsoFoo = fun(Z) -> (AlsoFoo(AlsoFoo))(Z) end,
    FactRec(AlsoFooAlsoFoo)
end,

Fact = fun(X) -> (Foo(Foo))(X) end

Simplifying Fact

We're getting close, but the definition of Fact still leaves something to be desired. However, in order to make it simpler, we have to first make it messier. Start by moving Foo inside the definition for Fact:


FactRec = fun(Self) ->
    fun(N) ->
        case N of
            1 -> 1;
            _ -> N * Self(N - 1)
        end
    end
end,

Fact = fun(X) ->
    Foo = fun(AlsoFoo) ->
        AlsoFooAlsoFoo = fun(Z) -> (AlsoFoo(AlsoFoo))(Z) end,
        FactRec(AlsoFooAlsoFoo)
    end,
    
    (Foo(Foo))(X)
end

Our goal is to build a general-purpose function Y that takes a function F and produces a self-recursive version of that function. Right now, the innermost part of Foo makes an explicit reference to FactRec. We want to eliminate that explicit reference:


FactRec = fun(Self) ->
    fun(N) ->
        case N of
            1 -> 1;
            _ -> N * Self(N - 1)
        end
    end
end,

Fact = fun(X) ->
    Y = fun(Proc) ->
        Foo = fun(AlsoFoo) ->
            AlsoFooAlsoFoo = fun(Z) -> (AlsoFoo(AlsoFoo))(Z) end,
            Proc(AlsoFooAlsoFoo)
        end,
        Foo(Foo)
    end,
    (Y(FactRec))(X)
end

Now that we've done this, Y no longer has any bound variables, so we can pull it out of Fact completely:


FactRec = fun(Self) ->
    fun(N) ->
        case N of
            1 -> 1;
            _ -> N * Self(N - 1)
        end
    end
end,

Y = fun(Proc) ->
    Foo = fun(AlsoFoo) ->
        AlsoFooAlsoFoo = fun(Z) -> (AlsoFoo(AlsoFoo))(Z) end,
        Proc(AlsoFooAlsoFoo)
    end,
    Foo(Foo)
end,

Fact = fun(X) ->
    (Y(FactRec))(X)
end

Of course, if we want, we can simplify some of these definitions. Y can become a normal Erlang module function (rather than a function value). Fact itself can be curried - we can eliminate the noise of the explicit parameter. Also, FactRec doesn't need to be named anymore - it can become the anonymous function that we originally intended:


y(F) ->
    G = fun(AlsoG) ->
        F(fun(Z) -> (AlsoG(AlsoG))(Z) end)
    end,
    G(G).


Fact = y(fun(Self) ->
    fun(N) ->
        case N of
            1 -> 1;
            _ -> N * Self(N - 1)
        end
    end
end)

Unfortunately, the y function only supports functions that take a single parameter. Some languages have a "splat" operator that can be used to represent "all the parameters;" unfortunately, Erlang does not. Instead, it can be useful to define a family of y functions that deal with functions taking more than one parameter:


y2(F) ->
    G = fun(AlsoG) ->
        F(fun(Y, Z) -> (AlsoG(AlsoG))(Y, Z) end)
    end,
    G(G).

Conclusion

We have shown the difficulty in defining recursive, anonymous functions in Erlang. We showed a simple solution to this problem, and then generalized the plumbing to make it easier to use. While this is not necessary in all functional languages, I hope that this is useful to anybody working in a strict language, such as Erlang.

I am planning more posts on this topic. One post will explain the strange step I took in Simplifying the Recursive Function Call. Others will explain just what a fixed point is, what the fixed point combinators are, and how the Y combinator satisfies the definition of a fixed point combinator.

Deriving the Y Combinator in Erlang - Part 1: Intuition

2010-01-18T14:18:00.004-05:00

This is the first of a series on the Y combinator. Part 2

When I heard about the fixed point combinators before, I didn't know what to make of them, so I filed the topic away in my brain. However, when I was working on implementing continuations in Erlang, I ended up building a small structure that reminded me of the Y combinator. With a little massaging, I extracted the actual Y combinator, and proceeded with what I was working on.

The actual definition of the Y combinator is insanely dense:

Y = λf·(λx·f (x x)) (λx·f (x x))

This is precisely why I didn't understand it at first - that definition means nothing to me. We need a more intuitive way to think about it.

Suppose you decide to write the factorial function in Erlang. A simple (by which I mean unoptimized) implementation might look like this:


fact(N) -> 
    case N of
        1 -> 1;
        _ -> N * fact(N - 1)
    end.

There's nothing particularly complicated here - we're just calling fact recursively. But what happens if you try to make fact into a fun (an anonymous function in Erlang). Watch:


Fact = fun(N) -> 
    case N of
        1 -> 1;
        _ -> N * ??? (N - 1) %%How do we call ourselves? Fact isn't yet bound!
    end
end.

In some languages, we could replace the ??? with Fact. Unfortunately, Erlang doesn't let you do this. If you tried, Erlang would say that Fact is unbound. This is true - until we've finished defining the fun, we can't assign it to Fact. Other languages provide you with a magic variable that represents the current function (Javascript has arguments.callee). Again, as far as I know, Erlang doesn't provide such a variable. Does that mean that we have no hope?

Let's look at this problem one step at a time. We need something to stand in for the ???. We need a name that represents the current, anonymous function. Where can we get that from? In functional Erlang, there are only three ways that names are bound - by closure, by parameter, or by local definition. We can't close over it, because the anonymous function isn't yet defined. We can't create a local definition, because the local scope is too narrow a scope for that. That leaves only one possibility - we need to pass the anonymous function to itself.


Foo = fun(AlsoFoo, N) ->
    case N of
        1 -> 1;
        _ -> N * AlsoFoo(AlsoFoo, N - 1)
    end
end.

Fact = fun(N) -> Foo(Foo, N) end.

OK, so we created a helper function - more on that in a minute. Foo (formerly Fact) now takes an extra parameter, which just creates another name for the current, anonymous function. Since we intend for that to be the same as Foo, we call it AlsoFoo. We know that Foo is a fun/2. Since AlsoFoo is supposed to be another name for Foo, then AlsoFoo must be a fun/2 as well. This means that, when we call AlsoFoo, we need to also tell it about itself - that is, we need to pass AlsoFoo along when we call AlsoFoo to recurse.

Now that leaves us to deal with the function Fact. Clearly, Fact needs to call Foo. We noted that Foo is a fun/2, so again, we need to call it with two parameters. The intent of the extra parameter to Foo was to be able to pass Foo to itself, so we do just that.

Believe it or not, we have just derived the concept behind the Y combinator. We have invented a scheme that allows an anonymous function to know itself, even in a language that doesn't directly support this. This is (I believe) the purpose behind the Y combinator. However, we're not yet done. There is still some cruft that we would like to eliminate. In particular, we hand-built the plumbing to route AlsoFoo around. We would like to use higher order functions to eliminate this. This is what the Y combinator does - it manages the plumbing of anonymous functions that refer to themselves.

In the next part, we will continue the derivation of the Y combinator in Erlang. Our goal is to eventually be able to write something like this:


Fact = y(fun(Self) ->
    fun(N) ->
        case N of
            1 -> 1;
            _ -> N * Self(N - 1)
        end
    end
end).

It's not perfect, but in a language that doesn't directly support anonymous function recursion, it's not too bad!

Even More Space Marine Painting

2009-12-07T14:52:00.000-05:00

I've been slowly working on my Space Marines. It's taken a while, but they almost look like a unit. I think I've spent between 10 and 20 hours on them, but much of that was spent learning. Most of the major painting is done, and now it's time for touchups and details. For example, I spent some time on that rocket launcher to make it appear to be metal, painted red, and then worn. I think it's pretty convincing. I intend to do the same with that red bolter. The one that's not wearing armor needs a lot of work. Enjoy.

Standard algorithms and boost::ptr_vector

2009-11-17T18:08:00.000-05:00

I did something bad the other day.

OK, I can't tell if it was bad. In another environment, it would have been bad, but since this was C++, perhaps it was OK. I was in the situation where I had a boost::ptr_vector, and I wanted to use a standard algorithm on it. Specifically, I wanted to use std::partition to separate the objects that were still "alive" from those that were "dead" (where alive and dead are domain concepts in our application). The complexity here is that ptr_vector is a crazy container.

Most containers deal with a specific type T. You add Ts to the container. Dereferencing an iterator gives you a T&. It's generally assumed that a container operates on a single type, and the standard algorithms make this assumption.

The ptr_vector, on the other hand, appears to be two containers at once. Semantically, it's analogous to a std::vector<managed_ptr_type<T> >. It is intended that, by adding a pointer to the ptr_vector, the ptr_vector takes ownership of the lifetime of the memory at the end of the pointer. So, it is a container of pointers. On the other hand, when iterating a ptr_vector, it appears that it is a container of Ts.

In my case, I wanted to rearrange my ptr_vector. In particular, I wanted to partition the pointers into those whose object was still "alive", and those whose object was "dead". Since a ptr_vector is semantically a container of pointers, it made sense that I should apply std::partition to the ptr_vector. However, ptr_vector::iterator removes a level of indirection: instead of iterating T*, it iterates T&.

In fact, ptr_vector doesn't seem to provide any ways to rearrange the pointers once they are put into the container. Sure, you can mutate the object on the end of the pointer. You could operate at that level. But there doesn't appear to be a safe way to treat the ptr_vector as a container of pointers.

Fortunately, ptr_vector provides a back door. Its iterators support a base() method, which will return an iterator over T* (instead of an iterator over T&). This allows us to treat the ptr_vector as a container of pointers, and to use standard algorithms to manipulate those pointers. Now, this is not without peril. While it seems to be OK to rearrange the pointers, it wouldn't be safe to change the set of pointers. I wouldn't trust using something like std::remove_if, because it might leave garbage in the container after it is done. The container might contain duplicate pointers. Some pointers might get dropped completely. If the container then goes out of scope, it will try to delete these pointers multiple times, which would be a bad thing. It might also fail to delete some pointers, because they were overwritten (and not preserved elsewhere in the container).

This whole thing felt like the best solution possible, while at the same time leaving a lot to be desired. I felt like I was violating the encapsulation of the ptr_vector. I suppose this is one of those cases for which they put in the base() methods on the iterators. Additionally, I don't see any clear way that they could do better. For example, I think an assumption of ptr_vector is that a given pointer only occurs inside it at most once. The standard algorithms don't necessarily respect this assumption; see my commentary on remove_if in the previous paragraph. The standard algorithms, in some cases, expect more freedom than ptr_vector can provide. This disconnect is unfortunate, but not without reason.

An important first step to helping with this problem would be to add methods to ptr_vector (and its siblings) that allow you to treat it as a container of pointers. You could add, remove, and re-arrange the container using these methods. In addition, they could maybe provide specializations of some of the standard algorithms for each container. This is difficult for third party developers to do, since the actual type of a ptr_vector::iterator is implementation defined. The boost guys can cleanly provide a specialization of std::partition for this kind of iterator, but I can't. Now, this isn't perfect. It would help with the standard algorithms, but not third-party algorithms. Still, it would be a great step in the right direction.

So, did I do something bad, or did I do something necessary?

Why Google Experience phones are pretty awesome

2009-11-02T12:49:00.002-05:00

As Android has grown, devices fall into one of two major classifications. Some devices are so-called "Google Experience" devices (featuring the phrase "with Google" somewhere on the device). Other devices are, well, not Google Experience devices. What is the difference? I've had a hard time figuring it out.

I think that Google Experience phones are updated by Google itself, while the rest of the devices are supported by the phone's manufacturer. I have an original G1 (a Google Experience phone), and I've gotten prompt updates as each new Android OS version has been released. This is similar to the experience that iPhone users enjoy.

Some devices, such as the HTC Hero and the Motorola Cliq (and the HTC Magic in certain regions), are not Google Experience phones. These phones were released with heavily customized software (such as HTC's Sense UI or Motorola's Motoblur). These customizations, while attractive to some users, also make it much harder for the phone manufacturer to update to a new version of the base Android OS. Both the Hero and the Cliq shipped with Android 1.5, and I don't believe that there are announced plans to update either to 1.6 (or 2.0, for that matter).

At first, I thought that the notion of a Google Experience phone was silly. At the time, the Magic was launching on Rogers with Exchange support, and that somehow disqualified the phone as being a Google Experience device. I now understand that Google Experience really means "unforked code base". In order to add Exchange support, I suspect that HTC had to fork and modify the standard Mail app. While they were able to add a feature that people wanted, it really just makes these phones into some sort of mutant Android device. No thank you. Google should really make it clear to users that the Google Experience is a feature in and of itself.

Android, at this point, is a rapidly evolving platform. Google Experience phones seem to be the best way to keep up with this evolution. I was pleased when I heard a Verizon rep say that the Droid will be a Google Experience phone. Now they just need to release a T-Mobile US GSM version, and I'll be happy. Over time, Android evolution will slow down, and then it might make sense for a manufacturer to fork the Android code base. Maybe they would even be willing to contribute back to the core distribution. But, until then, I'm sticking with Google Experience devices.

Fixing hard disk clicking / aggressive head parking on Mac OS X

2009-11-02T12:13:00.000-05:00

I recently bought a Western Digital Scorpio notebook hard drive to put into my 2007-vintage Macbook Pro. Everything seemed fine at first. However, as I used my laptop, I noticed that it would frequently make a quiet clicking noise. At first, I thought that I had gotten a bad disk. However, after doing a little research, it became clear that this is a common problem. This clicking is a "normal" operational noise - it is the sound of the heads parking.

People say that you should just get used to the noise. However, this blog post makes the argument that every one of these clicks is killing your hard disk. Some people claim that this is related to the sudden motion sensor that's built into most (if not all) Apple portables. However, this is a red herring. The disk still clicks even if it is sitting on a table. It is the hard disk's own built-in power management that is causing the head parking. The disk's SMART statistics record the number of head parking cycles. If you want to see this for yourself, you can use either this menu extra or this command line tool (MacPorts). You are looking for the Load Cycle Count value.

To explain the problem (as I understand it), modern hard disks have some responsibility to manage their power consumption. One manifestation of this is to spin down the platters and to park the read/write heads. The operating system can influence the time before the heads are parked by setting the "APM Level" of the drive to a value between 0x00 and 0xfe. Each drive manufacturer is free to interpret this value as they see fit. Mac OS X seems to set a default APM Level for all disks, and I think this value is 0x80. This is fine with Apple-shipped disks, but not necessarily for third party disks.

But wait! Perhaps you have bought the same kind of drive that Apple ships in their laptops. Are you safe? Not necessarily. Allegedly, Apple flashes their own firmware onto the the hard disks that they install at the factory. That's right, you're not running stock disk firmware. My suspicion is that this firmware changes the drive's interpretation of the default APM level. Recently, there was a firmware update from Apple that fixed this problem on disks that were shipped by Apple. Unfortunately, you can't use this utility to flash the new firmware onto a non-Apple drive.

Right, so the two solutions that I see are either:

Write our own firmware
Set a different APM Level value

Obviously, option 2 looks much more attractive. Bryce McKinlay wrote a utility called hdapm for doing just that. He even includes a launchd configuration to run hdapm as the system starts. One thing not mentioned in the readme is that you need to get the permissions of the launchd config file correct. The file needs to be owned by root (preferably root:wheel), and must not be group- or world-writeable. I also changed the config file a little; here is my version:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
 <key>Label</key>
 <string>hdapm</string>
 <key>ProgramArguments</key>
 <array>
  <string>/usr/local/bin/hdapm</string>
  <string>disk0</string>
  <string>max</string>
 </array>
 <key>RunAtLoad</key>
 <true/>
</dict>
</plist>

The biggest change is that I removed the "LaunchOnlyOnce" and "ServiceDescription" keys. I didn't see a reason to load it only once, and ServiceDescription seemed undocumented. This solution isn't perfect, however. First of all, hdapm uses a seemingly undocumented back door to adjust the APM setting. Ideally, we would actually spawn a daemon that continuously monitors and adjusts the drive's APM level. I'm not yet convinced that Mac OS X won't override my setting. Still, I have been running with this configuration for a couple of days, and things seem to be working well.

I have an open support issue with Western Digital to see if they have a fix for this issue. If there were some way that we could change the way the disk behaves under OSX, we could forego the additional software, which would be great. I've also heard of a utility called wdidle, which allegedly lets you write new idle settings to the hard drive. However, I was unable to find any official site for this software, so I'm not using it.

Finally, I would like to thank two people. First, Doug Aghassi's post really explained the symptoms that he was experiencing and put me on the right track for solving the problem. Thanks, Doug. Also, Bryce McKinlay was kind enough not only to write the hdapm utility, but also to answer the questions that I emailed to him. Thanks, Bryce.

More Painting Space Marines

2009-09-20T11:57:00.000-04:00

I have an update to my Space Marine painting. I've continued to shade the marines. After the previous coat of 1:1 Regal Blue to Ultramarine Blue, I added the following:

1:2 Regal Blue to Ultramarine Blue
Ultramarine Blue
2:1 Ultramarine Blue to Codex Grey

Each coat is painted in a slightly smaller area. The goal is to shade the model to match an imaginary light source. In my case, my light source is directly above the model. Here are the results:

I'm now only painting 2 marines. I'm going to finish them before starting others, so that I can improve on my technique for those later models.

I also started painting my Tyranid spores. This has been basecoated with Chaos Black spray, then painted with Blood Red, then washed with Chestnut Ink. I had tried Red Ink, but unfortunately, that color is too close to Blood Red.

Painting Space Marines

2009-09-16T22:00:00.000-04:00

For some reason, I got an itch to paint some Warhammer 40k figures. I've been slowly assembling them over... I don't know, maybe 2 years. Hey, I have a lot of things that I do in my free time. Anyway, I finally got around to priming them the other day, and I've been painting like a crazy person since then.

Now, I should mention that these aren't the first figures I've painted. I had painted a squad of 5 marines that came in a box along with 6 paints and a brush. Here's one of them.

That was a good learning experience. This time, I have quality glue, basing material, spray primer, a variety of brushes, and lots of paints and inks. I'm posting some photos of various steps of the process. Hopefully, by putting them here, I will actually finish painting them.

This space marine has been assembled, based, and primed. The basing material is sand and rock, glued into place with regular white glue. The whole model is sprayed with a flat black primer, to help the other paint stick better.

This marine has had his armor painted with Ultramarine Blue. Actually, his feet are missing some paint, but I'll get around to that. Additionally, the black ground cover has been drybrushed to look like sand and rocks. I started with rocky sand, painted it black, then painted it to look like rocky sand again. Crazy? Probably.

This marine has been washed with a blue ink (which I think has been replaced with Asurmen Blue Wash). Ink is used because it settles in the crevasses and provides great depth.

This marine has had his armor panels painted with a 50/50 mix of Regal Blue and Ultramarine Blue. By leaving a slight gap around the edges, the blue wash peeks through the armor panels, and this looks great.

For those who don't know much about 40k, these figures are pretty small. Here's a comparison shot.

Now imagine trying to paint those eye lenses. Yeah, I'm not looking forward to it, either. Besides the eyes, I still have a lot to do. I plan to put another few layers of blue on the armor, paint the shoulder pad edges, drybrush the metal pieces, and so on. If anybody reads this and has feedback or suggestions, I'd love to hear from you!

I wish Blogger would let me rename tags

2009-08-07T13:36:00.003-04:00

Occasionally, if you read this via RSS, you will notice that I repost old articles. This isn't intentional - any time I edit or retag a post in Blogger, it puts it back on the feed. I don't see any way to say "quietly republish this post". The same problem occurs if I want to rename a tag - I have to remove the tag from all posts, and then re-add it to all posts. Please, Google, add features to Blogger to make this less painful.

T-Mobile Visual Voicemail Problems

2009-08-07T12:49:00.005-04:00

T-Mobile recently released their Visual Voicemail application in the Android market. It had some launch problems, but those are mostly smoothed out at this point. The app works pretty well, and I'm glad that they have finally implemented it. However, the app does have its share of first-release problems. They are listed here, in the order that I hope T-Mobile addresses them.

VVM doesn't work with Wifi. Most people probably have Wifi enabled on their phones. After all, it's the most efficient (both bytes / time and power / byte) way to tranfer data. However, VVM doesn't work with Wifi. It will neither notify you of new voicemails, nor will it download new messages. In order to make it work, you need to

Turn off Wifi
Wait or click "Synchronize Voicemails"
Turn Wifi back on

Call me crazy, but that's just stupid. At the very least, the "Synchronize Voicemails" button should do those steps for you, similar to the My Account app. There has been some FUD about the reason for this omission. Some people claim that it's for "security". I'm going to make this perfectly clear: there is no security-related reason to prevent people from getting their voicemails over Wifi. It's easy to encrypt data that is transmitted over the internet. There are a number of possible reasons they don't support Wifi. It might just be too much work. Maybe they haven't had time yet. It might be cost prohibitive. Perhaps there is a technical restriction - they would need to read the SIM's IMSI into the app, which Android might not allow. Whatever the case, it's not a security issue.

VVM has a separate notification icon. Every time you get a new voicemail, you get the standard voicemail icon. In addition, you get a new VVM icon. For now, this is fine. If I have Wifi enabled, I still get notified of a new voicemail (via the standard voicemail icon). When the Wifi issue is fixed, however, I would like to see the new icon go away. The notification bar is crowded enough.

VMM uses the wrong audio stream. VVM uses the "media" audio stream. Many people complain that prevents you from using a bluetooth headset to listen to your VM. I don't have a bluetooth headset, so I can't confirm this. It should use the "phone" audio stream.

The UI needs polish. There are some small look-and-feel issues:

The VVM status bar icon doesn't match the Android UI Guidelines.
After pressing the "Synchronize Voicemails" button, there is no feedback. No spinner, no progress bar, nothing.
The long-press context menu on a VM does not include a "delete" option (only Open, Reply As, and Copy To)
The buttons that appear when you press the "Menu" button have no icons.
The "Copy to" screen is a little too technical. The file name defaults to vmn (i.e. vm0, vm1, vm2). It should instead default to something like "Voicemail from John Smith on 22 Jul, 2009". In addition, the save directory defaults to "/sdcard". Should users really be exposed to UNIX pathnames? Clicking the Save in Directory dropdown presents me with a file browser for my SD card. For me, this lists locations like ".Trashes" (I use a Mac), espeak-data (the data files for the Text-To-Speech engine), "where" (the data for Where), and other places that I probably shouldn't be saving random files. Do I really need to be able to specify the location to save the voicemail? Why not just save everything to /sdcard/voicemails? Or at least, why not assume that all voicemails get saved to /sdcard/voicemails or a subdirectory (i.e. you can't save a voicemail outside this directory, only inside)?

Initial, first-run experience is lousy. When I first installed and ran the app, it wasn't able to connect to the server. After disabling Wifi, it worked. I was taken to a set-up screen, but then got distracted by something and hit the back button. When I relaunched the application, the set-up screen wasn't presented. This worried me (is there some setup that needed to occur), so I uninstalled the app and re-installed it. I don't think I did any harm, but the app didn't behave as I expected, so I didn't know what to think.

Deleting doesn't always work. I'm going to chalk this up to glitch behavior. The first time I used the app, I went through and deleted some old messages. Then I went into the analog voicemail system, and they were back! I deleted them a second time, and now they're really gone. shrug

Now, I don't mind all of those problems. I'm glad that T-Mobile finally released a VVM app. I'm glad that they released it early, warts and all. I hope that they are not done working on it. For me, the Wifi issue is huge. I'm connected to Wifi 90% of the time, and that means that the VVM app doesn't function as a voicemail app 90% of the time. I suspect many other people are in the same boat as me. Furthermore, Google Voice is coming. If the Wifi issue isn't fixed by the time GV is generally available, I might just jump ship, and T-Mobile doesn't want me to do that. I understand if T-Mobile can't fix this on their own - they might need support from Google. Still, every carrier is going to want to provide VVM, and it would behoove Google to provide whatever support necessary.

Getting Started with Stack Overflow

2009-07-18T20:37:00.004-04:00

I joined Stack Overflow shortly after it launched, but I didn't do anything with it. I found it in search results here and there, but I never asked any questions. I would have done more, but new users are pretty helpless. You can't vote up or down, you can't comment on answers, you can't post an answer with more than 1 link, etc. It's almost like you're not wanted. Compared to the relative freedom of Wikipedia, it was really demoralizing to me.

I decided tonight to actually try to get some reputation. Most of the interesting stuff happens around 50 reputation, so that's my goal. I answered 2 questions this evening. Suddenly, my rep is skyrocketing. I'm at 31 right now, and I bet that will continue to climb on its own. It seems that people are very willing to vote your answers up if they are relevant. As you can see, it shouldn't be hard to get to the point of actually being able to contribute.

So, if you want to get started with Stack Overflow, here are my suggestions.

Go to the newest questions page.
Find something that you know something about. Don't troll, and don't post to random topics about which you know nothing.
Write an answer.

That should just about do it. Don't despair, it's easier than it initially seems.

Fixing the Xbox 360 's Grinding Noise / Tray Ejection Problem

2009-07-16T02:06:00.003-04:00

I spent an evening performing unexpected surgery on my Xbox 360. When I put a game in, the drive made the most horrible grinding noise. On top of that, the drive would not stay closed. The tray would almost always eject seconds after being closed. Research led me to conclude that the rare earth magnet that is part of the disc clamp had probably become unglued. Since my initial warranty has long since expired and the red ring of death warranty only has another year, I decided to crack the case myself.

It's not worth going through the details, but I did find two useful videos. The first is an overview of the problem. The second is a good tutorial on opening the 360's case. I used some Zap-A-Gap brand contact adhesive that I had laying around to actually reattach the magnet.

I felt quite proud to have diagnosed, researched, and fixed the problem on my own (without sending my console to Microsoft for repairs). $100 plus shipping just to have some intern apply some glue is a little extreme. So many people have had this problem that YouTube videos just refer to it as "the grinding noise problem." Either Hitachi (the drive manufacturer) just made a lousy drive, or Microsoft didn't correctly anticipate the effect that their game furnace would have on the glue that Hitachi used. I don't know who is to blame, but Microsoft should extend their warranty on the 360 to 3 years for all defects, not just those that cause the red LEDs to light up in a circular fashion. I don't expect my car to wear out in 2 years, and I use it every day. My game console shouldn't wear out, either.

Feedback Works!

2009-07-15T00:02:00.003-04:00

It's hard to say if this is coincidence, but it's worth mentioning. A few weeks ago, I sent eBay feedback on their new site design. My gripe was that they were nesting scrollable areas within scrollable areas. As I was using the site today, it dawned upon me that the bad behavior was gone. Somebody actually fixed it. Perhaps it was my email; perhaps it was the combined voice of thousands of users; perhaps some developer just realized that there was a better way. Whatever the case, I'm glad that eBay made the change, and I like to think that I helped in some way. Thanks, eBay!

Before, things just felt wrong. I would be scrolling the page, and it would mysteriously stop. This happened on every single page, it seemed. When they switched to good design, I didn't even notice at first. It just felt natural. This brings to light a sort of design axiom: design successes are invisible, design flaws are glaring.

Excellent Customer Service at Adafruit

2009-06-28T19:40:00.001-04:00

I've been meaning to write this post for a while, but I just haven't gotten around to it. Well, that ends now! The time has come!

A little over a month ago, I ordered a couple of kits from Adafruit Industries. When the box arrived at my doorstep, it seemed a little too small. Once I opened the package, my fears were confirmed: they had only included one of the two kits.

I typed up a quick email to customer support that night, and figured I'd hear back in a few days time. After all, they are a small company (I think they are small. I don't really know.). The next morning, I was surprised to find this email in my mailbox:

hi daniel -

that's odd - we're going to send out a starter pack immediately - and
we will check the order to see what happened.

cheers,
adafruit

That email was sent at 3:03AM. They immediately sent out the second kit. It arrived a couple of days later.

I was surprised and pleased with their customer service. They didn't question my claim, didn't beat around the bush, and didn't make me wait. They corrected their mistake as quickly as they could. Adafruit Industries is a top-class outfit. I will be ordering (many things) from them in the future.

Arduino librxtxserial.jnilib on MacOS

2009-06-27T12:18:00.006-04:00

After re-installing MacOS from scratch, I found that I was having problems launching Arduino. I got the following error:

In order to solve this, I followed the instructions at rancidbacon.com. The solution was to use Finder to configure the application to launch in 32-bit mode. I quickly looked to see if I could modify the Info.plist file to enforce this behavior, but I don't see any way to do that.

Another solution from technobabble is to replace the library inside the Arduino app bundle with a 1.6 compatible version.

Quick, Over There!

2009-06-18T00:09:00.003-04:00

I'm going to try to continue posting stuff here, but it's worth mentioning that I will also be posting over at the Gents with Beards blog. Anything related to the iPhone will go there (though I will make brief mention of it here). Anything else might go there, might come here, or might get cross-posted.

I made a couple of posts so far, related to my experiences with OpenAL in the iPhone Simulator and differences in the Initial Developer Experience between iPhone and Android. I hope to write one or two about Git, and who knows what after that.

JTextArea and javax.swing.text.Document

2009-05-15T00:13:00.003-04:00

I was playing with a JTextArea in my Swing project today. My underlying data is actually quite simple (a list of names and quotes), and my JTextArea is read-only. Therefore, I decided to try to make my own class that implements javax.swing.text.Document, but whose internal structure more closely models my real... model.

I had a hard time getting started. From poking around in Document's Javadocs and in related Java source code, it appears that Document is a very general concept. It seems to have originally been intended to model an SGML document's structure - or something like that. The Document interface lets you navigate a tree structure that represents the document. However, when it comes to modeling data for a JTextArea, things are much simpler. Instead, it appears that JTextComponent (the base class of JTextArea) assumes that the Document models a list of lines of text. That is to say, it seems to expect that the Document has one root element which contains a separate element for each line. This is most obvious from JTextArea's getLineCount() method:

public int getLineCount() {
    Element map = getDocument().getDefaultRootElement();
    return map.getElementCount();
}

Now, the problem with all of this is that Document is a pretty complicated interface. On top of that, since SGML is a tree-oriented structure, it is not quite intuitive to map the concepts of "stream of text" to "tree-shaped structure". More problematically, I didn't see anything in the documentation to even begin to shed some light on this unspoken relationship. I stumbled upon it while examining stack traces from my first cut. I vaguely recall this being much easier in the little bit of Cocoa that I dabbled in, but I can't be sure. Maybe I'm just crazy - there are certainly a number of other concrete, Document-derived classes out there.

So remember, if you want to implement Document, make sure to model your document as a list of lines of text.

Something Nice

2009-04-07T20:56:00.003-04:00

I don't like that most of my blog posts have a negative tone. I'd like to balance that by talking a bit about something that was really cool. Let me wax lyrically about MacHeist.

After buying the last MacHeist bundle a year ago, I got put on the MacHeist mailing list. As a result, I was able to participate in the missions that they've been putting out for the past few months. Between the free apps I got from completing the missions, the apps in the actual heist, referring 2 friends, and tweeting once, I managed to score 40 apps worth a theoretical value of almost $1700. It cost me $33 (I got a $6 discount by completing in the missions), and a few hours of my time. Let me go into a little more detail about the whole thing.

In order to drum up support, the MacHeist team starts by putting out challenges for several months before the actual sale. In order to solve these riddles, you need to do a little internet sleuthing. This year, they got the cooperation of Veronica Belmont, Lisa Bettany, and Chris Pirillo. People work hard to figure them out but, if you're lazy, you can just follow the walkthroughs that people post. I usually do about 50/50 - working on the challenge until I'm out of ideas, then go read what smarter people than me wrote. By doing this, I got $541 of software (and a $6 off rebate on the actual bundle) for just a few hours of time. Lisa Bettany also starred in a handful of mission briefings as Sophia, the Eastern European secret agent trying to save the world from time glitches. A little campy, but entertaining nonetheless.

Of course, there's also the heist itself. The whole bundle was introduced this year in a 90ish minute live streaming show with an overview of every app. They started this one by selling 8 apps for $39. From each sale, 25% of all sales go to charity. When they reach certain charity milestones, they add more apps. This year, there were 3 milestones. The last was $500,000 for charity, and that unlocked the final 2 apps. It was a little scary this year, because sales stagnated after a couple of days. The last two apps, one of which was Espresso, were still locked. Fortunately, there was a last-minute sales frenzy, and now they're waaay over their final unlock milestone. To try to incentivize people who were sitting on the fence, they added two more apps during the sale, for a total of 14.

Since Twitter is popular with the kids these days, MacHeist took the opportunity to reach tons of people by hyjacking the social experience. If you were willing to tweet a short message to get the word out about MacHeist, they gave you two more applications. Was I just helping The Man by giving free advertising? No! I wanted other people to have the chance to get these apps. This is the tweet they asked me to post:

I bought the @MacHeist 3 Bundle. 12 Top Mac apps worth $900+ for just $39 AND I just got Delicious Library 2 FREE!

That's not too bad!

Finally, I managed to refer two of my friends to the sale, and I got an additional 2 apps from that.

Now, I realize that I didn't actually get $1700 of worth from this bundle. I will probably never even start some of these apps. However, if you can find even 2 apps that you want, this deal makes sense. In my case, I had my eye on several of these apps for a while, and just never got around to buying them. Also, keep in mind that a big pile of them were totally free. I could have walked away with a bunch of software for just a few hours of puzzle solving (or walkthrough-reading). This sale always amazes me. Past sales have included Delicious Library 1, TextMate, CSSEdit, PixelMator, and other great apps. Good job, MacHeist.

Bundle apps

Acorn
Cro-Mag Rally
BoinxTV Sponsored Edition (which means that you have to indicate that you used Boinx to record your video, which kinda sucks)
Espresso
iSale
Kinemac
LittleSnapper
PhoneView
Picturesque
SousChef
The Hit List
Times
WireTap Studio
World of Goo

Apps for completing missions (these only took a few hours to get, but were only available for a limited time)

Apps for tweeting about the bundle (only available after buying the bundle)

Apps for referring others (only available after buying the bundle)

Android SDK + eBay SDK

2009-02-25T22:44:00.004-05:00

I haven't done anything with Android in a while, so I thought I would see whether I could get my phone talking to eBay. I grabbed the eBay SDK for Java, stuck it in my project, wrote a small demo app, tried it out, and BOOM!

VFY: unable to resolve new-instance 1516 (Ljavax/swing/event/EventListenerList;) in Lcom/ebay/sdk/ApiCredential;
VFY:  rejecting opcode 0x22 at 0x0015
VFY:  rejected Lcom/ebay/sdk/ApiCredential;.<init> ()V
Verifier rejected class Lcom/ebay/sdk/ApiCredential;
Shutting down VM
threadid=3: thread exiting with uncaught exception (group=0x4000fe68)
Uncaught handler: thread main exiting due to uncaught exception
java.lang.VerifyError: com.ebay.sdk.ApiCredential
    at com.ebay.sdk.ApiContext.<init>(ApiContext.java:41)
    at org.balefrost.bodacious.Bodacious.onCreate(Bodacious.java:75)
    at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1122)
    at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2104)
    at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2157)
    at android.app.ActivityThread.access$1800(ActivityThread.java:112)
    at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1581)
    at android.os.Handler.dispatchMessage(Handler.java:88)
    at android.os.Looper.loop(Looper.java:123)
    at android.app.ActivityThread.main(ActivityThread.java:3739)
    at java.lang.reflect.Method.invokeNative(Native Method)
    at java.lang.reflect.Method.invoke(Method.java:515)
    at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:739)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:497)
    at dalvik.system.NativeStart.main(Native Method)

It looks like it depends on a Swing class (EventListenerList), but Swing isn't available on Android. I guess I'm rolling it by hand then. It's actually unfortunate that EventListenerList is a part of Swing - it looks generic enough that maybe it should be promoted to the core library. For that matter, I'm a little surprised that this made it past the apk builder. Perhaps that is outside the scope of the apk builder.

Looking For a Laptop Table

2009-01-28T19:14:00.002-05:00

I'm looking to get a small side table so that I can use my laptop in my living room without it burning my lap. I'm considering a LapDawg, but the $130 price tag isn't too appealing. Whatever I end up with, it won't be a Freedom Furniture Laptop Table II Mobile.

Does anybody have any suggestions?

Vista Taskbar Transparency

2009-01-18T17:06:00.002-05:00

One quick Vista UI tidbit. Window decorations (titlebars and borders) in Vista are normally translucent. When you maximize a window, its titlebar become opaque. The Vista taskbar is also normally translucent and, like window titlebars, also turns opaque whenever any window is maximized.

Interestingly, this is done even if the window is maximized on another screen. I recently switched from a dark background picture to a nearly white picture, and it's getting frustrating to see my taskbar alternate between black and gray for what seems like no good reason.

Ultimately, this is a small gripe. It's doesn't really get in the way. It's just odd. It reflects a lack of focus on fit and finish at Microsoft. I hope Windows 7 ends up being better.

G1 Camera Tricks

2008-12-02T23:05:00.006-05:00

The G1's camera seems to be slow. Really slow. As in several-seconds slow. I always wondered why; then I started to look at my pictures from the Macy's Thanksgiving Day Parade.

Ah. I see. I don't think that's because of the curvature of the lens (it's not curved THAT much) I was apparently panning the camera when I took this picture (to keep up with the float, though I didn't correctly account for the very long shutter delay, and the float ended up behind the pole). You are seeing the effect of the CCD "scanning". This reminds me of this project, in which a dude builds a digital camera from a flatbed scanner. This is a picture of his garage door opening and closing while he takes a single exposure.

I hope this isn't an actual limitation of the G1's hardware. My cheap Nokia 6267, a feature phone, had an excellent 2MP camera that took pictures nearly instantaneously. So far, the G1's camera reminds me of my short experience with Windows Mobile 5's camera app - slow and clunky.

Using a Horizontal Progress Bar in Android

2008-12-02T22:35:00.006-05:00

I was working on my Android downloader application yesterday, when I ran into a bit of a snag. I wanted to display a list of downloads (similar to the Firefox downloads window). Each item in the list should show the file name and a horizontal progress bar. I started to implement this, and several hours later stumbled upon the arcane solution. I wanted to share this, both to help anybody who may be trying to do something similar, and also to illustrate how the lack of good documentation is hurting aspiring Android developers.

I thought that I could put a ProgressBar instance in my xml, set it to be indeterminate, give it a max and a value, and be done. However, it wasn't nearly that easy. After a few hours of hunting on Google and the Android Groups, I stumbled upon the solution in an online preview of The Busy Coder's Guide to Android Development. In short, I had to set the following on my ProgressBar's XML:

style="?android:attr/progressBarStyleHorizontal"

WTF?

Let's try to break this down. The ? on the front indicates that this is a reference to something in the current theme. What is a theme? Well, Android's UI infrastructure is similar to Swing or HTML in that it tries to accommodate components whose size is not known until runtime. Android's components are also themeable (similar to the Swing Synth LAF). A theme is a collection of related component styles. You can control the theme used by your application in the AndroidManifest.xml file. So, I think we can safely conclude that attr/progressBarStyleHorizontal is something whose actual value is defined in a particular theme.

If you take a peek in the android source code, inside frameworks/base/core/res/res/values/attrs.xml, you will find the following definition:

<resources>
    <declare-styleable name="Theme">
        <!-- snip -->
        <attr name="progressBarStyleHorizontal" format="reference" />

OK, not very useful. However, in frameworks/base/core/res/res/values/themes.xml, we find:

<resources>
    <style name="Theme">
        <item name="progressBarStyleHorizontal">@android:style/Widget.ProgressBar.Horizontal</item>

Nowe we're getting somewhere. Finally, inside frameworks/base/core/res/res/values/styles.xml, we see:

<resources>
    <style name="Widget.ProgressBar.Horizontal">
        <item name="android:indeterminateOnly">false</item>
        <item name="android:progressDrawable">@android:drawable/progress_horizontal</item>
        <item name="android:indeterminateDrawable">@android:drawable/progress_indeterminate_horizontal</item>
        <item name="android:minHeight">20dip</item>
        <item name="android:maxHeight">20dip</item>
    </style>

What's going on here? Well, it looks like themes.xml defines a theme as a collection of styles. Along with that, styles.xml specifies a set of attributes that should be applied to a particular XML element. By saying

<ProgressBar style="?android:attr/progressBarStyleHorizontal">

we are really saying

<ProgressBar style="@android:style/Widget.ProgressBar.Horizontal">

which is like saying

<ProgressBar 
    android:indeterminateOnly="false" 
    android:progressDrawable="@android:drawable/progress_horizontal" 
    android:indeterminateDrawable="@android:drawable/progress_indeterminate_horizontal" 
    android:minHeight="20dip" 
    android:maxHeight="20dip">

Spiffy.

For bonus points, we can look in frameworks/base/core/res/res/drawable/progress_horizontal.xml. This file apparently contains the declarative expression of the horizontal scrollbar graphic. I don't know how extensive this syntax is, but a quick peek at progress_indeterminate_horizontal.xml indicates that it is possible to specify animations in this XML file format.

All this seems really powerful, only somewhat useful, and totally confusing. All I wanted was to create a horizontal progress bar. I can't tell why the Android widget collection includes only one ProgressBar class that is meant to be used for horizontal, indeterminate horizontal, indeterminate circular, and every other kind of possible progress indicator. Perhaps there would be duplication of code if those were separated into different classes. Whatever. I actually don't mind that they stuffed all those progress indicators into a single class. There are some questionable decisions (like having both indeterminate and indeterminateOnly), but I can deal with it. On the other hand, the incantations that you have to use to get the Progress Bar to behave are truly arcane. I think I have proven in this blog post that there is some method to the madness and, as is often the case, easy access to the source code is a must. What bothers me most is that there was absolutely no documentation on the subject. Google seems silent on this topic, and even The Busy Coder's Guide mentions that the style attribute won't be covered until a later edition of the book (though it is possible that the Google Books sample chapter is out of date at this point).

I still maintain that Android is one of the most unique application systems that I've seen. However, as I've tried to develop for it, I have encountered a general lack of polish. Many things that you want to do as a developer are confusing and poorly documented. Familiarity with existing technologies (such as Swing or Applet programming) is pretty much useless here. On the other hand, the core concepts (such as the use of Intents to start applications and communicate between processes, the HEAVY use of message passing, and the declarative nature of the AndroidManifest.xml file) seem to be an excellent base upon which to build not just an open operating system, but a truly open user experience. A power user can really replace virtually any aspect of the system. Perhaps developers will create third party libraries that make Android a little easier within which to develop, or maybe Google itself will provide simpler interfaces for developers. Or perhaps Android will die out because developers had a difficult time building anything particularly complex. Whatever the outcome, I'm excited for the ride.

Grep is fun

2008-11-13T21:23:00.003-05:00

I'm not what you would call a UNIX power user. I'm more of a UNIX casual user. I don't agree with the zealots who claim that the command line is king, and shell scripts keep the world spinning. It's a pain to remember the myriad of switches, and to remember the differences in those switches between platforms (BSD find requires a pathname, GNU find does not), etc. Maybe my distaste of the command line is why my primary computer is a Mac.

In any case, I was rooting around in the Android source code today, looking for examples involving the NotificationManager. I managed to whip up the following pipeline, of which I am proud.

grep -rl --include=*.java NotificationManager . | xargs grep -l "\.clear" | xargs mate

To break that down,

grep -rl --include=*.java NotificationManager . This will recurse in the current directory, finding all .java files which contain the word NotificationManager. The resulting list of files will be passed to the next step.

xargs grep -l "\.clear" This looks for the string ".clear" in any of the files that came from the first command. It outputs the list of matching files to the next command.

xargs mate This launches TextMate, my editor of choice, with all of the files found in the previous step.

To summarize, this pipeline opens a text editor with every .java file in the android source code that contains both of the strings "NotificationManager" and ".clear". In my case, I was looking for all invocations of NotificationManager.clear(). Since the Android source code seems to be pretty consistent about explicitly importing all classes used, this scheme works pretty well.

If anybody knows of any command-line tools for searching a code base for Java-aware constructs (such as method definition, method invocation, etc.), please let me know! Something like that would be really, really handy. Eclipse provides all that, but it doesn't look trivial to get all of the Android source code into Eclipse projects.

AnyCut's "Make Your Own" Demystified

2008-11-11T00:49:00.004-05:00

I installed AnyCut on my G1 a little while ago. AnyCut is a small app that allows you to create shortcuts to, well, almost anything. It has a mysterious "Make Your Own" option that isn't really documented at all. I started to play with it a bit, and it's not as bad as you would think.

The Make Your Own screen has three fields: Action, Data, and Type. I believe that these line up with the constructor parameters of public Intent(String action, Uri uri, Context packageContext, Class<?> cls). The Action parameter is one of the string consts in the Intent class (such as android.intent.action.VIEW, not Intent.ACTION_VIEW), and the Data parameter should really be called Uri, because it includes the Uri of the item upon which you want to act. See also the list of common intents. I imagine that the Type parameter is a fully-qualified class name, but it can be left blank for most uses. Finally, I suspect that the packageContext parameter is supplied by AnyCut itself.

I haven't yet had a chance to verify any of this against the source code (which I came across while writing this post), but it might be worth a look.

2 Days with Google Gears

2008-10-30T21:11:00.002-04:00

I had a chance to play with Google Gears on a mockup project recently. I was surprised to learn that my understanding of Gears was not the same as the reality of Gears. I had expected it to be a JS library to facilitate rich HTML applications to handle spotty network connections. It turned out that Gears is a browser plugin that adds useful tools that any rich web app developer would find useful, though they are all aligned to handle an app losing its network connection. In the two days that I spent playing with Gears, I was pretty much blown away by the power and simplicity of it.

Besides understanding what Gears is, it's important to also understand what Gears is not. Gears doesn't try to be a UI library, or a general "glue" library (see JQuery, Prototype, MochiKit, MooTools, or any other Javascript framework). It doesn't really force you to code in any particular way (this is a bit of a lie, but more on that later). Just as a sampling, Gears includes:

A cross-browser compatible XMLHttpRequest
Local database storage
A web resource cache, so that items can still be fetched even if the network connection goes down. This is nearly completely transparent to the developer, and works for resources loaded by Gears or by the browser itself.
A mechanism to run scripts in the background (for real).

In our little demo, we have a set of checkboxes that can be checked. Initially, these would perform an asynchronous post to the server, where they would update some server-side state. However, if the app goes offline, nothing will ever reach the server. We modified this to enqueue changes into a local database, with a background script pulling items out of the database and sending them to the server. If the server ever goes down, that thread simply stops pulling items out of the database. In addition, we set our app up so that its resources (HTML, Javascript, CSS, and images) were initially cached when the app is first loaded. A neat feature of Gears seems to be that it will monitor the apps that it knows about and will automatically update its cache if the cache ever gets stale. Unfortunately, it's not perfect. It depends on the developer updating a version string, which causes Gears to update its cache from the source.

One problem that we had is that the HTML that we served up would include information about what items were checked. That is to say, when you would load the page, we would serve up some <input type="checkbox" /> and <input type="checkbox" checked="checked" /> elements. This makes total sense in a traditional web app. The client requests the page, and you serve it, and everybody is happy. Every time the page is served, it is reconstructed with the current state of the data. As you might imagine, this caused all kinds of problems for us. Concretely, we noticed that every time the page was reloaded (whether the network connection is up or down), the browser would display the state of the page as it was when the cache first acquired it. In a real application, that could mean that you are seeing data that is several months out of date. Now you see how I lied earlier. Gears does influence the way you code your application, but its requirements are about the same as those of any Javascript-heavy web app. As long as you separate your application presentation from your data, you should be fine.

Another thing that surprised and greatly pleased me was Gears' WorkerPool implementation. As everybody knows, it is impossible to run Javascript in the background in a normal web browser. I think that's because multi-threaded programming is hard, and Javascript can be pretty hairy as it is. I think that the browser designers have held off on implementing a threading solution out of fear that multithreaded Javascript would cause the apocalypse. As it turns out, though, Gears' implementation is both simple and powerful. Gears uses a message-passing mechanism for communication, with absolutely no shared state. This is great news. As far as I can tell, just as your main JS code has an event loop, each worker also has an event loop. Whenever a message is sent from the main JS code to a worker, that message is copied and onMessage is invoked on by that worker's event loop. Likewise, when a worker sends a message back to the main JS, the message is copied and onMessage is invoked on the main event loop. This has some interesting implications. For one, none of the workers have access to the DOM, or to global variables defined on the page, and cannot participate in closures with mainline Javascript code. By placing a concrete wall between your page and your workers, Gears forces you to think about the interactions that the page and the worker will have, and that's a Good Thing. I'm sure that it's still possible for threading to ruin you, it's just a lot harder with a scheme like this.

And that's it. There's more to Gears that what I described (though not much more). It also includes some geolocation bits (presumably for Android, and maybe Safari Mobile, integration), desktop integration stuff, a standards-compliant timer, a file multi-chooser (yay!), and a binary data type (as opposed to String, which is for textual content). It's a shame that Gears is still in beta. I would really like to see some sites that use it. Of course, since I just recently installed Gears, there might be some sites that do and I never realized it.

User-Visible Permissions in Android

2008-10-29T23:37:00.005-04:00

I picked up a T-Mobile G1 (danger: Flash-heavy site) at the local T-Mobile store. For those that don't know, the G1 is the first device to run Google's Android platform. So far I like it a lot, and I'll probably post a lot more about it in the near future.

Like the iPhone, Android has its own app store. Unlike the iPhone, nobody moderates apps submitted to the Android app store. If an app tries to do anything of consequence (i.e. anything that a user might want to know about), it must explicitly request that permission. When you start to download an app from the marketplace, it tells you what permissions that app will require. Most apps are well behaved, but some ask for way too much.

For example, I wanted a weather app. I saw that there is a Weather Channel app. When I went to download it, however, I was very surprised. Here is a list of the permissions that it requested.

Network communication	full Internet access

Your location	coarse (network-based) location, fine (GPS) location
System tools	change network communication, change your UI settings, modify global system settings
Your messages	edit SMS or MMS
Services that cost you money	send SMS messages
Your personal information	read contact data

What? Why does this app need access to my contacts, or to send text messages? I hope that this was just a lazy developer who requested more permissions that he actually needed, but I'm suspicious. It's entirely possible that The Weather Channel intends to compile a list of all my contacts. Not cool. Especially since it makes no mention of that.

I think it's great that Android provides some ability for the end user to judge the software that they might install on their phone. I'll wait until The Weather Channel updates their app.

Scala's Model of Functions

2008-10-14T01:01:00.003-04:00

I was a little dismayed to learn that Scala models functions with different numbers of parameters as instances of distinct function trait definitions. A practical upshot of this is that you can't really work with functions that take more than 22 parameters.

def crazy(a:Int, b:Int, c:Int, d:Int, e:Int, f:Int, g:Int, h:Int, i:Int, j:Int, 
          k:Int, l:Int, m:Int, n:Int, o:Int, p:Int, q:Int, r:Int, s:Int, t:Int, 
          u:Int, v:Int) = 0
(crazy _).curry    :    (Int) => (Int) => (Int) => (Int) => (Int) => (Int) => (Int) => (Int) => (Int) => (Int) => 
                        (Int) => (Int) => (Int) => (Int) => (Int) => (Int) => (Int) => (Int) => (Int) => (Int) => 
                        (Int) => (Int) => Int = <function>

def crazier(a:Int, b:Int, c:Int, d:Int, e:Int, f:Int, g:Int, h:Int, i:Int, j:Int, 
            k:Int, l:Int, m:Int, n:Int, o:Int, p:Int, q:Int, r:Int, s:Int, t:Int, 
            u:Int, v:Int, w:Int) = 0
(crazier _).curry    :    <error>

The scala runtime apparently has traits Function0 through Function22 defined. I guess this is so that they can have call methods that take a statically known list of parameters (rather than, say, an array). That's all well and good, and probably necesary for proper Java interop, but it's still a little sad. Still, I don't expect to run into that limit any time soon. Oh wait, I have already worked on projects with functions that take more than 20 parameters. Maybe this was added just for me. Now I'm sad.

Partial Application in Haskell and Scala

2008-10-13T23:49:00.004-04:00

This is an attempt to squeeze out a blog post while I wait for my laundry to finish.

Functional languages are fun. Fun in ways that Java (and, for that matter, Ruby) are not. Take Haskell. In that language, we can take any operator and turn it into a function. Normally, we use the symbol + to represent addition. If we enclose it in parentheses, we instead have a function.

(+) :: (Num a) => a -> a -> a

In this case, (+) is a function of 2 number parameters, which returns a number. Now that we have a function, we can apply all of the standard Haskell magic to it. Since Haskell is automatically curried (no function really ever takes more than one parameter), we chain calls to fully evaluate our (+) function.

(+) 2 3 => 5

We can also partially apply this operator.

add5 :: Integer -> Integer
add5 = (+) 5
add5 3 => 8

In this case, we have created an alias for the partially bound + operator. Rather than jump through so many hoops, we could specify add5 more directly.

add5 = (5+)

Finally, a slightly more complicated example.

simple :: Integer -> Integer -> Integer -> Integer
simple x y z = x * (y + z)

simpler :: Integer -> Integer -> Integer
simpler = simple 2

simplest :: Integer
simplest = simpler 3 4 => 14

All functions are also values in Haskell.

easy = simple
easy 2 3 4 => 14

As you can see, in Haskell, we can turn any operator into a function. Functions are curried, and can be partially evaluated from the left. Functions are also values that can be assigned and passed around as needed.

Scala takes a different approach. In Scala, operators are actually methods on values. There is no global + operator. Instead, you invoke the + method on the left hand parameter.

5 + 3 //is the same as...
(5).+(3)

If you want to refer to a function as a value in Scala, you must "partially apply" it to zero parameters.

val output = println //will result in a compilation error
val output = println _
output "Oh Hai, World!"

The underscore is the Scala placeholder operator. If used as we did with println, it stands in for the whole argument list, effectively turning the function into a function value. It is also the mechanism by which we can partially apply a function.

def simple(x:Int, y:Int, z:Int) = x * (y + z)
val simpler = simple(2, _:Int, _:Int)
simpler(3, 4) => 14

The underscores, when used this way, compel the result of the expression to itself be a function that takes n parameters, where n is the number of placeholders. Sometimes, it is possible to infer the type of the missing parameters; other times, it isn't. It depends on how the parameters are used.

It is very important to notice that, unlike Haskell, it is very easy to bind only the parameter in the middle of this expression.

val sample = simple(_:Int, 3, _:Int)
sample(2, 4) => 14

By combining placeholder syntax with operators, it is possible to turn an operator into a function, even a function that takes its left operand as a parameter.

List(1, 2, 3).map(_ + 2) => List(3, 4, 5)
List(1, 2, 3).reduceLeft(_ + _) => 6

As you can see, Haskell and Scala have a lot in common. Haskell's syntax is a bit more concise (and its inference rules much better), but Scala's ability to bind any parameter is pretty handy, too. There's something both cluttered and clean about Haskell's use of underscores, especially when types aren't required. Of course, I'm not an expert (or, in fact, experienced at all) with either language, so please correct me if I got any of my facts wrong.

Looks like I failed. My laundry was done 30 minutes ago.

The Machine

2008-10-04T10:43:00.003-04:00

Two weeks ago, I had the opportunity to see The Machine with my family. The Machine is a Pink Floyd tribute band. That is to say, at their shows, they play nothing but Pink Floyd music. All of the musicians are clearly extreme Floyd fans. I mean, why else would you spend 20 years of your life playing somebody else's music? Now, some people don't like tribute bands. I had a hard time getting people to see The Australian Pink Floyd Show when they came to New York (playing literally a few blocks from where we were staying). Who cares that these aren't the original musicians? Would you also refuse to go to a performance of Beethoven's 5th because it wasn't being conducted by the man himself? Of course not! The music is just as good, and the musicians are going to make it special and awesome anyway. But I digress...

It was interesting to see the variety in people in the theater. Obviously, many of the patrons were my parents' age, but there were also some college kids and folks whose heads were completely gray. What was perhaps more interesting to me is that the 50 year olds were more animated and crazy than the college kids. They had some smoke machines up on stage, but I don't think that was the source of all the smoke in the hall. It's fun to watch adults relive their youth.

The set was Dark Side of the Moon (with the Wizard of Oz projected onto their own version of Mr. Screen), followed by an intermission, followed by The Wall. Not a bad setlist at all. As they launched into the beginning songs from Dark Side, I was carefully listening for any variation from the album tracks that I know so well. I couldn't help it. These guys were playing well-known and well-loved music, so it's only natural to compare their performance to the original. By The Great Gig in the Sky, though, I was totally sold. The woman that belted out those notes was simply amazing. She absolutely hit every note. It was surreal. The keyboardist was younger than the rest and totally crazy, with a maniacal grin that was somehow larger than his actual face. The drummer hid behind the drums for most of the show, but did a very good job. The bassist seemed detached, standing apart from the others. I suspect that was completely intentional. The saxophone player was decent, but wasn't very memorable (after all, he only played on a few songs). Rounding out the group is the lead guitarist / lead singer. His ability to mimic both David Gilmour and Roger Waters was spooky. The man knew his guitar well, and made it sound just like the original.

By the time they were playing The Wall, people in the crowd were singing along. Performing Dark Side first was a good idea. People were more mellow when the entered the theater than when they left, and Dark Side is best appreciated without whoops and cheers. The Wall, on the other hand, is great with audience participation. In the end, they ended up getting 4 standing ovations (after Dark Side, after (I think) Comfortably Numb, after The Wall, and after their encore of Run Like Hell). They deserved each and every one of them. They probably played for 2.5 hours all told.

I never got a chance to see Pink Floyd live. As one of the people sitting next to us pointed it, this is the closest you can get at this point. While I agree with him, it is wrong to think of these guys as a facsimile of that famous band. These are all very talented musicians who love this music so much that they have dedicated a big chunk of their lives to it. As a fan, I'm grateful to them for doing that.

No More Statics! Part 2

2008-09-06T01:30:00.007-04:00

In a previous post, I explained how Scala's use of singleton objects is better than Java's use of static members. I was asked for some sample code after that post, so I thought I would throw some together. Let's look at a simple Java class.

class Foo {
    private static int number = 1;
    
    public static Foo create(String a) {
        return new Foo("Some " + a);
    }
    
    private String a;
    private int id = number++;
    
    public Foo(String a) {
        this.a = a;
    }
    
    @Override
    public String toString() {
        return "Foo #" + id + " is " + a;
    }
}

This class keeps track of how many instances have ever been created. You construct a Foo with a name, and the Foo's name and id are part of its string representation. In addition, there is a create method that has been defined on the Foo class itself.

Scala doesn't have a "static" keyword. Instead, members that would otherwise be static are placed onto the so-called companion object.

object Foo {
  var number:Int = 1;

  def create(a:String) = new Foo("Some " + a)
}

class Foo(a:String) {
  private val id:Int = Foo.number
  Foo.number = Foo.number + 1

  override def toString() = {
    "Foo #" + id + " is " + a
  }
}

Because Foo is the name of both an object and a class in the same package, they are allowed to access each other's private members. Basically, this makes the instance members of a singleton object equivalent to static members of an ordinary Java class. However, since the singleton object is a fully fledged object, it can be passed around in a way that Java classes normally can't be.

def createList(f : Foo.type) = {
  List(f.create("One"), f.create("Two"))
}

Have you ever wanted a Java class' static members to obey an interface? Well, the singleton object can mix in Scala traits (Scala traits seem to take the place of both interfaces and mixins from other languages).

trait Creatable[A] {
  def create(a:String) : A

  def createDefault() : A = {
    return create("Default")
  }
}

object Foo extends Creatable[Foo] {
  var number:Int = 1;

  override def create(a:String) = new Foo(a)
}

And here's a whole sample program:

trait Creatable[A] {
  def create(a:String) : A
  
  def createDefault() : A = {
    return create("Default")
  }
}

object Foo extends Creatable[Foo] {
  var number:Int = 1;
  
  override def create(a:String) = new Foo(a)
}

class Foo(a:String) {
  private val id:Int = Foo.number
  Foo.number = Foo.number + 1

  override def toString() = {
    "Foo #" + id + " is " + a
  }
}

def createList(f : Creatable[Foo]) = {
  List(f.create("Three"), f.create("Four"))
}

println(Foo.create("One"))
println(Foo.create("Two"))
println(createList(Foo))
println(Foo.createDefault())

----------

Foo #1 is One
Foo #2 is Two
List(Foo #3 is Three, Foo #4 is Four)
Foo #5 is Default

Why are singleton objects better than static members? To begin with, Scala's singleton objects are at least as expressive as static class members, so you're not losing anything from Java. You define a singleton object differently that you define static members in Java, but you access them using notation identical to Java (i.e. Foo.bar(5) in both languages). In addition, you get some other nice features - first class object status and the ability to participate in the normal class hierarchy. As an added bonus, Scala's simpler syntax actually made the class/singleton-object pair shorter than the equivalent Java solution. Not bad!

On Architecture

2008-09-02T19:30:00.003-04:00

The other day, a coworker told me that they want to improve their architectural skills. That brought up a number of questions for me. What is architecture and how does it differ from programming? What are traits of a good architecture? What about a good architect? Am I a good architect?

"Traits of a good architecture" is something that gets covered in a lot of CS textbooks: extensibility, security, resilience, and so on. However, concentrating on those traits won't make you a better architect.

I don't know what will improve your architectural skills, but I do know what I think about when I'm working on architecture. I don't necessarily architect by reason. I have "good code" values, and I evaluate all the code that I write or encounter against them. For example, I think that exceptions (if available) should be used to signal the presence of and to recover from erroneous conditions. Status codes are straight out, and "fixups"[1] are a bad idea, too. Only in rare performance-critical areas are other mechanisms appropriate.

Where did I get my "good code" values? I've built them up from my personal and professional experience. I've been writing toy programs for 18 years. A lot of that is useless today, but some of those experiences have taught me valuable lessons. Just like a master carpenter didn't acquire his skills overnight, a software architect needs to develop his skills over a period of time. Trying, failing, reading, watching, discussing, and reflecting are all useful for developing this sense.

Just like other developers, I like to program. In fact, for an architect, it's essential to get one's hands dirty. However, it's also important to know when to step back and look at the big picture. For me, it's all about relying on that design "sense". When people talk about code smells, it's because something smells rank to their design sense. On the other hand, it's important not to get paralyzed by that sense. If you're unsure what direction you should be heading, it might be time to try something. Commit your current work (or, if using Git or Mercurial, create a new branch) and try going down a path. Keep your eyes open along the way, and learn what works and what does not. If you have time, try the other path. It might also make sense to start a new "toy" project to try some ideas without being burdened by the current code base. Also, be ready to revert all of your changes. Most of the work in programming is wrapped up in the thinking. Once you know how to do something, it should be reasonably simple to reproduce your work[2].

Finally, I'm a big fan of deciding on a direction and pursuing it. Sometimes, you don't know what to do, and it's too much work to mock up or try out even one approach. In this case, I ask myself how I would want it to work, and I try to do that. I ignore technical constraints unless they are insurmountable. Blindly following a path because it is "right" can get you into a lot of trouble, but so can meandering about without any goal in sight.

What do other people think? What is architecture to you, and how do you improve those skills?

[1] In this case, I'm talking about code which detects invalid input or state (which is a good thing), but then changes the input or state to be valid. An example would be a function with a parameter whose value needs to be in the range [1, 1000], but which will assume 0 for any invalid value. This makes the function always terminate normally (after all, there are no longer any invalid inputs), but it doesn't necessarily ensure correctness of the program. Who's to say that 24124142 and 0 are equivalent, anyway? Exceptions are better because they allow the caller, not the callee, to determine the result in the case of an invalid condition.

[2] If, on the other hand, most of your programming effort is spent on pushing keys on a keyboard, you're in a difficult position. Programming is primarily a thinking man's game, not a typing man's game. Your goal should not be to generate the most code in a day, but rather to generate the most value per line of code. Partially, then, your goal should be to keep the number of lines of code to a minimum. Removing code while retaining functionality, flexibility, and performance is always a good thing.

No More Statics!

2008-09-02T19:00:00.005-04:00

As I read more about Scala, I'm running across a lot of things that I like. In Scala, there are no static members: no static methods; no static fields. Instead, Scala has so-called "singleton" objects. These singleton objects are globally accessible, though their instance methods and fields are still subject to access restriction. This is great because it exposes what we all knew all along: that static fields and methods in Java are really just global variables and functions. Granted, they are access-controlled, namespaced globals, but they're still globals.

Since each class' singleton object is in fact an object, it can subclass another object or mix in traits, just like objects that are spawned by a class. The singleton object has the same rights as any other object in the system.

In addition, a singleton object can share a name with a class; if it does so, they can access each other's private data. I'm not sure yet, but I assume that this is how Scala accesses static members of Java classes - it creates a singleton object that doesn't derive or mix in anything, but turns all the static methods and fields of the Java class into instance members of the singleton object.

Terseness for Terseness' Sake

2008-08-31T22:43:00.005-04:00

I've been reading up on Scala, since it seems like it may be a better Java than Java itself. As I was reading through the pre-release PDF of Programming in Scala, I came across something goofy.

Scala, like (as I understand it) F#, tries to nestle itself comfortably between the functional and imperative camps. It has a syntax that supports both schools of though. So, as you might expect, some functions in Scala will behave nicely and will return a value without any side effects. Other functions will be executed solely for the side effects, and will return nothing (the Unit type in Scala). To further the Functional mindset, Scala does not require an explicit return statement at the end of a function. Instead, the last value in the function is used as the value of the function. Programming in Scala is quick to point out that, if you want, you can just as easily use explicit return statements (if that floats your boat).

The functional and imperative worlds collide in a shower of fireworks. From Programming in Scala:

One puzzler to watch out for is that whenever you leave off the equals sign before the body of a function, its result type will definitely be Unit. This is true no matter what the body contains, because the Scala compiler can convert any type to Unit. For example, if the last result of a method is a String, but the method’s result type is declared to be Unit, the String will be converted to Unit and its value lost.

The book then goes on to provide an example where a function's value is accidentally lost.

Now, I'm all for shortening my programs. The less I have to type, the better. This is, in fact, one of the big advantages Scala has over Java. But wait just a minute! I thought that our compilers were supposed to help us, not trip us up! Here's a situation where 2 different things (a function's return value and the function's return statement) are optional. If they are not specified, they are inferred. In that case, the only difference between retaining and losing your return value is a single character - a '='.

To get all concrete, here are a pair of Scala programs that do different things.

package org.balefrost.demo

object Sample {
  def foo {
    "bar"
  }
  
  def main(args : Array[String]) : Unit = {
    val baz = foo
    println(baz)
  }
}

=> ()

package org.balefrost.demo

object Sample {
  def foo = {
    "bar"
  }
  
  def main(args : Array[String]) : Unit = {
    val baz = foo
    println(baz)
  }
}

=> "bar"

I don't know. To me, that's goofy. Other people might find it completely reasonable. Of course, you can protect yourself with explicit types.

package org.balefrost.demo

object Sample {
  def foo {
    "bar"
  }
  
  def main(args : Array[String]) : Unit = {
    val baz:String = foo  //compiler error: can't assign Unit to String
    println(baz)
  }
}

Anyway, kudos to Programming in Scala for pointing out the potential cause of a hair-yankingly-frustrating bug. Now that I understand what's going on, I will probably be better able to handle it when it comes up in a real program.

Should I Squash Underhanded Corporate Comments or Let Them Live?

2008-08-14T22:48:00.005-04:00

At this point, my Memeo Autosync post has gotten a few comments that clearly originate from somebody who works for(or otherwise has a stake in) Memeo. On one hand, I really dislike this corporate intrusion in an otherwise pristine blog. They have masqueraded as a genuine user, which is misleading and underhanded. On the other hand, it appears that they have offered a discount on Memeo software.

What do other bloggers do with these situations? Do they squash comments that are subversive like this? Do they just allow them, realizing that blog readers are intelligent individuals and will notice the obvious deception? What do you think?

Why You Can Throw Away "If", and Why You Shouldn't

2008-08-14T21:40:00.005-04:00

Introduction

Most reasonably experienced object-oriented programmers have probably stumbled upon the same realization; namely, that it's possible to replace if statements with polymorphism. Polymorphism is simply a way to delay a decision until runtime. The if statement does the same thing. In fact, procedural programmers need to resort to things like if and switch statements because they have no other tool. Functional programmers, on the other hand, simply toss functions around willy nilly.

This realization can be powerful. It can also really hurt a code base (I know - I've smashed my share of algorithmic china with this hammer). I recently ran into a place on a project where it was a great idea, and I thought I would share why I thought it worked so well.

Current Implementation

Suppose you have 2 methods:

public void storeNewItem() {
    Item item = new Item();
    item.name = request["name"];
    item.description = request["description"];
    item.quantity = request["quantity"];
    item.value = someComplexCalculation();
    item.totalValue = item.quantity * item.value;
    // calculate and store some more fields here
    items.addNewItem(item);
}

public void storeExistingItem() {
    Item item = items.get(request["itemId"]);
    item.name = request["name"];
    item.description = request["description"];
    item.quantity = request["quantity"];
    item.value = someComplexCalculation();
    item.totalValue = item.quantity * item.value;
    // calculate and store some more fields here
    item.update();
}

These two functions should look pretty similar. In fact, they are nearly identical. Both acquire an item, populate it with data, and then store it. The only difference is the way that the item is acquired and the way that the item is stored.

First Attempt

I wanted to merge these methods, and this was my first attempt.

public void storeItem() {
    Item item;
    if (request["itemId"] == null) {
        item = new Item();
    } else {
        item = items.get(request["itemId"]);
    }

    item.name = request["name"];
    item.description = request["description"];
    item.quantity = request["quantity"];
    item.value = someComplexCalculation();
    item.totalValue = item.quantity * item.value;
    // calculate and store some more fields here

    if (request["itemId"] == null) {
        items.addNewItem(item);
    } else {
        item.update();
    }
}

This works, but is obvious crap. I found myself saying "I wish Item were able to handle those details by itself".

Second Attempt

Well, I wasn't brave enough to change Item, so I instead wrapped it.

public void storeItem() {
    Persister persister;
    if (request["itemId"] == null) {
        persister = new NewItemPersister();
    } else {
        persister = new ExistingItemPersister(request["itemId"]);
    }

    Item item = persister.getItem();
    item.name = request["name"];
    item.description = request["description"];
    item.quantity = request["quantity"];
    item.value = someComplexCalculation();
    item.totalValue = item.quantity * item.value;
    // calculate and store some more fields here
    persister.persist();
}

interface Persister {
    Item getItem();
    void persist();
}

class NewItemPersister implements Persister {
    private Item item = new Item();
    
    public Item getItem() { return item; }
    
    public void persist() { items.addNewItem(item); }
}

class ExistingItemPersister implements Persister {
    private Item item;
    
    public ExistingItemPersister(String itemId) {
        item = items.get(request["itemId"]);
    }
    
    public Item getItem() { return item; }
    
    public void persist() { item.update(); }
}

We still have an ugly if at the top of the function, and we have certainly ballooned the code. I still think that this is better than what we started with.

There is less duplication, which will make maintenance here much easier.
The Persister interface could be made into a generic class, and the implementations could be re-used all over the system. Some reflection here could really simplify your life.
A good web framework would allow you to remove that pesky initial if statement. In a less good framework, you could hide this behind some sort of object that knows how to generate a persister from an itemId (or null).

The practical upshot is that these changes should make it easier to apply metaprogramming techniques to this chunk of code. The only code that can't really be made declarative is some of the code which assigns values to fields.

There is one thing that bothers me, though. We have made the Persister implementors responsible for the lifetime of the Item. That's not at all clear from the interface, but it is obvious from the use. The tell-tale sign is that we have a getItem() method. Getters that expose their class' internals like this are evil, and if you don't believe me, you're just plain wrong. I won't try to justify that statement in this post, but trust me.

Third Attempt

To solve this, we could change the interface yet again (and I will switch to Javascript, because Java doesn't have any convenient lambda syntax).

function storeItem() {
    if (request["itemId"] == null) {
        var persister = newItemPersister;
    } else {
        var persister = new ExistingItemPersister(request["itemId"]);
    }
    
    persister.update(function(item) {
        item.name = request["name"];
        item.description = request["description"];
        item.quantity = request["quantity"];
        item.value = someComplexCalculation();
        item.totalValue = item.quantity * item.value;
        // calculate and store some more fields here
    });
}

var newItemPersister {
    update:function(f) {
        var item = new Item();
        f(item);
        items.addNewItem(item);
    }
}

function ExistingItemPersister(itemId) {
    this.itemId = itemId;
}

ExistingItemPersister.prototype.update = function(f) {
    var item = items.get(request["itemId"]);
    f(item);
    item.update();
}

Now, the item's lifetime is only as long as a call to update() is on the stack. This is a common idiom in Ruby, as well.

Conclusion

In the end, I wasn't completely happy with any of these solutions. I think that things are better than they were before. There are also a number of other permutations that will get it marginally closer to ideal. I think that the real solution is to update Item so that you can create a new item and save an existing item with a single method. After that, the code to choose whether to create a new object or fetch an existing object should be minimal and extractable.

I did learn a rule of thumb for deciding when to replace an if statement with polymorphism. If you find yourself saying "I don't want to deal with this, I wish it were handled for me," there's a good chance that you could benefit from some polymorphism. Also, if you find yourself checking the same condition multiple times in a function (as we had in the original implementation), you might want to consider whether polymorphism will help you.

Experiments in Firmware Hacking

2008-08-07T00:53:00.006-04:00

I got a new ethernet-ready printer today, and wanted to add it to my existing wireless network. This is not the usual home wireless network use case - most people want to share an upstream connection with a bunch of wireless clients. I wanted to connect a wired device to a wireless network. I first tried using a spare Airport Express. That worked perfectly. Then, I decided to try getting my Linksys WRT54G v2 to work. When I realized that the stock firmware was definitely not up to the task, I grabbed Tomato. It claims to be solid, fast, and AJAX-y with realtime, SVG charts. How could I resist. However, the settings that I needed weren't obvious at first. After some fiddling, I think I've made it work. I'll share them here in case they're useful to somebody.

The most important setting is Wireless Mode (under Basic/Network). Here's my current best understanding of these modes:

Access Point	This is what the Linksys router would do with the default firmware. It allows wireless clients to connect to it in infrastructure mode, and will route packets between the wireless network, the LAN network, and the WAN network (with NAT).
Access Point + WDS	I think this may work like the Airport Express' WDS Remote mode. That would mean that it can accept wireless clients and simultaneously connect to a WDS network.
Wireless Client	This mode appeared to work like the Wireless Ethernet Bridge mode, except with NAT. It appeared that the router will can run a DHCP server on the LAN interfaces. It also requires that the WAN port be configured, which seems very strange to me.
Wireless Ethernet Bridge	This is the one that ended up doing what I need. As far as I can tell, the WAN port is disabled. The device connects to an existing wireless network. It will then route packets between the wired and wireless network without NAT. Furthermore, contrary to other reports, it appears that you can connect devices to more than one of the LAN ports. I had both my printer and my laptop connected to LAN ports, and things still seemed to be working.
WDS	I think this may be similar to the Airport Express' "WDS Relay" mode.

It might be fun to pick up a WRT54GL. (The 54G has been simplified and will no longer work with most custom firmware. The 54GL restores the missing features. It appears to have been created specifically so that people can continue to use alternative firmware.)

Date Formatting in Javascript

2008-08-03T22:25:00.006-04:00

Problem

I found myself doing some stuff in Javascript. In particular, I needed to be able to turn a Date into HTML specific to the application that I'm working on. Let's say that the server sends us a task's creation date. We need to format it:

var fromServer = Date.parse("Sun Aug 03 2008 23:12:52 GMT-0400 (EDT)");

td.innerHTML = format_date(fromServer);

<td>
    <span class="date">2008-08-03</span><span class="time">11:12 PM</span>
</td>

However, and I have no guarantee that it will always be a parseable date. In fact, the server sends '-' for dates that don't exist. We still need to output something.

var fromServer = "-";

td.innerHTML = format_date(fromServer);

<td>-</td>

I would like format_date to accept either a Date that should be formatted, or a string that should be passed along verbatim (with only necessary HTML character entity escaping). How can we do this in an object-oriented fashion?

Attempt 1

The default object-oriented mindset would encourage us to use polymorphism. We have an object, and we want to be able to call a method on that object. Well, we have a Date object.

Since this is Javascript, we could stick an extra method onto Date.prototype that would let us do this. While we're at it, we can put a similar function onto String.prototype:

Date.prototype.formatAsAppSpecificHTML = function() {
    return "<span>" + this.getFullYear() + ... + "</span><span>" + this.getHours() ... + "</span>";
}

String.prototype.formatAsAppSpecificHTML = function() {
    return this;
}

function format_date(o) {
    return o.formatAsAppSpecificHTML();
}

There are some problems with this.

We have an ugly, ugly method name. This is because we're mixing abstractions. A Date, in general, shouldn't know how to format itself in this way. Why not? Because it probably doesn't apply to most usages of Date. It may be a common behavior in my application, but its a nonsensical behavior in your application. Since the method is very context-specific, the name has to be equally specific.
This only works in an "open" language that lets us add methods to an existing class/prototype (Ruby and Lua (and arguably C#) fall into this camp, Java and C++ do not). Even if your language has the necessary support, you still have to wonder whether it's a good idea to handle stuff this magical.
It's not obvious. You don't normally connect Date and String. You don't expect to see methods shared between them. They are very orthogonal primitives. Yet we've tied them together in an unnatural way. In order for somebody to discover this, they need to think to look in two (potentially distant) places in the code.

Attempt 2

Polymorphism is a form of runtime decision making. Rather than use language constructs (such as 'if' and '? :'), polymorphism leverages the power of pointers. Since Polymorphism created some problems in this example, what if we switch to use a more traditional (i.e. not object-oriented) solution?

function format_date(o) {
    if (o.constructor === Date) {
        return "<span>" + o.getFullYear() + ... + "</span><span>" + o.getHours() ... + "</span>";
    } else {
        return o.toString();
    }
}

This approach also creates some problems. We've simply moved the complexity further up the ladder. Before, the knowledge that a task may or may not have a creation date was strewn across 2 types: Date and String. Looking at either type in isolation, you only see half of the picture. You might not realize that you can take anything that comes from the server and format it correctly. Now, however, that knowledge is pushed in your face. We're making explicit decisions about concrete types wherever we need to. It's easy to get it right once. So far, we're only handling formatting. What if we also want to draw a timeline? What if we want to find the earliest task in a list of tasks? What if you want to relate a task to source control submissions that occurred while the task was active. In all of these cases, you will need to deal with the fact that a task might or might not have a starting date. At some point, somebody's going to forget that they need to check this, and there's going to be a bug.

Attempt 3

If object-oriented didn't work, and procedural didn't work, what are we going to do? Well, actually, I lied. The first attempt used one form of object-oriented abstraction. There are many more. Both of the attempts so far have suffered from primitive obsession. They dealt with both Strings and Dates. In actuality, we don't want to concern ourselves with either of these. We actually have something different - we have an OptionalDate. An OptionalDate knows whether it represents an actual date or whether it represents no date at all. It can format itself correctly in either case, and can be compared to other OptionalDates for sorting purposes. In fact, OptionalDate handles any operation that needs to work with both actual dates and "not dates".

function format_date(o) {
    return o.format();
}

function OptionalDate(d) {
    this.d = d;
}

OptionalDate.prototype.format() {
    if (d) {
        return "<span>" + this.d.getFullYear() + ... + "</span><span>" + this..getHours() ... + "</span>"
    } else {
        return "-";
    }
}

What makes this a better solution? After all, the code looks very similar to the code in attempt 2.

It localizes the code better than attempt 2. Rather than checking for the presence of a date all throughout your code base, you can collect all those if/else statements in one place. You also get the chance for some pretty cool higher-order programming, where OptionalDate has a method that takes a function to be called if the OptionalDate actually has a date.
It also gives us a better place to hang domain-specific code. Hey, business logic has to live somewhere. It never seems right to put it on the primitive objects, and it also doesn't make sense to put it at the highest level of abstraction. Business logic is the foundation upon which you build an application. As a foundation, it needs a separate place to live.
It makes more sense. When a new programmer is brought onto the team, they will be able to better understand just what is going on. This is extremely important. I believe that, if a person ever has a question about the code, it's a good sign that the code should change. That doesn't mean that you actually take the time to refactor the code, but it's a sign that this is a place that could use some attention.

My Solution

In the end, I went with something close to Attempt 2. This was actually my first choice; Attempt 1 was purely synthetic. I'm working with a legacy code base, and I'm a little wary about introducing big changes just yet. I'm also very conscious about time. In any case, this is an improvement over what was there before (it just treated the date/time as an opaque string, which wouldn't work at all for my requirements).

Conclusion

There are definitely some problems for which the object-oriented noun/verb paradigm breaks down. Or, perhaps stated more precisely, there are problems where that paradigm confuses more than it helps. However, that isn't true in this case. We were able to introduce some good refactorings even while staying true to the spirit of good design.

You may wonder why I care so much. I mean, any of the attempts would have solved the problem perfectly well. Why spend time even thinking about it? I believe that pragmatism is an important trait in programmers, but so too is learning. Whenever you start working on a problem, you need to choose a direction to pursue. Until you start walking, you'll never make progress. You may find yourself at a dead-end, but you wouldn't have known if you hadn't gone that way. My goal is to develop a strong enough design sense that the path that I choose with little thought tends to be one that will work out in the long run. Programming is both tactical and strategic. Most programmers develop their tactical skill as a natural part of writing code. I'm trying to sharpen my strategic skill.

Blogger Timestamping

2008-08-03T21:59:00.003-04:00

Interesting thing about Blogger - it looks like the "Posted by... at..." clause uses the time the post was started, not the time that you eventually push the big Publish button.

F is for Fail

2008-08-03T20:27:00.003-04:00

It's really sad to see code copied-and-pasted. I feel like a teacher who is grading a test, only to find that two kids have exactly the same answers. It's even sadder to see copied-and-pasted comments. That's like seeing that two kids have exactly the same answers, and both kids' answers are wrong.

Object Creation in Ext JS

2008-07-21T20:19:00.002-04:00

Ext JS has a useful object creation pattern. Most constructors can be passed a hash of configuration parameters. This can be used instead of the prototype creation pattern (not to be confused with Javascript's prototypical inheritance). Rather than create new objects that are copies of an existing object, you create objects based on a set of configuration data.

You want an example? OK. Today, I was creating context menus for items in a tree. There is a global pool of possible actions, and each node responds to a different subset of them. When the user right-clicks on a node, I need to

Create a Menu instance
Add all of the appropriate menu items to it
Show the menu

I had hoped that I could create one menu item instance per possible action (20 or so), and then re-use them in different Menu instances.

var actionMenuMap = {
    addChild: new Ext.menu.Item({text:"Add child", icon:"add.png"}),
    delete: new Ext.menu.Item({text:"Delete", icon:"delete.png"}),
    fireZeMissiles: new Ext.menu.Item({text:"Fire ze Missiles!", icon:"fire.png"})
}

var nodeActions = ["addChild", "delete"];   //in practice, this would come from the node itself

//dangerous nesting ahead!
new Menu({
    items: nodeActions.map(function(a) {
        return actionMenuMap[a];
    })
}).showAt(event.getXY());

That didn't work - it seemed like I couldn't share a menu item instance between menu instances. I might be able to get it to work by removing menu items after the menu is dismissed, but I don't actually need to. I can simply hold on to the configuration information.

var actionMenuMap = {
    addChild: {text:"Add child", icon:"add.png"},
    delete: {text:"Delete", icon:"delete.png"},
    fireZeMissiles: {text:"Fire ze Missiles!", icon:"fire.png"}
}

var nodeActions = ["addChild", "delete"];   //in practice, this would come from the node itself

//now we're cooking with functional programming!
new Menu({
    items: nodeActions.map(function(a) {
        return new Ext.menu.Item(actionMenuMap[a]);
    })
}).showAt(event.getXY());

In case you are not familiar, map is a function that is present in every functional language and many dynamic languages. It does not exist in Javascript natively, but is added by Prototype, jQuery, dojo, Mochikit, and probably every other Javascript framework. Here's a sample implementation for reference:

Array.prototype.map = function(f) {
    if (typeof(f) !== "function") {
        throw new Error("map takes a function");
    }
    
    var result = new Array(this.length);
    for (var i = 0; i < this.length; ++i) {
        result[i] = f(this[i]);
    }
    return result;
}

What could possibly make this better? In addition to taking a hash, allow the constructor to take a function. That function manipulates the object after the rest of the construction runs, allowing you to add children or manipulate settings or calculate values. This would be pure icing, of course. Factory functions also feel a lot more lightweight to me than factory objects. See also Rails' version of the K combinator.

Please encode the patterns that you see

2008-07-12T00:36:00.002-04:00

As my software career has advanced, I've steadily become more cynical and bitter. This isn't good, and it's something that I'm trying to work on. It's hard to go a day without having one of those "you've got to be kidding me" moments. "I can't believe they did it like that." "What were they thinking?" "Hasn't anybody ever tried to do this before?" "There has to be an easier way."

Case in point: I recently read that SWFObject was THE way to embed swf files in your HTML. Fine, I though. I'll give it a try. Within minutes, I ran into something simple that it couldn't do. When you call the function that embeds the swf, it might not actually do the operation immediately - it might wait until the page initialization has progressed beyond a certain point, and then inject the code. I needed to run some code AFTER the swf had been embedded in the page (for sure). There's no way that I could see to do this with the stock SWFobject. Despite the fact that SWFObject inherently depends on event dispatch to work, it doesn't dispatch enough of its own events to be useful to me.

As I started to look at the code to see if I could work it in, I stumbled across the following snippet:

var att = {};
if (attObj && typeof attObj === OBJECT) {
    for (var i in attObj) {
        if (attObj[i] != Object.prototype[i]) { // Filter out prototype additions from other potential libraries
            att[i] = attObj[i];
        }
    }
}

"Wow!" I thought. "That looks like it would be useful to lots of people." As it turns out, it is very useful - even to the SWFObject developers, who used the same pattern twice more in the same 58-line function:

var par = {}; 
if (parObj && typeof parObj === OBJECT) {
    for (var j in parObj) {
        if (parObj[j] != Object.prototype[j]) { // Filter out prototype additions from other potential libraries
            par[j] = parObj[j];
        }
    }
}
if (flashvarsObj && typeof flashvarsObj === OBJECT) {
    for (var k in flashvarsObj) {
        if (flashvarsObj[k] != Object.prototype[k]) { // Filter out prototype additions from other potential libraries
            if (typeof par.flashvars != UNDEF) {
                par.flashvars += "&" + k + "=" + flashvarsObj[k];
            }
            else {
                par.flashvars = k + "=" + flashvarsObj[k];
            }
        }
    }
}

That general pattern appears at least 9 times in the code. It's a good thing they put it in a reusable functio... oh wait, they didn't. Instead, they dutifully pasted the code (comment and all) 8 times.

OK, something's going on. Maybe I've become a much better programmer in the past year; maybe everybody else is getting worse; maybe I'm off my rocker; maybe there's some strange performance or compatibility tweak that prevented them from doing what seems plainly obvious to me. Removing the obvious duplication would have simplified their code, making it more readable (and possibly obviating the need for the accompanying comment).

It's the little things like this that make me wonder if we're actually moving the state-of-the-art forward at all. If we can't see (or don't care about) the duplication at the micro scale, how will we re-use software at any broader level?

Planescape: Effortless

2008-06-26T19:50:00.002-04:00

I was amazed to learn that Planescape: Torment, a game released in 1999, is still playable under Vista. I installed it the other night (intending to finish it this time), patched it (using the most recent official patch, which is probably 8 years old), and started it without a hitch. I'm sure some combination of Microsoft's fervent dedication to backwards compatibility, nVidia's seemingly rock-solid drivers, and Black Isle's crack team of developers made this possible.

Recently, Scott Hanselman interviewed Steven Frank of Panic (a Mac software developer). In the interview, it was revealed that Mac users are less worried about backwards compatibility. Apple has completely axed support for Classic mode, which pretty much kills any software more than 8 years old. As a Mac user, I agree. I'm not that worried about old applications not working. But that's because I'm talking about applications. If an old application stops working, there's a good chance that somebody has written a more recent, and probably better, replacement.

Games are a different beast, though. Games are closer to movies or novels. How would people feel if they couldn't read Les Misérables anymore? Or if somebody went and mucked with Star Wars, and then threatened to not publish the theatrical release anymore? Well, we know what happens then: thousands of fans get really really angry. People crave old stuff. Really great content is timeless. Games should be treated the same way. Ever since "game designer" has become an official title, games have been steadily becoming more than children's amusements. There's often some great narrative buried in there, wrapped up tight inside a husk of decaying code. Eventually, that code won't run on new computers, and the narrative will be lost forever.

Of course, now we have virtualization, so maybe it's not such a big problem.

On Succinctness

2008-06-26T19:47:00.002-04:00

One of the reasons that I've been blogging has been to help me communicate succinctly. I've been trying to keep my posts small without being lifeless. I want to balance terseness with expressiveness. If you read this blog, how am I doing? What could I do better?

On Usability

2008-06-26T19:07:00.004-04:00

I've been slowly learning that, in my heart, I'm a usability guy. I'm always concerned for the experience that end users will have with software. I care about making it easier for people to do what they want to do. When it comes to user interfaces, I like bling, but only when it helps the user accomplish a task or understand an interaction. I love Expose. It's such a simple but powerful concept. It's also made for a great demo. I hate Flip 3D. It's pretty, but also pretty pointless.

I've been interested in API design for a long time, too. As a developer, I have used some libraries that were an absolute joy, and other libraries that were a total mess. My recent foray into Flex (specifically, getting the Tree control to lazily load its data) reminded me what it's like to use a bad API.

I mentioned to a coworker the other day that, if you're developing a library, you should be at last as good a developer as the people that will end up using it. To me, it looked like Adobe (or Macromedia - I don't really know who's responsible) put all of their junior developers on the component library project. As a result, we're blessed with the interface IDataDescriptor, whose methods include isBranch() and hasChildren(). What's the difference? Should they ever be different? Who knows?!

I realize that I misspoke to my coworker. I should have said that people developing libraries should have a code aesthetic sense. I don't mean code aesthetics in terms of indentation or brace style or what have you. I mean that API designers need to consider the people that will end up using the library. Just as the hallmark of an Apple Design Award winner is that users "just get it", a good API should also be so easy to understand that it seems obvious to you. As in, "why would anybody make this API any different?"

Design is hard. It takes a conscious effort (well, for most of us). That's why it's important to do. If you're a developer, ask yourself: is this easy enough to understand. Remember, you're writing for an audience - your code does not exist for your eyes only. Even if it did, you're liable to forget everything before you come back to work on it again. If you're a manager, make sure there's enough space for developers to focus on this. If it looks like you're going to miss a deadline, that doesn't give you carte blanche to pressure developers out of finishing their work. Just because the code works doesn't mean it's done.

If you can't make an API grokkable, at least document it well. The standard Java and MSDN docs are full of nonsense.

setFoo(IFurple f) - sets foo to f

Really? I wouldn't have guessed. This tells me nothing. What is foo and why do I care. Why is foo a furple? If anybody ever asks a question about your API, it's a sign that something needs to change. Maybe you just need to add some documentation. Maybe you need a high-level wiki page explaining what the library is trying to do. Maybe you need to rename a method. Maybe you really do need to change the API. Or maybe you just need to smack the developer on the head and tell them to read the Javadocs first. But if you don't change anything, the problem is just going to repeat itself. Who knows - maybe you'll discover a bug or race condition along the way.

Vista Usabaility... It's Got Issues

2008-06-23T21:26:00.006-04:00

I bought a Dell with Vista on it a few months ago. I figured that it was high time that I upgraded to Microsoft's newfangled operating system. It hasn't been too bad. There are some hiccups, but every complex piece of software has its share of those. I'm sure Microsoft will try its hardest to fix those with time.

On the other hand, Aero is pretty much terrible. Sure, it's pretty (at least at first). My impression is that the rendering infrastructure is also quite good. But holy cow, Microsoft needs to hire some actual UI designers. Not just artists, but people who understand user interfaces. There are some reasonably amateurish mistakes in Aero. Even if they seemed like good ideas at the time, after using Vista for about a week, you really start to notice them.

I was first surprised to see how mousovers have invaded Explorer. Look at the following picture. Which icon is selected, and which icon am I pointing my mouse at?
Can you believe that it's easy to delete the wrong file?

The new, unified look for the windows isn't too bad. The window titlebars are translucent, with a frosted glass appearance. At first I though that the effect would be distracting. You know what, I was right.

Explorer windows look a little bare, but the whole top of the window is available for your dragging pleasure. Well, almost all of it. If you manage to click in the upper-left of the window's title-less-bar, you get your friendly system menu.

Finally, my absolute least favorite thing. For whatever reason, the rollovers that cause me to shred the wrong file have also made it into the standard tree control. Here, when your mouse leaves the tree control's bounds, the tree "fades away..." It literally disappears. Which can leave the following unfortunate situation:

Wait, I think maybe there's a bit more to it...

Ah, I bet we can expand that.

There we go!
Now, you might be asking yourself "who cares?" It doesn't seem like it's that big of a problem. True, in Explorer, the only inconvenience is that I don't know at a glance if my folder contains any items. It's a problem, but not a huge one. So, let's take a look at the Java programmer's friend, Eclipse.

Search results, you say. Search results, indeed. And people are trying to emulate it! Little hint: just because Microsoft does it, doesn't mean it's a good idea.

Even the Windows Classic theme isn't safe. The "expandos" in the classic TreeView fade out too.

If only Microsoft supported third-party themes... oh wait, you need to replace system DLLs to do that. Still, I don't know if I can stand the horrible usability of the stock UI. Hey Microsoft: ever wonder why so many themes and theming tools exist for your platform? I can think of a reason.

Even with all my bitching, I think Microsoft's heart was in the right place here. They realized that they needed a better rendering infrastructure (with features like compositing on the video card) and a more slick UI. Apple had been stealing the show on this issue for the past ~~seven~~ twenty-three years. Microsoft needs to go back to the drawing board. They need a fresher UI in the next version of Windows. And they need fewer people working on it. I think Windows is going to buckle under its own weight within 10 years. Apple continues to embellish their own UI, but it's often both functional and pretty. That's why people go ga-ga over it. Expose is a great demo, but I use it daily. Same with Dashboard.

I can only imagine what it must have been like to have seen this live:

... especially when this is what people were used to...

Flex Builder 3 Debugger on Mac OSX

2008-06-18T00:07:00.003-04:00

I was having a hard time getting Flex Builder on my Mac to behave. I came across a post at Big Spaceship Labs that put me on the right path. I tried things, and came up with a reasonable solution, which I added as a comment to the original post.

This technique is useful for keeping several Flash Players on your system (or anything, really). I think I read that MultiSafari uses this trick. The trick is to change an application's bundle identifier. You may have noticed that Mac OSX seems to magically know about all the applications on your system, even though you didn't use an installer for most of them. It uses the bundle identifier (and I believe the bundle version) to keep track of them. I believe that Max OSX treats all applications with the same bundle identifier and version as the same. Internally, they probably form a composite key in a map. Furthermore, most associations (such as "Open With") are made to the bundle identifier, not the application's path. This is so that you can move an application to a new directory without breaking anything. It also means that, if you have 2 applications with the same bundle identifier and version, there is no way to choose which one will launch.

It's surprising how well the whole application list works on Mac OSX. I still had some problems in 10.4, but 10.5 seems to be rock solid in this area. And, when application developers know what they're doing, it works pretty well.

Exceptions in Multithreaded Programming

2008-06-08T23:11:00.002-04:00

Exceptions are great. I've used them since C++. It's even sweeter that more modern languages (Java, C#, Ruby, etc.) have enhanced them with information like stack traces and nested exceptions. Procedural programming will never go back to error codes.

However, exceptions are increasingly insufficient for the work that I do. I suspect that this is true of everyone. Anybody who has done multithreaded programming in Java knows this problem:

class WorkItem implements Runnable {
    @Override
    public void run() {
        try {
            //do some work
        } catch (SomeException e) {
            //WHAT DO WE DO HERE?
        }
    }
}

Exceptions fall apart in a multithreaded environment. Take the example. What are our choices? We could silently swallow the exception (bad style, and also probably wrong). We could write an error to stderr and quit (bad for users). We could throw a RuntimeException (which will forcefully terminate the thread, which might not be too bad). In practice, we should probably notify somebody. But whom? Exceptions fit so nicely into the framework of the call stack - simply unwind the stack until you find an appropriate exception handler. In a multithreaded environment, you have several independent call stacks. It's likely that the code which will respond to the error is running in a different thread. It's insufficient to unwind one thread's call stack - you need a cross-thread error handling mechanism.

What's the solution in Java? I don't know. Erlang's approach is interesting. In Erlang, you can link processes together. By default, any error in one process will kill every linked process. That sounds bad, but it's a reasonable default. You can instead register yourself as a "system" process. Rather than being outright killed by friendly fire, system processes are informed of their comrades' fate. They are able to decide what to do about it. They could restart the failed process, or send a notification, or log it, or whatever. The important thing is that it's automatic. Once you've linked the processes together, the Erlang runtime takes care of notifying other processes about fatal errors and so forth. It's pretty neat

Maybe I'll eventually get to write something in Erlang. I don't know how easy it is to use in day-to-day life, but it seems pretty awesome in theory.

Aspect Oriented Programming in Javascript

2008-06-04T18:03:00.003-04:00

Working with YUI today, I found that I wanted to augment a built-in method. I am using a custom subclass of TreeView. My TreeView registers some mouse callbacks for the generated HTML elements that compose the tree. My TreeView needs to know whenever a Node is refreshed (i.e. whenever a Node regenerates its HTML) so that I can re-register all of my callbacks. The standard YUI Node base class doesn't do this.

This sounded like the sort of thing that could be solved using Aspect Oriented Programming. I wanted to run some of my own code after every invocation of some predefined method. It turns out that AOP in Javascript is pretty trivial. There might be a framework out there, but it wasn't even worth looking. My first implementation simply replaced the existing method with one of my own design, which delegated to the original method. After doing this in a few other places, I clobbered together the following function, whose name sucks, to simplify the whole mess.

function augment(obj, fn, options) {
    var original = obj[fn];
    
    //make copies of the fields in options in case it changes
    //between now and when the generated method is called 
    //(which may be a long time).
    var before = options.before;
    var replace = options.replace;
    var after = options.after;
    
    obj[fn] = function() {
        if (before) {
            before.apply(this, arguments);
        }
        
        if (replace) {
            //TODO: we should probably send original along with
            //this call, possibly embedded in arguments
            replace.apply(this, arguments);
        } else {
            original.apply(this, arguments);
        }
        
        if (after) {
            after.apply(this, arguments);
        }
    }
}


//samples:
augment(YAHOO.widget.TreeView.prototype, "draw", {
   after:function() {
       this.nodeRefreshed(this.getRoot());
   } 
});

augment(YAHOO.widget.Node.prototype, "refresh", {
    after:function() {
        this.tree.nodeRefreshed(this);
    }
});

//also needed (for those playing along at home):
YAHOO.widget.TreeView.prototype.nodeRefreshed = function() { };

It's actually pretty simple. I grab a reference to the original function. We create a new, closed function. It can refer to the original function and the augmentations, but they are not visible outside the closure. That was the primary driver for me - my previous implementation was storing the original functions in global variables, which polluted the global namespace. And we use Function's handy apply(), which allows us to be ignorant of the parameters that the original function takes. In the sample, I also register a dummy implementation of nodeRefreshed() so that I don't need to perform a null check in my augmented refresh().

One improvement that I would like to make, but is probably impossible, is to make the environment for replace() be identical to the environment for the original function. As it is, any variables that are closed by the original function are unreferenceable by the new function. This is particularly problematic when you only want to make minor changes to a function. You copy the original function's source code into replace, and make the minor changes that you need, but it doesn't work because the variable bindings are different. I think you can do this in Lua. I know you can do it in Ruby. I don't think Javascript has the necessary support yet (and, since we always need to support IE, the necessary support will likely never be available).

So, it's good to know that AOP is so easy in Javascript. I've never really used AOP before, though I work with people who have, and who seem to like it. However, AOP's utility here is only because of YUI's heavy use of inheritance. I've ranted against that before. If YUI's TreeControl didn't try to do everything itself, there would be a lot of natural seams that I could use to inject my own logic.

I hope to post soon about my experiences with YUI's TreeControl, Drag and Drop library, and other components.

Swing Does Components Better Than Component Libraries Do

2008-05-25T10:49:00.005-04:00

I'm currently trying to get a handle on ~~Macromedia~~ Adobe Flex using the free, open-source Flex SDK. Flex is a set of tools to create flash-based "applications" using XML UI definition and Actionscript for everything else.

It seems that Adobe has adopted some Microsoft conventions for programmatic UI creation and management. In particular, components in Flex have a long and convoluted lifecycle. In Web Forms, I never understood why I shouldn't create child controls in my Control's constructor; Flex's UIComponent has a createChildren() method. I still don't know why I shouldn't create child controls in the constructor.

In my Swing programming, on the other hand, I don't seem to run across these problems. As far as I know, a Swing component has one and only one step in its initialization process - the constructor. That certainly makes things simpler. I don't have to wonder "What happens if I put this here?" or, more commonly, "Why does this thing not work? Oh, somebody messed up the initialization code."

Actually, most of my frustration here is directed to ASP.NET 1.1. Things may have gotten better in later versions (though I doubt it), and I don't really know how good or bad it is in Flex land. Still, this dredged up bad memories for me. Thick base classes suck!

Why Inheritance Is Bad

2008-05-14T22:05:00.004-04:00

3D graphics programmers, if there are any that read this blog, are familiar with gimbal lock. The Wikipedia article makes an analogy to compass directions. If you tell somebody at the north pole to face south, which direction are they facing? You have no way of knowing, because south loses its meaning at the north pole. At any other place on the planet, your location and direction are completely independent. At the poles, direction has no meaning in the cartographic coordinate system. Two previously independent variables have now become linked.

In programming, when you derive from a base class, the classes are in a state of gimbal lock. We can vary the subclass as much as we want without affecting the base class. However, a change to the base class will not only affect our subclass, but every other subclass out there. Suppose that our base class provides some core data processing functionality, and its various subclasses encapsulate different data sources. So far, so good. We've managed to avoid duplication. If we come up with a new data source, we can simply create a new subclass. However, what happens if we want an existing data source to send data to a different set of processing rules? A class would need to have more than one base class, except in a way different from normal multiple inheritance. No programming language that I've seen can support this scenario.

A hallmark of good object-oriented design is that classes have one and only one responsibility. Why is this important? It makes the code simple, it makes the code testable, it decreases coupling, it might increase cohesion, and it's just plain nicer to read. When you derive from a base class, your class has all of its explicit responsibilities, and the responsibilities of each of its ancestor classes. In our example, each subclass is itself responsible for knowing the ins and outs of a particular data source. Since they all derive from a common base class, they are also responsible for knowing how to process the data. This example only deals with 2 responsibilities, but real systems deal with hundreds.

Remember that inheritance is often used to represent taxonomies. You might have classes such as Animal > Mammal > Giraffe or Widget > Button > Pushbutton. You might say that things on the right are defined in terms of things on the left, but that's not exactly true. A giraffe doesn't have lungs because it's a mammal; we call it a mammal because it has lungs. Using inheritance to describe something (using, for example, interface inheritance) makes a lot of sense. Using inheritance to define something doesn't.

Adventures with the Scriptaculous Ajax.Autocompleter

2008-05-14T21:17:00.004-04:00

I had a chance to use the Scriptaculous autocompleter control at work the other day. Since we are using Ruby on Rails, it was easy enough to use text_field_with_auto_complete() to generate the relevant HTML.

The client noted that it was difficult to enter a new word if it was too close to an existing word. For example, he would type "expensive", and the system would suggest "not expensive". When he would hit Enter, it would replace what he had typed with "not expensive". This is because the Scriptaculous control doesn't support the case where no item is selected - one item is always selected, and it defaults to being the first item.

I decided to have a peek at the source code. In maybe 15 minutes, I had managed to convince it that "nothing selected" is a valid case. Unfortunately, I didn't refactor the code in any way - now every autocompleter behaves this way.

Scriptaculous uses prototype's Class.create() to generate 2 classes: Autocompleter.Base and its sole subclass: Ajax.Autocompleter. From the source code:

Autocompleter.Base handles all the autocompletion functionality that's independent of the data source for autocompletion. This includes drawing the autocompletion menu, observing keyboard and mouse events, and similar.

I appreciate their desire to re-use code. Unfortunately, the use of inheritance has caused the whole mess to suffer some sort of object-oriented gimbal lock. It's easy to create a new Autocompleter that gets its data from a different source; it's much more work to create an Ajax autocompleter with different behavior.

If I have some time (that is, when I'm not busy complaining about something), I might see if I can refactor the scriptaculous source to further separate the data source of an autocompleter from its interaction controller, using some method other than inheritance.

Ruby in LISP clothing

2008-05-06T18:26:00.005-04:00

As it turns out, Ruby's syntax is somewhat flexible. As an example, parentheses are sometimes optional when calling a method.

puts "Oh hai!"
puts("Oh hai!")

link_to_remote "Go there", :url => { :controller => :there, :action => :go }
link_to_remote("Go there", { :url => { :controller => :there, :action => :go } })

The only time the parentheses are needed is when the invocation would otherwise be ambiguous:

link_to_remote h text, :url => { ... }

(h is the HTML escaping function. link_to_remote may actually automatically escape the HTML, but I'm not sure.)

Ruby can't tell what the user is trying to do. The normal way to fix this is to add some parentheses to the call:

link_to_remote(h(text), :url => { ... })
link_to_remote(h text, :url => { ... })

however, it is also possible to use LISP-like parentheses with the same result:

link_to_remote (h text), :url => {...}
(link_to_remote (h text), :url => {...})

Which is better? Certainly one of the first pair is more conventional. For some reason, though, I found myself migrating to the second pair. I suspect it's a combination of

I try to avoid using parentheses in method calls in Ruby as much as possible
LISP is awesome
I have done some Objective-C programming, in which you [object send:message]

Consistency on Rails

2008-05-01T22:18:00.002-04:00

The Ruby on Rails crowd makes a big deal about consistency. They claim that Rails is easy and fast for development partially because it favors convention over configuration. I guess the thought is that you don't need to write any configuration. Plus, once you've worked on a few Rails apps, you pretty much know where everything is.

So far, I've found a lot of that to be pretty much true. Migrations, routes, controllers, views, database configuration - there's a place for all of them, and they're pretty much exactly where I would expect them. However, I've often found myself struggling when actually writing the code. I'm either going to the documentation constantly, or even peeking in the Rails source. (It turns out that some options to methods aren't documented, and the rails plugins that we are using have poorer documentation than the core).

I only have 2 small examples at hand, but I could write about many more annoyances.

link_to and link_to_remote both seem similar on the surface. One will generate a normal hyperlink that will navigate the browser; the other will generate an AJAX link that will interpret the resultant response as either HTML to be embedded or Javascript to be executed.

Both allow you to specify the url using :controller and :action. However, they have very different signatures, and it trips me up everytime.
link_to "Help", { :controller => :help, :action => :sample }
link_to_remote "Help", :url => { :action => :help, :action => :sample }
Did you see that? The only difference between the two is that one requires the URL-related options to be packaged in a hash named :url, while the other disallows the hash
When using the form helper methods such as text_field and text_field_tag, you must be careful. Here is an example:
text_field :post, :name <input id="foo_bar" name="foo[bar]" size="30" type="text" />
text_field_tag 'post[name]' <input id="foo[bar]" name="foo[bar]" type="text" />
As you can see, if you change from one to the other, you might need to also update you RJS files (or remember to set an explicit ID). This would have all been avoided if text_field had simply generated id and name attributes with the same value.

Rails certainly has its good points, but the small details are sometimes infuriating.

Dream Programming Language: Optional Interface Methods

2008-04-10T21:30:00.010-04:00

Yet again, I find myself disliking abstract classes. I was creating a subclass of Java's OutputStream called PathalogicallyBufferedOutputStream (for testing purposes). I wanted to implement flush, which Eclipse nicely autocompleted to look like this:

@Override
public void flush() throws IOException {
// TODO Auto-generated method stub
super.flush();
}

That gave me enough pause to wonder... do I actually need to call the super method. It seems that I have 2 choices:

Check the documentation
Read the source code

Fortunately, Eclipse makes it easy to browse the JDK source code. It only took 2 seconds to verify that the base implementation, in fact, does nothing. Without Eclipse, I probably would have ignored it.

The no-op method stub is in the base class so that consumers of any OutputStream can indicate when the bytes that they have written so far form a semantically meaningful chunk. Somebody might be waiting for some data; this allows you to get it to her. The problem that I have as a deriver of OutputStream is that, without source code or good documentation, I have no idea whether I should or should not call the base method. This is not a problem with JDK-provided classes, but it could be a real bear with a third party library.

Some people might claim that you should "always call the base method". If course, that raises the question of at what point one should call the base method. Should you call it before or after your code? Perhaps you call it in the middle. I prefer to assume that I never have to call the base method, because I shouldn't be deriving from concrete classes, and I like to assume that method that I am overriding is simply a stub that I am free to customize.

If I were designing a language, I would want some notion of "optional interface methods." I want methods that I can declare on an interface (or perhaps an abstract base class) which implicitly have a no-op implementation. If I choose to implement one in a subclass or interface implementor (because I want to be notified when something happens), I would get the benefit of code completion. The caller is not burdened with the need to check to see if I've implemented the method, and the implementor is not burdened with the uncertainty of whether or not to call the base method.

Why Message Passing?

2008-04-10T18:51:00.003-04:00

Back at the end of March, I was starting some work on Ruby on Rails. I thought that a good April Fool's Day joke would be to release "Lua on a Luge" or "Lua on Skiis" (somebody came up with something even better). As the people that I work with know, I'm a big fan of Lua. I appreciate the things that it does well (prototype-based programming, and having very simple, consistent syntax). Even more interesting, though, are the things that it doesn't try to do:

Lua does not try to help you write programs with hundreds of thousands of lines. Instead, Lua tries to help you solve your problem with only hundreds of lines, or even less. - Programming in Lua - Preface

In any case, Lua doesn't directly support multithreaded programming. Normally, this would be fine. However, I've worked on some .NET and Java web apps that:

Spawn a background thread to do some work
Handle multiple requests at once
Share some object / data between all of the request-handling threads

What kinds of things might we share? We might share configuration information, we might share connection pools (of connections, or threads, or whatever), we might share session information. Anything that is shared must be concurrency-safe.
Java and .NET provide synchronization mechanisms. Lua does not.

I started to think about various ways to solve this problem. One could expose a C threading library to Lua. You could expose both the thread management and synchronization primitives. Then, you could only share objects that are specifically written to use that threading library. You might also be able to inject metatables into arbitrary objects, but that would only go so far.

Then I realized that I already knew a better approach. If we relied on simple message passing between Lua processes, we could avoid the need to synchronize every shared objects - no objects would actually be shared (we might copy all values that get sent along on a message, or restrict the kinds of values that may be sent on a message).

It would probably not be as efficient as Erlang's message passing. Also, I don't know of any Lua implementations other than the reference implementation and World of Warcraft's implementation, but I doubt that either support lightweight processes (instead, both implementations provide coroutines). Incidentally, Ruby on Rails is currently not thread-safe (other Ruby web frameworks apparently are, according to the Mongrel FAQ). At least one Rails guy thinks that it should remain that way (see slide 115).

Dream Programming Language: No Static Fields

2008-04-10T18:50:00.004-04:00

If I were to implement my own OO programming language, static data would be the first thing to go. I can't find any compelling reason to keep it around.

Something that people may or may not undestand is that static really means "global". Seriously, I wish that languages like Java and C# simply used global as the keyword instead. Static methods are simply global methods that have a long name with a period in the middle (well, I suppose they are also scoped and are subject to access control). Static, final references to immutable objects are just that - they're global constants. Keeping a single definition of PI around is generally a good thing. And, you guessed it, static fields are simply global variables in disguise.

"But!" I hear you say. "But I need static fields to implement singletons!" Why do you want singletons? Sure, you might think that you will only ever want a single instance of a class. I think you're wrong. What happens when you want to do some unit testing? What happens when the requirements change.

At one point, I'm sure that Windows assumed that it had exactly one display. If singletons had been widely used then, I would not have been surprised if there were some singleton which kept the display's information (in actuality, it may have simply been a global variable). Today, we might not only have several physical displays, but we might also have multiple active windowing sessions on a single Windows server. How about mice? Surely, a system has no more than one pointer. Well, that's kinda true, but my coworkers and I wish that we could get a second pointer on the screen for our pairing sessions. What about my Wacom tablet? And my friend prototyped a game that was controlled by using two mice. Singletons are the antithesis of what the Dependency Injection folks have been telling us.

If I were designing a language, I would want the programming language to encourage good programming practice. Since I'm the one designing this language, I get to choose what is good and bad. And I say, "down with singletons!"

Message Passing

2008-04-06T22:17:00.006-04:00

Anybody who has tried to do multithreaded programming in Java, C#, or most any C-based language finds that it can quickly become nightmarish. Deadlocks, race conditions, lock management - all of these issues will occur. In Java, all threads are able to share objects. As a result, the programmer is responsible for ensuring that his objects can't be put into an invalid state. Java provides per-object monitors to enable this. However, working with such monitors can be a nightmare. Using notifyall when you should have used notify, or forgetting to add the synchronized keyword to a method declaration, or incorrectly assuming that a class was thread-safe when it wasn't; these are all examples of simple problems that can cause your application to fail (perhaps only sometimes).

Erlang, on the other hand, does not suffer the same problems. Instead of sharing things, Erlang processes (which are the Erlang unit of concurrency) do not share anything. As a result, the need for synchronization goes away. Erlang processes send messages to each other. Each process has a mailbox. Each process can retrieve and process messages from its mailbox. You can imagine that a message is copied when it is put into another process' mailbox, but in actuality, since data in Erlang is immutable, the system is probably able optimize that away.

One of Java's founding principles was that manual memory management is complex, and thus easy to get wrong. Far simpler was to provide a system where the programmer didn't need to worry about such things. Erlang does for concurrency what garbage collection did for memory management - it removes that responsibility from the developer. Instead of asking every developer to handle any threading issues that might arise, it provides an abstraction that hides all of them. Behind the scenes, Erlang's mailbox system does whatever synchronization it needs to do in order to be completely thread safe.

Of course, Erlang's approach isn't perfect. Just as Java's garbage collection can sometimes allow memory to leak (when you are holding onto a reference longer than you need to), Erlang's system could still be susceptible to problems such as deadlocks. However, it is less likely to occur, and should be more obviously solvable when it happens.

Erlang's approach is so radical, yet so simple, that it's definitely worth learning more about. I hope to see Java and C# projects adopt Erlang's philosophy when appropriate. In performance-critical applications, manual concurrency management might be the way to go. For most purposes, though, the safety provided by Erlang's approach outweighs any negative side effects.

GameStop culture

2008-04-02T00:25:00.002-04:00

Unlike some people, I actually have a decent GameStop nearby. The people who work there are intelligent, useful, and just generally pleasent. I was there tonight, picking up a copy of Condemned, which I have heard good things about.

In any case, the clerk was telling me that its sequel is also good, and noted that it is more of a horror game (Condemned being more of a thriller). As I was leaving, he suggested that, if I beat it in less than a week, I could trade it in towards its big brother. I thought he said something about full credit.

This struck me as odd. Here was a Gamestop employee essentially offering to loan me a game for a week, with no risk to me. Well, I suppose I would be turning $17 of hard currency in GameStop store credit, which is good for them, but it still seems odd. Then again, their whole business model is odd.

On the other hand, perhaps I simply misheard him.

Ruby on Rails after 3 days

2008-03-31T18:49:00.006-04:00

I've been learning Ruby on Rails at work for the past 3 days. I had never really worked with Rails before, though I had certainly read enough to form misguided opinions about what was good and what was bad. After some hands-on experience, I have a better idea of what I do and don't like.

Routes

Routes are Rails' way to find the handler for a given request. You can establish routes by editing routes.rb. I don't know whether this is common practice, but an advantage of using a Ruby source file over XML is that you can use Ruby language and library features. This makes it easier to add conditional or generated routes, something that the Rails REST support does with abandon.

Routes are also 2-way. They are also used when you want to know the URL to a particular controller/action. This makes virtually impossible for your URL recognition and URL creation code to get out of sync.

RJS

RJS is Ruby's javascript-generation library. It's used as part of infrastructure for AJAX applications. Most of the Ruby AJAX tutorials that I've seen do something that I consider to be very bad practice - they run code on the server that doesn't need to be. The typical pattern seems to be to use an RJS helper method to emit Javascript that attaches a listener to some other HTML element. This listener then issues an XmlHttpRequest to the server, which produces Javascript code. This code is then sent to the client, where it is eval()'d. While there is nothing wrong with that pattern per se, imagine doing that for something as simple as clearing a text box. Imagine thousands of clients contacting the server just so that they can clear their text boxes.

Of course, in typical Rails fashion, there are many ways to use RJS. Rather than use it in response to a lightweight request to the server, you can also use it when you are building up the original HTML page. This can both simplify your application and also increase its perceived performance. Of course, you need to know that you are able to do this.

Unfortunately, the Javascript that is generated isn't that much different from the original Ruby. Compare this RJS Ruby code:

page['query_results'].value = ''

to this hand-crafted Javascript:

window['query_results'].value = '';

I'm generally against code-which-generates-code, and especially when used to avoid learning a new programming language. Web developers usually end up developing a reasonable understanding of HTML, and those in the Ruby community seem to also be reasonably proficient with CSS. I would hate for them to miss out on a fantastic language like Javascript.

Ruby Itself

Finally, Ruby as a language creates some interesting situations. I find its dynamism to be both a blessing and a curse. It's pretty amazing that Aptana Studio seems to be able to "go to declaration" reasonably well. Of course, this only works for methods which are defined statically, in the source code. Some methods are added dynamically (via method_missing or some other black magic of which I am unaware). I believe that it is impossible to even find out what these are - you simply have to know what methods ActiveRecord may have added to your model objects. I'm sure that this gets easier with exposure.

Incidentally, this very dynamism has spawned some very interesting workarounds. Since the routing tables are stored as data, it's hard if not impossible to see the set of routes by looking at the code. However, there is a rake target to see the contents of the routing table:

rake routes

Similarly, the migration nature of Rails database management makes it quite hard to quickly grok the structure of the database. However, Rails generates a schema.rb file which theoretically mirrors the current state of your database schema. Perhaps not strictly necessary, but awfully convenient.

Conclusion

In all, I'm reasonably happy with Rails so far. It's quite different from previous experiences that I've had developing web applications. In a way, it hearkens back to when I was first learning HTML. It was trivially easy to write some HTML and then see what it looked like in a web browser. The code/test cycle was very small. Rails brings the same sort of simplicity to web application development. Rather than spend time writing code, you spend time learning how to write the code in the correct Rails mindset.

I don't know if Rails will scale well. I don't know if Ruby will be able to keep up. I don't know if ActiveRecord will strangle us or liberate us. However, I do know myself, and I know that I will stay skeptical for quite some time. Hopefully, one day I will realize that I've been happily writing Ruby code for years and find that neither it nor Rails have failed me.

Windows Home Server

2008-03-11T22:38:00.004-04:00

I just received my HP MediaSmart Server (actually, the EX470). I've just finished setting it up and playing with it a bit.

Here are some problems that I encountered:

To get my VMWare Windows XP instance on my Macbook Pro to see the WHS, I needed to put the network connection into Bridged mode.
It didn't initially work with my Airport Extreme base station, but there is a solution (and not a bad solution at that).

Here are some of my findings:

It appeared to be cheaper to buy an extra 500GB drive (even the exact same drive as is in the server) than to buy the EX475. The drive that is in the first slot is a 500GB Seagate ST3500630AS (Newegg search).
It runs IIS 6.0, ASP.net 2.0.50727, PostgreSQL 8.2.1.7007, Firefly media server (build 1.1 svn-1601), .NET 2.0 SP1, and Java 1.4.2_13
Though it is preferable to administer the server only via the Home Server Console and the shared folders (for adding Add-Ins), it is also possible to remote desktop to the server and get a full Windows shell, start menu and all.
The HP Photo WebShare web application appears to be written in Java, and is being served by Jetty under IIS using isapi_redirect.dll. If I knew anything about Jetty or about using a Java application server under IIS, I would tell you more. Otherwise, the app looks pretty much crap. It works, but looks very amateurish. Much like most OEM-supplied software that comes pre-installed on your computer. On the bright side, it seems to have an ActiveX control to bulk upload images, which is a very nice (and essential) touch.

If you don't get giddy about the idea of an extensible, autonomous, unobtrusive computing henchman in your domicile, then I suppose Windows Home Server isn't for you. But I personally relish the idea of making my home network a little more than a web browsing gateway. I have great faith in the ability for computers to improve the way we live, and I see this as a very small step towards a seemingly distant, but bright, future.

Changing Microsoft Office 2008 for Mac Splash Screen Registration Information

2008-03-10T19:47:00.005-04:00

I stumbled around a bit before finding this solution. From some post:

Quit all Office programs
Delete these files:
- /Users/<user name>/Library/Preferences/Microsoft/Office 2008/Microsoft Office 2008 Settings.plist
- /Applications/Microsoft Office 2008/Office/OfficePID.plist
Start any Office program
Retype your name, company, and product key

It seems that, if you only delete one of them, it regenerates the other.

Memeo AutoSync is extremely slow

2008-02-14T19:49:00.001-05:00

I just bought a Western Digital MyBook Home Edition as a backup hard drive. The drive itself is small, pleasing to the eye, and not too noisy. It comes with WD Anywhere Backup, which itself is a rebranding of Memeo LifeAgent AutoBackup (I assume the standard edition). This software is supposed to automatically back up your files as they are changed on your hard drive.

I set the software to backup my WHOLE drive (from the C: directory down, with no exclusions). That's about 208GB of data. I've left it running overnight and over a day at work, and this is the progress so far.

(I didn't take a snapshot of the other screen, which indicates that this process was started at 12:32 am, so it's actually been running for about 19.5 hours at this point, though it apparently was idle for a while)

That's right: after 19.5 hours of work, it has only backed up about 20 GB of my data. At this rate, it will take about eight and a half days to complete. USB 2.0 may be slow, but it ain't that slow! The whole time, it's been using between 30% and 50% of one of my CPU cores.

To be fair, it is verifying the contents of all of the backed-up files as it goes. I do appreciate that. But seriously!

To ensure that it wasn't a problem with the drive, I copied my 1.3GB Guild Wars data file (a single file) to the drive. That took 1'09". I copied my nearly 8GB Unreal Tournament 3 install to the drive (1900 files exactly), and that took 10'21". Memeo copies files at 2% of the speed that Windows Explorer does. Another way to look at it is that Memeo is 50x slower than Windows Explorer.

I recommend that you don't buy Memeo software. They might eventually get it right, but you should wait until then. Western Digital should find a new company to provide their backup software - this Memeo junk just makes them look bad.

Removing Credit Card from Xbox Live

2008-02-10T01:54:00.001-05:00

I accidentally charged an old credit card on XBox Live tonight. I started looking around to figure out how to remove the old card. Half an hour later, I stumbled upon the solution. Since it's so obtuse, I will describe it here.

Notes: You must have a Windows Live ID associated with your XBox account, and you must have another credit card tied to your XBox Live account. You will be transferring all charges FROM one card TO another card. This apparently also has the side effect of "removing" the old card from the system. If you need to do something different, like to cancel XBox Live, you might need to call support instead.

Go to Microsoft Billing and Account Management. Log in.

You will see a list of credit cards attached to your Windows Live ID. Note that this will only work if you have associated your gamertag with a Windows Live ID. (In my case, you can see that I have removed one of my cards).

Click the card you want to delete. This will take you to a screen that includes this:

Choose "Go to payment method information". You will be at a screen that includes this:

Click "Use a different payment method," and select the card that you want to use.

I could make jokes about how Microsoft is just looking for an excuse to keep your money. Instead, I'll joke that they can't design user interfaces. I mean, it's not like anybody else has solved the cancel-a-credit-card interface problem.

Oh wait. This is from the iTunes music store:

Of course, this should be no surprise to any 360 owner. The 360's interface is among the worst things ever created by Redmond. Yes, I think it's worse than Clippy. Quickly - where do you go to get a list of your active downloads? If you're in a game, the only way is by pressing the "XBox" button, and selecting PERSONAL SETTINGS! Clearly, downloads are a setting. There are a number of settings which you change in 2 different places. Themes are one example. Also, the 360 is the only console whose menus are affected by the speed of your internet connection. Quite often, I can't start a game for 30 seconds or more because the menus haven't yet loaded (presumably, downloading ads off the XBox servers).

It's worth noting that the XBox 360 has been out for OVER 2 YEARS! The original XBox was only around for 4 years before the 360 was introduced. This console is half dead, and the user interface is only beta quality. Nice play, Microsoft.

Update:

I noticed that Microsoft added an item to their help center in the XBox interface. You can instead go to xbox.com/accounts to manage your credit cards. The interface is a little nicer, though they still dump you back to the old system to transfer your XBox Live subscription from one card to another.

The Dangers of Flash

2007-07-23T17:22:00.000-04:00

As I sit here and use my computer (2.4 GHz MacBook Pro), I noticed that the laptop was getting quite hot. I looked up at smcFanControl, and noticed temperatures between 165 and 175 Fahrenheit. The fans had spun up to nearly 6000 RPM (from a base of 2000). Trying to figure out just what was going on, I noticed that Safari was using 66% of my CPU time.

As I closed some tabs, I noticed that my temperature was dropping... fast. In about 20 seconds, the temperature was down to 140-145 Fahrenheit. the fans had slowed as well.

It looks like the problem tab was Verizon Wireless' Chocolate Site. I had clicked the commercial link because I was trying to identify the song. I had paused the movie while I went to some other site to do some searching.

I fired up Quartz Debug, and learned that the entire contents of the HTML document were getting redrawn continuously. That's a lot of pixels to be pushing.

So, if you ever use flash for your website, remember to take it easy on your end user's laps.

Objects, just not Object Oriented

2007-04-23T22:31:00.000-04:00

I've been on a real programming language kick for the past year. First, I picked up Prolog again. Then Javascript, then Ruby, then Lua, then Erlang. This is in addition to lots of programming in C#, some in Java, and a little C++ on the side. I'm starting to notice lots of similarities between languages. I don't mean superficial similarities (Java has classes. Whoa, so does C#. Dude...). I mean deeper, conceptual similarities.

As an example, take objects. C++ has objects. Smalltalk does too. Also Java and C# and a whole lot of other languages. However, all of the objects in those languages are bound together by sharing a single class.

Javascript and Lua don't share that requirement. They're still object-based (you have entities which hide implementation and expose functionality), but object's don't necessarily share a common class. In fact, an object in these languages can (and often does) have no class at all.

Further still, some languages have object-like things. Erlang, for example, allows you to create processes. Processes can send messages to each other. Processes can even have private data (since a process is just a function, and functions can always have local variables, processes can have private data). Is this an object? I think so!

Some languages even give you object-like functionality in even stranger ways. Take purely functional languages like Lisp (or Scheme) or ML. These languages let you create "closures". A closure is a function that exists within (is closed to) an environment. In Javascript (not a functional language, but one which supports closures):

function make_add(left_addend) {
  var f = function(right_addend) {
    return left_addend + right_addend;
  }
  
  //At this point, f is bound to left_addend.  Whatever
  //we passed in as left_addend is now known by f.
  //We can return f, and it will correctly remember
  //the value of left_addend.
  
  return f;
}

var add10 = make_add(10);
var add1000 = make_add(1000);

print_to_console(add10(5));
print_to_console(add10(50));

print_to_console(add1000(5));
print_to_console(add1000(50));



15
60
1005
1050

Why is this cool? Well, we have effectively created an entity which has private data. We could go a step further and make 'f' into some sort of dispatch method. It could take, as its first parameter, an operation. So, perhaps we could have this:

function make_number(value) {
  var f = function(operation) {
  if (operation == "add") {
    return make_number(value + arguments[1]);
  } else if (operation == "sub") {
    return make_number(value - arguments[1]);
  } else if (operation == "toString") {
    return value;
  } else {
    return null;
  }
}

var number10 = make_number(10);
var number1000 = make_number(1000);

print_to_console(number10("add", 5)("toString"));
print_to_console(number10("sub", 50)("toString"));

print_to_console(number1000("sub", 5)("toString"));
print_to_console(number1000("add", 50)("toString"));



15
-40
995
1050

Not too pretty, but it should work. Object-oriented programming in functional languages. Who knew?

Big, Bigger, Biggest Design Up Front

2007-04-23T22:20:00.000-04:00

This was originally posted at Ternary Software. It is mirrored here because I wrote it and I want it on my personal blog as well as my work blog.

Background

Extreme Programming encourages its practitioners to do away with a technique known as Big Design Up Front. BDUF is a software development process that has existed for nearly as long as software development. The general idea is to design a software system completely on paper before writing a line of code (experiments and proof-of-concepts aside). XP claims that this is a bad idea. In my recollection, the XP book claims that you wouldn’t get the design right anyway. You really can’t get the design right until you have finished writing the code.

Reality

In theory, this doesn’t sound too bad. I mean, the best way to understand a system is to try to formalize it in code. You’ll spot all of the inconsistencies and problems pretty quickly. (In fact, I think that’s the real skill that a software developer brings to the table - the ability to analyze a system and find the problems). But still, to do no design up front? Clients don’t come to you with no design. In fact, if one did, would you really take their business?

Prospective Client: Boy, do I have a project for you!
Development Team: Great, what do you have in mind?
Prospective Client: Well, see, it’s a thing. It does stuff. It’s a lot like That Web Site, only different. By different, I mean really different. But I’m not quite sure how. Also, it transforms into a boat.
Development Team: What? That makes no sense!
No Longer Prospective Client: Well, I thought I’d design it as I go.
Development Team: Well, we’re really booked right now, but we’ll call you as soon as we have some free time.

So I suspect that developers, too, need to do some design up front. I can think cases where NOT designing up front has cost us a lot of effort in re-design. Developing a 2-tier system when we knew that the client really wanted a 3-tier system is but one example. (We fixed that problem within a month, but it was still a hassle that could have been avoided). But how much design is too much? The XP book says something about doing “enough” design, but that doesn’t really provide a lot of guidance.

Disclaimer

Now we get into the part that I don’t really understand well enough. These are my thoughts, but they’re surely wrong. My thoughts will probably change with time, and I would appreciate any feedback that I can get.

3 Kinds

Here’s an over-generalization: systems that you design can fall into 3 categories:

Systems that perfectly implement the requirements of today with no thought to the future. Any attempt to use them in a way that is different from the original intent will require massive amounts of redesign. I’ve seen a lot of Javascript code that seems to fall into this category. For that matter, a lot of code written on consulting projects seems to fall into this category.
Systems that implement any conceivable requirement for the next 5 years. Sometimes, they do a good job of this; sometimes, they fail miserably. Yes, ZZZManager, I’m talking about you! A lot of libraries tend to lean in this direction.
Systems that, while implementing the requirements of today, also have a lot of hooks for the requirements of tomorrow. The assumptions that they make (and the hooks that they provide) aren’t necessarily perfect, but it’s not TOO hard to change them to support new requirements. Some really good libraries and really good frameworks do this. When I saw how simple it was to write a Java Servlet, I remember thinking that Servlets strike an excellent balance of providing functionality to the programmer without making too many assumptions.

I would furthermore attempt to characterize those three kinds of designs as:

Not Enough Design
Too Much Design
A design that’s juuuuust right.

The Source of All Design that is Good

Let’s assume that I’m correct in my characterization. How do you get this kind of design? Well, I think it’s a little like Chess. For chess masters, the match is won by the person who can “see” further into the future. I think that good software design comes from people who can (correctly) guess the future. There are a number of contributing factors:

You yourself. I think some people just naturally think more about the future than others. I don’t think that people who think about the future are better at designing (see Too Much Design). But I do think that it’s important to be thinking about both the present and the likely future.
Your experience. If you’ve seen a problem before, there’s a chance that you will try to avoid it in the future. I think that experience can help you avoid problems without even realizing that you are avoiding them - you just make really good choices.
The pressures on your team. I think teams that have some healthy amount of slack time will be more likely to think about things that are a little broader than their current task.
Your level of attunement. The more in-tune you are with your client and their needs, the more likely you are to guess correctly about their future requirement needs. You need to understand not only their product, but to a certain extent, their business model and their users.

Tips

Those factors are often hard to change. Are there more immediate guidelines for avoiding problems? I certainly don’t know how to avoid all problems, but here are some general thoughts that have helped me so far:

If, while implementing a feature, you think you will be ripping out the code in a month, you probably need to design a little more. Changing code in a month is OK, but replacing it is scary. Replacing it in 3 years is probably OK.
If you can’t explain to the client what you are working on and why they need it, you’ve probably designed the wrong thing (i.e. over-design)
Don’t take shortcuts. If there’s an easy way and a hard way to design a system, make sure that the easy way isn’t really the “really hard later” way. It might be perfectly fine to take the easy approach. In fact, you probably should. Just make sure that you’re not going to get screwed in the near future.
If you are writing very similar (but also slightly different) code in several places, you probably need more design. For example, you use an Excel writing library in 20 places in the code base, something is screaming to be extracted into its own class (or set of classes).
If your code has no layers, you probably need to design some more. Just as classes need to be small and simple, so too must groups of classes. If you have a class that is referenced by code in several namespaces, you should wonder why everybody needs to know about him. (Perhaps everyone really does need to know, but you should still think about it).
If you end up writing a lot of static methods, you need to extract some classes. A few static methods (especially named constructors) are OK, but if you have too many, you’re just writing procedural code.
If you end up writing a lot of private methods, you need to extract some classes. A few private methods are OK, but if you have too many, you’re just writing procedural code.
If you know that some of the code that you are writing is more general than your specific task, break it into 2 pieces. Write some of the code in a general way (using Template Method, aggregation, or something cooler), and write your specific code in terms of the general pattern. LISPers (and other functional programming dudes) know how to take this to the extreme. (Every functional language that I’ve encountered comes with a built-in function that can apply a function to every element of a list. Finding the sum of a set of numbers is often as simple as applying “+” to every element of a list). Be very wary of this, however, as this can lead to some scary over-design.
If you don’t know what you are going to be doing later this week, you can’t anticipate the future. Have a plan.
If your plan describes what you will be doing for 6 months, you will be using a different plan in 6 months. You’ll probably write code that won’t make sense in that new plan. Spend less time designing, and focus on short, iterative releases.
If you write code that everybody uses, and which affects code throughout the system, and your code needs to be changed whenever some other requirement changes, you have too much centralization. You’ve designed a fancy system, but it’s going to cause you problems.

Conclusion

Those are all of my ideas for now. Actually, I could keep going, but I would just be rambling more. I want to again stress that this area is very unclear to me, and what I have typed here are only my current thoughts on the subject. I would love to hear other people’s opinions!

On Windows CE, Microsoft, the framebuffer, backwards compatibility, and demons

2007-04-23T22:10:00.000-04:00

This was originally posted at Ternary Software. It is mirrored here because I wrote it and I want it on my personal blog as well as my work blog.

I spent the better part of today struggling to get some fancy graphics on my O2 XDA Exec (read a review if you are interested in the device). I know that I can use the Windows CE GDI to draw the graphics on the screen. I also know that will be slow. Not necessarily too slow, but certainly not an API that any self-respecting game developer would use. So, moving along, I knew that there was the GAPI - the Game API. It sounded great - provide access to the framebuffer and to low-level keyboard functions. So I read and implemented.

Everything was working fine in the Pocket PC emulator. But on a real device, no luck. It ran, but incorrectly. For one, it pretended that my device had a resolution of 240×320 pixels, which is 1/4 of the actual number. It used pixel doubling to fill the whole screen. On top of that, it wouldn’t draw anything in the leftmost pixel column. I had an ugly blue column extending from the top of the screen to the bottom.

I spend hours looking for a solution. I looked to see if it was a clipping problem, an API misunderstanding, a device issue. I couldn’t find any information. Then, I started reading support forums for other applications. It was there that I discovered that the problem is that, at this point, GAPI is provided on new devices as an emulation library. Apparently, in the Pocket PC 2002 / Windows Mobile 2003 days, GAPI was the way to make games. However, a lot of game developers made a lot of assumptions about transient things like, say, the screen resolution of devices. Microsoft, fearing that high resolution screens would break a lot of existing applications, decided to release a GAPI implementation for more recent devices that doesn’t return a pointer to the ACTUAL framebuffer. Instead, the application gets a pointer to a back buffer. Upon calling GXEndDraw(), a GAPI function whose purpose should be reasonably obvious (it flags that drawing operations are complete), this emulation library flipped the buffers and (on my device), upscaled all of the pixels.

HOWEVER, certain crafty game developers apparently thought that GXEndDraw was a useless function. After all, it didn’t seem to do anything on their Windows Mobile 2003 devices. And, if it doesn’t do anything, why should they take the time to call it? I believe that this is why I was having trouble with Pocket Quake. I plan to try to fix it and then tell the developer.

As for the single unwritable column of pixels, I simply think that this is a bug in the copy of gx.dll (the GAPI runtime library) that is on my device. The pixel doubling routine is probably wrong. I believe this because I think I have solved my problem. I can write all 480×640 pixels, including that annoying leftmost column.

To perhaps save somebody else the time that I just spent, let me share my findings. Microsoft suggests using the GETRAWFRAMEBUFFER escape operation in the GAPI Legacy Support section of “Developing DPI-Aware Applications”. There’s a good example in that article. This seems to be the new way to really, definitely, for sure this time, get raw access to the framebuffer. Until some developer takes too many liberties with it, and Microsoft has to invent yet another hack to keep other people’s applications working.

I should probably just bite the bullet and use DirectDraw. I thought that I had read on the GAPI for Handheld PC’s page that DirectDraw was not supported on the PocketPC platform, but they were perhaps referring to the 2002/2003 platform. I don’t see DirectX dlls on my device, but perhaps they are hiding.

Perhaps I’ve read so many of Raymond Chen’s posts that I’ve started to sound a little like him.

The Double If

2006-10-26T01:49:00.000-04:00

This is a anti-pattern that I have started to see when people attempt to think in terms of objects, but don't think in terms of messages. The case is that you have one bit of code that will do a test and will return either a real value or null (in this case, that method is GetParentName). Then, anybody who uses GetParentName needs to either be OK with getting a null back, or needs to test to make sure that the value returned is not null. This code illustrates that anti-pattern:


    class Item
    {
        //Return either a real string or null
        string GetParentName()
        {
             Item parent = GetParent();
             if (parent == null)
             {
                 return null;
             }
             else
             {
                 return parent.GetName();
             }
        }
    }

    void SetupTextBox(Item myItem, TextBox someTextBox)
    {
        string parentName = myItem.GetParentName();
        if (parentName != null)
        {
            //TextBox::SetText(string) will throw an exception
            //if it is passed a null
            someTextBox.SetText(parentName);
        }
    }

A way to fix that is to change the class' interface. Instead of getting values from the class, let's tell the class to tell its information to somebody.


    class Item
    {
        string TellParentNameTo(StringReceiver receiver)
        {
             Item parent = GetParent();
             if (parent != null)
             {
                 receiver.SetString(parent);
             }
        }
    }

    void SetupTextBox(Item myItem, TextBox someTextBox)
    {
        //myItem will do the best that he can do, even if he 
        //has a null
        myItem.TellParentNameTo(someTextBox);
    }

The biggest problem with the second example is that you need to have really, REALLY broad interfaces that are used everywhere. In this example, TextBox would need to be a StringReceiver. In fact, anybody that could conceivably get a string from an Item would need to be a StringReceiver. If TextBox were written so that it is a StringReceiver from the beginning, everything works great. However, suppose that StringReceiver was written later than TextBox. Either TextBox will need to get it retrofitted, or an adapter class will need to be written:


    class Item
    {
        string TellParentNameTo(StringReceiver receiver)
        {
             Item parent = GetParent();
             if (parent != null)
             {
                 receiver.SetString(parent);
             }
        }
    }

    class TextBoxStringReceiverAdapter
    {
        TextBoxStringReceiverAdapter(TextBox textBox) {}

        void SetString(string str)
        {
            textBox.SetText(str);
        }
    }

    void SetupTextBox(Item myItem, TextBox someTextBox)
    {
        //myItem will do the best that he can do, even if he 
        //has a null
        myItem.TellParentNameTo(
            new TextBoxStringReceiverAdapter(
                someTextBox)
            );
    }

Concurrency

2006-10-05T23:16:00.000-04:00

There has been a lot of talk recently about how the exponential growth in CPU power is tapering off. Almost everybody is saying that the future will be multicore processors and multiprocessor systems. A lot of consumer PCs are shipping with dual core processors. The Xbox 360 uses a custom, triple-core CPU. The PlayStation 3 will have 7 active cores. This isn't theory; it's quickly becoming fact.

Everybody who has ever tried knows that multithreaded programming is hard. The programmer is required to do a lot of mental juggling to try to imagine what the system will do. I'm not a fan of making a programmer do a lot of mental gymnastics if the computer can do a better job of it.

One of the biggest challenges in multiprocessing is in dividing an application into multiple threads. To this day, most applications run in a single thread. Even when an application has multiple threads, it's usually only when performing some lengthy operation, such as a find. If multiple threads were not used, the entire application would freeze up.

Interestingly, programmers have been dividing their programs into pieces for a long time. Instead of carrying out a process by performing 1 million discrete operations, we perform 10 high-level operations. These operations need to be carried out in order, because the output of one is used as the input to the next. Also, each of those high-level operations is implemented in terms of 15 mid-level operations, and each of those is implemented in terms of another 10 mid-level operations, and so on. These subdivisions are usually used to take what is essentially one long process and split the code for that process into multiple, bite-sized chunks that an individual programmer can understand.

Unfortunately, This doesn't help with the concurrency problem. However, a lot of programmers have started programming in an object-oriented way. Making classes is a very different process from making functions. The purpose of functions is allow the programmer to focus on a small section of code. However, the code that the programmer is working on is still sitting inside of some larger whole. The purpose of classes is to let the programmer work on a piece of the program in complete isolation. This is why a class contains both state and behavior. In fact, the class that a programmer writes for one program should be able to be dropped, without change, into a completely different program. A BlogPost class should be reusable, as long as it provides all of the behavior that everybody needs.

Good OO design is characterized by a principle that I recently learned. The principle is "Tell, don't ask". This has nothing to do with bullying your objects around. To translate, to "tell" your objects is to send messages to them. To "ask" your objects is to write them such that they cough up their internal state to whomever asks. So, the principle is really "write your objects so that they expose behavior to the outside world, but don't write them so that they expose state to the world". I'd love to go into more detail, but I don't have space right now.

If we take TDA to the extreme, we can make some radical assumptions. First of all, we can completely remove return values. Return values, I claim, are simply an artifact of the procedural days. If you need to have an object tell you something, send a callback object to him. You TELL him to TELL you something. Crazy, no? Now, you might point out that this is a lot of work for something that used to be handled by something as simple as a return value. This is a valid concern. However, my gut tells me that the callback scheme would actually simplify a lot of code. Again, I would love to go more into this topic, but I don't have space. Please just assume (as I am assuming) that callbacks are preferable to return values in a lot of cases.

So, with this mindset, we can really think of method calls on an object instead as sending messages to the object. Since the object can't send anything back (except by sending US a message), there's no guarantee that the target object will process the message right away. In fact, there's no guarantee that the target object will process the message at all. For example, when you're working on a networked application, you need to realize that nothing is ever guaranteed. You might have an object that represents a remote computer. You might send a SendPacket message to that object, but that object might not send a packet out right away. In fact, if the remote computer goes down, the packet might never get sent at all.

Most importantly, this shifts our thinking. In the procedural world, it is vital that all steps be carried out in the correct order. This is because each step will update data somewhere and return information to the caller. In the TDA world, this can never happen. Of course, SOME ordering is still needed. But that ordering can be captured in a different way. Only the significant order is enforced.

A trend that I am starting to see in systems that are designed with TDA is that they seem to be "configured" more than "programmed". I mean that main() is pretty much responsible for instantiating a bunch of instances and wiring them up correctly. Then, main() sends one message to one object and, like a bunch of dominos, eventually something comes out of some other object. In the meantime, all of the objects involved were furiously chatting. The important thing is that they were all basically dormant until a message was sent to them. Then, they did a whole lot of stuff and sent a whole lot of messages to other objects, but then they went dormant again.

So, what does all of this have to do with concurrency? Suppose that each object in the system had its own thread and message queue. The queue is pumped by the thread, and each incoming message is dispatched to the appropriate message handler. Now, when an object needs to collaborate with other objects, it not only doesn't HAVE to wait until the other object is done, it simply DOESN'T wait. It is executing in a completely different thread that may be running on a different processor (or even a different machine!). Also, since the object doesn't have to wait, it can run through all of its code very quickly and return to a dormant state. While dormant, the object's thread would not get scheduled.

What this effectively does is to turn a processor's execution cores into a simple resource, like physical ram. Just as a program does not need to decide where in physical ram to store its data (it stores its data in virtual address space, which the processor maps to physical ram), the program will ALSO not need to worry about how many threads it has.

There are so many details that I have left out. This post could turn into a whole lot of smaller posts with a lot of detail about various small parts of this scheme. But this is a decent overview. I hope that people read this and have good ideas.