Fuzzy Unit Testing, Performance Unit Testing

Because of the topic of testing coming up in my recent post "Is Morality Eating Your Own Dogfood?", it made me finally publish the following notebook entry from 2007...

In reading Philosophy 101, about Truth with a capital "T", and the non-traditional logics that use new notions of truth, we of course arrive at Fuzzy Logic with its departure from simple binary true/false values, and embrace of an arbitrarily wide range of values in between.

Contemplating this gave me a small AHA moment: Unit Testing is an area where there is an implicit assumption that "Test Passes" has either a true or false value.  How about Fuzzy Unit Testing where there is some numeric value in the 0...1 range which reports a degree of pass/fail-ness? i.e. a percentage pass/fail for each test.  For example, testing algorithms that predict something could be given a percentage pass/fail based on how well the prediction matched the actual value.  Stock market predictions, bank customer credit default prediction, etc come to mind.  This sort of testing of predictions about future defaults (i.e. credit grades) is just the sort of thing that the BASEL II accords are forcing banks to start doing.

Another great idea (if I do say so myself) that I had a few years ago was the notion that there is extra meta-data that could/should be gathered as a part of running unit test suites; specifically, the performance characteristics of each test run.  The fact that a test still passes, but is 10 times slower than the previous test run, is a very important piece of information that we don't usually get.  Archiving and reporting on this meta-data about each test run can give very interesting metrics on how the code changes are improving/degrading performance on various application features/behavior over time.  I can now see that this comparative performance data would be a form of fuzzy testing.

Your Pipe Inventory Record is not a Pipe

In this blog, I recently posted: Reality is the System of Record. Unfortunately, I just found a lost reminder to myself about a really good introductory example to use. Even though it didn't make it into the original, it seems worth mentioning here anyway...

La trahison des images is a famous painting by Magritte which seems to pose a riddle.  It contains nothing but a pipe and the phrase, "This is not a pipe". However, it only seems enigmatic because the solution is too obvious.

As Art critic Robert Hughes explains in The Shock of the New, "This, indeed, is not a pipe. It is a painting; a work of art; a sign that denotes an object and triggers memory". As Magritte himself once remarked, "Of course it's not a pipe. Just try to fill it with tobacco".

In other words, it is a representation of a pipe, not to be mistaken for an actual one. And as obvious as that seems, computer system developers make the same mistake all the time. They do so when they forget that their Customer data table & business domain objects are not actual customers, but only representations, i.e. memories, i.e. copies of information.  So, as with all cached copies of external data, it is the duty of your so-called "system of record" to keep in sync with the real customer, in the real world, because Reality is the System of Record ...

A hole for every component, and every component in its hole

Amongst the surprisingly simple ideas that aren't so simple when thought about Philosophically are holes.  There is a small library of publications on what exactly holes are.  A summary can be found online in this Stanford Encyclopedia of Philosophy article on the metaphysics of holes.  One of the viewpoints it cites is: ‘There is no such thing as a hole by itself’ (Tucholsky, 1930).  This reminded me of one of my very first blog posts from 2000 which I reprint here...

There is no such thing as a Component

I maintain that there is no such thing as a Component in the same way that there is no such thing as a donut hole. Just as the donut hole doesn't exist without a donut to define it, a Component doesn't exist without a Framework to define it. Using a printed circuit board as a metaphor for a framework, it's the "sockets", into which IC chips are meant to be plugged, that define components. So called universal or standalone components are meaningless (and certainly useless) without some framework that expects components of the same purpose and interface.

Ok, so what's your point? The point is that too many developers (and books on the subject) think about components as standalone chunks of functionality that can be "glued together" after the fact. They don't realize that the framework has to come first and foremost in conception and design. Szyperski doesn't get around to talking about frameworks until chapter 21 of his Component Software book for heaven's sake.

Even physical components are like this. The prototypical component, the IC chip, always was designed within a family of chips that were meant to work together. They all needed the same voltage levels for zeroes and ones and tri-states, the same amperage levels, the same clock rates, etc, etc. Other families used other voltage levels. The first reusable, interchangeable parts in history were for rifles. They were meant to be easy and quick to replace (as opposed to the hand crafted muskets they were replacing) but they were meant specifically to make rifles!

Rummaging around a garage, you could find all sorts of "widgets" and "gizmos" that you might guess are components of something, but unless you know what framework they were meant to be a part of, they are not good for anything but door stops or paperweights. In other words, random components don't tend to fit together or work together.

Too many people are trying to make "universal" components without realizing that those components still work within some framework that allows them to be put together and communicate with each other. The problem is that other people doing the same thing have defined other "generic" frameworks that are none the less incompatible.

For example, the toys that baby boomers played with when they were young abounded with generic frameworks of universal components: Tinker Toys, Lincoln Logs, Erector Sets, LEGOs. They all had universal components within a generic framework that let you build anything. BUT, you couldn't mix Tinker Toy parts with Erector Set parts (without glue or duct tape).

Ah, you say. That's why I like duct tape, weakly typed, languages like Perl that lets me glue together parts. Also, what about Play-doh?! You could stick anything together with that! Yes, but there was a reason you made bridges out of Erector Sets instead of Play-doh, and the same reasons apply to software systems (but Strong versus Weak typing is another discussion).

Objects versus Components

Until I had this epiphany about components as donut holes, I didn't have a good answer to the question "what's the difference between an object and a component?". I now understand that all objects ARE components, but not all components are objects. The framework that defines a set of components does not have to be an object oriented framework. But all object oriented languages define an object framework. They are generic enough frameworks that any objects programmed in that language may interoperate with each other. Unfortunately though, as with Tinker Toys and Lincoln Logs, Java objects typically can't interact with Smalltalk objects.

In the Java language there are at least two levels of object framework. There are plain old Java objects (POJOs) and there are so-called JavaBeans. Whereas any property of a POJO can be accessed (assuming its not protected by the "private" keyword) via a fooObject.barProperty syntax, only special properties may be accessed via the JavaBeans framework. JavaBeans are those objects that have defined special property accessor methods of the form: getBarProperty() and setBarProperty(). "JavaBean" is the name given to any component that works within that specialized framework. To make matters confusing however, it turns out that Javasoft called more than one framework "JavaBeans" (arrgh!). There are even more specialized versions of JavaBeans that are made to work with fancy GUI toolkits.  SO, WITHOUT CLEARLY FOCUSING ON FRAMEWORKS, EVEN JAVASOFT CONFUSES DIFFERENT COMPONENT TYPES WITH EACH OTHER!

The moral? Don't fret that there is no such thing as a truly "universal" component. Don't spend energy trying to build them, or building "single universal" frameworks. Focus on what is needed for your situation and design a well crafted framework first and foremost. If it needs to work with other frameworks (like whatever Microsoft builds that won't integrate with anybody else), understand that Frameworks need bridges to each other rather than simplistic component socket adapters.