Because of the topic of testing coming up in my recent post "Is Morality Eating Your Own Dogfood?", it made me finally publish the following notebook entry from 2007...
In reading Philosophy 101, about Truth with a capital "T", and the non-traditional logics that use new notions of truth, we of course arrive at Fuzzy Logic with its departure from simple binary true/false values, and embrace of an arbitrarily wide range of values in between.
Contemplating this gave me a small AHA moment: Unit Testing is an area where there is an implicit assumption that "Test Passes" has either a true or false value. How about Fuzzy Unit Testing where there is some numeric value in the 0...1 range which reports a degree of pass/fail-ness? i.e. a percentage pass/fail for each test. For example, testing algorithms that predict something could be given a percentage pass/fail based on how well the prediction matched the actual value. Stock market predictions, bank customer credit default prediction, etc come to mind. This sort of testing of predictions about future defaults (i.e. credit grades) is just the sort of thing that the BASEL II accords are forcing banks to start doing.
Another great idea (if I do say so myself) that I had a few years ago was the notion that there is extra meta-data that could/should be gathered as a part of running unit test suites; specifically, the performance characteristics of each test run. The fact that a test still passes, but is 10 times slower than the previous test run, is a very important piece of information that we don't usually get. Archiving and reporting on this meta-data about each test run can give very interesting metrics on how the code changes are improving/degrading performance on various application features/behavior over time. I can now see that this comparative performance data would be a form of fuzzy testing.
Showing posts with label test driven. Show all posts
Showing posts with label test driven. Show all posts
Tuesday, September 21, 2010
Sunday, June 6, 2010
Fuzzy Unit Testing, Performance Unit Testing
Because of the topic of testing coming up in my recent post "Is Morality Eating Your Own Dogfood?", it made me finally publish the following notebook entry from 2007...
In reading Philosophy 101, about Truth with a capital "T", and the non-traditional logics that use new notions of truth, we of course arrive at Fuzzy Logic with its departure from simple binary true/false values, and embrace of an arbitrarily wide range of values in between.
Contemplating this gave me a small AHA moment: Unit Testing is an area where there is an implicit assumption that "Test Passes" has either a true or false value. How about Fuzzy Unit Testing where there is some numeric value in the 0...1 range which reports a degree of pass/fail-ness? i.e. a percentage pass/fail for each test. For example, testing algorithms that predict something could be given a percentage pass/fail based on how well the prediction matched the actual value. Stock market predictions, bank customer credit default prediction, etc come to mind. This sort of testing of predictions about future defaults (i.e. credit grades) is just the sort of thing that the BASEL II accords are forcing banks to start doing.
Another great idea (if I do say so myself) that I had a few years ago was the notion that there is extra meta-data that could/should be gathered as a part of running unit test suites; specifically, the performance characteristics of each test run. The fact that a test still passes, but is 10 times slower than the previous test run, is a very important piece of information that we don't usually get. Archiving and reporting on this meta-data about each test run can give very interesting metrics on how the code changes are improving/degrading performance on various application features/behavior over time. I can now see that this comparative performance data would be a form of fuzzy testing.
In reading Philosophy 101, about Truth with a capital "T", and the non-traditional logics that use new notions of truth, we of course arrive at Fuzzy Logic with its departure from simple binary true/false values, and embrace of an arbitrarily wide range of values in between.
Contemplating this gave me a small AHA moment: Unit Testing is an area where there is an implicit assumption that "Test Passes" has either a true or false value. How about Fuzzy Unit Testing where there is some numeric value in the 0...1 range which reports a degree of pass/fail-ness? i.e. a percentage pass/fail for each test. For example, testing algorithms that predict something could be given a percentage pass/fail based on how well the prediction matched the actual value. Stock market predictions, bank customer credit default prediction, etc come to mind. This sort of testing of predictions about future defaults (i.e. credit grades) is just the sort of thing that the BASEL II accords are forcing banks to start doing.
Another great idea (if I do say so myself) that I had a few years ago was the notion that there is extra meta-data that could/should be gathered as a part of running unit test suites; specifically, the performance characteristics of each test run. The fact that a test still passes, but is 10 times slower than the previous test run, is a very important piece of information that we don't usually get. Archiving and reporting on this meta-data about each test run can give very interesting metrics on how the code changes are improving/degrading performance on various application features/behavior over time. I can now see that this comparative performance data would be a form of fuzzy testing.
Sunday, May 16, 2010
Is Morality Eating Your Own Dogfood?
There are two schools of thought about whether programmers should have to write tests to verify their own code (in addition to writing the code itself). The philosophy of economics, and psychology, and morality, all overlap in studies that show how people will readily abandon moral responsibilities if they are given ways to avoid the stigma of doing so. This leads me to feel more justified in my belief that programmers do a poorer job of reading, understanding, and implementing a specification when someone else has the responsibility of verification.
Changing the rules changes people’s attitudes
There is a 1998 experiment[1] that keeps popping up in the new “freakonomics
”-type literature[2][3] (e.g. Economics 2.0
[4]), where a controlled subset of Israeli day-care centers started charging a fine for parents who came late to pick up their children. To everyone’s surprise, the number of people showing up late almost doubled. Additionally, when the fine was later dropped, the number of late parents stayed at the high level. It is theorized that the moral responsibility parents felt to be on time was much stronger than the economic cost of paying a fine, which was rationalized by the parents as a fee, thus removing the stigma of being late. The fee made it “just business”. As Professor Michael Sandel summarized[5]…
The blind experiment is a well established doctrine in science requiring that people should not know too much about something they are testing, otherwise the results are often biased. Scientists gathering their own raw data (not to mention interpreting their own data) often get the results they expected to get, where objective outsiders don’t. With that theory, it is argued that software development projects should engage external, objective, “QA testers” to develop and administer test suites against the code produced by the “programmers”. Since many programmers don’t like to eat their spinach, ahem, write their own tests (or documentation for that matter), there are not usually arguments against the idea.
From my experience though…
I say, both independent testers AND the original programmers should each write independent test suites. This way you get the power of both perspectives. This is an old lesson from the days of computer punch-cards; two different people key-punch the same data so that they can later be compared, thus eliminating most typos.
[1] A Fine is a Price, URI GNEEZY and ALDO RUSTICHINI, Journal of Legal Studies, vol. XXIX ( January 2000)
http://rady.ucsd.edu/faculty/directory/gneezy/docs/fine.pdf
[2] Brain food: when does a fine become a fee?, Aditya Chakrabortty, The Guardian, Tuesday 23 February 2010
http://www.guardian.co.uk/science/2010/feb/23/brain-food-fines-and-fees
[3] Why an L.A. Times wikitorial effort went wrong, Clay Shirky, O'Reilly Media Gov 2.0 Summit, 2009-09-09
http://itc.conversationsnetwork.org/shows/detail4411.html
[4] pg 7, Economics 2.0, Norbert Haring, Olaf Storbeck, Palgrave Macmillan, 2009
[5] Michael Sandel, The Reith Lectures 2009, BBC, 9th JUNE 2009
http://www.bbc.co.uk/programmes/b00kt7rg
Changing the rules changes people’s attitudes
There is a 1998 experiment[1] that keeps popping up in the new “freakonomics
”-type literature[2][3] (e.g. Economics 2.0 [4]), where a controlled subset of Israeli day-care centers started charging a fine for parents who came late to pick up their children. To everyone’s surprise, the number of people showing up late almost doubled. Additionally, when the fine was later dropped, the number of late parents stayed at the high level. It is theorized that the moral responsibility parents felt to be on time was much stronger than the economic cost of paying a fine, which was rationalized by the parents as a fee, thus removing the stigma of being late. The fee made it “just business”. As Professor Michael Sandel summarized[5]…“So what happened? Introducing the fine changed the norms. Before, parents who came late felt guilty; they were imposing an inconvenience on the teachers. Now parents considered a late arrival a service for which they were willing to pay. Rather than imposing on the teacher, they were simply paying her to stay longer. Part of the problem here is that the parents treated the fine as a fee. It’s worth pondering the distinction. Fines register moral disapproval, whereas fees are simply prices that imply no moral judgement.”Don’t make “not my job” “just business”
The blind experiment is a well established doctrine in science requiring that people should not know too much about something they are testing, otherwise the results are often biased. Scientists gathering their own raw data (not to mention interpreting their own data) often get the results they expected to get, where objective outsiders don’t. With that theory, it is argued that software development projects should engage external, objective, “QA testers” to develop and administer test suites against the code produced by the “programmers”. Since many programmers don’t like to eat their spinach, ahem, write their own tests (or documentation for that matter), there are not usually arguments against the idea.
From my experience though…
- Programmers will suffer peer pressure and social costs if they fail their own tests (which is a good thing).
- Programmers who understand that they are obligated to deliver testable components, will do so more often if they must actually produce the tests themselves, compared to those where testing is “not my job”.
- The act of writing a test forces a clearer understanding of both the interface and the implementation of the tested component compared to just programming it.
- Programmers who fail their own tests will be much more likely to change that component’s implementation if needed, rather than obstinately maintaining that the externally-produced test is wrong.
- Writing your own tests is the most systematic method of “eating your own dogfood”
I say, both independent testers AND the original programmers should each write independent test suites. This way you get the power of both perspectives. This is an old lesson from the days of computer punch-cards; two different people key-punch the same data so that they can later be compared, thus eliminating most typos.
[1] A Fine is a Price, URI GNEEZY and ALDO RUSTICHINI, Journal of Legal Studies, vol. XXIX ( January 2000)
http://rady.ucsd.edu/faculty/directory/gneezy/docs/fine.pdf
[2] Brain food: when does a fine become a fee?, Aditya Chakrabortty, The Guardian, Tuesday 23 February 2010
http://www.guardian.co.uk/science/2010/feb/23/brain-food-fines-and-fees
[3] Why an L.A. Times wikitorial effort went wrong, Clay Shirky, O'Reilly Media Gov 2.0 Summit, 2009-09-09
http://itc.conversationsnetwork.org/shows/detail4411.html
[4] pg 7, Economics 2.0, Norbert Haring, Olaf Storbeck, Palgrave Macmillan, 2009
[5] Michael Sandel, The Reith Lectures 2009, BBC, 9th JUNE 2009
http://www.bbc.co.uk/programmes/b00kt7rg
Thursday, March 20, 2008
The Logical Positivists were Test-Infected
(Is that a good thing?)
While reading Looking at Philosophy
[1], I came across the section about Logical Positivists[2] whom I immediately recognized as "test infected"[3]! The Logical Positivists (circa 1910s-1930s) promoted the idea British philosopher, A. J. Ayer, named "the principle of verification"[8][9] i.e. "the meaning of a proposition is its method of verification". In other words, a statement is meaningless if it can't be objectively tested. This is the same big idea of Test Driven Development[5] and more specifically Test Driven Specifications[6] (aka Test Driven Requirements[7]); namely, don't create a requirement (or interface definition) that is so vague, subjective, or contradictory that an automated computer program can't be written to test for compliance.
[1], I came across the section about Logical Positivists[2] whom I immediately recognized as "test infected"[3]! The Logical Positivists (circa 1910s-1930s) promoted the idea British philosopher, A. J. Ayer, named "the principle of verification"[8][9] i.e. "the meaning of a proposition is its method of verification". In other words, a statement is meaningless if it can't be objectively tested. This is the same big idea of Test Driven Development[5] and more specifically Test Driven Specifications[6] (aka Test Driven Requirements[7]); namely, don't create a requirement (or interface definition) that is so vague, subjective, or contradictory that an automated computer program can't be written to test for compliance.In other words, all that verbage[12] that usually passes for a specification/requirements document is really just commentary to the REAL specification which is encoded in a comprehensive compliance test suite. [BTW, this isn't just a software concept since IC chips have long had not only "testbed" circuit boards to test chips, but also ICE (in circuit emulators) to test circuit boards via simulated chips (which software copied with "mock objects").]
The goal of both Logical Positivists and test-driven engineers is to weed out statements that are so poorly conceived and worded as to be effectively meaningless. The way to ensure this is to submit each statement to this rigorous method and rework any statements that come up short.
What did Logical Positivists have to say about "how" to do tests?
My whole excitement about finding philosophy movements that parallel aspects of software engineering is the notion that "top men"[11] have already thought and argued long & hard about this stuff, and therefore we programmers can benefit from what they've already learned the hard way. So, what method did they advocate? Basically, it is the idea that all statements should be broken down into "protocol sentences" plus logical conclusions built on top of them. Protocol Sentences were to be simply observation reports (based on first hand direct sensory experience). This was ultimately unworkable for humans, but everything in a computer test is based on its first hand direct sensory experience (of electrical signals anyway).
Karl Popper, among the most influential philosophers of science of the 20th century, said that claims could only be considered "scientific" if they were falsifiable, meaning that there should be some observation or experiment specified that, if it were verified, would prove the claim false.
Logical Positivists claimed that unverifiable statements were LITERALLY meaningless, and therefore they dismissed entire disciplines like metaphysics, morality, and ethics. Other, more moderate philosophers held that untestable propositions were merely unproductive to work with. Popper claimed that his demand for falsifiability was not meant as a theory of meaning, but rather as a methodological norm for the sciences.
Uh oh, Logical Positivism was considered a failure...Will Test Driven be too?!
I was all happy about a philosophical foundation to being test driven (since I was test infected years ago), until I got to the part where Logical Positivism has been so devalued by other philosophers that one wrote "Logical positivism is one of the very few philosophical positions which can be easily shown to be dead wrong, and that is its principle claim to fame."[10]
OUCH!
OUCH!By saying that so-called metaphysical claims were meaningless, they unwittingly said that the very claim "metaphysical claims are meaningless" was itself meaningless! I.E. there was no way to prove the statement that "only provable statements had meaning".
Lessons for the Test Driven Approach
Luckily, Logical Positivism was only an extreme position in the spectrum of ideas under the Verificationism[4] umbrella. In the conclusion of "Verificationism: Its History and Prospects
"[8], some points were noted...
"[8], some points were noted...- one can't reduce every statement to be a logically equivalent statement about sensory experiences. (i.e. not every thought comes under the umbrella of "natural science")
- one can say that "a statement lacks legitimacy or objectivity if there would be no evidence for or against it; if it is insulated from reason, where reason is linked to the possibility of public evidence for or against the statement from other, already established statements"
- one must admit that some more abstract statements are understandable even if not testable.
So, the bottom line seems to be that the Test Driven approach is a good one if you...
- concentrate on testing what can be tested,
- create tests that would actively prove statements to be false, and
- know that you probably can't prove (with tests) that the test driven approach itself is correct.
[1] Looking at Philosophy, Donald Palmer, 4th Ed. 2005
[2] ibid, pg 327
[6] Acceptance Tests and the Test Driven Specification
[7] Test Driven Development or Test Driven Requirements?
[8] Verificationism: Its History and Prospects, C. J. Misak, 1995
[9] Language, Truth and Logic, Alfred Jules Ayer, 1952
[10] Prolegomena to Philosophy, Jon Wheatley, 1970
[11] "top men", Raiders of the Lost Arc
[12] verbage vs verbiage
Friday, August 31, 2007
Fuzzy Unit Testing, Performance Unit Testing
In reading Philosophy 101, about Truth with a capital "T", and the non-traditional logics that use new notions of truth, we of course arrive at Fuzzy Logic with its departure from simple binary true/false values, and embrace of an arbitrarily wide range of values in between.
Contemplating this gave me a small AHA moment: Unit Testing is an area where there is an implicit assumption that "Test Passes" has either a true or false value. How about Fuzzy Unit Testing where there is some numeric value in the 0...1 range which reports a degree of pass/fail-ness? i.e. a percentage pass/fail for each test. For example, testing algorithms that predict something could be given a percentage pass/fail based on how well the prediction matched the actual value. Stock market predictions, bank customer credit default prediction, etc come to mind. This sort of testing of predictions about future defaults (i.e. credit grades) is just the sort of thing that the BASEL II accords are forcing banks to start doing.
Another great idea (if I do say so myself) that I had a few years ago was the notion that there is extra meta-data that could/should be gathered as a part of running unit test suites; specifically, the performance characteristics of each test run. The fact that a test still passes, but is 10 times slower than the previous test run, is a very important piece of information that we don't usually get. Archiving and reporting on this meta-data about each test run can give very interesting metrics on how the code changes are improving/degrading performance on various application features/behavior over time. I can now see that this comparative performance data would be a form of fuzzy testing.
Contemplating this gave me a small AHA moment: Unit Testing is an area where there is an implicit assumption that "Test Passes" has either a true or false value. How about Fuzzy Unit Testing where there is some numeric value in the 0...1 range which reports a degree of pass/fail-ness? i.e. a percentage pass/fail for each test. For example, testing algorithms that predict something could be given a percentage pass/fail based on how well the prediction matched the actual value. Stock market predictions, bank customer credit default prediction, etc come to mind. This sort of testing of predictions about future defaults (i.e. credit grades) is just the sort of thing that the BASEL II accords are forcing banks to start doing.
Another great idea (if I do say so myself) that I had a few years ago was the notion that there is extra meta-data that could/should be gathered as a part of running unit test suites; specifically, the performance characteristics of each test run. The fact that a test still passes, but is 10 times slower than the previous test run, is a very important piece of information that we don't usually get. Archiving and reporting on this meta-data about each test run can give very interesting metrics on how the code changes are improving/degrading performance on various application features/behavior over time. I can now see that this comparative performance data would be a form of fuzzy testing.
Subscribe to:
Posts (Atom)
Posts
