Some further reading on experimental methods

For the reader eager to learn about the role of experimentation in general, I suggest the following literature:

1. Chalmers, A.F., What Is This Thing Called Science? The Open University Press, Buckingham, England, 1988. Addresses the philosophical underpinnings of the scientific process, including inductivism, Popper’s falsificationism, Kuhn’s paradigms, objectivism, and the theory dependence of observation.
2. Latour, B., Science in Action: How to Follow Scientists and Engineers through Society, Harvard University Press, Cambridge, Mass., 1987. Describes the social processes of science-in-the-making as opposed to ready-made science. Latour illustrates the fact-building and convincing power of laboratories with fascinating examples.
3. Basili, V.R., “The Role of Experimentation in Software Engineering: Past, Current, and Future.” Proc. 18th Int. Conf. Software Eng., IEEE Computer Soc. Press, Los Alamitos, Calif., March 1996.
4. Frankl, P.G., and S.N. Weiss, “An Experimental Comparison of the Effectiveness of Branch Testing and Data Flow Testing,” IEEE Trans. Software Eng., Aug. 1993, pp. 774-787.
5. Brett, B., “Comments on The Cost of Selective Recompilation and Environment Processing,” ACM Trans. Software Eng. and Methodology, 1995, pp. 214-216. A good example of a repeated experiment in compiling.
6. Denning, P.J., “Performance Evaluation: Experimental Computer Science at Its Best,” ACM Performance Evaluation
   Review, ACM Press, New York, 1981, pp. 106-109. Argues that performance evaluation is an excellent form of experimentation in computer science.
7. Hennessy, J.L., and D.A. Patterson, Computer Architecture: A Quantitative Approach, Morgan Kaufmann, San Mateo, Calif., 1990. A landmark in making computer architecture research quantitative.
8. Cohen, P.R., Empirical Methods for Artificial Intelligence, MIT Press, Cambridge, Mass., 1995. Covers empirical methods in AI, but a large part applies to all of computer science.
9. Fenton, N.E., and S.L. Pfleeger. Software Metrics: A Rigorous and Practical Approach (2nd edition), Thomson Computer Press, New York, 1997. Excellent discussion of experimental designs as well as a wealth
   of material on experimentation with software.
10. Christensen, L.B., Experimental Methodology, Allyn and Bacon, New York, 1994.
11. Judd, C.M., E.R. Smith, and L.H. Kidder, Research Methods in Social Relations, Holt, Rinehart, and Winston,
    1991. General experimental methods.
12. Moore, D.S., and G.P. McCabe, Introduction to the Practice of Statistics, W.H. Freeman and Co., New York, 1993. Excellent introductory text on statistics.
13. Venables, W.N. and B.D. Ripley, Modern Applied Statistics with S-PLUS, Springer Verlag, New York, 1997. One of the best statistical packages available today is SPlus. Venables and Ripley’s book is both a guide to using S-Plus and a course in modern statistical methods. Keep in mind, however, that sophisticated statistical analysis is no substitute for good experimental design.

From Tichy, Walter (1998). Should Computer Scientists Experiment More?