(notes in progress)

An experiment can be seen as a structured method for collecting evidence to test a hypothesis. The process of designing a good experiment involves reducing unknowns and establishing a clear relationship between the hypothesis and any expected results. is there a causal relationship that can be established between the initial conditions and the result? does the collected data actually support (or refute) the hypothesis? is there enough data to draw a conclusion? are there other factors effecting the outcome?

From Labcraft on experimentation:

The concept of experimentation in the “hard” sciences is widely understood to involve these steps: look at the evidence; propose a hypothesis that explains that evidence; create a trial that tests the ability of your hypothesis to confirm, predict, or explain the evidence; and use the results of your trial to refine your hypothesis. (…) We create hypotheses about how best to intervene in these situations to realize a desired social change, as well as hypotheses concerning the interactions of our solution with and within connected systems. We translate our hypotheses into prototypes for new or improved solutions to social challenges, often in the form of products, processes, policies, or services. We test those solutions through their application, often in the form of pilots or trials with users. And we use the results of our tests to iterate and to inform the creation of still-better solutions. And we develop our own strategies and programs through a trial-and-error process of experimenting and prototyping. -pp-89-90

From Labcraft on prototyping:

The prototype emerges as a central feature of our approaches. Whether it’s that new, real-time water quality testing device, that multi-stakeholder spatial planning policy, a new pricing model for electricity, or the SMS-based vaccine stock-out reporting tool, a prototype is our hypothesis made real. It’s vital to experimentation that we introduce some thing you can test—something real that can succeed or fail, that can go off the rails, that can have unintended outcomes, that can break! That test allows us to learn. (…) Create them quickly and cheaply to make your thinking tangible, get it into the hands of users and stakeholders to test it, and throw it out when you’ve extracted what you need to know in order to make a better version. Iteration is what we do with that learning: we take our lessons from trials and pilots and feedback loops built around our prototypes, consolidate them into a refined hypothesis, and build a new and improved version of that prototype. -pp91

From Labcraft on lessons learned about experimenting:

Know what you’re trying to discover. There’s a lot to be said for insights that emerge from pilots, and even more to be said about being open to being surprised. But our experience suggests that our efforts are best served when we define from the outset what we hope to learn from a pilot or trial. If it isn’t working, stop doing it. This may sound obvious, but continuing on with something when it’s clearly not working happens more often than you might think in almost every type of organization. One of the key aspects of rapid-cycle prototyping is that you simply stop doing something when you realize it’s not working, learn from that, and move on. Don’t take it personally. Labs take risks, so failure will happen. As much as we fetishize failure in social innovation, it can still hurt when it happens. Make sure the culture in your organization genuinely supports the notion that things won’t always work, and backs up the individuals who lead experiments. Be strict about learning. Experimentation isn’t a substitute for deeper learning. There’s no point in failing for the sake of it. It’s crucial that no matter how much you may want to forget a failed experiment, you reflect after every activity that went wrong on what went well, what didn’t, and what you’d do differently. -pp 96

Venkatesh Rao on Extraordinary Laboratories

Civilization is the world we build within these extraordinary laboratories. A contingent state of affairs that is only stable to the extent that the truths it is built on are sufficiently sequestered. Understood this way, all of civilization is one giant laboratory instrument, poking at the unknown.

On unstructured tinkering and statistical validity…

• https://en.wikipedia.org/wiki/Quasi-experiment
• https://en.wikipedia.org/wiki/Natural_experiment
• https://en.wikipedia.org/wiki/Design_of_experiments
• https://en.wikipedia.org/wiki/Concept_development_and_experimentation
• http://www.dummies.com/how-to/content/designing-experiments-using-the-scientific-method.html

Demonstration vs. experiment

• A demonstration is a way to explain a principle by an example. it provides evidence of the the principle under investigation.
• An experiment is a test performed to discover an unknown or to validate (or refute) a theoretical principle.

How can we frame a feasibility study, or pilot project as an experiment? Is it more, or less than an existence proof? How does it help?

Replicability […]

(to be continued)