Hypothesis-based thinking

"The first principle is that you must not fool yourself – and you are the easiest person to fool."

- Richard Feynman

As far back as elementary school, I remember being taught this concept of the "scientific method", which seemed like a pointless codification of what was basically the common sense process of trial-and-error. We were told that it was this single method that separated the scientific modern age from the superstitious past (maybe exaggerating a bit), which struck me as a bit odd, since the ability of the human mind to apply logical reasoning was obviously not suddenly invented by Francis Bacon in 1620. There is probably more to the philosophy of the scientific method than what is taught to primary and secondary school students, but by the same token they are only formally exposed to the "dumbed-down" version. Whether teaching the more hardcore version would make the average man more rational is debatable, but more importantly for us, there may be a lesson to be learned by taking a step back to see the rationale for having a formal scientific method in the first place.

As you may know, the first step in the scientific method is to in fact accept that there is no right model of reality (though there are wrong, i.e. inconsistent ones), only some that are 1) simpler than others and 2) more compatible with other models concerning other parts of reality than others, and so some models are "more right" in a sense. The problem arises when there are two or models that seem roughly equal in merit when it comes to 1) and 2); how can you know which model is "more right" in that case?

The obvious answer is to perform an experiment that tests an edge case where the competing models predict different results. But experiments can be biased, not just because those who perform them can be biased, but the design of the experiment itself can be structured, even unintentionally, to favor one model over another. The solution to this conundrum turns out to be the concept of the hypothesis -  a concept unfortunately misunderstood as the dreadful boilerplate you had to write in the beginning of your physics lab report.

The actual point of the hypothesis is that the experiment should then be structured in a way not just to possibly prove the hypothesis, but also to disprove it. It must be possible to look at the results of the experiment, consider whether they support the hypothesis, but also consider whether they support the negation of the hypothesis. This nuance is important because it is conceivable that, after performing an experiment, you find that the results weakly support the hypothesis, but upon reversing your initial assumption, you find that the results strongly support the negative hypothesis. If you neglected to consider the negative hypothesis, you might draw an overall conclusion of weak support for the hypothesis, when that was not the correct conclusion to make at all.

More relevantly to a layman like myself, this kind of hypothesis-based reasoning is important not just in formal scientific studies. This video by Julia Galef is about a more complicated topic known as "Bayesian methods" but its gist is more general and contains an important point: a major reason why people fail to be rational in their everyday thinking is that when faced with new information, people tend to try to fit it into their existing belief framework, and if the new information conflicts with their beliefs, they tend to reject the new information, when instead they should also consider the possibility that they need to change their beliefs to accommodate the new information. Or in Galef's words, it important not just to estimate the likelihood of new information assuming their existing beliefs are true, but to estimate the likelihood assuming they are false.

The connection to our earlier discussion about the need for hypotheses appears if we replace "existing belief system" with "hypothesis" and "new information" with "experimental results"; another way of explaining the usefulness of hypothesis, then, is that by affirming a hypothesis at the outset, you are openly declaring your "belief system"/hypothesis, and trying to judge whether 1) the experimental results can be accommodated into the hypothesis or 2) the hypothesis needs to be negated for the results to be accommodated. So in an informal sense, the point of having hypotheses is to enforce a kind of humility on the person judging any kind of evidence, making sure that they consider not only the possibility that the evidence is wrong, but the possibility that their assumed model of reality was wrong.

This understanding of hypothesis-based thinking might be useful for your next debate with a hollow-earther, or more importantly, might prove helpful in making sure you do not make the same mistakes as one.

On Understanding

What does it mean to understand something?

Here is a naive proposal for a definition: you understand a concept if you can model it in any situation. That is, viewing every situation as a set of inputs to a system, you can predict the state of the system after it responds to those inputs. This is a good start, but can we come up with any situations where this definition does not hold up to our intuitive idea?

Consider the digital computer. At its core, a computer is really a device that reads a machine code instruction from memory, interprets the instruction, executes the instruction by making changes to registers and RAM, and then moves on to the next instruction. Undergraduates in computer science and engineering are taught to not just know about this model but to actually emulate the process; given the state of the CPU and RAM, the student has to know how to do the same thing that the computer does, painstakingly reading the code line by line and manually handling changes to memory (presumably it is less painstaking for the computer). It is probably fair to say, then, that the student has learned to model a computer, in some sense. But can we say that the student understands computers?

The obvious answer is that no, the student might not understand everything about computers well enough to say that they understand computers, but they certainly understand something about computers, and a fairly fundamental fact of their operation at that. But what else, then, would you have to learn in order to say that you understand computers in their entirety?

Staying with our example, there are a few possible directions to go in. You could learn about the hardware of the computer: all the circuits down to the level of logic gates, further down to the level of transistors, and maybe even down to the level of particle physics. Or you could learn about software, about the OS and applications that run on the computer, about the programming languages used to write those programs, and so on. But what do all these have in common, that ties them together as the concept of "computer"?

Let us introduce the concept of a level of abstraction, a somewhat arbitrary subdivision of the thing we wish to understand. For understanding how to execute machine instructions like a CPU, it might not be necessary to understand the physics of a semiconductor junction, so those details could be omitted from our model, and those that remain make up the level of abstraction. The model of a CPU as something that reads and executes instructions might be deemed the CPU level of abstraction, the model of logic gates and transistors the circuit level of abstraction, and so on. Perhaps, then, we can refine our definition of understanding as follows: you understand something if you can model it at every level of abstraction, using the most appropriate (i.e. elegant) model for each of those levels of abstraction.

But we run into an immediate problem: Isn't the definition of what a particular level of abstraction is completely arbitrary? We have defined a "physics level", "circuit level", and "CPU level" but those are man-made categories, and we could have divided them up some other way. In fact, nothing is stopping us from modeling one level of abstraction with the models we developed for another. We could model software at the machine code level perfectly fine in the sense that we could replicate outputs from inputs, or we could model the operation of the computer at the level of transistors if we wanted with some horrendously complicated differential equation that would probably take the lifetime of the universe to solve. Hell, we might as well say everything is fundamentally physics at the bottom, and posit that whoever understands quantum electrodynamics can model the computer at any level of abstraction, and since quantum electrodynamics is more fundamental in a sense, they understand everything there is to know about computer.

Of course, this is at odds with our intuitive concept of what it means to understand something. Someone might be a master of modeling the computer as a bunch of interactions between quantum-level wavefunctions, but be unable to write a single line of Python code. So how we write our definition to resolve this conundrum, between finding the most elegant model for something and satisfying all the possible levels of analysis?

Here is a crazy idea: our confusion about this definition of understanding things is not a difficulty of defining understanding, but a difficulty of defining things. From one point of view, the software on a computer is not actually there; it is just binary data we think of in a certain way to make it easier to interact with the computer. Strictly speaking, we do not need to know anything about software to build a computer out of circuits and run some code that someone else gave us. It is at least somewhat reasonable to say that we know everything to know about "computers" because writing the code does not fall under our job description, but rather some other field called "programming". Another, more realistically debatable example is to define the cut-off point between "circuit design" and "physics"; my guess is that there is considerable overlap, and the division is quite arbitrary. More generally speaking, it is not just the subdivision of concepts into levels of abstraction that is arbitrary, but the composition of sub-concepts into larger concepts as well, all the way up to reality as a whole. Saying that you understand how a computer works is really saying that you are capable of modeling a somewhat arbitrary level of abstraction of reality that you call "the computer", which you have yourself subdivided into arbitrary levels of abstraction you call "the CPU", "software", and so on, and you can model each of those levels with the most appropriate model.

So the problem with our definition of understanding something is that it depends on our definition of something. It seems that this should imply that our difficulties resulted from attempts to divide up reality into nice sub-concepts or levels of abstraction, but this was in fact a mistake. On the other hand, it seems like these divisions should not be completely arbitrary: the division of the computer into the customary hardware/software levels of analysis is for good reason, is it not? Or perhaps the levels of abstraction we made are simply an accident of how we first decided to approach problems, and if history had turned out slightly differently, we could be talking about dry/wetware or hot/coldware instead of hard/software. But that's a headache for another time.

A Flat Earth Crisis

A while ago I had the opportunity to have an earnest conversation with a flat-earther. At the end of the conversation I was psychologically shaken; not because what I knew had really changed, but because what I knew about how I could know anything at all had been seriously challenged.

After a great deal of questioning, it became clear that the flat-earther's view of the world was not at all a sign of stupidity; his model of the world was in fact entirely consistent. He had rebuttals to the common counter-arguments against the flat earth theory: the photographs taken of Earth from space were faked, NASA was compromised and trying to hide the truth, and so on. On the other hand, I had no such responses to his evidence, and it is unlikely I would be able to come up with any. After all, his evidence was the intuitive conclusion one would make from daily experience: that the Earth's surface is flat, and that the sky is a dome.

But surely, you say, I could simply point out that the round Earth theory is the one compatible with scientific theories of the universe, the ones people use for everything from GPS satellites to calculating ballistic trajectories. The problem is that these theories are still only as true as the results of experimental testing - experiments that I have never done personally. In fact, the flat-earther claimed to have performed a balloon experiment himself, where a weather balloon was released into the atmosphere, and came down not far from where it was released. On the other hand, I have done no such experiment, and instead put my faith in the theory that has been fed to me in school and in the media, without ever having doubted the story. Is it not the case that, in this situation, I am the blind follower, while he is the skeptic?

Okay, but surely, you say, the faith in the theory given to me by society is justified, because the theory also explains a great deal many other things and essentially underpins the modern world, with all its technology. This makes sense, but once again how can I be really sure that the theory is what everything is based on? I could try to re-create the theory, starting from the basics and making sure I understand every step of the derivation until I have reached the conclusion that I believed in the first place - but this would take a huge amount of time and effort I will probably never expend. And because I won't ever do that, there is always room for doubt, and thus my belief that the world is round is based at least partly on a rough intuition that the model is probably right - in other words, based on blind faith.

So then do I now believe that the earth is flat? Well, no; but the realization I did make is arguably more important: the realization that everything you believe is based on faith when you dig deep enough. In a world where the products of science are ubiquitous and so many things seem to be explained perfectly, it is a surprising lesson, but one that should lead you to a deeper understanding of science, not to reject it.