I wrote this in Feb 2009 and emailed it to Yudkowsky. He didn't reply.

---

Dear Eliezer Yudkowsky,

I am writing to criticize some of your statements regarding Karl Popper. I hope this will be of interest.

http://yudkowsky.net/rational/bayes

Previously, the most popular philosophy of science was probably Karl Popper's falsificationism - this is the old philosophy that the Bayesian revolution is currently dethroning. Karl Popper's idea that theories can be definitely falsified, but never definitely confirmed, is yet another special case of the Bayesian rules

That isn't Popper's idea because he doesn't believe in definite falsifications. Falsifications are themselves tentative conjectures which must be held open to criticism and reconsidering.

Popper also doesn't assert that confirmations are never definite, rather he denies there is confirmation at all. The reason is that any given confirming evidence for theory T is logically consistent with T being false.

More generally, Popper's philosophy is not about what we can do definitely. He does not address himself to the traditional philosophical problem of what we can and can't be certain of, or what is and isn't a justified, true belief. While he did comment on those issues, his epistemic philosophy is not an alternative answer to those questions. Rather, his positive contributions focus on a more fruitful issue: conjectural knowledge. How do people acquire conjectural knowledge? What is its nature? And so on.

BTW, conjectural knowledge does not mean the probabilistic knowledge that Bayesians are fond of. Probabilistic knowledge is just as much of an anathema to Popper as certain knowledge, because the same criticisms (for example that attempting justification leads to regress or circularity) apply equally well to each.

Your claim at the end of the quote that Popperian epistemology is a special case of Bayesian epistemology is especially striking. Popper considered the Bayesian approach and told us where he stands on it. On page 141 of Objective Knowledge he states, "I have combated [Bayesian epistemology] for thirty-three years."

To say that something which Popper combatted for over three decades is a more general version of his own work is an extraordinary claim. It should be accompanied with extraordinary substantiation, and some account of where Popper's arguments on the subject go wrong, but it is not.

Popper was a hardworking, academic person who read and thought about philosophy extensively, including ideas he disagreed with. He would often try to present the best possible version of an idea, as well as a history of the problem in question, before offering his criticism of it. I would ask that a similar approach be taken in criticizing Popper. Both as a matter of respect, and because it improves discussion.

In my understanding, there’s no one who speaks for Less Wrong (LW), as its representative, and is responsible for addressing questions and criticisms. LW, as a school of thought, has no agents, no representatives – or at least none who are open to discussion.

The people I’ve found interested in discussion on the website and slack have diverse views which disagree with LW on various points. None claim LW is true. They all admit it has some weaknesses, some unanswered criticisms. They have their own personal views which aren’t written down, and which they don’t claim to be correct anyway.

This is problematic. Suppose I wrote some criticisms of the sequences, or some Bayesian book. Who will answer me? Who will fix the mistakes I point out, or canonically address my criticisms with counter-arguments? No one. This makes it hard to learn LW’s ideas in addition to making it hard to improve them.

My school of thought (Fallible Ideas – FI) has representatives and claims to be correct as far as is known (like LW, it’s fallibilist, so of course we may discover flaws and improve it in the future). It claims to be the best current knowledge, which is currently non-refuted, and has refutations of its rivals. There are other schools of thought which say the same thing – they actually think they’re right and have people who will address challenges. But LW just has individuals who individually chat about whatever interests them without there being any organized school of thought to engage with. No one is responsible for defining an LW school of thought and dealing with intellectual challenges.

So how is progress to be made? Suppose LW, vaguely defined as it may be, is mistaken on some major points. E.g. Karl Popper refuted induction. How will LW find out about its mistake and change? FI has a forum where its representatives take responsibility for seeing challenges addressed, and have done so continuously for over 20 years (as some representatives stopped being available, others stepped up).

Which challenges are addressed? All of them. You can’t just ignore a challenge because it could be correct. If you misjudge something and then ignore it, you will stay wrong. Silence doesn’t facilitate error correction. For information on this methodology, which I call Paths Forward. BTW if you want to take this challenge seriously, you’ll need to click the link; I don’t repeat all of it. In general, having much knowledge is incompatible with saying all of it (even on one topic) upfront in forum posts without using references.

My criticism of LW as a whole is that it lacks Paths Forward (and lacks some alternative of its own to fulfill the same purpose). In that context, my criticisms regarding specific points don’t really matter (or aren’t yet ready to be discussed) because there’s no mechanism for them to be rationally resolved.

One thing FI has done, which is part of Paths Forward, is it has surveyed and addressed other schools of thought. LW hasn’t done this comparably – LW has no answer to Critical Rationalism (CR). People who chat at LW have individually made some non-canonical arguments on the matter that LW doesn’t take responsibility for (and which often involve conceding LW is wrong on some points). And they have told me that CR has critics – true. But which criticism(s) of CR does LW claim are correct and take responsibility for the correctness of? (Taking responsibility for something involves doing some major rethinking if it’s refuted – addressing criticism of it and fixing your beliefs if you can’t. Which criticisms of CR would LW be shocked to discover are mistaken, and then be eager to reevaluate the whole matter?) There is no answer to this, and there’s no way for it to be answered because LW has no representatives who can speak for it and who are participating in discussion and who consider it their responsibility to see that issues like this are addressed. CR is well known, relevant, and makes some clear LW-contradicting claims like that induction doesn’t work, so if LW had representatives surveying and responding to rival ideas, they would have addressed CR.

BTW I’m not asking for all this stuff to be perfectly organized. I’m just asking for it to exist at all so that progress can be made.

Anecdotally, I’ve found substantial opposition to discussing/considering methodology from LW people so far. I think that’s a mistake because we use methods when discussing or doing other activities. I’ve also found substantial resistance to the use of references (including to my own material) – but why should I rewrite a new version of something that’s already written? Text is text and should be treated the same whether it was written in the past or today, and whether it was written by someone else or by me (either way, I’m taking responsibility. I think that’s something people don’t understand and they’re used to people throwing references around both vaguely and irresponsibly – but they haven’t pointed out any instance where I made that mistake). Ideas should be judged by the idea, not by attributes of the source (reference or non-reference).

The Paths Forward methodology is also what I think individuals should personally do – it works the same for a school of thought or an individual. Figure out what you think is true and take responsibility for it. For parts that are already written down, endorse that and take responsibility for it. If you use something to speak for you, then if it’s mistaken you are mistaken – you need to treat that the same as your own writing being refuted. For stuff that isn’t written down adequately by anyone (in your opinion), it’s your responsibility to write it (either from scratch or using existing material plus your commentary/improvements). This writing needs to be put in public and exposed to criticism, and the criticism needs to actually get addressed (not silently ignored) so there are good Paths Forward. I hoped to find a person using this method, or interested in it, at LW; so far I haven’t. Nor have I found someone who suggested a superior method (or even any alternative method to address the same issues) or pointed out a reason Paths Forward doesn’t work.

Some people I talked with at LW seem to still be developing as intellectuals. For lots of issues, they just haven’t thought about it yet. That’s totally understandable. However I was hoping to find some developed thought which could point out any mistakes in FI or change its mind. I’m seeking primarily peer discussion. (If anyone wants to learn from me, btw, they are welcome to come to my forum. It can also be used to criticize FI.) Some people also indicated they thought it’d be too much effort to learn about and address rival ideas like CR. But if no one has done that (so there’s no answer to CR they can endorse), then how do they know CR is mistaken? If CR is correct, it’s worth the effort to study! If CR is incorrect, someone better write that down in public (so CR people can learn about their errors and reform; and so perhaps they could improve CR to no longer be mistaken or point out errors in the criticism of CR.)

One of the issues related to this dispute is I believe we can always proceed with non-refuted ideas (there is a long answer for how this works, but I don’t know how to give a short answer that I expect LW people to understand – especially in the context of the currently-unresolved methodology dispute about Paths Forward). In contrast, LW people typically seem to accept mistakes as just something to put up with, rather than something to try to always fix. So I disagree with ignoring some known mistakes, whereas LW people seem to take it for granted that they’re mistaken in known ways. Part of the point of Paths Forward is not to be mistaken in known ways.

Paths Forward is a methodology for organizing schools of thought, ideas, discussion, etc, to allow for unbounded error correction (as opposed to typical things people do like putting bounds on discussions, with discussion of the bounds themselves being out of bounds). I believe the lack of Paths Forward at LW is preventing the resolution of other issues like about the correctness of induction, the right approach to AGI, and the solution to the fundamental problem of epistemology (how new knowledge can be created).

I emailed this to some MIRI people and others related to Less Wrong.

---

I believe I know some important things you don't, such as that induction is impossible, and that your approach to AGI is incorrect due to epistemological issues which were explained decades ago by Karl Popper. How do you propose to resolve that, if at all?

I think methodology for how to handle disagreements comes prior to the content of the disagreements. I have writing about my proposed methodology, Paths Forward, and about how Less Wrong doesn't work because of the lack of Paths Forward:

http://curi.us/1898-paths-forward-short-summary

http://curi.us/2064-less-wrong-lacks-representatives-and-paths-forward

Can anyone tell me that I'm mistaken about any of this? Do you have a criticism of Paths Forward? Will any of you take responsibility for doing Paths Forward?

Have any of you written a serious answer to Karl Popper (the philosopher who refuted induction – http://fallibleideas.com/books#popper )? That's important to address, not ignore, since if he's correct then lots of your research approaches are mistakes.

In general, if someone knows a mistake you're making, what are the mechanisms for telling you and having someone take responsibility for addressing the matter well and addressing followup points? Or if someone has comments/questions/criticism, what are the mechanisms available for getting those addressed? Preferably this should be done in public with permalinks at a venue which supports nested quoting. And whatever your answer to this, is it written down in public somewhere?

Do you have public writing detailing your ideas which anyone is taking responsibility for the correctness of? People at Less Wrong often say "read the sequences" but none of them take responsibility for addressing issues with the sequences, including answering questions or publishing fixes if there are problems. Nor do they want to address existing writing (e.g. by David Deutsch – http://fallibleideas.com/books#deutsch ) which contains arguments refuting major aspects of the sequences.

Your forum ( https://agentfoundations.org ) says it's topic-limited to AGI math, so it's not appropriate for discussing criticism of the philosophical assumptions behind your approach (which, if correct, imply the AGI math you're doing is a mistake). And it states ( https://agentfoundations.org/how-to-contribute ):

It’s important for us to keep the forum focused, though; there are other good places to talk about subjects that are more indirectly related to MIRI’s research, and the moderators here may close down discussions on subjects that aren’t a good fit for this forum.

But you do not link those other good places. Can you tell me any Paths-Forward-compatible other places to use, particularly ones where discussion could reasonably result in MIRI changing?

If you disagree with Paths Forward, will you say why? And do you have some alternative approach written in public?

Also, more broadly, whether you will address these issues or not, do you know of anyone that will?

If the answers to these matters are basically "no", then if you're mistaken, won't you stay that way, despite some better ideas being known and people being willing to tell you?

The (Popperian) Fallible Ideas philosophy community ( http://fallibleideas.com ) is set up to facilitate Paths Forward (here is our forum which does this http://fallibleideas.com/discussion-info ), and has knowledge of epistemology which implies you're making big mistakes. We address all known criticisms of our positions (which is achievable without using too much resources like time and attention, as Paths Forward explains); do you?

---

Update (Dec 2019):

One person from MIRI responded the day I sent out the letter (Nov 9, 2017). He didn't answer anything I asked, but I decided to add the quotes for better completeness and record keeping. Below are Rob Bensinger's 3 emails, quoted, and my replies. After that he stopped responding.

Hi, Elliot. My short answer is that I think Popper is wrong; inductive reasoning works just as well as deductive in principle, though in practice we often have to rely on heuristic approximations of ideal inductive and deductive reasoning. The traditional problems with in-principle inductive reasoning (e.g., infinite hypothesis spaces) are well-addressed by Solomonoff's theory of algorithmic probability (http://world.std.com/~rjs/tributes/rathmannerhutter.pdf).

Have you written a serious and reasonably complete answer to Popper, or do you know of one that you will endorse, take responsibility for (if it's mistaken, you were mistaken), and address questions/criticisms/etc regarding?

And where is the Path Forward if you're mistaken?

I feel comfortable endorsing Solomonoff induction and Garrabrant induction (https://intelligence.org/2016/09/12/new-paper-logical-induction/) as philosophically unproblematic demonstrations that inductive reasoning works well in principle.

So you're disagreeing with Popper, but without addressing his arguments. If you're mistaken, and your mistakes have already been explained decades ago, you'll stay mistaken. No Paths Forward. Right?

I've read Popper before, and I believe the SEP when it says that he considered infinite hypothesis spaces a major underlying problem for induction (if not the core problem):

Popper gave two formulations of the problem of induction; the first is the establishment of the truth of a theory by empirical evidence; the second, slightly weaker, is the justification of a preference for one theory over another as better supported by empirical evidence. Both of these he declared insoluble, on the grounds, roughly put, that scientific theories have infinite scope and no finite evidence can ever adjudicate among them (LSD, 253–254; Grattan-Guiness 2004).

My claim is that Solomonoff induction addresses this problem handily, and that it more generally provides a good formal framework for understanding how and why inductive reasoning works well in practice.

I think we're getting off-topic. Do you agree or disagree with Paths Forward? Why? Do you have alternative written procedures for having issues like this addressed? Do you have e.g. a forum which would be a good place for me to reply to what you've said?

If I point out that you're mistaken about Popper's arguments and how they may be addressed, what happens next? BTW this would be much easier if there was a direct written answer to Popper by you, or by anyone else, that you were willing to take responsibility for. Why isn't there? That would also save effort for both of us – because responding to your unwritten views will require back-and-forth emails where I ask questions to find out what they are and get clarifications on what you're actually claiming. If your reasoning is that Popper is mistaken, so you don't want to bother properly answering him ... then your fallible criticism of Popper isn't itself being exposed to error correction very well.

Gyrodiot wrote at the Less Wrong Slack Philosophy chatroom:

I was waiting for an appropriate moment to discuss epistemology. I think I understood something about curi's reasoning about induction After reading a good chunk of the FI website. Basically, it starts from this:

He quotes from: http://fallibleideas.com/objective-truth

There is an objective truth. It's one truth that's the same for all people. This is the common sense view. It means there is one answer per question.

The definition of truth here is not the same as The Simple Truth as described in LW. Here, the important part is:

Relativism provides an argument that the context is important, but no argument that the truth can change if we keep the context constant.

If you fixate the context around a statement, then the statement ought to have an objective truth value

Yeah. (The Simple Truth essay link.)

In LW terms that's equivalent to "reality has states and you don't change the territory by thinking differently about the map"

Yeah.

From that, FI posits the existence of universal truths that aren't dependent on context, like the laws of physics.

More broadly, many ideas apply to many contexts (even without being universal). This is very important. DD calls this "reach" in BoI (how many contexts does an idea reach to?), I sometimes go with "generality" or "broader applicability".

The ability for the same knowledge to solve multiple problems is crucial to our ability to deal with the world, and for helping with objectivity, and for some other things. It's what enabled humans to even exist – biological evolution created knowledge to solve some problems related to survival and mating, and that knowledge had reach which lets us be intelligent, do philosophy, build skyscrapers, etc. Even animals like cats couldn't exist, like they do today, without reach – they have things like behavioral algorithms which work well in more than one situation, rather than having to specify different behavior for every single situation.

The problem with induction, with this view is that you're taking truths about some contexts to apply them to other contexts and derive truths about them, which is complete nonsense when you put it like that

Some truths do apply to multiple contexts. But some don't. You shouldn't just assume they do – you need to critically consider the matter (which isn't induction).

From a Bayesian perspective you're just computing probabilities, updating your map, you're not trying to attain perfect truth

Infinitely many patterns both do and don't apply to other contexts (such as patterns that worked in some past time range applying tomorrow). So you can't just generalize patterns to the future (or to other contexts more generally) and expect that to work, ala induction. You have to think about which patterns to pay attention to and care about, and which of those patterns will hold in what ranges of contexts, and why, and use critical arguments to improve your understanding of all this.

We do [live in our own map], which is why this mode of thought with absolute truth isn't practical at all

Can you give an example of some practical situation you don't understand how to address with FI thinking, and I'll tell you how or concede? And after we go through a few examples, perhaps you'll better understand how it works and agree with me.

So, if induction is out of the way, the other means to know truth may be by deduction, building on truth we know to create more. Except that leads to infinite regress, because you need a foundation

CR's view is induction is not replaced with more deduction. It's replaced with evolution – guesses and criticism.

So the best we can do is generate new ideas, and put them through empirical test, removing what is false as it gets contradicted

And we can use non-empirical criticism.

But contradicted by what? Universal truths! The thing is, universal truths are used as a tool to test what is true or false in any context since they don't depend on context

Not just contradicted by universal truths, but contradicted by any of our knowledge (lots of which has some significant but non-universal reach). If an idea contradicts some of our knowledge, it should say why that knowledge is mistaken – there's a challenge there. See also my "library of criticism" concept in Yes or No Philosophy (discussed below) which, in short, says that we build up a set of known criticisms that have some multi-context applicability, and then whenever we try to invent a new idea we should check it against this existing library of known criticisms. It needs to either not be contradicted by any of the criticisms or include a counter-argument.

But they are so general that you can't generate new idea from them easily

The LW view would completely disagree with that: laws of physics are statements like every other, they are solid because they map to observation and have predictive power

CR says to judge ideas by criticism. Failure to map to observation and lack of predictive power are types of criticism (absolutely not the only ones), which apply in some important range of contexts (not all contexts – some ideas are non-empirical).

Prediction is great and valuable but, despite being great, it's also overrated. See chapter 1 of The Fabric of Reality by David Deutsch and the discussion of the predictive oracle and instrumentalism.

http://www.daviddeutsch.org.uk/books/the-fabric-of-reality/excerpt/

Also you can use them to explain stuff (reductionism) and generate new ideas (bottom-up scientific research)

From FI:

When we consider a new idea, the main question should be: "Do you (or anyone else) see anything wrong with it? And do you (or anyone else) have a better idea?" If the answers are 'no'and 'no' then we can accept it as our best idea for now.

The problem is that by having a "pool of statements from which falsehoods are gradually removed" you also build a best candidate for truth. Which is not, at all, how the Bayesian view works.

FI suggests evolution is a reliable way to suggest new ideas. It ties well into the framework of "generate by increments and select by truth-value"

It also highlights how humans are universal knowledge machines, that anything (in particular, an AGI) created by a human would have knowledge than humans can attain too

Humans as universal knowledge creators is an idea of my colleague David Deutsch which is discussed in his book, The Beginning of Infinity (BoI).

http://beginningofinfinity.com

But that's not an operational definition : if an AGI creates knowledge much faster than any human, they won't ever catch up and the point is moot

Yes, AGI could be faster. But, given the universality argument, AGI's won't be more rational and won't be capable of modes of reasoning that humans can't do.

The value of faster is questionable. I think no humans currently maximally use their computational power. So adding more wouldn't necessarily help if people don't want to use it. And an AGI would be capable of all the same human flaws like irrationalities, anti-rational memes (see BoI), dumb emotions, being bored, being lazy, etc.

I think the primary cause of these flaws, in short, is authoritarian educational methods which try to teach the kid existing knowledge rather than facilitate error correction. I don't think an AGI would automatically be anything like a rational adult. It'd have to think about things and engage with existing knowledge traditions, and perhaps even educators. Thinking faster (but not better) won't save it from picking up lots of bad ideas just like new humans do.

That sums up the basics, I think The Paths Forwards thing is another matter... and it is very, very demanding

Yes, but I think it's basically what effective truth-seeking requires. I think most truth-seeking people do is not very effective, and the flaws can actually be pointed out as not meeting Paths Forward (PF) standards.

There's an objective truth about what it takes to make progress. And separate truths depending on how effectively you want to make progress. FI and PF talk about what it takes to make a lot of progress and be highly effective. You can fudge a lot of things and still, maybe, make some progress instead of going backwards.

If you just wanna make a few tiny contributions which are 80% likely to be false, maybe you don't need Paths Forward. And some progress gets made that way – a bunch of mediocre people do a bunch of small things, and the bulk of it is wrong, but they have some ability to detect errors so they end up figuring out which are the good ideas with enough accuracy to slowly inch forwards. But, meanwhile, I think a ton of progress comes from a few great (wo)men who have higher standards and better methods. (For more arguments about the importance of a few great men, I particularly recommend Objectivism. E.g. Roark discusses this in his courtroom speech at the end of The Fountainhead.)

Also, FYI, Paths Forward allows you to say you're not interested in something. It's just, if you don't put the work into knowing something, don't claim that you did. Also you should keep your interests themselves open to criticism and error correction. Don't be an AGI researcher who is "not interested in philosophy" and won't listen to arguments about why philosophy is relevant to your work. More generally, it's OK to cut off a discussion with a meta comment (e.g. "not interested" or "that is off topic" or "I think it'd be a better use of my time to do this other thing...") as long as the meta level is itself open to error correction and has Paths Forward.

Oh also, btw, the demandingness of Paths Forward lowers the resource requirements for doing it, in a way. If you're interested in what someone is saying, you can be lenient and put in a lot of effort. But if you think it's bad, then you can be more demanding – so things only continue if they meet the high standards of PF. This is win/win for you. Either you get rid of the idiots with minimal effort, or else they actually start meeting high standards of discussion (so they aren't idiots, and they're worth discussing with). And note that, crucially, things still turn out OK even if you misjudge who is an idiot or who is badly mistaken – b/c if you misjudge them all you do is invest less resources initially but you don't block finding out what they know. You still offer a Path Forward (specifically that they meet some high discussion standards) and if they're actually good and have a good point, then they can go ahead and say it with a permalink, in public, with all quotes being sourced and accurate, etc. (I particularly like asking for simple things which are easy to judge objectively like those, but there are other harder things you can reasonably ask for, which I think you picked up on in some ways your judgement of PF as demanding. Like you can ask people to address a reference that you take responsibility for.)

BTW I find that merely asking people to format email quoting correctly is enough barrier to entry to keep most idiots out of the FI forum. (Forum culture is important too.) I like this type of gating because, contrary to moderators making arbitrary/subjective/debatable judgements about things like discussion quality, it's a very objective issue. Anyone who cares to post can post correctly and say any ideas they want. And it lacks the unpredictability of moderation (it can be hard to guess what moderators won't like). This doesn't filter on ideas, just on being willing to put in a bit of effort for something that is productive and useful anyway – proper use of nested quoting improves discussions and is worth doing and is something all the regulars actively want to do. (And btw if someone really wants to discuss without dealing with formatting they can use e.g. my blog comments which are unmoderated and don't expect email quoting, so there are still other options.)

It is written very clearly, and also wants to make me scream inside

Why does it make you want to scream?

Is it related to moral judgement? I'm an Objectivist in addition to a Critical Rationalist. Ayn Rand wrote in The Virtue of Selfishness, ch8, How Does One Lead a Rational Life in an Irrational Society?, the first paragraph:

There's a lot of reasoning for this which goes beyond the one essay. At present, I'm just raising it as a possible area of disagreement.

There are also reasons about objective truth (which are part of both CR and Objectivism, rather than only Objectivism).

The issue isn't just moral judgement but also what Objectivism calls "sanction": I'm unwilling to say things like "It's ok if you don't do Paths Forward, you're only human, I forgive you." My refusal to actively do anti-judgement stuff, and approve of PF alternatives, is maybe more important than any negative judgements I've made, implied or stated.

It hits all the right notes motivation-wise, and a very high number of Rationality Virtues. Curiosity, check. Relinquishment, check. Lightness, check. Argument, triple-check.

Yudkowsky writes about rational virtues:

Haha, yeah, no wonder a triple check on that one :)

Simplicity, check. Perfectionism, check. Precision, check. Scholarship, check. Evenness, humility, precision, Void... nope nope nope PF is much harsher than needed when presented with negative evidence, treating them as irreparable flaws (that's for evenness)

They are not treated as irreparable – you can try to create a variant idea which has the flaw fixed. Sometimes you will succeed at this pretty easily, sometimes it’s hard but you manage it, and sometimes you decide to give up on fixing an idea and try another approach. You don’t know in advance how fixable ideas are (you can’t predict the future growth of knowledge) – you have to actually try to create a correct variant idea to see how doable that is.

Some mistakes are quite easy and fast to fix – and it’s good to actually fix those, not just assume they don’t matter much. You can’t reliably predict mistake fixability in advance of fixing it. Also the fixed idea is better and this sometimes helps leads to new progress, and you can’t predict in advance how helpful that will be. If you fix a bunch of “small” mistakes, you have a different idea now and a new problem situation. That’s better (to some unknown degree) for building on, and there’s basically no reason not to do this. The benefit of fixing mistakes in general, while unpredictable, seems to be roughly proportional to the effort (if it’s hard to fix, then it’s more important, so fixing it has more value). Typically, the small mistakes are a small effort to fix, so they’re still cost-effective to fix.

That fixing mistakes creates a better situation fits with Yudkowsky’s virtue of perfectionism.

(If you think you know how to fix a mistake but it’d be too resource expensive and unimportant, what you can do instead is change the problem. Say “You know what, we don’t need to solve that with infinite precision. Let’s just define the problem we’re solving as being to get this right within +/- 10%. Then the idea we already have is a correct solution with no additional effort. And solving this easier problem is good enough for our goal. If no one has any criticism of that, then we’ll proceed with it...")

Sometimes I talk about variant ideas as new ideas (so the original is refuted, but the new one is separate) rather than as modifying and rescuing a previous idea. This is a terminology and perspective issue – “modifying" and “creating" are actually basically the same thing with different emphasis. Regardless of terminology, substantively, some criticized flaws in ideas are repairable via either modifying or creating to get a variant idea with the same main points but without the flaw.

PF expects to have errors all other the place and act to correct them, but places a burden on everyone else that doesn't (that's for humility)

Is saying people should be rational burdensome and unhumble?

According to Yudkowsky's essay on rational virtues, the point of humility is to take concrete steps to deal with your own fallibility. That is the main point of PF!

PF shifts from True to False by sorting everything through contexts in a discrete way.

The binary (true or false) viewpoint is my main modification to Popper and Deutsch. They both have elements of it mixed in, but I make it comprehensive and emphasized. I consider this modification to improve Critical Rationalism (CR) according to CR's own framework. It's a reform within the tradition rather than a rival view. I think it fits the goals and intentions of CR, while fixing some problems.

I made educational material (6 hours of video, 75 pages of writing) explaining this stuff which I sell for $400. Info here:

https://yesornophilosophy.com

I also have many relevant, free blog posts gathered at:

http://curi.us/1595-rationally-resolving-conflicts-of-ideas

Gyrodiot, since I appreciated the thought you put into FI and PF, I'll make you an offer to facilitate further discussion:

If you'd like to come discuss Yes or No Philosophy at the FI forum, and you want to understand more about my thinking, I will give you a 90% discount code for Yes or No Philosophy. Email [email protected] if interested.

Incertitude is lack of knowledge, which is problematic (that's for precision)

The clarity/precision/certitude you need is dependent on the problem (or the context if you don’t bundle all of the context into the problem). What is your goal and what are the appropriate standards for achieving that goal? Good enough may be good enough, depending on what you’re doing.

Extra precision (or something else) is generally bad b/c it takes extra work for no benefit.

Frequently, things like lack of clarity are bad and ruin problem solving (cuz e.g. it’s ambiguous whether the solution means to take action X or action Y). But some limited lack of clarity, lower precision, hesitation, whatever, can be fine if it’s restricted to some bounded areas that don’t need to be better for solving this particular problem.

Also, about the precision virtue, Yudkowsky writes,

FI/PF has no issue with this. You can specify required precision (e.g. within plus or minus ten) in the problem. Or you can find you have multiple correct solutions, and then consider some more ambitious problems to help you differentiate between them. (See the decision chart stuff in Yes or No Philosophy.)

PF posits time and again that "if you're not achieving your goals, well first that's because you're not faillibilist". Which is... quite too meta-level a claim (that's for the Void)

Please don't put non-quotes in quote marks. The word "goal" isn't even in the main PF essay.

I'll offer you a kinda similar but different claim: there's no need to be stuck and not make progress in life. That's unnecessary, tragic, and avoidable. Knowing about fallibilism, PF, and some other already-known things is adequate that you don't have to be stuck. That doesn't mean you will achieve any particular goal in any particular timeframe. But what you can do is have a good life: keep learning things, making progress, achieving some goals, acting on non-refuted ideas. And there's no need to suffer.

For more on these topics, see the FI discussion of coercion and the BoI view on unbounded progress:

http://beginningofinfinity.com

(David Deutsch, author of BoI, is a Popperian and is a founder of Taking Children Seriously (TCS), a parenting/education philosophy created by applying Critical Rationalism and which is where the the ideas about coercion come from. I developed the specific method of creating a succession of meta problems to help formalize and clarify some TCS ideas.)

I don't see how PF violates the void virtue (aspects of which, btw, relate to Popper's comments on Who Should Rule? cuz part of what Yudkowsky is saying in that section is don't enshrine some criteria of rationality to rule. My perspective is, instead of enshrining a ruler or ruling idea, the most primary thing is error correction itself. Yudkowsky says something that sorta sounds like you need to care about the truth instead of your current conception of the truth – which happily does help keep it possible to correct errors in your current conception.)

(this last line is awkward. The rationalist view may consider that rationalists should win, but not winning isn't necessarily a failure of rationality)

That depends on what you mean by winning. I'm guessing I agree with it the way you mean it. I agree that all kinds of bad things can happen to you, and stuff can go wrong in your life, without it necessarily being your fault.

(this needs unpacking the definition of winning and I'm digging myself deeper I should stop)

Why should you stop?

---

Justin Mallone replied to Gyrodiot:

hey gyrodiot feel free to join Fallible Ideas list and post your thoughts on PF. also, could i have your permission to share your thoughts with Elliot? (I can delete what other ppl said). note that I imagine elliot would want to reply publicly so keep that in mind.

Gyrodiot replied:

@JUSTINCEO You can share my words (only mine) if you want, with this addition: I'm positive I didn't do justice to FI (particularly in the last part, which isn't clear at all). I'll be happy to read Elliot's comments on this and update in consequence, but I'm not sure I will take time to answer further.

I find we are motivated by the same "burning desire to know" (sounds very corny) and disagree strongly about method. I find, personally, the LW "school" more practically useful, strikes a good balance for me between rigor, ease of use, and ability to coordinate around.

Gyrodiot, I hope you'll reconsider and reply in blog comments, on FI, or on Less Wrong's forum. Also note: if Paths Forward is correct, then the LW way does not work well. Isn't that risk of error worth some serious attention? Plus isn't it fun to take some time to seriously understand a rival philosophy which you see some rational merit in, and see what you can learn from it (even if you end up disagreeing, you could still take away some parts)?

---

For those interested, here are more sources on the rationality virtues. I think they're interesting and mostly good:

https://wiki.lesswrong.com/wiki/Virtues_of_rationality

https://alexvermeer.com/the-twelve-virtues-of-rationality/

http://madmikesamerica.com/2011/05/the-twelve-virtues-of-rationality/

That last one says, of Evenness:

Maybe. Depends on how "cautious" would be clarified with more precision. This could be interpreted to mean something I agree with, but also there are a lot of ways to interpret it that I disagree with.

I also think Occam's Razor (mentioned in that last link, not explicitly in the Yudkowsky essay), while having some significant correctness to it, is overrated and is open to specifications of details that I disagree with.

And I disagree with the "burden of proof" idea (I cover this in Yes or No Philosophy) which Yudkowsky mentions in Evenness.

The biggest disagreement is empiricism. (See the criticism of that in BoI, and FoR ch1. You may have picked up on this disagreement already from the CR stuff.)

View this post, with discussion, on Less Wrong.

---

Consider an idea consisting of a group of strongly connected sub-ideas. If any sub-idea is an error (doesn’t work), then the whole idea is an error (doesn’t work). We can metaphorically model this as a metal chain made of links. How strong is a chain? How hard can you pull on it before it breaks? It’s as strong as its weakest link. If you measure the strength of every link in the chain, and try to combine them into an overall strength score for the chain, you will get a bad answer. The appropriate weight to give the non-weakest links, in your analysis of chain strength, is ~zero.

There are special cases. Maybe the links are all equally strong to high precision. But that’s unusual. Variance (statistical fluctuations) is usual. Perhaps there is a bell curve of link strengths. Having two links approximately tied for weakest is more realistic, though still uncommon.

(A group of linked ideas may not be a chain (linear) because of branching (tree structure). But that doesn’t matter to my point. Stress the non-linear system of chain links and something will break first.)

The weakest link of the chain is the bottleneck or constraint. The other links have excess capacity – more strength than they need to stay unbroken when the chain gets pulled on hard enough to break the weakest link.

Optimization of non-bottlenecks is ~wasted effort. In other words, if you pick random chain links, and then you reinforce them, it (probably) doesn’t make the chain stronger. Reinforcing non-weakest links is misallocating effort.

So how good is an idea made of sub-ideas? It’s as strong as its weakest link (sub-idea). Most ideas have excess capacity. So it’d be a mistake to measure how good each sub-idea is, including more points for excess capacity, and then combine all the scores into an overall goodness score.

Excess capacity is a general feature and requirement of stable systems. Either most components have excess capacity or the system is unstable. Why? Because of variance. If lots of components were within the margin of error (max expected or common variance) of breaking, stuff would break all over the place on a regular basis. You’d have chaos. Stable systems mostly include parts which remain stable despite variance. That means that in most circumstances, when they aren’t currently dealing with high levels of negative variances, then they have excess capacity.

This is why manufacturing plants should not be designed as a balanced series of workstations, all with equal production capacity. A balanced plant (code) lacks excess capacity on any workstations (chain links), which makes it unstable to variance.

Abstractly, bottlenecks and excess capacity are key issues whenever there are dependency links plus variance. (Source.)

Applied to Software

This is similar to how, when optimizing computer programs for speed, you should look for bottlenecks and focus on improving those. Find the really slow part and work on that. Don’t just speed up any random piece of code. Most of the code is plenty fast. Which means, if you want to assign an overall optimization score to the code, it’d be misleading to look at how well optimized every function is and then average them. What you should actually do is a lot more like scoring the bottleneck(s) and ignoring how optimized the other functions are.

Just as optimizing the non-bottlenecks with lots of excess capacity would be wasted effort, any optimization already present at a non-bottleneck shouldn’t be counted when evaluating how optimized the codebase is, because it doesn’t matter. (To a reasonable approximation. Yes, as the code changes, the bottlenecks could move. A function could suddenly be called a million times more often than before and need optimizing. If it was pre-optimized, that’d be a benefit. But most functions will never become bottlenecks, so pre-optimizing just in case has a low value.)

Suppose a piece of software consists of one function which calls many sub-functions which call sub-sub-functions. How many speed bottlenecks does it have? Approximately one, just like a chain has one weakest link. In this case we’re adding up time taken by different components. The vast majority of sub-functions will be too fast to matter much. One or a small number of sub-functions use most of the time. So it’s a small number of bottlenecks but not necessarily one. (Note: there are never zero bottlenecks: no matter how much you speed stuff up, there will be a slowest sub-function. However, once the overall speed is fast enough, you can stop optimizing.) Software systems don’t necessarily have to be this way, but they usually are, and more balanced systems don’t work well.

Applied to Ideas

I propose viewing ideas from the perspective of chains with weakest links or bottlenecks. Focus on a few key issues. Don’t try to optimize the rest. Don’t update your beliefs using evidence, increasing your confidence in some ideas, when the evidence deals with non-bottlenecks. In other words, don’t add more plausibility to an idea when you improve a sub-component that already had excess capacity. Don’t evaluate the quality of all the components of an idea and combine them into a weighted average which comes out higher when there’s more excess capacity for non-bottlenecks.

BTW, what is excess capacity for an idea? Ideas have purposes. They’re meant to accomplish some goal such as solving a problem. Excess capacity means the idea is more than adequate to accomplish its purpose. The idea is more powerful than necessary to do its job. This lets it deal with variance, and may help with using the idea for other jobs.

Besides the relevance to adding up the weight of the evidence or arguments, this perspective explains why thinking is tractable in general: we’re able to focus our attention on a few key issues instead of being overwhelmed by the ~infinite complexity of reality (because most sub-issues we deal with have excess capacity, so they require little attention or optimization).

Note: In some ways, I have different background knowledge and perspective than the typical poster here (and in some ways I’m similar). I expect large inferential distance. I don’t expect my intended meaning to be transparent to readers here. (More links about this: one, two.) I hope to get feedback about which ideas people here accept, reject or want more elaboration on.

Acknowledgments: The ideas about chains, bottlenecks, etc., were developed by Eliyahu Goldratt, who developed the Theory of Constraints. He was known especially for applying the methods of the hard sciences to the field of business management. Above, I’ve summarized some Goldratt ideas and begun relating them to Bayesian epistemology.

View discussion of this post at Less Wrong.

---

This post follows my Chains, Bottlenecks and Optimization. The goal is to give hypothetical examples of bottlenecks and non-bottlenecks (things with excess capacity), and to answer johnswentworth, who helpfully commented:

I really like what this post is trying to do. The idea is a valuable one. But this explanation could use some work - not just because inferential distances are large, but because the presentation itself is too abstract to clearly communicate the intended point. In particular, I'd strongly recommend walking through at least 2-3 concrete examples of bottlenecks in ideas.

I’ll give a variety of examples starting with simpler ones. If you want a different type, let me know.

Note: The term “bottleneck” has synonyms like “constraint” or “limiting factor”. I’ll often use “key factor”. This contrasts with a non-bottleneck, or secondary factor, which is something with excess capacity (above a margin of error), so improving it isn’t very useful. Doing better at a bottleneck makes a significant difference to doing better at your goal; doing better at a non-bottleneck doesn’t. My basic point is that we should focus our attention on key factors.

Oven

In The Goal by Eli Goldratt, the main example is a factory. One of the bottlenecks is the heat treat oven: the rate of baking parts in the oven was limiting the overall output of the factory.

A non-bottleneck example is quality assurance. It was possible to check parts for defects significantly faster than they came out of the oven. So hiring more QA people wouldn’t result in more finished products.

One of the main points of Goldratt’s book is that trying to have a balanced production line (no excess capacity at any workstation) is a bad idea.

Software

Focusing on key factors or bottlenecks is well known in software: To speed up a program, measure where most of the run time is being spent, then speed up that part(s). Don’t just optimize any function. Most functions have excess capacity (they are more than fast enough already to get a satisfactory result, and their impact is orders of magnitude less than the bottleneck’s impact).

Chair

I weigh 150lbs and buy an office chair that can hold someone up to 300lbs. It has excess capacity. Would a chair that can hold 400lbs be 33% better (regarding this factor)? Nope, that wouldn’t be useful to me. Everything else being equal, a sturdier chair is better, but I should focus my attention elsewhere: other factors are going to matter orders of magnitude more than having more excess capacity on sturdiness.

I have a budget. Price is a key factor. If a buy a cheaper chair, I can buy more Fortnite skins. So when I’m chair shopping, I focus on variations in price, but I don’t pay attention to variations in weight capacity. (Every chair in the store holds my weight plus a margin of error, so it’s a non-issue.)

Another non-bottleneck is smoothness. I want a chair that doesn’t poke me. Every chair in the store is far more than smooth enough. If I measured the bumps, I’d fine one chair has 50 micrometers bumps, another has 100 micrometer bumps, and so on, but it’d take 4000 micrometer bumps to poke me uncomfortably. I shouldn’t assign a higher score to the chair with smaller bumps when both have plenty small enough bumps. And there’s so much excess capacity here that I don’t need to and shouldn’t even do those measurements – that’d be wasteful.

Ideas

These examples involve ideas. E.g. “I’ll buy the Aeron chair” is an idea about how to proceed in a life situation. It has excess capacity on chair smoothness and sturdiness. It unfortunately fails horribly on the price bottleneck.

Factories are designed according to ideas. Someone’s design plan (or someone’s ideas about how to modify the factory) created that bottleneck at the oven.

Computer code corresponds to ideas that programmers have about what steps should be used to accomplish tasks. A programmer’s idea about how to design a program can have a speed bottleneck for one sub-idea and excess speed capacity for many other sub ideas.

“I should go to Stanford” is an idea with excess capacity on distance because it’s more than far enough away from my parents. It also does great on the prestige key factor.

Another type of idea is a skill. E.g. I have ideas about how to play chess. They have excess capacity for the goal of beating a 1000 rated player – they are more than good enough to do the job. For the goal of getting a higher rating, my endgame knowledge is a bottleneck, but my opening knowledge has excess capacity. The positions I get out of the opening are more than good enough to move up in the chess world, but I lose too many drawn endgames.

When constructing a birdhouse, I have excess capacity for reading and understanding a guide, but a bottleneck for patience to go slowly and carefully enough given my poor skill at making wood come out the right shape. The wood has excess capacity for strength, but not for weight because I want to hang the birdhouse from a thin branch.

Evolution

We’re debating selfish gene, group selection or Lamarckism as the primary driver of biological evolution. The key factors involve causal explanations.

Lamarckism lacks specifics about the mechanism for transmitting change to the next generation (it’s also experimentally questionable). Sure you can hypothetically imagine a system which saves information about bodily system usage during a lifetime and then puts information into eggs or sperm. But that system hasn’t been found in reality, studied, observed under a microscope, etc. Genes have been, e.g. we’ve studied the shape, chemical composition and copying mechanisms of DNA.

A key issue with group selection was what happens with traits which help the group but harm the individual. What are the causal mechanisms by which those traits would end up in the next generation at higher rather than lower rates (lower due to the harm to the holders of the trait)? No good answer is known for the general case.

These theories all have excess capacity at being able to tell a high level story to account for the animals we observe. Their ability to do that could survive infinitely many variations of the animals to be explained (e.g. if giraffes were 1.1 inches taller on average, or 1.11, or 1.111…). They could also still tell their stories successfully given an infinity of additional constraints, e.g. that the story doesn’t use the number 888111, or the constraint it doesn’t use 888112, or a constraint on 888113, etc.

It’d be an error to pick some evidence, e.g. observations of spiders, and then try to estimate how well each theory fits the evidence, and assign them differing scores. Each theory, if it was assumed to be right about the key issues, would be able to explain spiders fine. (Our view of how well an idea deals with a non-bottleneck factor is often a proxy for our judgment of a key factor – I don’t like Lamarckism’s explanation of the origin of spiders because I don’t think acquired traits are inherited in genes.)

College Rankings

College rankings are discussed in The Order of Things, an article by Malcom Gladwell about why it’s hard to usefully combine many factors into a single overall ranking score.

Many dimensions, like class size, graduation rate or prestige, come in different units with no conversions (and some dimensions are hard to measure at all). It’s not like converting inches to meters, it’s like trying to convert inches to minutes (or converting both inches and minutes to something else, e.g. grams).

The key factors for colleges vary by person/context. I want a college which is at least 1000 miles away from my parents, but you strongly prefer a local college so you can save money by not moving out. And neither of those factors can be taken into account by one-size-fits-all college rankings published nationally, even if they wanted to include them, because college seekers live in different places.

Joe has excess capacity on graduation rate. He doesn’t mind going to a school where 80% of people graduate over a school where 90% of people graduate. He’s a great student and is confident that he can graduate regardless. His parents have PhDs and he’s had exposure to professors, to what type of skills are needed to graduate, etc., so he’s in a good position to make this judgment.

Steve will be the first person in his family to go to college. He struggled in high school, both with the academics and with communicating in English with his teachers. For Steve, a college with a 99% graduation rate looks way less risky – that’s a key factor.

Key factors are situational. Kate wants a prestige degree, but Sue wants any degree at all just to satisfy her parents. Sue also wants somewhere she can live on campus with her dog.

Kate and Sue have excess capacity in different areas. Kate is so good at basketball that she can get a full scholarship anywhere, so she doesn’t care about tuition price. Sue is way less bothered by dirt and bad smells than most people, so she has excess capacity on attending a dirty, smelly college.

Some factors are about the same for everyone. They all want a college with plenty of air available to breathe. Fortunately, every single college has excess capacity on air. Even if you came and took some air away, or the college had a bad air day (where, due to the motion of gas atoms and statistical fluctuations, there were an unusually low number of air molecules on campus that day), there’d still be plenty of air left. This example is a reminder of the importance of focusing on only a few factors out of infinite factors that could be evaluated.

Physics

In science, we want our empirical theories to match our observations but not match a ton of other, logically possible observations. A law like E=hf (the energy of a photon is Plank’s constant times the photon’s frequency) is valuable in large part because of how much it excludes. It’s pretty specific. We don’t want excess capacity for the set of physical events and states allowed by the law; we prefer a minimal and highly accurate set. So that’s a key factor where we want as much as we can get (more of it translates to more success at our goal).

E=hf has excess capacity on shortness. It could be a longer formula and we’d still accept it.

E=hf has excess capacity on experimental data. We could have less data and still accept it. The data is also much more precise than necessary to accept E=hf. And we have excess documented counter examples to E=hf^7, E=hf^8, E=hf^9, and to infinitely many other rival theories.

E=hf has excess capacity on ease of use. It could be more of a hassle to do the calculation and we’d still accept it.

E=hf has excess capacity of rhetorical value. It could be less persuasive in speeches and we’d still accept it. This would remain true even if it’s rhetorical value was ~zero. We don’t judge science that way (at least that’s the aspiration).

Peter tries to debate me. No, E=Gd, he claims. What’s Gd I ask? God’s decision. But that’s not even a multiplication between G and d! This reminds me that E=hf does great on the “actually math” criterion, which normally isn’t a key factor in my discussions or thinking, but it becomes a key factor when I’m talking with Peter. Related to this, I have a bunch of excess capacity that Peter doesn’t: I could be really tired and distracted but I’d still remember the importance of math in scientific laws.

As long as Peter disagrees re using math, many other issues that I’d normally talk about are irrelevant. I shouldn’t try to debate with Peter how significant figures and error bars for measurements work. That wouldn’t address his no-math perspective; it’d be the wrong focus in the situation. It’d be a mistake for me to say that my approach has a really great, nuanced approach to measurement precision, so Peter should increase his confidence that I’m right. If I said that, he should actually become more doubtful about me because I’d be showing inflexible thinking that’s bad at understanding what’s relevant to other contexts that I’m not used to.

Minimum Wage Debate

We’re debating minimum wage. We agree that low skill workers shouldn’t get screwed over. I say minimum wage laws screw over workers by reducing the supply of jobs. You say minimum wage laws prevent workers from being screwed over by outlawing exploitative jobs.

The key factor for my claim is economics (specifically the logic and math of supply and demand in simple hypothetical scenarios). When I convince you about that, you change your mind. I should focus on optimizing for that issue. During the debate, I have excess capacity on many dimensions, such as theism, astrology or racism. I’m not even close to causing you to think my position is based on God, the stars, or race. I don’t need to worry about that. When I’m considering what argument to use next, I don’t need to avoid arguments associated with Christianity; I can ignore that factor. Similarly, I don’t need to factor in the race of the economists I cite.

There are many factors which could be seen positively in some way. E.g. economics books with more pages and more footnotes are more impressive, in some sense. This is contextual: some people would be more impressed instead by a compact, very clear book.

But we actually have tons of excess capacity on page count and footnotes. You’re tolerant of a wide variety of books. I don’t need to worry about optimizing this factor. I can focus on other factors like choosing the book with the best clarity and relevance (key factors).

If you were picky about dozens of factors, our discussion would fail. Your tolerance lets me focus on optimizing only a few things, which makes productive discussion possible.

So I convince you that I’m right about minimum wage. But next year you come back with a new argument.

Don’t government regulations make it harder to start a business and to hire people? There’s lots of paperwork that discourages entrepreneurship. This artificially reduces the supply of jobs. It prevents the supply and demand of jobs from reaching the proper equilibrium (market clearing price). Therefore, workers are actually being underpaid because they’re competing for too few jobs, which drives wages down.

Now what? You’re right that my simplified market model didn’t fully correspond to reality. The bottleneck is no longer your ignorance of basic economics. You’ve actually read a bunch and now have excess capacity there: you know more than enough for me to bring up some economics concepts without confusing you. Also you’re very patient and highly motivated, so I don’t have to keep things really short. However, you’re sensitive to insults against less fortunate people, so I have to check for something potentially offensive when I do an editing pass. I only want to share arguments with you that have excess capacity for that – they are more than inoffensive enough.

What should I do? I could defend my model and tell you all kinds of merits it has. The model is useful in many ways. There are many different ways to argue for its correctness given its premises. But those aren’t bottlenecks. You aren’t denying that. That won’t change your mind because the problem you brought up focuses on a different issue.

I judge that the bottleneck is your understanding of what effect minimum wage has on a scenario where the supply of jobs is artificially suppressed. Yes that’s a real problem, but does minimum wage help fix it? I need to focus on that. When I’m considering candidate arguments to tell you, I should look at which one will best address that (and then check offensiveness in editing), while not worrying about factors with excess capacity like your patience and motivation. All the arguments I’m considering will work OK given the available patience and motivation (yes, I could make up a tangled argument that takes so much patience to get through that it turns patience into a bottleneck, but it doesn’t require conscious attention for me to avoid that). Improvements to those factors (like requiring one less unit of patience) are orders of magnitude less important than the key factors (like creating one more unit of understanding of the effects of minimum wage on a system with a government-constrained job supply).

Read this post, with replies, on Less Wrong.

---

Replies to comments on my Chains, Bottlenecks and Optimization:

abramdemski and Hypothesis Generation

Following the venerated method of multiple working hypotheses, then, we are well-advised to come up with as many hypotheses as we can to explain the data.

I think come up with as many hypotheses as we can is intended within the context of some background knowledge (some of which you and I don’t share). There are infinitely many hypotheses that we could come up with. We’d die of old age while brainstorming about just one issue that way. We must consider which hypotheses to consider. I think you have background knowledge filtering out most hypotheses.

Rather than consider as many ideas as we can, we have to focus our limited attention. I propose that this is a major epistemological problem meriting attention and discussion, and that thinking about bottlenecks and excess capacity can help with focusing.

If you’ve already thought through this issue, would you please link to or state your preferred focusing criteria or methodology?

I did check your link (in the quote above) to see if it answered my question. Instead I read:

Now we've got it: we see the need to enumerate every hypothesis we can in order to test even one hypothesis properly. […]

It's like... optimizing is always about evaluating more and more alternatives so that you can find better and better things.

Maybe we have a major disagreement here?

abramdemski and Disjunction

The way you are reasoning about systems of interconnected ideas is conjunctive: every individual thing needs to be true. But some things are disjunctive: some one thing needs to be true. […]

A conjunction of a number of statements is -- at most -- as strong as its weakest element, as you suggest. However, a disjunction of a number of statements is -- at worst-- as strong as its strongest element.

Yes, introducing optional parts to a system (they can fail, but it still succeeds overall) adds complexity to the analysis. I think we can, should and generally do limit their use.

(BTW, disjunction is conjunction with some inversions thrown in, not something fundamentally different.)

Consider a case where we need to combine 3 components to reach our goal and they all have to work. That’s:

A & B & C -> G

And we can calculate whether it works with multiplication: ABC.

What if there are two other ways to accomplish the same sub-goal that C accomplishes? Then we have:

A & B & (C | D | E ) -> G

Using a binary pass/fail model, what’s the result for G? It passes if A, B and at least one of {C, D, E} pass.

What about using a probability model? Problematically assuming independent probabilities, then G is:

AB(1 - (1-C)(1-D)(1-E)))

Or more conveniently:

AB!(!C!D!E)

Or a different way to conceptualize it:

AB(C + D(1 - C) + E(1 - C - D(1 - C)))

Or simplified in a different way:

ABC + ABD + ABE - ABCD - ABCE - ABDE + ABCDE

None of this analysis stops e.g. B from being the bottleneck. It does give some indication of greater complexity that comes from using disjunctions.

There are infinitely many hypotheses available to generate about how to accomplish the same sub-goal that C accomplishes. Should we or together all of them and infinitely increase complexity, or should we focus our attention on a few key areas? This gets into the same issue as the previous section about which hypotheses merit attention.

Donald Hobson and Disjunction

Disjunctive arguments are stronger than the strongest link.

On the other hand, [conjunctive] arguments are weaker than the weakest link.

I don’t think this is problematic for my claims regarding looking at bottlenecks and excess capacity to help us focus our attention where it’ll do the most good.

You can imagine a chain with backup links that can only replace a particular link. So e.g. link1 has 3 backups: if it fails, it’ll be instantly replaced with one of its backups, until they run out. Link2 doesn’t have any backups. Link3 has 8 backups. Backups are disjunctions.

Then we can consider the weakest link_and_backups group and focus our attention there. And we’ll often find it isn’t close: we’re very unevenly concerned about the different groups failing. This unevenness is important for designing systems in the first place (don’t try to design a balanced chain; those are bad) and for focusing our attention.

Structures can also be considerably more complicated than this expanded chain model, but I don’t see that that should change my conclusions.

Dagon and Feasibility

I think I've given away over 20 copies of _The Goal_ by Goldratt, and recommended it to coworkers hundreds of times.

…

The limit is on feasibility of mapping to most real-world situations, and complexity of calculation to determine how big a bottleneck in what conditions something is.

Optimizing software by finding bottlenecks is a counter example to this feasibility claim. We do that successfully, routinely.

Since you’re a Goldratt fan too, I’ll quote a little of what he said about whether the world is too complex to deal with using his methods. From The Choice:

"Inherent Simplicity. In a nutshell, it is at the foundation of all modern science as put by Newton: 'Natura valde simplex est et sibi consona.' And, in understandable language, it means, 'nature is exceedingly simple and harmonious with itself.'"

…

"What Newton tells us is that […] the system converges; common causes appear as we dive down. If we dive deep enough we'll find that there are very few elements at the base—the root causes—which through cause-and-effect connections are governing the whole system. The result of systematically applying the question "why" is not enormous complexity, but rather wonderful simplicity. Newton had the intuition and the conviction to make the leap of faith that convergence happens, not just for the section of nature he examined in depth, but for any section of nature. Reality is built in wonderful simplicity."

(I also posted this at Less Wrong.)

This post follows my Chains, Bottlenecks and Optimization (which has the followups Bottleneck Examples and Comment Replies for Chains, Bottlenecks and Optimization). This post expands on how to think about bottlenecks.

---

There are deeper concepts behind bottlenecks (aka constraints, limiting factors, or key factors).

First, different factors contribute different amounts to goal success. Second, there’s major variation in the amounts contributed.

E.g. I’m adding new features to my software. My goal is profit. Some new features will contribute way more to my profit than others There are lots of features my (potential) customers don’t care about. There are a few features that tons of customers would pay a bunch for.

A bottleneck is basically just a new feature that matters several orders of magnitude more than most others. So most features are approximately irrelevant if the bottleneck isn’t improved.

Put another way: improving the bottleneck translates fairly directly to more goal success, while improving non-bottlenecks translates poorly, e.g. only at 1/1000th effectiveness, or sometimes 0. (It’s possible, but I think uncommon, to have many factors that contribute similarly effectively to goal success. Designing stuff that way doesn’t work well. It’s the same issue as balanced production lines being bad, which Eli Goldratt explains in The Goal: A Process of Ongoing Improvement. It’s also similar to the Pareto Principle which says 80% of effects come from 20% of the causes – meaning most factors aren’t very important.)

What about the software not crashing, not corrupting saved data, and not phoning home with location tracking data? People want those things but I could have them and easily still make zero profit.

A good, typical model for viewing goal pursuit is:

1. There are many factors that would help, and just one or a few of them are the most important to focus on. This is because most factors have a significantly smaller impact. This is focusing on the key positives.
2. There are also many dealbreaker factors that cause failure if screwed up. This is avoiding major negatives.

People care about (1) conditional on (2) not being broken. Avoid anything awful, then optimize in the right places.

When buying a cat, I might try to optimize cuteness and cheapness, while also making sure the cat has 4 paws, a tail, no rabies, is tame, and isn’t too old. I want to do well on a couple key factors and also a bunch of easy factors need to be non-broken. It’s generally not that hard to brainstorm dozens of dealbreakers, many of which are quite easy to avoid in your current situation, even to the point of sounding silly to mention it at all.

(Dealbreakers are also contextual. If there were no cats available meeting all my criteria, I might lower my standards.)

The type (2) factors don’t require much attention. If a factor did need attention, it’d switch categories. (2) is just for failure conditions which are pretty easy to handle. This means most of our attention is available to focus on a few key issues.

I think this model is more effective than e.g. something like “consider all the factors; find out it’s way too complicated; try to approximate what you’d do if you had enough attention for all the factors”.

The model I’m proposing can be thought of as a method of organized, effective approximation from a more complex “take everything fully into account” approach. It tells us how to approximate. Thought of another way, I’m saying don’t distribute your significant figures equally.

You might think “Why not just weight all the factors relevant to my goal, then distribute my attention and significant figures according to the weightings?” The difficulty with that is how to weight things. Having a cat that doesn’t attack me and give me rabies is really important. If I’m just weighting factors normally, I’ll give that high weight because I want to reject any cat purchase which fails at that issue.

So if you just start assigning weights straightforwardly, you’ll give the type (2) factors high weights, e.g. 50,000 each, and if they all pass then the type (1) factors will function as tiebreakers worth e.g. 1-50 points each (minor detail: you can scale the weights so they add up to 1, but it’s easier to do that after you have all the factors with weights assigned – I don’t know what fraction of 1 a big factor should be until I know how many big factors there are). But the high value type (1) factors are actually the best place to put a bunch of significant figures. We don’t need a bunch of precision to address our cat having a tail, 4 legs, and no rabies. So attention allocation shouldn’t correspond to weighting.

In general, when we pursue a goal, there are many important but easy factors, and a few important but hard factors. For goals which are achievable but not easy, it has to be this way. If there were dozens of hard factors, that basically means we’re not ready to do it (though with a huge budget and a big team, sometimes it can be done – that lets you have specialists each working on just one hard factor each, plus some additional people figuring out how to coordinate and combine the results). But the standard progression is: if a project has 10 hard factors, that’s too many for me to focus my attention on at once, so I need to work on some easier sub-projects first – e.g. learning about some of those issues in isolation or doing smaller projects that help build up to the bigger one.

Another way to view the difference is that an increase in the key factors increases our success at the goal. E.g. adding the right new feature will increase profit. Or getting a cuter cat will increase enjoyment. Loosely, the more the better (there’s sometimes an upper limit, at which point it stops helping or is even actively harmful, or it keeps helping but now some other factors matter more). But for type (2) factors, the attitude isn’t anything like “more is better”; it’s just “don’t screw it up”.

In this analysis, I’ve basically assumed that type (1) factors and goal success come in matters of degree (can have more or less of them), but type (2) factors have a binary, pass/fail evaluation. The analysis needs extending for how to deal with binary goals, binary type 1 factors, and matters of degree for type (2). Those issues will come up and we need some way to think about them. I’ll leave that extension for a future post.

That extension is part of a broader issue of how binary and degree issues come up in life, how to think about them, how to convert from one to the other (and when that’s possible or not), when one type is preferable to the other, and so on. They’re both important tools to know how to think about and work with.

Factory Example

Now let’s go through an extended example to clarify how some of these issues work.

In my factory, I’m combining foos and bars to make foobars, which I sell. I have more bars than foos. So foos are the bottleneck. Getting even more bars won’t result in producing additional foobars. I already have an excess capacity of bars.

I also have excess capacity for assembly and QA. My current work area and team could produce many more foobars without hiring new people, getting more space, or getting new tools. And they could already check more foobars for defects.

And I have excess capacity in the market: I could sell more foobars if only I could produce them.

I also have excess capacity on foobar quality. I could redesign them to be nicer, but they’re good enough. Customers are satisfied. They do the job.

And I have excess capacity on price. Cheaper would be nicer, sure, but there isn’t much competition and my customers are people with a good reason to get a foobar. They get benefits from the foobar which are well above the price I’m charging.

Excess capacity means non-bottleneck.

Supply of foos is the bottleneck and the other issues are non-bottlenecks.

Using bars is limited by the availability of foos. That’s a traditional, standard bottleneck.

I call niceness a non-bottleneck because, as with foos, there is excess capacity. It won’t make much difference to achieving more of my goal (profit via foobar sales).

Key factor and secondary factor may be better terminology. It has some advantages, mostly because 1) foo supply isn’t blocking niceness from mattering in the way it’s blocking more supply of bars from mattering 2) niceness would help a little (a few orders of magnitude less than getting more foos, but not zero), which contrasts with bars – getting more bars wouldn’t help at all (in current circumstances).

Bottlenecks can be changed. E.g. I find a new supplier who can deliver far more foos than I need. Foos are no longer a bottleneck. Now what’s the bottleneck?What limits my profit? Maybe I’ll start running out of bars now. Maybe I won’t have enough customers and I’ll need better marketing. Maybe I’ll need to hire more workers. Maybe price will become the crucial issue: if I could lower the price, it’d get me a million new customers. Maybe price is key to breaking into the hobbyist market whereas price isn’t so important for the business market I currently serve.

To break into the hobbyist market, I might need to expand production capacity and lower the price and do a new marketing campaign. There could be several key factors. Doing three things at once is realistic (though not ideal), but we can’t split our focus too much. It’d be nice to find a way to improve things more incrementally. Maybe I’ll figure out how to produce foobars more cheaply first while leaving my price the same, and I’ll get some immediate benefit from higher profit margins. Then once I have the price low enough I’ll try to start selling some to hobbyists as a test (sell in some small stores instead of the big chains, or I could try online sales), and only if that works will I try to ramp up production and hobbyist marketing together.

I can also view the new project (selling to hobbyists, via expanding production, producing more cheaply, and a new marketing campaign) as a whole and then look at what the bottleneck(s) and excess capacity are. They might be quite unequal between the different parts of the new project.

(This is just a toy example. I didn’t worry about new distribution for hobbyists nor about designing a different version of the product for them which better meets the needs of a different market, nor did I worry about market segmentation and how to maintain my higher prices for business customers (a separate production version is one way to do that, using different regions is another, e.g. I could do my hobbyist sales in a different country than my existing business sales.))

Category (1) above (key positives) is the bottlenecks, the things that are valuable to pay attention to and optimize. Category (2) above (avoiding major negatives) is the non-bottlenecks, the things with excess capacity, which I can view as either “good enough” or “failure”. Relevant non-bottlenecks are important. I can’t just ignore them. They need to work. They’re in a position to potentially cause failure. But I’m not very worried about getting them to work and I don’t need to optimize them.

(I also posted this on Less Wrong.)

Below I quote from Process of On Going Improvement forum, letter 6. Eli Goldratt shares a letter he received. I added a few notes to help people follow acronyms.

My question is: Does anyone know of any applications of Less Wrong philosophy to a situation like this? How can LW ideas about rationality explain or fix this sort of problem? The scenario is that someone tried to use rational thinking to make business improvements, was highly successful (which was measured and isn't in dispute), but nevertheless has met so much ongoing resistance that he's at the point of giving up.

I am no expert in TOC but I believe my recent experiences have impact as to what you are writing about.

TOC = Theory Of Constraints. Summary.

About 2 years ago I started on my TOC adventure. Read everything I could get a hold off etc. Tried to get the company interested, etc. In fact, I finally got them interested enough that we had multiple locations participate in the satellite sessions and had enough for three facilitators (myself included). However, I could never get the company to spend for training at AGI. So, in the old air cav fashion, I felt it was up to me to make it happen.

Last year we had real problems with cost, service, high inventory, etc. My plant, I am the plant manager, was being analyzed for a possible shutdown or sell off. We were asking for 17 machines at about $300,000 each due to "lack of capacity" and we were being supplemented by outside producers.

Again, I am not a TOC expert. Basically my exposure has been reading and researching and building computer simulations to understand. But I put on TOC classes for all of my associates (200). I spent 8 hours with each of these associates in multiple classes. We talked about the goal, TIOE, we played the dice game (push, KanBan, DBR) with poker chips, paper clips, and different variations of multiple sided dice and talked about its impact, etc.

The Goal (summary) is a book by Eli Goldratt that has sold over 6 million copies.

TIOE = Throughput, Inventory and Operating Expense. These are the measurements Goldratt recommends.

The dice game is explained in The Goal. It's also now taught by e.g. MIT (section 3-2).

DBR = drum buffer rope. It's about coordinating activities around the bottleneck/constraint.

Last summer we started development on DBR and a new distribution strategy based on what I have read and researched on TOC. I used Bill Dettmer‘s book to develop trees and the clouds. I check our plan against some presentations last November in Memphis when we attended the TOC symposium there.

We had many in the company who doubted but we stuck our necks out and started at the beginning of this year. And we knew we would not be perfect.

YTD results:

YTD = Year to date

Achieved Company President‘s Award for Safety (First Plant to do so) and the planning was based on things I had read about TOC and techniques on establishing teamwork.
Service is up from high 80 to low 90 percentile to averaging above 98.5%
Costs are under budget for the first time in some years
Total Inventory has decreased over 30% and is still dropping
No Longer being supplemented by outside companies for our production
No longer need additional machines to supply demand
We do need additional business to fill our machines
Plant is no longer being considered for close, in fact production from other
facilities are being transferred in.

The chief concern when we told the big wigs we were going to this, was that the cost of freight would go up because our transfer batch sizes would get small. I told them correct but we would stop shipping product back and forth between distribution centers and repacking of product would be almost non-existant. YTD: Our total freight dollars spent is 10% less than the previous year but they look at $/ lb of freight which has gone up. I know this is wrong, they state they know it is wrong, but it still gets measured and used for evaluations.

Anyway, as we shipped more often but smaller quantities our distribution centers complained that we were costing them too much. I have tried for 9 months to get them to quantify this to me. "If I increase batch size of the transfer how many people will it reduce or how much overtime will it reduce" or any other real incremental cost will it get rid off? The general response is, it is hard to quantify but we know it is there. Maybe their intuition is correct, but maybe it is not.

So finally, I am at my end. The DCs continue to insist that we are driving their costs up with small transfer batch sizes. They have complained greatly to my boss and my bosses boss. I am growing weary of the continual fight, which has cost me and my family so much time and effort. I have chosen togive up. I have grown tired of the comments, "Well it was said in a meeting that the concept did not deliver what we expected." Then I show them the numbers and ask, "What else was expected." The reply, "That is what I heard at the meeting."

DCs = distribution centers.

Maybe I made a mistake trying to bring TOC to my plant myself. I would have loved to hire a consultant who really knew what they were doing, but any mention of that brought long talks about cost, etc. I hate to give up but my frustration level has impacted my family, which is something I cannot let happen.

In the end, I have decided this week to give them their large transfer batch sizes while I begin to look for somewhere else to go.

I did not mean for this to be a bitch session. But I can not believe the sheer level of frustration on trying to achieve buy in, even when:
1) Prior to going to our concept we had meetings with our leadership where I presented the UDES from the previous year, and all agreed,

UDES = UnDesirable EffectS. He's saying that before starting he discussed what problems the company was facing with leadership and got unanimous agreement.

2) Showed our potential solution, not all agreed but they were willing to try it
3) Now showing the best numbers the plant has ever turned out.
I just cannot understand the skepticism.

What insight can LW bring to this problem of negative response to rational improvement?

This topic is for discussing anything related to Less Wrong.

I have read Harry Potter and the Methods of Rationality and Inadequate Equilibria by Eliezer Yudkowsky. I'm over half way through reading his Rationality: From AI to Zombies. I also plan to read The Codex by Scott Alexander.

I've recently dual posted several articles on Less Wrong and curi blog. In this thread I'll post copies of comments I write on Less Wrong, and you can too.