# Understanding is the Deliverable Bojan Tunguz (a data scientist) is quoted as saying: I treat data science as Science. This might seem like a tautology, but within the practicing community of data scientists this is a very, very contentious issue. What treating data science as Science boils down to (at least for me) is: 1. Make the focus of your work on designing and running experiments 2. Spend a lot of time analyzing and interpreting those experiments 3. Exploratory mindset over exploitative mindset 4. Understanding as the ultimate “deliverable” Now, all of the above bullets are good reminders, but I found (4) really struck a chord: > **Understanding is the ultimate deliverable.** In a world that is so results oriented, it is easy to squirm in your chair at the thought of telling your boss/team that while you didn’t achieve the outcome you wanted, you did manage to build your understanding of the problem a bit more and therefore it is a success. There are three reasons why this feels uncomfortable: 1. Often we don’t *really* increase our understanding of the problem or its mechanics. 2. At the same time society and schooling frame problems as a directed graph with 2 nodes. You either move from the `start` node to the `success` node, or you don’t. 3. Even if we do, we don’t couple this increased understanding with clear language and a strong plan for what to do next. ### We don’t really increase our understanding Consider the following example: A team has been tasked with building a new software product. They are given 6 months to do so, with the goal of then releasing and having 100K daily active users in the next 2 months. They spend 6 months doing everything from scoping to implementing to releasing. When it is finally released the 2 months pass and they only end up 20K daily active users. The question is why? We can come up with effectively infinite hypotheses, moving along a scale of *actionability*:  The challenge is figuring out which hypothesis is the true cause^[Of course, it could be that a *combination* of hypotheses are the true cause. I am not making a huge deal of this combinatoric challenge because we already stated that our set of hypotheses is infinite.]. At its core this is a *scientific* problem; one in which we are trying to come up with the best [explanation](Explanations.md) about why our software didn’t get traction. But coming up with good explanations is *hard*. This would require another post altogether (for those interested, check out David Deutsch’s books), but the key idea is that good explanations are *testable* and *hard to vary*. That is a tall order if the environment you are operating in moves quickly. So what do you do? You try and make a set of simplifying assumptions and pick the first hypothesis that pops out the other side, especially when it is in line with your *incentives*. For instance if we have two hypotheses: 1. The software didn’t gain traction because it isn’t solving a genuine need. 2. The software didn’t gain traction because it wasn’t responsive enough. As a team member your incentive is to pick hypothesis (2), otherwise you may be out of a job! The entire process of picking the best hypothesis is complicated by this (frequently subconscious) incentive structure. This brings us back to the scale on the graph above: “actionability”. We have this strong incentive to pick a hypothesis that allows us to *feel* that we are doing something (this is known as [Doing Something Syndrome](Doing%20Something%20Syndrome.md)). But, as you may have already put together, this doesn’t necessarily coincide with understanding. Now this will always be a problem, but the problem is amplified in the world of startups. This is because understanding has an outsized impact when you have yet to find your niche. When you are [iterating from plan A to a plan that works](https://www.amazon.com/Running-Lean-Iterate-Plan-Works/dp/1449305172), the effectiveness of the iteration process is *directly tied* to your ability to improve your understanding over time. At this point I hope we can agree that: 1. Not improving our understanding is the most likely outcome due to a handful of different forces at play 2. Our ability to improve our understanding is important, particularly in the world of startups How do we combat this? Again, it could take on an entire post of its own, but I’ll leave you with a few ideas I have found most effective. **Create a team composed of *smart, independent* thinkers** Ray Dalio does a great job of illustrating this in his book [Principles](https://www.amazon.com/gp/product/1501124021), and it is outlined well in [this article](https://fs.blog/ray-dalio-not-knowing/). This excerpt captures the idea perfectly: > I began seeking out the smartest people I could find who disagreed with me so that I could understand their reasoning. Only after I fully grasped their points of view could I decide to reject or accept them. By doing this again and again over the years, not only have I increased my chances of being right, but I have also learned a huge amount. > > There’s an art to this process of seeking out thoughtful disagreement. People who are successful at it realize that there is always some probability they might be wrong and that it’s worth the effort to consider what others are saying — not simply the others’ conclusions, but the reasoning behind them — to be assured that they aren’t making a mistake themselves. They approach disagreement with curiosity, not antagonism, and are what I call “open-minded and assertive at the same time.” This means that they possess the ability to calmly take in what other people are thinking rather than block it out, and to clearly lay out the reasons why they haven’t reached the same conclusion. They are able to listen carefully and objectively to the reasoning behind differing opinions. > > When most people hear me describe this approach, they typically say, “No problem, I’m open-minded!” But what they really mean is that they’re open to being wrong. True open-mindedness is an entirely different mind-set. It is a process of being intensely worried about being wrong and asking questions instead of defending a position. It demands that you get over your ego-driven desire to have whatever answer you happen to have in your head be right. Instead, you need to actively question all of your opinions and seek out the reasoning behind alternative points of view. Notice a few things about the way Dalio outlines this process: 1. It is not easy! If at any point in your life you have ever interacted with humans (this should be everyone, unless an alien is reading this) you know that they are complicated and full of unique emotions. Fostering an environment of thoughtful disagreement is akin to alchemy. It can be done, but it requires great care to be done *well*. 2. It is heavily related to increasing your *understanding*. I like to think of this process as taking place in the Roman Colosseum, except that instead of gladiators entering the arena it is a set of independent ideas. These ideas battle it out on their own merits, where the weapons at their disposal are logic and reasoning. The battle highlights each ideas strengths and weaknesses, with only one to be crowned the winner. This entire process bolsters your understanding. **Daily practice: Reflect on what you have learned** Every single day we should be learning something about the problem we are trying to solve. Take 15 minutes to reflect and (in writing) synthesize what that days bounty of understanding consisted of. This will likely be uncomfortable at first. If you aren’t used to understanding being the main deliverable then there is a good chance that many days you won’t have learned anything new; you level of understanding will not have changed. Push past this discomfort and let it guide you. What [lens](360%20Framing%20and%20Multiple%20Lenses.md) have you not viewed the problem through? You may start structuring your days around increasing your understanding. Some may feel that this is a luxury (especially in a startup) that they simply cannot afford. They may say: “We have already determined we need to work on widget `X`, and that will take two weeks to build, so for two weeks we won’t be learning much about our problem. The learning will need to be put on pause until we have released widget `X` and we can see how users respond”. This is false. You can, and *should*, increase your understanding during the two week building phase. But how? If you are heads down sprinting to a release how can you increase understanding? By always be in a state of *empathizing* with you end user. As you are building, consider their usage patterns, objectives, goals and workflows. It may be that something doesn’t feel right as the implementation progresses, or there is a snag you cannot get past. You don’t need to *act* on these things. That would indeed slow you down, and simply getting customer feedback is likely the best way to address these concerns anyways. However, you should be *writing these concerns and thoughts down*. Each day during the two weeks of building you will improve your understanding of where the customer may be coming from by coming up with a set of ideas and hypotheses, none of which have been validated yet. These may be wrong, but they will still force you to engage and create a finer fabric of the problem you are trying to solve. Then, once you get customer feedback, these ideas can be validated and your understanding can improve further. Oh so the customers didn’t really need bell `Y` and whistle `Z` that you felt strongly they would? Why was your understanding of the user incorrect? Did you have incorrect assumptions? Turns out your users only use the app for 10 minutes a day instead of 30 (which was the goal)? You thought this may be the case; you intuition was right on the money. Is this a problem? Do they need to use it for 30 minutes each day? This process will *strengthen* your understanding and intuitions, over time ensuring that you have a unique and independent way of viewing your problem. If your entire team is doing this, your ability will improve exponentially. **** ### Real world problems are DAGs with many paths TODO ### TODO --- Date: 20221127 Links to: Tags: #review References: * [Exclusive: Grandmaster Bojan Tunguz on what it takes to break Kaggle’s Top 10 | The Business of Business](https://www.businessofbusiness.com/articles/Kaggle-grandmaster-top-10-bojan-tunguz/)