# Machine Learning in Startups ### Principles 1. Start *simple* and *iterate*. The goal is to *derisk* our model. 2. Business goals should drive modeling goals. 3. Work in public, embrace accountability. Be clear on goals and solutions, welcome critical feedback. 4. Maximize learning about customers & problem per unit time. 5. Risk mitigation should be a part of all decisions. Goal: Systematically derisk over time. 6. Focus on bottlenecks and biggest risks, combined with biggest areas of learning and information gain. 7. Form falsifiable hypothesis. # 1. In the first month... ## 1.1 Product * **Product Market Fit**: Do customers want this? How did they determine this? Do we feel we have product market fit? Who is our customer (merchant or their customers)? * **Define Product Problem & ML Problem**: Define the product level problem, the product level goal in clear writing. Then define the problem in an ML paradigm (clearly, working in public for others to see). * **Data Collection**: Think about data collection from day one. Better data operations means better data analytics, allowing the organization to improve more effectively. This would involve building a dashboard. * **Product Definitions, Modeling Metrics & AC**: Bridge gap between the product definitions and modeling metrics. Work with product to determine Acceptance Criteria. I.e. translate business requirements to clear modeling metrics. This just needs to be done at a high level. * **Business Value of a Prediction**: Can we increase the value of our prediction? For instance, by selling our prediction (via exposing API) to shoppify. Shoppify can then use that prediction (or acquire Kover...). * **Business goals must drive modeling goals** * **Consider the [Product Data Fit Strategy](Product%20Data%20Fit%20Strategy.md)** * **Maximize learning about customers per unit time**: This is the goal of a lean startup, and should always be in the back of your mind. * ## 1.2 Product & ML * **Part of product that would benefit from ML**: Identify which parts of a product would benefit from ML and how to frame a learning goal in a way that minimizes the risk of users having a poor experience. * **Handle Failures Gracefully**: How can we design product to handle ML failures gracefully? * **Latency**: What latency is acceptable from product perspective (i.e. how long with model take to make an inference)? * **Expectations/What metrics am I judged on**: What is expectation of me/the ML model? Are we supposed to be profitable? How profitable? How many users? tldr: What metrics am I being judged on? Do I agree with those metrics? (if not speak up, you are being paid for your judgment. Your goal is to help the company succeed). * **Guard Rails**: Should there be guardrails? If so, what are some good first approaches? For instance, maybe if order is over $1,000 we say that we *must* charge 10% minimum. Or we say that we can never charge less than 1% of order and greater than 30%. * **Error Tolerance**: What is our error tolerance? * **Business Value of prediction**: What is the business value of a prediction? * **Derisk via Baseline/Heuristic Solution**: What is our current baseline (start with a straw man baseline to evaluate worst case performance if possible)? Assuming our model is not much better than this baseline, would our product still be valuable? * **Make use of the Backend**: What application logic would be useful on the backend? Does a user hover over button but not click? * **Deployment Speed as Bottleneck**: What effort should be spent lowering effort for deployment? This has a good chance of being the bottleneck. If it is, it is likely a good place to invest time and resources. * **Conservative or Permissive Model**: This is a business question! How to balance false positive vs. false negative (in this case, pricing low vs high)? Part of this factors into *burn rate* and hypothesis testing. Have they already determined that customers want this? How did they determine this? E.g. if we price policies very low, most customers would accept. This could be a nice baseline for if customers want the product. But at the same time it could be misleading! You would likely want to see how customers responded to a *high policy price*. This would give us two baselines for customer demand. E.g. 20% of customers purchase insurance with it is 1% of order price, 5% of customers purchase when policy is 20% of order price. Note: this is likely dependent on size of order and item * **Be the Algorithm**: Spend some time *being the algorithm*. Without using any data, leverage any prior ideas to solve this problem (in this case, look at how insurance was traditionally priced) * **Clearly Articulate Technical Problem**: Once you have a good understanding of product problem, clearly articulate (in public) the technical problems that must subsequently be solved. Disentangle the different prediction problems (probability of return, probability of purchase policy at a given price, profitability of policy...) * **Initial Model is used to Gather Information**: When developing an initial model, the goal is that initial model and dataset produce informative results that guide further modeling and data gathering. The entire purpose is *to gather information* that allows us to make better decisions about where and how to invest time. ## 1.3 General Thinking & Questions * **Assumptions:** Keep track of all assumptions you are making. What assumptions am I making? What assumptions may I be taking for granted? Critically questions these regularly. * **Right Problem**: Are you working on the right problem? Are you at the right stage of growth for this? * **Identify Impact Bottle Neck**: To help identify, imagine the potential bottleneck(s) as being solved, then ask whether or not they were worth the effort spent? The ideal case is that you can pitch the results regardless of the outcome: if you do not get the best outcome, is this still impactful? Have you learned something or validated some assumptions? * **Constraints of Problem Space**What are the constraints of your problem space? Are they actually constraints? Could you reframe? * **Always work in public**: Communication here is key. * **Never forget, your business model is the product.** * **Prioritization**: Pick things to work on where if it's a success *it matters*. What you work on is more important than how hard you work. * Do a world class job of reflecting and thinking about what the most important things to work on are. * Startup employees are venture capitalists, but they invest their *time*. Think of your time as *capital*. ## 1.4 Miscellaneous * **Understand Tech Stack**: In order to go fast, you must understand the current technical stack, pipelines, architecture, etc. * **Make a name for yourself as an Independent Thinker**: Pick a few "arguments" in areas related to product. You want to demonstrate that you are valuable in those areas and should be taken seriously there. The first few weeks can set peoples impressions of you very strongly # 2. General Notes #### What I've learned working with 12 ML startups Checkout [this article](https://towardsdatascience.com/what-ive-learned-working-with-12-machine-learning-startups-a9a3026d2419). A few key themes: 1. It’s about building products, not about AI 2. It's about the problem, not the method 3. Look for synergies between data and product 4. Data first, AI later 5. Invest in effectively communication. * One key ingredient is maintaining an ongoing conversation about business metrics and how they translate into modeling metrics 6. Quick and dirty isn't actually that dirty 7. Get market feedback * The most important activity in early stage product development is getting market feedback. How often do users click the button? When are they more likely to click? I.e. how do they engage? #### Build an AI moat Checkout this article [here](https://towardsdatascience.com/how-to-build-an-ai-moat-386c81a79900). 1. Build a graph of model value vs. prediction accuracy. Ideally this is somewhat linear, where as the prediction becomes more accurate the value of the model keeps increasing. In the case of Kover, as accuracy increases, we can become more profitable. The error tolerance may be bad since a bad prediction likely results in loss. However, we can increase model value even further by offering to *sell* our prediction (exposed via an API for instance) to shopify. This could allow shoppify to build features for there merchants to alert *prior* to sale of high probability return carts, and then shopify can take some sort of action. Kover would of course charge for this. 2. See [Stacking S-Curves](Stacking%20S%20Curves.md) ### Product Driven Machine Learning Checkout this article [here](https://hackernoon.com/product-driven-machine-learning-and-parking-tickets-in-nyc-4a3b74cfe496). 1. This communication challenge can create a *gap* between the **product definitions** and **modeling metrics**. 2. Define **Acceptance Criteria**. They must translate business requirements to clear modeling metrics. You can be *invaluable* in translating business goals to model goals. 3. **Business goals must drive modeling goals**. ### Product data fit Checkout this article [here](https://hackernoon.com/the-challenge-of-product-data-fit-92543078551b). 1. The fundamental challenge of finding product/market fit: identifying a market need and building a product that satisfies the need. Why is this so hard? Because the market is a source of _uncertainty_. You don’t know what the exact market need is and if your product really solves it. 2. In a data product — a product that relies on large amounts of data or involve machine learning and AI — you run into the challenge of product/data fit. It arises from the uncertainty about the ability of data to satisfy product needs: - Machine learning models make predictions that aren’t always correct. You don’t know when they’re right and when they’re wrong. - You can’t guarantee in advance the level of performance of a model. Let’s say you can afford up to 10% wrong predictions. You may not be able to build a model that’s 90% accurate. - It’s entirely possible that you don’t have enough data or that there’s not enough signal, but you don’t know that in advance. - Even if the model performs well on training data, its performance in production may vary significantly. 3. So when building a data product, you are tackling at least two sources of uncertainty _simultaneously_: the market and the data. 4. See [Lean Startup Methodology](Lean%20Startup%20Methodology.md) ### Random Notes 1. [Data Leakage](Data%20Leakage.md) --- Date: 20211230 Links to: [Building Machine Learning Powered Applications](Building%20Machine%20Learning%20Powered%20Applications.md) [Startups MOC](Startups%20MOC.md) Tags: References: * []()