How To Make Great ML Products

Machine learning (ML) has been widely recognized as a powerful tool for solving complex problems in various industries, from retail and banking to healthcare and insurance. However, many organizations still view their ML projects and data science teams as siloed domains within their business, leading to potential suboptimal results and limited impact. This is where the concept of the ML product comes into play. In this article, we’ll explore what it means to transform a model into a product and provide a framework for building great ML products that drive value for your users.

What’s Considered a Successful ML Model?

A successful machine learning model is one that is able to accurately make predictions or decisions based on data inputs. In order to be considered successful, the model should be able to perform well on both training data and unseen, real-world data. Additionally, a successful ML model should be able to adapt and improve over time as it is exposed to new data. This ability to adapt to changing inputs is crucial for the long-term success of the model.

The model should also be able to run efficiently in a production environment so that it can be used in real-world applications with minimal computational resources and time. It should also be able to handle various types of inputs and edge cases, and be able to run efficiently in production environments. 

An Example of ML Success
Let’s say that one day, a product manager at an eCommerce company – let’s call it “Coolshirts.net” – knocks on the data science and ML engineering teams’ door, tasking them to boost sales and increase revenue. How? By employing a recommender system to suggest relevant products to their users, based on purchase history, engagement, user profiles, etc. The idea is that a recommendation system will lead to a higher conversion rate, an increase in average checkout size, and improved customer retention and loyalty.

Obviously, the data science and ML teams stand up to the challenge. Step one is to gather historical data, capture them on CSV files, and load them into a python notebook. Then some exploratory data analysis (EDA) fun time, fine-tuning the data, cleaning it up, and we’re ready to go. Let’s start building a model. 

After weeks of experimenting, parameter tuning, and feature engineering, a recommender system is born, and let’s say it achieves an nDCG score of 0.92. Great success! Save to a pickle file, and the job is done. Now the data science and ML teams can move on to the next cool project. Or can they?

As our grumpy cat correctly questions – after working and experimenting for weeks, is it considered a success to have 0.92 nDCG score for our model? If so, is this benchmark agreed upon throughout the company? How has the business defined success? What was the goal? And for whom? These are basic questions that businesses should ask themselves because we’ll quickly learn that data science milestones aren’t enough to declare success. 

Why Data Science Metrics Aren’t Enough 

It’s impossible yet to say that this model will drive more visitors or increase revenue in any way. At this point, all that exists is a recommender system with a high nDCG score, and that’s just not enough to pop the champagne bottles. 

Let’s say a user visiting our eCommerce store, Coolshirts.net, is looking for a white polo t-shirt in the $40-$50 range. They find one they like. Then the recommender system populates another similar polo shirt, but this time the price point is upwards of $700. Would this be considered a good recommendation? Probably not. 

This “bad” recommendation essentially wasted valuable, revenue-driving real estate. And, we can see a similar problem if it is the other way around.

This time the user is looking for an elegant and expensive polo for his cousin’s Bar Mitzvah, something in the high-end range. The recommender system populates another white polo that looks similar to the $720 one, but it only costs $45. No brainer, right? In this case, the recommender did the opposite of its mission to generate the maximum amount of revenue by convincing the customer to spend less than they originally wanted. 

These simple examples perfectly amplify why just creating a high-performing model isn’t a guaranteed path to success. Here is where you need to shift your mindset from building machine learning models to building ML products. 

The Definition of an ML Product 

A software product that facilitates an end goal of a user base through the use of machine learning. 

Shifting Your Mindset From ML Models to ML Products

It’s not just about what’s under the hood of your ML models, but rather how and ML-driven product can bring tangible benefits to its users.

The rise of big data and advancements in AI technology has revolutionized various industries from insurance policy optimization to eCommerce web personalization, credit card fraud detection, song or movie recommendations, and many more.

However, an ML product goes beyond just the technology. It’s about understanding the needs of the end-users and how they will interact with the product Where they touch? What they look at? A great ML product is one that seamlessly integrates into the user’s workflow, delivering valuable insights and outcomes that enhance the user experience. 

In essence, an ML product is a solution that brings together the technical aspects of machine learning and the human elements of user experience, creativity, and design, to create a powerful product that helps users achieve their goals more efficiently.

Now, the moment you’ve been waiting for – Let’s take a look at what it takes to build great ML products. 

The Framework – 7 Principles for Building Great ML Products

1. User-focused – It’s not about YOU. ML Products are built for the benefit of groups or individuals. To strive for success, organizations must understand their users, their challenges, and what the ML product will do for them. In the end, it’s not just about how the model works behind the scenes, but if it’s driving value for the end users. 

2. Goal-driven – We can always be better. Data science and ML engineering teams consistently strive to achieve better results, but it’s not about just tweaking the model for higher accuracy. 

What is the goal, and what goal will provide our users with the most value? The different goals need to be defined, iterated, tracked, and improved to ensure all ML and business stakeholders understand their responsibility and contribution to the end product. 

Clear goals assist in prioritization, facilitate better collaboration, and ensure you’re on the right track to success.

3. Visualize – It’s crucial to envision the world with your ML Product in production. What does it look like? Where is it on the screen? What’s the first touchpoint with the user? Without visualizing the end result, it’s easy to miss some essential parts along the way. This is true for both customer-facing ML products as well as the models running in the back end. 

Draw it out, bring your ML product to life with visualization tools, and gain different perspectives. Is it a nice web application for internal users? Is it a REST API endpoint that another service pulls from? If so, what does that REST API look like? List the endpoints, define the parameters, and visualize model interaction. 

Pro Tip: Use visualization tools like Excalidraw to mock up your vision

4. Align – Different people in the organization have different definitions of success. It’s vital to get all stakeholders aligned on the goals and metrics for their ML Product. Next on the to-do list, write everything down and share it across the organization.

This will act as a single source of truth for perfecting your ML product. Remember to use common language so everyone in the organization can understand – nDCG won’t mean a lot to most product managers and business stakeholders. 

5. Ask Questions – Assume nothing. Instead, make sure to ask questions that extract objective facts and not subjective opinions. This will flush out any confusion about the goals while elevating the discussion about how to perfect the product for the end users. What do your users care about the most? What’s their current workflow? How will they work with your ML Product? You don’t want to waste valuable time and resources on a project that people think is just “nice”.

You need questions that raise real concerns to drive objective facts that lead to a product people will actually use and benefit from. 
 
Pro Tip: “The Mom Test” is an excellent book that shows how to get past the pleasantries and subjective feelings, and drill down to the core essentials to find the real pain points.

6. Measure – Be data-driven. We talked about setting goals and aligning stakeholders, but how do you measure the success of our ML Product? By measuring everything. It’s time to define, track and monitor: Usage metrics, Business KPIs, Data Science metrics, etc. These metrics will tell if you’ve moved the needle, and help quickly derive the next steps for your ML product. 

Once the key metrics have been measured, you should share them across your organization and make relevant stakeholders accountable for each action item that comes up. This will help create a coherent overview of the end goal and inform business stakeholders on the progress of the ML product.

7. Communicate – If a tree falls in the forest, and no one is around, does it still make a sound? Similarly, you can build an awesome model, but if it’s just chilling in your S3 bucket, have you made an impact? This is why communication is key to getting a great ML product off the ground. 
Questions and outcomes should constantly be shared between relevant stakeholders. This way there are fewer to no surprises along the way.

Pro Tip: Schedule a weekly or quarterly meeting with all relevant stakeholders to discuss the progress of the ML product – ask questions, raise concerns, and communicate facts.

🕺The Framework for Great ML Products

Click the link and review the framework. Share it with your team and see how you can start transforming your ML model into a true ML product that makes your users’ lives better. Click here.

Make ML Products, not ML Models

To ensure that your machine learning efforts drive real world impact — creating a successful ML product requires a holistic approach that considers the business goals, the target audience, and the collaboration of different stakeholders, alongside some badass algorithms. By breaking down silos and working together, you can turn machine learning models into valuable ML products that drive real-world results.

Whether you’re approaching a new project, or already in the middle of one, it’s definitely worthwhile to gather all relevant stakeholders, open the framework, and discuss the objectives, metrics, and measurements to make sure that everyone is aligned on the goals. It’s time to move beyond traditional machine learning approaches and embrace a more integrated, product-driven approach. With the right mindset and framework in place, the possibilities for ML products are truly endless.

Seeing is Believing 

Check out my talk at TMLS (Toronto Machine Learning Summit) and feel free to send in questions and comments to learn more.

Liran Hason

Data ScienceHow-To