Only 15% of Machine learning projects succeed, or 1 out of every 10 make it to production. 55% of machine learning projects are not even completed. And, only 8% of AI projects are regarded as ‘very’ successful. These are some staggering statistics begging to be noticed by all the organizations out there aiming to launch an effective and bankable ML project.
Despite how hot of a topic AI and Machine Learning are right now, in reality these projects face a lot of challenges. The failure of these projects stem from both technical as well as non-technical issues. Gaps in organizational structure and processes along with insufficient knowledge about appropriate technology required for such projects are only some of the culprits.
Here are some sources who have quoted the failure rate of Machine Learning projects:
- Gartner has estimated that 85% of machine learning projects fail.
- Microsoft executive and ex Snowflake Computing CEO, Bob Muglia said that most customers are not successful with Hadoop
- VentureBeat AI says ‘87% of ML projects never make it to production’
- Tom Davenport, a senior advisor at Deloitte Analytics has said that, ‘the biggest problem in the analysis process is having no idea what you are looking for in the data.’
AI fails are not just limited to smaller projects or companies. In recent years there have been some major high-profile AI failures. Some of the famous failed machine learning projects include IBM’s cancelled project - Watson for Oncology, Apple’s failed Face ID, and Microsoft’s failed AI Chatbot.
Chapo, SVP of data and analytics at GAP, gives an example of how data science projects fail:
In one of their early data science projects, size profiles were created which could determine the range of sizes and their distribution, required to meet demand. The algorithm was handed over to an engineer by the data science team about 4 years ago. It was recorded in Java and implemented. But recently they discovered that the model has been broken for about 3½ years. The problem here was that nobody owned the project, and the data science team was unable to continually iterate on the model.
Here's a checklist to help you accelerate AI Vendor selection
What are the main reasons for Machine Learning projects to fail?
Successful digital transformations do not happen overnight. They take place in small steps, by evaluating every change and letting the business adjust to new technologies. Identifying the usual pitfalls can give you a better view of the problems laying ahead, which your implementation team may encounter. This analysis by us presents a mix of organizational and technical challenges that may hamper the success of your project.
Here are the key reasons why many Machine Learning projects fail along with suggestions on how to avoid these missteps.
1. Plan the machine learning project life-cycle
Planning the machine learning project carefully is the key to its success. Companies need to identify the right use cases, start small and then gradually expand their scope. It should be ensured that the ML project is small and achievable, with every step deliberately planned out.
A well planned, AI driven development lifecycle (SDLC) can affect the development process and success of a machine learning project. The 8 phases of SDLC in machine learning will make sure that the end product meets all the requirements.
An iterative process needs to be followed for machine learning projects which includes the eight stages of ML life-cycle - Data Collection, Data Labelling, Data Visualization, Feature Engineering, Model Training, Model Evaluation, Model Deployment, and Model Monitoring.
The process involves collecting unstructured data, through APIs or manually using a collaboration app, labeling or annotating the unlabeled data to make sure it is clean and analysing the dataset after labeling to make sure the data is accurately labeled. The next steps involve identifying the features best suited to train the model, then training the model using this featureset, and evaluating the model by testing it to see its accuracy. After training, the model is deployed into production and then finally it’s monitored to ensure that it runs accurately and gives expected results.
Here every development lifecycle represents an iterative workflow that can be used in machine learning projects to bring out the best outcomes.
2. Data problems
Data is often described as the most critical requirement in machine learning projects, and even then very few companies address the problems related to it. An ML project can be stalled due to data problems, and it can have a major impact on the success of a project.
Integrating cloud services into existing systems has been a problem for most companies, resulting in the creation of siloed data. The data is redundantly stored and managed. Another problem is data being scattered across various teams, due to which the consolidation of data becomes difficult.
Having a large amount of data is essential for a machine learning project, however it can also turn into a data swamp. These are useless, difficult to manage data. Machine learning projects are slow to start, as data needs to be collected and labeled from different sources and cleaned.
While gathering data you may end up collecting all kinds of data - useful as well as useless. There is also a chance of not collecting enough data. The solution is identifying the right kind of data which helps to make the correct decisions and improvements over time. Keeping data organized includes defining the goal, defining who is in charge of data collection, and keeping it updated as frequently as possible.
3. Lack of right skill set
It is difficult to come by people skilled in machine learning technology. The skill shortage is more apparent with smaller companies that have lower budget and are bringing technology in their processes for the first time. Most studies have revealed that the challenges are human rather than technical.
A successful machine learning project would require to employ a team that possesses a wide range of skill sets. They may include data scientists for data modeling and model performance, AI product managers, and data engineers for acquiring the correct data and production deployment. Apart from these there should be subject matter experts who bring in knowledge and focus to the project. A well rounded team ensures that mistakes are identified and their collective talent works towards making the project successful.
4. Culture of the organization
Good engagement of managers and business unit leaders is required for a successful project. It’s important to have an open line of communication. Everyone should be involved in the data strategy process, especially those who are directly dealing with the data on an everyday basis. Not consulting them may result in collection of unnecessary or incorrect data that may just slow down the process. It is important to involve critical employees in strategy meetings. They will mention concerns, ask questions and give opinions that may be important for the success of a machine learning project.
5. Excessive focus on technology
The model should not go in search of a problem, it should be the other way around. The machine learning model should be driven by the business problem/use case that needs to be solved. There are enough critically important problems to be solved, and opportunities to be analyzed. The model could help solve these problems that are already present. If you aim to apply technology to solve a problem that businesses don’t care about, you are aiming at the wrong KPI. Experimenting with methods and technologies, and trying to develop an ML model that is very advanced but doesn’t serve any practical purpose is unnecessary, and simply a waste of time and resources.
6. Maintaining the model: Avoid and reduce Drift and Bias
Machine learning models start to decline as the ground truth changes. The model needs to be monitored and maintained in order to provide consistent quality output. A Concept Drift may appear in the model when there is a change in data overtime, meaning the relation between input and output data changes due to unforeseen circumstances. Concept is the quantity that is predicted, which may change. This happens because the predictions are being made by a model which is trained on older data that is no longer valid.
The solution to this is periodically updating the model with latest and updated data. The update can be done monthly or yearly. Continuous training should be part of the ML workflow. This will result in more accurate prediction over a long period.
Models are developed by humans and therefore they reflect human biases. Machine learning models will reflect the biases of designers, data scientists, and the engineers who have worked on the model. There are few approaches to mitigating these biases. Maintaining a diverse team in demographics as well as skillset is an important factor. This makes the issues around unwanted bias more noticeable and they can be removed before the model is released into production. There are also a large number of debiasing and technical awareness tools that can support a team towards avoiding and removing machine learning bias.
7. Failure in promoting experiment to production
Building a machine learning model is one thing but getting that into production is a totally different ballgame. A lot of effort may go into building a good model but if the deployment is not done correctly, the machine learning project is sure to fail. Figuring out how to deploy the model into production is hard but following these steps with extreme care can ensure success:
- Document the model release, with its training data, objective, featureset used for the model, algorithms used, and the Inference API documentation. Proper documentation will help in the model’s development process, further iterations, debugging, and knowledge transfer.
- Evaluate the model for the quality and accuracy of its predictions.
- Integration of model inference or prediction within the product/service, which mainly includes Online prediction, Batch prediction, and Edge prediction.
- Packaging the model correctly based on the type of inference and ML framework. For example using Python function, deploying to Docker for serving as online inference, or by using Apache Spark.
- Ensure capacity planning to handle inference requests.
- Monitor and evaluate the performance of the model by comparing it to the true data. This step is essential to find out errors and latency, and to make sure that the model continues to give the desired level of performance.
- Model versioning for easy model deployment and rollback to production.
- Measuring the outcome to make sure that the system keeps performing well.
8. Other challenges
There are various challenges in promoting machine learning projects from experiment to production which leads to project failure. The initial models cannot scale or are too experimental to run consistently in production. Sometimes complex models are created by data scientists, when a simpler one can be as good. People have a tendency to complicate the problem statement, as a result the solutions are also complicated. Organizations should hire relevant MLOps engineers possessing good knowledge in this domain who can take care of scaling for production and deployments confidently.
Another issue faced is that the model may not meet the business requirements or may be too weak to respond to change in supporting data. When the project is finally pushed into the market there is usually minimal adoption. Even if the solution is able to meet a specific challenge, the user experience is not optimum. In response to this some organizations hire product marketers to market their concepts directly to customers.
Given the uncertainty of the result and success, many machine learning projects die even before they can begin. Also the dominance of companies like Amazon, Microsoft, and Google discourage smaller enterprises from attempting the impossible.
To summarize, for machine learning projects to succeed, the methodology needs to include planning the machine learning project by identifying the right use cases, implementing a well-planned AI driven software development lifecycle, addressing all issues related to data carefully, employing the right set of people for the machine learning project, and encouraging good engagement from them. And, most importantly maintaining and training the model well, and being prepared for the challenges faced during deployment of the machine learning model. Failure to take these strategies into consideration may just result in you becoming another one of the statistics mentioned above.
Skyl.ai is an end-to-end machine learning platform, incorporating all the features of an ideal ML automation tool. It helps in efficient data management with great accuracy and quality. By providing early visibility at each and every step, it assists in taking better decisions about data and detecting biases early on in the process. With the provision of templatized workflows, it immensely reduces the time, efforts and costs required to build ML solutions, from experimenting with machine learning models to taking them to production. Skyl also comprises a highly scalable automatically managed infrastructure while allowing uncompromised collaboration among team members and stakeholders, with effective collaboration tools.
Want to learn more? Visit us and try out our risk free trial today!