Developing ModelOps sophistication: choosing the right level
My previous articles talked about why organisations should choose ModelOps, and how to implement ModelOps using a holistic approach covering process, people and technology. This article explains the levels of sophistication available when implementing ModelOps, and how to choose the right level.
ModelOps is NOT a one-size-fits-all approach. It is important to identify the appropriate level of sophistication based on both the organisation’s current readiness, and its current and future business needs.
Three levels of sophistication
We have identified three main levels of ModelOps sophistication. However, in practice there is a fourth: many organisations may not yet even be at Level 1, especially if they do not use any predictive analytics. These organisations should therefore target Level 1 as a first step, even if their business needs suggest that a higher level would be appropriate. Other organisations will have higher levels of analytics skills and maturity, and can start from a more advanced level, if their business need demands it.
Level 1: Handcrafted
This is the first level. It is not very sophisticated and is perfect for organisations that are new to ModelOps and looking to experiment and learn before they consider including more automation in the process. It can also be the right solution when there is no real need to scale analytics. This can be because the business does not require many models or can accept longer deployment cycles. It can also be appropriate when organizations do not have the culture to support the agile methodology and effective collaboration across departments that is needed to support ModelOps.
In Level 1, every step of the process is manual, ad-hoc and usually delivered by one team. This is generally the data science team, with very little input from the IT/Ops functions. When deployment is required, the model is executed manually, and results are delivered manually to end points or users. However, this is still effective model operationalisation, because the model is being put into production at an end point and is working to achieve a business goal. Feedback is also provided to the model development process.
Level 2: Automated
At Level 2, automation is introduced. This level is appropriate for organisations who have some or all of the following criteria:
· Time sensitivity for responding to business conditions, which demands more efficient model deployment;
· Limited bandwidth to create, manage and deploy the models required to meet business goals; and
· A drive to challenge their culture, increase efficiency and make changes.
In Level 2, chosen steps are automated, usually scripted, and models are delivered to an end point such as an internal or external application or to support a decision. Responsibilities are distributed between different teams and model governance is starting to be required to ensure models are always up-to-date.
Level 3: Industrialised
The highest level of sophistication and efficiency is Level 3, where the process is industrialised. This level applies to organisations who have at least some of the following criteria:
· A well-established DevOps culture with agile/lean methodologies;
· A large number of models, probably over 100, with defined business indicators to guide automation processes; and
· A desire to automate where necessary, understanding the need for human intervention and checks during the process.
In Level 3, ModelOps is based on a standardized and enterprise-wide process model that encourages collaboration. Proactive fault handling is key. A culture of no-blame and failing fast are supported by automated and flexible software for fast iterations of models supporting business-critical applications. Automated integration and continual model governance mean new models are developed and deployed once any model monitoring business-led KPIs are triggered.
Conclusion
ModelOps is not for everyone. It provides benefits of a faster time to value, better and more justifiable business outcomes and the ability to scale analytics. However, some organisations need to increase their level of analytics maturity before they can really take advantage of a ModelOps framework. If ModelOps is the right approach, however, the level of sophistication required will depend on both the business objective, and the organisation’s level of analytical maturity.
If you have any questions, please post them in the comments or reach out to me directly. I would love to have a conversation about ModelOps.
