A few weeks ago, I reconnected with a former client who recently transitioned into a new role as the Director of Data Operations for a government agency. With decades of experience in data analysis, engineering, and management, this career move marked a significant and well-deserved promotion for her. During our conversation, she posed a challenging question to me: “Why do data projects still fail?” I sensed the concerns she expressed were due to an upcoming high-profile data project. Drawing on my almost three decades of engineering and architecture experience in the private and public sectors, coupled with my understanding of the organization’s culture and mission, I acknowledged the likely need for a data warehouse or data lakehouse solution and committed to providing strategic guidance to mitigate the risks associated with project failure.

From R&D to Operational Challenges: Navigating Data Projects in Government Agencies

Delving into the organization's background, it has evolved over forty years from a small government-funded R&D initiative to a major international agency. The transition from R&D to an operational focus is a common challenge, often leading to organizational struggles, especially when success results in increased reliance on solutions and services. Government constraints, whether budgetary, geographical, or security-related, drive creative solutions through the formation of specialized internal teams. While these teams play crucial roles, they inadvertently give rise to shadow IT and data silos. Perhaps you currently work in or can relate to an organization like this one. Perhaps you are a director facing a similar challenging project and can relate to her concerns.

Uncovering the Reasons Behind Data Project Failures

‘Why do data projects still fail…’ The rhetorical question was raised as though we should have learned from the decade plus of massive failures and published solutions to mitigate risks. There is no lack of recent studies siting current challenges that would lead anyone considering a large-scale data project to be a little nervous, including the need to design for scalability, security, accessibility, and supportability, not to mention the likelihood of needing to quickly adapt and support new requirements (potentially the penalty of success). Often organizations rush into data projects without clear objectives, without well-defined goals that lead to challenges in the implementation, adoption, and perceived value of the service. Data projects can be complicated to implement. Long implementation cycles without tangible value leads to skepticism among stakeholders and erosion of support for the data project initiatives. In my customer’s organization, a lack of adoption would result in more enclaves, data silos, and likely security issues. According to a study cited by Gerrit Kazmaier, Vice President and General Manager for Database, Data Analytics, and Looker at Google Cloud, more than two-thirds of companies say they’re not getting “lasting value” out of their data investments.

Lessons Learned: Addressing Data Quality, Integration, and Infrastructure in Large-Scale Projects

Technical and business issues (or a combination of both) contribute to project failures. Common examples include:

Technical examples:

• Poor data quality: Inaccurate or incomplete data. Leads to incorrect analysis or unreliable results. May also lead to a lack of adoption (business issue), shadow IT to create data solutions.
• Lack of integration: Difficulty or inability to integrate with data sources and systems in the enterprise. This will impede the ability to have a comprehensive view of the data, an essential requirement to attain valuable insights, thereby diminishing the relevance of the solution.
• Inadequate infrastructure: Insufficient hardware or software to support data projects.
• Data Breaches – Security issues such as data breaches will compromise sensitive information, damaging the project’s credibility. This may also lead to the advent of shadow IT and additional security issues.
• Complexity of Technology – Overly complex technology may lead to issues with implementation, maintenance, and adoption.
  

Business examples:

• Misalignment with business goals: The solution may be perceived as irrelevant or a waste of resources.
• Lack of user adoption: With a lack of adoption the perceived value diminishes.
• Inadequate communication: Poor communications about the project, benefits, progress may lead to lack of trust from stakeholders.
• Resistance to change: Organizational culture and resistance can impede implementation and adoption of new data initiatives.
• Unmet expectations: Unrealistic expectations of the data project may be perceived as a failure.

Managing Expectations in Data Initiatives

Reflecting on my own experience and observing both successful and unsuccessful IT projects, it's apparent that disappointment often arises from a lack of communication and the inability to manage expectations. The agreed-upon outcome, or expectations, is crucial for measuring success. Without a clear method to convey value, there is a high likelihood of a perceived failure, affecting both technical and business aspects. If expectations are poorly managed, providers should not be surprised when customers question value or ROI. I recently witnessed a project deemed a "failure" for this very reason. Despite agreeing on project scope, requirements, design, implementation schedule, and testable minimum viable products, the customer claimed there was no ROI, leading to a loss of faith in the provider. While the technical solution was solid from the provider's viewpoint, the customer's perspective deemed it irrelevant.

Overemphasizing tools without addressing organizational and cultural aspects can result in a misalignment between technology capabilities and business requirements. Failure to grasp the business value of a data project and establish both qualitative and quantitative methods for assessing service can lead to confusion and dissatisfaction. A value-oriented approach involves strategic planning that aligns business requirements with technical solutions. While this may sound basic, the reality is that perception can make or break any project, and failures often trace back to a lack of proactive communication and project definition.

Overcoming Data Project Challenges

NMR Consulting’s experience supporting private and public sector customers has taught us that technical capability alone is insufficient to meet our customers' needs. Instead, technical expertise must be applied through a holistic model that integrates all aspects of service delivery: people, process, and technology. With over two decades of experience creating and sustaining solutions with lasting long-term value, we welcome the opportunity to support your organization's IT mission.