By Simon Swan, Head of Field Solutions Strategy, Talend, winner of Best Cloud Data Management Solution at 2021/2022 Cloud Awards

In a cloud or hybrid enterprise setting, data access isn’t a straightforward arrangement. Data often needs to be pulled in near-real time from sources around the business to empower both line-of-business and technical users to contribute to major business goals. However, there’s a palpable tension between using information to power innovation and maintaining compliance and trust. For this reason, shadow IT further contributes to the data compliance crisis.

So how can businesses balance these tensions and provide access to everyone in a safe and governed way? By investing in self-service at scale in the cloud and offering dynamic data health tools that make it simple to use, personalize, and govern data.

Self-service for the reality of distributed work

While the self-service model isn’t new, it’s rapidly become popular thanks to the acceleration of cloud migration and shift to a distributed workforce model. Now, it’s not just early adopters unlocking the full benefits of the cloud, but the average user.

Self-service is becoming the norm in today’s distributed workforce, as company cultures shift to accommodate the need for line-of-business users to quickly access and work with data. After all, when not everyone works in-person, it’s not as simple as just walking over to IT to get questions addressed –– and it can be harder to meet those needs at scale. .

At the same time, employees are more technologically savvy than ever before and feel confident enough to proactively utilize data for their own needs. Years ago, employees were learning Excel, but now, the average non-IT employee might be learning Python as part of their job. That commitment to upskilling has turned into tangible business success. According to McKinsey, 72% of organizations with very effective responses to the pandemic were early investors in new technologies, and 67% also invested more than their industry peers in digitally-related capital expenditures.

As leaders and employees alike enable this trend in digital acceleration, application programming interfaces (API) have become more important than ever for self-service and data sharing. APIs enable applications, software, and systems to access data and interact with other parts of a business’s IT architecture. Thus, API’s ability to reduce complexity and distribute data sharing enables teams to become data-centric by taking an active role in shaping how data is used to benefit the business. This encompasses access to curated data sets and the ability to share and operationalize data across the enterprise.

With total control over data access points, IT can allow role-based access not just to the results gained from data, but also to the data itself. This means data can be integrated into workflows, processes, or even queried and updated through the same interface, without having to expose information beyond what is necessary for that use case.

This self-service model of granting access to data in various shapes and forms is intuitive, as well as easy to implement and extend to a full workforce, in today’s hybrid work environment. No-code and low-code software platforms offer easy points of entry for the average employee to become a citizen data scientist or their own IT resource.

Still, the use of APIs alone doesn’t guarantee that self-service can thrive. Getting improved time to data is important, but the right governance needs to be in place to help self-service reach its full potential.

Personalizing data with prudence

Democratizing data access allows workers to gain more insights that are beneficial to their roles and, ultimately, for business outcomes. However, there’s an inherent tension between democratization and ensuring compliance and trust in that data, which is vital to build confidence in the data sets and the insights derived from them.

To get a sense of how democratized data is being used by citizen data scientists, it’s important to measure:

  • The level of access users have to data for compliance purposes;
  • Its real-time uses across operations;
  • Its place in the IT estate.

The scale of data access and privileges is especially a concern, as self-service permissions can too often spill into creating shadow IT applications and data flows that can put trust and compliance at risk.

Say an IT team gives a sales department room to build its own data-gathering tool to support its CRM software. A steward still needs to be available to ensure that data is gathered, stored, and accessed in a compliant manner. Not doing so creates serious consequences, both legal and financial, for an organization.

However, businesses can ward off the biggest risks of shadow IT by finding the right balance of personalization and compliance. Creating a system for reporting the creation of new applications is one way that governance teams can be kept up to speed, without having to approve every single application or API in use across every department. Another key fix is telemetry and mapping for observability of the full enterprise. That way, IT teams can stay apprised of all new data flows and identify risks without slowing down time to access or value of the data.

Ultimately, democratization and control are the key benefits of self-service. To pull it off, businesses need to find a balance that allows employees to use, define, and deploy data in a way that helps them achieve business goals. It’s a multi-faceted process that requires investment, leadership, teamwork, and culture shifts.

Creating dynamic governance in the cloud

To ensure that compliance policies are flexible enough to give line-of-business users access to the data they need, without breaching governance standards, it’s not enough to “set and forget” control within self-service tools. As data is integrated into workflows or processes, or throughout a user interface, controls must be applied to avoid exposing information beyond the scope of specific use cases. Like those technologies themselves, businesses’ compliance policies need to evolve the way the business evolves: by being ever-adaptable to changing needs.

  • To start, leaders need to invest in technologies that are not only reactive to existing compliance policies, but also grant the ability to observe new patterns of behavior and data access, as well as create new controls and permissions for new patterns.
  • Second, these tools need to operate as fully integrated platforms, with central control and the ability to pull data from across the organization together. For example, with the right technologies and data available to a marketing organization, a company can execute a targeted advertising campaign quickly to give it a competitive advantage. But if this data doesn’t evolve dynamically as patterns of behavior change, that competitive advantage quickly becomes obsolete if the platform isn’t able to adapt often.
  • Business leaders also need to ensure these technologies can scale as their cloud use – and businesses – grow. Data at scale can be difficult to manage without a flexible, adaptable platform such as a data mesh or data fabric, which have emerged as critical tools for operationalizing data. Any self-service tool should fully enable the compliant sharing of data through one-click service. By choosing a consistent platform for managing data, metadata, and teams, technical teams can create a self-service environment that can be scaled and customized to fit the changing needs of both the data and users in the cloud.
  • Finally, every employee within an organization needs to create a data culture driven by trust, accessibility, and responsibility for proper data management. While leaders should advocate for this culture, employees also need to be equipped to build that culture and determine their team’s needs from the bottom-up. When every employee has ownership in their use of the data, they see governance as an asset. They not only embrace compliance, but also offer suggestions for improvements to protect the business while they innovate.

To unlock the full potential of data use in the cloud, empower every user with the tools and training they need to safely and compliantly innovate. Take advantage of the fact that people want to do this themselves, and give them the tools to unlock even more potential in the cloud.