Tackling the Challenges of Big Data for Federal Agencies

With the amount of global data growing exponentially through technologies such as social media, cloud computing, and the Internet of Things (IoT) devices, Big Data is not a conceptual buzz word anymore; it is a practical challenge every organization is facing.

Big Data is characterized through volume (consisting of enormous quantities of data), velocity (created in real-time), and variety (being structured, semi-structured, and unstructured). There is an ever-increasing need for data sharing.

Organizations must sift through this data to identify usable information that needs to be securely stored, shared, and accessed internally and externally. Federal agencies have additional security barriers with data residing in multiple domains, that must also be considered for data duplication, integrity, authority, and real time discovery.

Despite tremendous benefits achieved from these large data sets, agencies run into a number of challenges tackling big data. In this blog, we discuss our perspective of some of these challenges:

Data Storage Challenges: We begin with the most fundamental aspect of big data, which is size- and therefore storage is a critical issue. Traditionally, big data storage is handled with Data Lake, and Data Warehousing solutions and more recently storage in the cloud as an economical option.

However, with the ever-increasing volume of data, even these options are quickly becoming expensive, especially due to the data dump of every single byte prior to identifying its purpose. Constant streaming data from the IoT devices and social media will overflow the storage causing continuous expansion of additional storage and performance degradation. IoT platforms provide real-time data analytics to help decide whether to store the incoming data based on predefined rules, and social media data is too big to store without good reasons.

Data Discovery Challenges: Regardless of storage choices and challenges with data organization, the purpose of incoming data is not immediately identified. With multiple security domains, data is often simply duplicated further adding to the problem.

Discovering data, which includes the process of uncovering relevant insights, is a challenge as data is all over the place. Applications may not be using the most relevant data for the right business case. As a result, there is a lack of using right data at the right time causing inefficiencies. Data integrity and authenticity issues occur with duplication and false data on the rise.

Data Sharing Challenges: Data sharing is becoming a necessity amongst organizations due to globalization and global supply chain. Government agencies are required to share data with each other to provide citizen services. DoD, Intelligence, and Law Enforcement agencies are seeking to share data that can enhance their mission to fight against terrorism and crime.

But how do they share their data securely with each other without sacrificing ownership, authority, and maintain privacy? There are several challenges associated with the release of data concerning data generation, administration, and transmission. Data ownership comes with actively maintaining accountability of data and managing rights of the data.

Data privacy, on the other hand, is a heavy responsibility on organizations that store and share people’s data. The cultural barrier is alleged to be the biggest obstacle to sharing data due to fear and uncertainty of data quality.

Data Access Challenges: Because of the massive volume of data, it is virtually impossible to identify proper access control for the users and applications within and outside the agencies for every piece of data. Data marts, data lakes, and data warehouses provide tools to handle data access, but they are only as good as they are used.

DBAs, Data Architects, and Data Stewards are constantly facing challenges to provide proper access control due to the sheer volume of data. There are often granular data controls required with cell-level security that puts further restrictions on the data. Federal agencies are looking for zero trust data access control to protect their data from bad actors.

There is an appetite for data lineage that can help maintain data authenticity by tracing the data origin and identifying errors back to the root cause in a data analytics process. Without sufficient data access policies, security is compromised, and agencies become targets for data breaches.

So, How to Tackle These Challenges? While it is easier to list out some of the prominent challenges dealing with big data, tackling these challenges is not as easy as one can imagine. Agencies must take a comprehensive look at the data strategy on how to handle big data as it keeps getting bigger every day.

Governance is key to establishing the foundation of the organizational data strategy. There needs to be an enterprise-wide data strategy to handle data ingestion policies, data storage solution and all the sources data is coming through. Agencies need to consider a scalable and flexible data strategy that can handle this massive data instead of stove piped solutions that only adds to the problem.

Fortunately, there are tools and cloud native solutions that can solve these problems but only with proper vision and strategy along with sound governance, which must start with people.

• Data Virtualization can help tackle the data storage challenge by maintaining connectivity and retrieving real-time data only when needed.

• Data catalog can assist in providing business glossary and metadata to automate data processing. Tagging can be used for proper access control if used effectively.

• Data analytics and Visualization tools can support discovery challenges.

• Blockchain can be leveraged for data sharing with immutable footprint and data lineage history.

As technology continues to evolve with the latest tools and features, so do business needs. Unfortunately, there is no one-size-fits-all solution. It is up to the organization leaders to provide a top-down data strategy that considers all the above challenges. Agencies need to establish an enterprise-wide governance model to fit their business and mission objectives, and let the right people use the right tools to tackle these challenges.

Here at Makpar, we have addressed these challenges for our clients by utilizing the right tools and technologies, and by working closely with the stakeholders. What we discovered is: it is a journey, not a race to a quick solution.

Makpar has over a decade of experience providing high-level strategic and project management services, which are ideal for bringing a comprehensive data approach to our clients.

Please contact us here for more information about how Makpar can develop and implement data management strategies that advance mission success for your agency.