As organizations attempt to implement Big Data systems, they can be faced with a multitude of challenges. According to National Institute of Science and Technology (NIST) these can be nontechnical and technical.
Nontechnical challenges involve issues surrounding the technical components of a Big Data system, but not considered hardware or software issues. The nontechnical barriers could include issues related to workforce readiness and availability, high cost, too many or lack of regulations, and organizational culture.
Frequently cited nontechnical barriers include lack of stakeholder definition and product agreement, budget, expensive licenses, small return on investment (ROI) in comparison to Big Data project costs, and unclear ROI. Other major concerns are establishing processes to progress from proof-of-concept to production systems and compliance with privacy and other regulations.
Other nontechnical barriers to adoption of Big Data includes, the adoption of access technologies involves nontechnical organizational departments, for legal and security reasons; some silos of data and data access restriction policies are necessary. Poorly defined policies could result in inconsistent metadata standards within individual organizations, which can hinder interoperability. Workforce issues also affect the adoption of Big Data. The lack of practitioners with the ability to handle the complexities of software, and integration issues with existing infrastructure are frequently cited as the most significant difficulties.
Technical challenges encompass issues resulting from the hardware or software, and the interoperability between them, of a Big Data system. Technical barriers arise from various factors, which include functional components of a Big Data system, integration with those functional components, and the security of those components.
Parallel to market demand for self-service analytical application capabilities is a shift from centralized stewardship, toward a decentralized and granular model where user roles have structure for individual access rules. This shift presents barriers for search, including difficulties managing cloud sharing, mobile tech, and notetaking technologies. In addition, the cloud increases the challenges for governance.
Amongst privacy, security, and regulatory compliance concerns, governance appears to produce significant challenges. Often, privacy stakeholders may not need to be concerned with data in enterprise resource planning (ERP) systems, and security stakeholders may not need to be concerned with business intelligence and analytics systems; but governance stakeholders almost always need to be concerned with those systems, as well as with partner and financial data, and infrastructure components (e.g., database management system [DBMS] and networks).
Cloud technologies have facilitated some aspects of Big Data adoption; however, challenges have arisen as the prevalence of cloud grows. Big Data challenges stemming from cloud usage include concerns over liabilities, security, and performance; the significant constraint of physical connectivity bandwidth; and interoperability of mesh, cell, and Internet network components.
- Lack of stakeholder definition and product agreement
- Budget / expensive licenses
- Lack of established processes to go from proof-of-concept to production systems
- Compliance with privacy and regulations
- Inconsistent metadata standards
- Some silos of data and access restriction
- Shifting from centralized stewardship toward decentralized and granular model
- Legacy access methods present tremendous integration and compliance challenges
- Proprietary, patented access methods have been a barrier of contraction of connectors
- Organizational maturity
- Lack of practitioners with the ability to handle the complexity of software
- Integration with existing infrastructure
- Security of systems
- Cloud: concerns over liabilities, security, and performance
- Cloud: connectivity bandwidth is a most significant constraint
- Cloud: Mesh, cell, and Internet network components