Penetration Testing in Big Data Frameworks: A Systematic Review of Security Assessment for Hadoop and Spark

Luís Barata; Francisco Martins; Eurico Lopes

doi:10.22456/2175-2745.151182

Authors

Luís Barata Instituto Politécnico de Castelo Branco (IPCB) https://orcid.org/0000-0002-6471-4681
Francisco Martins Instituto Politécnico de Castelo Branco (IPCB) https://orcid.org/0009-0007-2015-1307
Eurico Lopes Instituto Politécnico de Castelo Branco (IPCB) https://orcid.org/0000-0002-1818-8203

DOI:

https://doi.org/10.22456/2175-2745.151182

Keywords:

hadoop, spark, vulnerabilities, big data, intrusion

Abstract

Big Data systems are expanding rapidly, and frameworks like Hadoop and Spark are now central to that growth. Yet this expansion also raises new security challenges, particularly regarding how vulnerabilities are identified, assessed, and mitigated in complex, distributed environments. While the existing literature predominantly focuses on defensive mechanisms, systematic evidence on offensive approaches such as penetration testing remains limited. This paper presents a systematic literature review of security assessment practices for Hadoop and Spark between 2015 and 2025, with an emphasis on penetration-testing techniques in Big Data frameworks. Our search across major digital libraries retrieved 1578 records. After title and abstract screening, 46 articles were selected. Of these, 20 were considered relevant and read in full; among these, only four explicitly applied penetration testing to Hadoop or Spark deployments. The review reveals a shortage of realistic testing environments and standardized metrics for evaluating the effectiveness of mitigation strategies, alongside a strong reliance on generic security tools rather than specialized offensive frameworks. These findings underscore the need for dedicated, framework-aware penetration-testing solutions for Hadoop and Spark, reproducible testing protocols, and tighter integration between offensive assessments and proactive mitigation practices.

Downloads

Download data is not yet available.

References

[1] BHATHAL, G.; SINGH, A. Big data: Hadoop framework vulnerabilities, security issues and attacks. Array, Elsevier, v. 1, 2019. Disponível em: ⟨https://www.sciencedirect.com/science/article/pii/S2590005619300025⟩.

[2] SPIVEY, B.; ECHEVERRIA, J. Hadoop Security: Protecting your big data platform. O’Reilly Media, 2015. Disponível em: ⟨https://books.google.com/books?id=VXEJCgAAQBAJ⟩.

[3] XU, C.; LI, J. Design of intelligent software security system based on spark big data computing. Wireless Personal Communications, Springer, 2025. Disponível em: ⟨https://link.springer.com/article/10.1007/s11277-024-11015-4⟩.

[4] KAUSHIK, P.; RATHORE, S.; RATHORE, R. Big data-powered analytics for fortifying virtualized infrastructure security in the cloud. Springer, 2023. Disponível em: ⟨https://link.springer.com/chapter/10.1007/978-3-031-80778-7_12⟩.

[5] HART, M.; DAVE, R.; RICHARDSON, E. Next-generation intrusion detection and prevention system performance in distributed big data network security architectures. International Journal of Emerging Technology and Advanced Engineering, 2023. Disponível em: ⟨https://search.proquest.com/openview/2b773d51151376afb674268b2d505ff8⟩.

[6] MA, L. Research on vulnerability exploitation and detection technology based on big data analysis. IEEE, 2021. Disponível em: ⟨https://ieeexplore.ieee.org/document/9699917/⟩.

[7] RAHMAN, M.; SHAHRIAR, H. Clustering enabled robust intrusion detection system for big data using hadoop–pyspark. In: IEEE Conference on Improving Quality of Life using AI. [s.n.], 2023. Disponível em: ⟨https://ieeexplore.ieee.org/document/10374747/⟩.

[8] BAGUI, S. et al. Introducing uwf-zeekdata22: A comprehensive network traffic dataset based on the mitre att&ck framework. Data, MDPI, v. 8, n. 1, 2023. Disponível em: ⟨https://www.mdpi.com/2306-5729/8/1/18⟩.

[9] BARATA, L.; MARTINS, F.; LOPES, E. Security Assessment in Big Data Frameworks: Systematic Review Dataset. 2025. Zenodo dataset. Disponível em: ⟨https://doi.org/10.5281/zenodo.19686185⟩.

[10] AHMAD, S.; YASIN, A.; SHAFI, Q. Ddos attacks analysis in bigdata (hadoop) environment. In: Proceedings of the 2018 15th International Bhurban Conference on Applied Sciences & Technology (IBCAST). Islamabad, Pakistan: IEEE, 2018. p. 495–501. Disponível em: ⟨https://ieeexplore.ieee.org/document/8312270⟩.

[11] SAMET, R.; AYDIN, A.; TOY, F. Big data security problem based on hadoop framework. In: Proceedings of the 2019 4th International Conference on Computer Science and Engineering (UBMK). Samsun, Turkey: IEEE, 2019. p. 525–530. Disponível em: ⟨https://ieeexplore.ieee.org/document/8907074⟩.

[12] TIAN, Y. et al. Non-authentication based checkpoint fault-tolerant vulnerability in spark streaming. In: Proceedings of the 2018 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid). IEEE, 2018. p. 783–786. Disponível em: ⟨https://ieeexplore.ieee.org/document/8538745⟩.

[13] KANG, J.; LEE, H.; KIM, D. A cybersecurity testbed for evaluating penetration attacks in hadoop ecosystems. In: 2021 IEEE International Conference on Big Data Security on Cloud. IEEE, 2021. p. 57–64. Disponível em: ⟨https://ieeexplore.ieee.org/document/9441082⟩.

[14] LATIF, A.; ABBAS, H.; KHAN, A. Adversarial threat modeling and simulation for hadoop security assessment. Journal of Information Security and Applications, Elsevier, v. 54, p. 102526, 2020. Disponível em: ⟨https://www.sciencedirect.com/science/article/pii/S221421262030059X⟩.

[15] SIDDIQUI, M.; UZMI, S. Offensive security assessment of apache spark: A penetration testing approach. In: Proceedings of the 2020 International Conference on Cyberworlds. IEEE, 2020. p. 45–52. Disponível em: ⟨https://ieeexplore.ieee.org/document/9287311⟩.

[16] MAHMOOD, K.; ZHOU, Z.; LYU, M. R. Sandbox-based vulnerability simulation for apache spark security evaluation. Future Generation Computer Systems, Elsevier, v. 127, p. 123–137, 2022. Disponível em: ⟨https://doi.org/10.1016/j.future.2021.09.020⟩.