Provision and Collection of Safety Evidence: A Systematic Literature Review

Lilandra Maria de Oliveira; Luiz Eduardo Galvão Martins; Johnny Cardoso Marques

doi:10.22456/2175-2745.126544

Authors

Lilandra Maria de Oliveira Federal University of Sao Paulo https://orcid.org/0000-0003-1428-2830
Luiz Eduardo Galvão Martins Federal University of Sao Paulo https://orcid.org/0000-0002-7266-5840
Johnny Cardoso Marques Technological Institute of Aeronautics https://orcid.org/0000-0002-1551-435X

DOI:

https://doi.org/10.22456/2175-2745.126544

Keywords:

safety evidence collection, safety evidence model, safety-critical systems certification, systematic literature review

Abstract

Safety-Critical Systems (SCS) are becoming more and more present in modern societies’ daily lives, increasing people’s dependence on them. Current SCS are firmly based on computational technology; possible failures in the operation of these systems can lead to accidents and endanger human life, as well as damage the environment and property. SCS are present in many areas such as avionics, automotive systems, industrial plants (chemical, oil & gas, and nuclear), medical devices, railroad control, defense, and aerospace systems. Companies that develop SCS must present evidence of their safety to obtain certification and authorization. This paper presents a Systematic Literature Review (SLR) to investigate processes, tools, and techniques for collecting and managing safety evidence in SCS. The authors conducted this SLR according to the guidelines proposed by Kitchenham and Charters. The SLR comprises seven (7) research questions that investigate essential aspects of collecting and managing safety evidence. The primary studies analyzed in this SLR were selected based on a search string applied into four data sources: ACM, IEEE Xplore, SpringerLink, and ScienceDirect. Data extraction considered (fifty-one) 51 primary studies. The authors identified eleven (11) different approaches covering processes, tools, and techniques for collecting and managing safety evidence. Despite other SLR works conducted about safety evidence, none of them focused on the details related to safety evidence collection. We found that very few approaches focused specifically on the process of collecting safety evidence.

Downloads

Download data is not yet available.

References

CARDOSO, M. J. S. M. Modelo de processo de testes para sistemas de software críticos. Dissertação (Mestrado) — Pontifícia Universidade Católica de Minas Gerais, Belo Horizonte. p. 171. 2010.

NAIR, S. et al. Evidence management for compliance of critical systems with safety standards: A survey on the state of practice. Information and Software Technology, London, v. 60, p. 1–15, apr. 2015.

MARTINS, L. E. G. ao. Desenvolvimento de um modelo de processo baseado em stamp para coleta e gerenciamento de evidências de segurança de sistemas críticos. (em preparação.). 2019.

MARTINS, L. E. G. ao; GORSCHEK, T. A process model based on stamp for collecting and management of safety evidence. In: SAFECOMP2020 - Position Papers International Conference on Computer Safety, Reliability and Security. Lisbon, Portugal: HAL, 2020.

PANESAR-WALAWEGE, R. K.; SABETZADEH, M.; BRIAND, L. Using model-driven engineering for managing safety evidence: Challenges, vision and experience. In: 2011 First International Workshop on Software Certification. New York: IEEE, 2011. p. 7–12.

KELLY, T. P. Arguing safety - A systematic approach to managing safety cases. 341 p. Tese (Doutorado) — University of York, York, 1998. Disponível em: ⟨https://www-users.cs.york.ac.uk/∼tpk/tpkthesis.pdf⟩.

PANESAR-WALAWEGE, R. K.; SABETZADEH, M.; BRIAND, L. A model-driven engineering approach to support the verification of compliance to safety standards. In: 22nd International Symposium on Software Reliability Engineering. New York: IEEE, 2011. p. 30–39.

WALKINSHAW, N. Software inspections, code reviews, and safety arguments. In: Software Quality Assurance: Consistency in the Face of Complexity and Change. Cham, Switzerland: Springer International Publishing, 2017. p. 127–140.

HUBER, M. et al. Roadblocks on the highway to secure cars: An exploratory survey on the current safety and security practice of the automotive industry. Springer International Publishing, Cham, Switzerland, p. 157–171, 2018.

NAIR, S. Evidence management for evolutionary safety assurance and certification. In: 21st IEEE International Requirements Engineering Conference (RE). New York: IEEE, 2013. p. 385–388.

NAIR, S. et al. An extended systematic literature review on provision of evidence for safety certification. Information and Software Technology, London, v. 56, n. 7, p. 689–717, jul. 2014.

BATE, I.; BURNS, A. An integrated approach to scheduling in safety-critical embedded control systems. Real-Time Systems, Netherlands, v. 25, n. 1, p. 5–37, jul. 2003.

LEVESON, N. G. Engineering a Safer World: Systems Thinking Applied to Safety. Cambridge, USA: Massachusetts Institute of Technology, 2011.

SHBOUL, B. A.; PETRIU, D. C. Pattern-based transformation of sysml models into fault tree models. In: Proceedings of the 29th Annual International Conference on Computer Science and Software Engineering. Riverton, USA: IBM Corp., 2019. p. 214–223.

WOLSCHKE, C. et al. Industrial perspective on reuse of safety artifacts in software product lines. In: Proceedings of the 23rd International Systems and Software Product Line Conference - Volume A. New York, NY, USA: Association for Computing Machinery, 2019. p. 143–154.

LUO, Y.; SABERI, A. K.; BRAND, M. v. den. Safety-driven development and iso 26262. In: DAJSUREN, Y.; BRAND, M. van den (Ed.). Automotive Systems and Software Engineering: State of the Art and Future Trends. Cham, Switzerland: Springer International Publishing, 2019. p. 225–254.

MACGREGOR, J.; BURTON, S. Challenges in assuring highly complex, high volume safety-critical software. In: SAFECOMP 2018: Computer Safety, Reliability, and Security. Cham, Switzerland: Springer International Publishing, 2018. p. 252–264.

VARA, J. L. de la et al. An industrial survey of safety evidence change impact analysis practice. IEEE Transactions on Software Engineering, Piscataway, USA, v. 42, n. 12, p. 1095–1117, apr. 2016.

SABETZADEH, M. et al. Using sysml for modeling of safety-critical software-hardware interfaces: Guidelines and industry experience. In: 2011 IEEE 13th International Symposium on High-Assurance Systems Engineering. New York: IEEE, 2011. p. 193–201.

WU, W.; KELLY, T. Towards evidence-based architectural design for safety-critical software applications. In: LEMOS, R. de; GACEK, C.; ROMANOVSKY, A. (Ed.). Architecting Dependable Systems IV. Berlin, Heidelberg: Springer Berlin Heidelberg, 2007. p. 383–408.

JARADAT, O.; BATE, I. Systematic maintenance of safety cases to reduce risk. In: 35th International Conference on Computer Safety, Reliability, and Security. Cham, Switzerland: Springer International Publishing, 2016. p. 17–29.

BRESSAN, L. et al. A systematic process for applying the chess methodology in the creation of certifiable evidence. In: 2018 14th European Dependable Computing Conference (EDCC). New York: IEEE, 2018. p. 49–56.

PANESAR-WALAWEGE, R. K. et al. Cresco: Construction of evidence repositories for managing standards compliance. In: ER 2011: Advances in Conceptual Modeling. Recent Developments and New Directions. Berlin, Heidelberg: Springer Berlin Heidelberg, 2011. p. 338–342.

SILVA, N.; VIEIRA, M. Certification of embedded systems: Quantitative analysis and irrefutable evidences. In: 2013 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW). New York: IEEE, 2013. p. 15–16.

REMPEL, P.; M ̈aDER, P. Continuous assessment of software traceability. In: Proceedings of the 38th International Conference on Software Engineering Companion. New York, USA: Association for Computing Machinery, 2016. p. 747–748.

HAWKINS, R. et al. Assurance cases and prescriptive software safety certification: A comparative study. Safety Science, Amsterdam, v. 59, p. 55–71, nov. 2013.

LIN, C.-L.; SHEN, W.; DRAGER, S. A framework to support generation and maintenance of an assurance case. In: 2016 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW). New York: IEEE, 2016. p. 21–24.

̈OFIG, K. H.; ZELLER, M.; HEILMANN, R. Alfred: A methodology to enable component fault trees for layered architectures. In: 41st Euromicro Conference on Software Engineering and Advanced Applications. New York: IEEE, 2015. p. 167–176.

VARA, J. L. de la; PANESAR-WALAWEGE, R. K. Safetymet: A metamodel for safety standards. In: MODELS 2013: Model-Driven Engineering Languages and Systems. Berlin, Heidelberg: Springer Berlin Heidelberg, 2013. p. 69–86.

NAIR, S. et al. Classification, structuring, and assessment of evidence for safety – a systematic literature review. In: 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation. New York: IEEE, 2013. p. 94–103.

ALMENDRA, C.; SILVA, C.; VILELA, J. Incremental development of safety cases: A mapping study. In: Proceedings of the 34th Brazilian Symposium on Software Engineering. New York, USA: Association for Computing Machinery, 2020. (SBES’20), p. 538–547.

NAIR, S. et al. Safety evidence traceability: Problem analysis and model. In: SALINESI, C.; WEERD, I. van de (Ed.). Requirements Engineering: Foundation for Software Quality: 20th International Working Conference, REFSQ. Cham, Switzerland: Springer International Publishing, 2014. p. 309–324.

VARA, J. L. de la et al. Towards a model-based evolutionary chain of evidence for compliance with safety standards. In: Computer Safety, Reliability, and Security: 31st International Conference, Safecomp 2012. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012. p. 64–78.

LARRUCEA, X. et al. Analyzing a ros based architecture for its cross reuse in iso26262 settings. In: MEDI 2018: New Trends in Model and Data Engineering. Cham, Switzerland: Springer International Publishing, 2018. p. 167–180.

KITCHENHAM, B.; CHARTERS, S. Guidelines for performing Systematic Literature Reviews in Software Engineering. Durham, UK, 2007. 65 p.

DENNEY, E.; PAI, G. Evidence arguments for using formal methods in software certification. In: IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW). New York: IEEE, 2013. p. 375–380.

GROZA, A.; MARC, N. Consistency checking of safety arguments in the goal structuring notation standard. In: 10th International Conference on Intelligent Computer Communication and Processing (ICCP). New York: IEEE, 2014. p. 59–66.

MURAM, F. U.; GALLINA, B.; RODR ́IGUEZ, L. G. Preventing omission of key evidence fallacy in process-based argumentations. In: 11th International Conference on the Quality of Information and Communications Technology (QUATIC). New York: IEEE, 2018. p. 65–73.

ŠLJIVO, I. et al. A method to generate reusable safety case argument-fragments from compositional safety analysis. Journal of Systems and Software, New York, v. 131, p. 570–590, sep. 2017.

PANESAR-WALAWEGE, R. K.; SABETZADEH, M.; BRIAND, L. Using uml profiles for sector-specific tailoring of safety evidence information. In: Conceptual Modeling – ER 2011. Berlin, Heidelberg: Springer Berlin Heidelberg, 2011. p. 362–378.

CÂRLAN, C. et al. Explicitcase: Integrated model-based development of system and safety cases. In: The 36th International Conference on Computer Safety, Reliability and Security. Cham, Switzerland: Springer International Publishing, 2017. p. 52–63.

SUN, L.; SILVA, N.; KELLY, T. Rethinking of strategy for safety argument development. In: The 33rd International Conference on Computer Safety, Reliability and Security. Cham, Switzerland: Springer International Publishing, 2014. p. 384–395.

MARTINS, L. E. G. a.; OLIVEIRA, T. de. A case study using a protocol to derive safety functional requirements from fault tree analysis. In: 2014 IEEE 22nd International Requirements Engineering Conference (RE). New York: IEEE, 2014. p. 412–419.

ARANDA, A.; DIESTE, O.; JURISTO, N. Evidence of the presence of bias in subjective metrics: Analysis within a family of experiments. In: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering. New York, NY, USA: Association for Computing Machinery, 2014. p. 1–4.

SOBRINHO, A. et al. Formal modeling of biomedical signal acquisition systems: source of evidence for certification. Software & Systems Modeling, Heidelberg, v. 18, n. 2, p. 1467–1485, apr. 2019.

DECHEV, D.; STROUSTRUP, B. Model-based product-oriented certification. In: 16th Annual IEEE International Conference and Workshop on the Engineering of Computer Based Systems. New York: IEEE, 2009. p. 295–304.

ROMANSKI, G. Combined safety and security certification. In: 7th IET International Conference on System Safety, incorporating the Cyber Security Conference. New York: IEEE, 2012. p. 1–5.

VARA, J. L. de la et al. An analysis of safety evidence management with the structured assurance case metamodel. Computer Standards & Interfaces, Amsterdam, v. 50, p. 179–198, feb. 2017.

LIN, H. et al. A systematic approach for safety evidence collection in the safety-critical domain. In: 2015 Annual IEEE Systems Conference (SysCon) Proceedings. New York, USA: IEEE, 2015. p. 194–199.

FALESSI, D. et al. Planning for safety standards compliance: A model-based tool-supported approach. IEEE Software, Los Alamitos, USA, v. 29, n. 3, p. 64–70, may 2012.

PANESAR-WALAWEGE, R. K. et al. Characterizing the chain of evidence for software safety cases: A conceptual model based on the iec 61508 standard. In: Third International Conference on Software Testing, Verification and Validation. New York: IEEE, 2010. p. 335–344.

LUO, Y. et al. From conceptual models to safety assurance. In: YU, E. et al. (Ed.). ER 2014: Conceptual Modeling. Cham, Switzerland: Springer International Publishing, 2014. p. 195–208.

GANNOUS, A.; ANDREWS, A.; GALLINA, B. Toward a systematic and safety evidence productive verification approach for safety-critical systems. In: 2018 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW). New York: IEEE, 2018. p. 329–336.

LOVRI ́C, T. Requirements for the certification of safety critical railway systems. In: WIECZOREK, M.; MEYERHOFF, D. (Ed.). Software Quality: State of the Art in Management, Testing, and Tools. Berlin, Heidelberg: Springer Berlin Heidelberg, 2001. p. 225–240.

BERTIERI, D. et al. Development and validation of a safe communication protocol compliant to railway standards. Journal of the Brazilian Computer Society, Heidelberg, Germany, v. 27, n. 1, p. 1–26, mar. 2021.

ROMANSKI, G. Certification of an operating system as a reusable component. In: Proceedings. The 21st Digital Avionics Systems Conference. New York: IEEE, 2002. p. 5D3–5D3.