Deep Learning-Based Segmentation of Nanomaterial Images Acquired via Scanning Electron Microscopy

Ivânia Maria Lucinda De Donato; Juliana do Nascimento Lunz; Fernanda Davi Marques; Bráulio Soares Archanjo; Francisco José Pereira Lopes; Gilson Antônio Giraldi

doi:10.22456/2175-2745.143516

Authors

Ivânia Maria Lucinda De Donato Universidade Federal do Rio de Janeiro - (UFRJ) https://orcid.org/0009-0008-7639-6423
Juliana do Nascimento Lunz Divisão de Metrologia de Materiais - INMETRO https://orcid.org/0000-0002-0341-5418
Fernanda Davi Marques Divisão de Metrologia de Materiais - INMETRO https://orcid.org/0000-0003-2652-557X
Bráulio Soares Archanjo Divisão de Metrologia de Materiais - INMETRO
Francisco José Pereira Lopes Universidade Federal do Rio de Janeiro - (UFRJ) https://orcid.org/0000-0002-2715-2341
Gilson Antônio Giraldi Laboratório Nacional de Computação Científica (LNCC) https://orcid.org/0000-0003-0623-9461

DOI:

https://doi.org/10.22456/2175-2745.143516

Keywords:

Deep learning, U-Net, nanomaterial, scanning electron microscopy

Abstract

This work investigates the application of deep learning techniques to analyze nanomaterial images acquired by scanning electron microscopy (SEM). The main objective is to improve the segmentation of zinc oxide (ZnO) nanoparticles and graphene oxide (GO) nanosheets. For this purpose, we developed a dedicated database for the supervised training of deep neural networks, using the U-Net architecture. Given the high spatial resolution of the original images, we investigated two approaches for image subdivision during the training and testing stages. Due to the high cost of acquiring and annotating SEM images of nanomaterials, we applied transfer learning through fine-tuning, adapting the network previously trained on ZnO images to segment GO images. The results were promising, with accuracies and recalls exceeding 95% for both ZnO and GO images.

Downloads

Download data is not yet available.

References

WANG, Y. et al. Generalizing from a few examples: A survey on few-shot learning. ACM Computing Surveys (CSUR), ACM New York, NY, USA, v. 53, n. 3, p. 1–34, 2020. Disponível em: ⟨https://doi.org/10.1145/3386252⟩.

DU, G. et al. Medical image segmentation based on U-Net: A review. Journal of Imaging Science & Technology, Society for Imaging Science and Technology, v. 64, n. 2, 2020. Disponível em: ⟨10.2352/J.ImagingSci.Technol.2020.64.2.020508⟩.

SCHNEIDER, C. A.; RASBAND, W. S.; ELICEIRI, K. W. NIH Image to ImageJ: 25 years of image analysis. Nature Methods, Nature Publishing Group, v. 9, n. 7, p. 671–675, 2012. Disponível em: ⟨https://doi.org/10.1038/nmeth.2089⟩.

BALS, J.; EPPLE, M. Deep learning for automated size and shape analysis of nanoparticles in scanning electron microscopy. RSC Advances, Royal Society of Chemistry, v. 13, n. 5, p. 2795–2802, 2023. Disponível em: ⟨10.1039/D2RA07812K⟩.

SUN, Z. et al. A deep learning-based framework for automatic analysis of the nanoparticle morphology in SEM/TEM images. Nanoscale, Royal Society of Chemistry, v. 14, n. 30, p. 10761–10772, 2022. Disponível em: ⟨https://pubs.rsc.org/en/content/articlelanding/2022/NR/D2NR01029A⟩.

AKERS, S. et al. Rapid and flexible segmentation of electron microscopy data using few-shot machine learning. npj Computational Materials, Nature Publishing Group, v. 7, n. 1, p. 187, 2021. Disponível em: ⟨https://doi.org/10.1038/s41524-021-00652-z⟩.

SIAO, H.-Y. et al. Machine learning-based automatic graphene detection with color correction for optical microscope images. arXiv preprint arXiv:2103.13495, 2021. Disponível em: ⟨https://doi.org/10.48550/arXiv.2103.13495⟩.

XIAO, C. et al. Automatic SARS-CoV-2 segmentation in electron microscopy based on few-shot learning. International Journal of Wavelets, Multiresolution and Information Processing, World Scientific Publishing Company, v. 22, n. 02, p. 2350047, 2024. Disponível em: ⟨https://doi.org/10.1142/S0219691323500479⟩.

DELERUYELLE, A.; VERSARI, C.; KLEIN, J. Self-mentoring: A new deep learning pipeline to train a self-supervised U-Net for few-shot learning of bio-artificial capsule segmentation. Computers in Biology and Medicine, Elsevier, v. 152, p. 106454, 2023. Disponível em: ⟨https://doi.org/10.1016/j.compbiomed.2022.106454⟩.

PROTONOTARIOS, N. E. et al. A few-shot U-Net deep learning model for lung cancer lesion segmentation via PET/CT imaging. Biomedical Physics & Engineering Express, v. 8, n. 2, p. 025019, 2022. Disponível em: ⟨https://iopscience.iop.org/article/10.1088/2057-1976/ac53bd/meta⟩.

LI, L. ZnO SEM image segmentation based on deep learning. In: IOP Conference Series: Materials Science and Engineering. IOP Publishing, 2020. v. 022035. Disponível em: ⟨https://iopscience.iop.org/article/10.1088/1757-899X/782/2/022035/meta⟩.

SAAIM, K. M. et al. In search of best automated model: Explaining nanoparticle TEM image segmentation. Ultramicroscopy, v. 233, p. 113437, 2022. Disponível em: ⟨https://doi.org/10.1016/j.ultramic.2021.113437⟩.

GUTIÉRREZ, J.; BARRERA, I.; GÓMEZ, N. Nanoparticle detection on SEM images using a neural network and semi-synthetic training data. Nanomaterials, v. 12, n. 11, p. 1818, 2022. Disponível em: ⟨https://doi.org/10.3390/nano12111818⟩.

IEEE. Few-Shot Learning for Segmentation of Yeast Cell Microscopy Images. 1–4 p. Disponível em: ⟨https://doi.org/10.1109/SIU53274.2021.9477988⟩.

CHEN, Z. et al. Deep learning-based method for SEM image segmentation in mineral characterization, an example from Duvernay Shale samples in Western Canada Sedimentary Basin. Computers & Geosciences, v. 138, p. 104450, 2020. Disponível em: ⟨https://doi.org/10.1016/j.cageo.2020.104450⟩.

ELIASSON, H.; ERNI, R. Localization and segmentation of atomic columns in supported nanoparticles for fast scanning transmission electron microscopy. npj Computational Materials, v. 10, n. 1, p. 168, 2024. Disponível em: ⟨https://doi.org/10.1038/s41524-024-01360-0⟩.

FARAZ, K. et al. Deep learning detection of nanoparticles and multiple object tracking of their dynamic evolution during in situ ETEM studies. Scientific Reports, v. 12, n. 1, p. 2484, 2022. Disponível em: ⟨https://doi.org/10.1038/s41598-022-06466-3⟩.

OZTURK, O.; SARITÜRK, B.; SEKER, D. Z. Comparison of fully convolutional networks (FCN) and U-Net for road segmentation from high-resolution imageries. International Journal of Environment and Geoinformatics, v. 7, n. 3, p. 272–279, 2020.

MARSLAND, S. Machine Learning: An Algorithmic Perspective. Chapman & Hall/CRC, 2009. Disponível em: ⟨https://doi.org/10.1201/9781420067194⟩.

GLOROT, X.; BENGIO, Y. Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. Sardinia, Italy: PMLR, 2010. v. 9, p. 249–256. Disponível em: ⟨https://proceedings.mlr.press/v9/glorot10a⟩.

KINGMA, D. P.; BA, J. A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2017. Disponível em: ⟨https://arxiv.org/abs/1412.6980⟩.

ZHANG, Z.; SABUNCU, M. Generalized cross entropy loss for training deep neural networks with noisy labels. In: Advances in Neural Information Processing Systems. Curran Associates, Inc., 2018. v. 31. Disponível em: ⟨https://proceedings.neurips.cc/paper/2018/hash/f2925f97bc13ad2852a7a551802feea0-Abstract.html⟩.

HOSSIN, M.; SULAIMAN, M. N. A review on evaluation metrics for data classification evaluations. International Journal of Data Mining & Knowledge Management Process (IJDKP), v. 5, n. 2, p. 1–11, 2019. Disponível em: ⟨https://doi.org/10.5121/ijdkp.2015.5201⟩.

ALMEIDA, L. R. de et al. Pairwise rigid registration based on Riemannian geometry and Lie structures of orientation tensors. Journal of Mathematical Imaging and Vision, Springer, v. 63, n. 7, p. 894–916, 2021. Disponível em: ⟨https://doi.org/10.1007/s10851-021-01037-z⟩.