Portal to Other Dimensions: use of Computer Vision to create art work from day life images

Luísa Ferreira; Ana Clara Fraga Osias; Lucas Vieira dos Santos; Michel Melo da Silva

doi:10.22456/2175-2745.143513

Authors

Luísa Ferreira Universidade Federal de Viçosa (UFV) https://orcid.org/0009-0002-9587-9603
Ana Osias Universidade Federal de Viçosa (UFV) https://orcid.org/0009-0004-2562-372X
Lucas Vieira Universidade Federal de Viçosa (UFV) https://orcid.org/0009-0000-2132-1282
Michel Silva Universidade Federal de Viçosa (UFV) https://orcid.org/0000-0002-2499-9619

DOI:

https://doi.org/10.22456/2175-2745.143513

Keywords:

Component, Formatting, Style, Styling

Abstract

This work aims to combine classic and contemporary paintings with the external world by altering the interior of a door into another style in real-time photos and/or videos, creating a portal effect to provide an interactive experience with artworks. To achieve this, various techniques from Artificial Intelligence and Computer Vision were employed, primarily focusing on convolutional neural networks (CNNs) with supervised training. Models proposed for semantic segmentation problems were used for door detection and style transfer to stylize the input image. Finally, we conclude with an analysis of the results for each network individually and in combination, ensuring that the metrics and approaches were satisfactory. With our results, we were able to offer an accessible and interactive way to bring art into the daily lives of many people.

Downloads

Download data is not yet available.

References

CHOU, J.-P.; STORK, D. G. Computational tracking of head pose through 500 years of fine-art portraiture. Electronic Imaging, Society for Imaging Science and Technology, v. 35, p. 1–13, 2023. Disponível em: ⟨https://doi.org/10.2352/ei.2023.35.13.cvaa-211⟩.

ERIKSSON, J. et al. Recovering lost artworks by deep neural networks: Motivations, methodology, and proof-of-concept simulations. Electronic Imaging, Society for Imaging Science and Technology, v. 35, p. 1–7, 2023. Disponível em: ⟨https://doi.org/10.2352/ei.2023.35.13.cvaa-210⟩.

YANG, J. Design of architectural decoration based on smart home system. In: 2022 14th International Conference on Measuring Technology and Mechatronics Automation (ICMTMA). [s.n.], 2022. p. 899–902. Disponível em: ⟨https://doi.org/10.1109/ICMTMA54903.2022.00183⟩.

ARTHUR, P.; PASSINI, R. Wayfinding: people, signs, and architecture. [s.n.], 1992. Disponível em: ⟨https://doi.org/10.5860/choice.30-1301⟩.

GADSDEN, V. L. The arts and education: Knowledge generation, pedagogy, and the discourse of learning. Review of Research in Education, Sage Publications Sage CA: Los Angeles, CA, v. 32, n. 1, p. 29–61, 2008. Disponível em: ⟨https://doi.org/10.3102/0091732x07309691⟩.

GIRSHICK, R. et al. Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. [s.n.], 2014. p. 580–587. Disponível em: ⟨https://doi.org/10.1109/cvpr.2014.81⟩.

GIRSHICK, R. Fast R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision. [s.n.], 2015. p. 1440–1448. Disponível em: ⟨https://doi.org/10.1109/iccv.2015.169⟩.

REN, S. et al. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE, v. 39, n. 6, p. 1137–1149, 2016. Disponível em: ⟨http://arxiv.org/abs/1506.01497⟩.

REDMON, J. et al. You only look once: Unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. [s.n.], 2016. p. 779–788. Disponível em: ⟨https://arxiv.org/abs/1506.02640⟩.

JOCHER, G.; CHAURASIA, A.; QIU, J. Ultralytics YOLOv8. 2023. Disponível em: ⟨https://github.com/ultralytics/ultralytics⟩.

HE, K. et al. Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision. [s.n.], 2017. p. 2961–2969. Disponível em: ⟨https://doi.org/10.1109/ICCV.2017.322⟩.

RONNEBERGER, O.; FISCHER, P.; BROX, T. U-Net: Convolutional networks for biomedical image segmentation. In: SPRINGER. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. 2015. p. 234–241. Disponível em: ⟨https://arxiv.org/abs/1505.04597⟩.

ARDUENGO, M.; TORRAS, C.; SENTIS, L. Robust and adaptive door operation with a mobile robot. Intelligent Service Robotics, Springer Science and Business Media LLC, May 2021. ISSN 1861-2784. Disponível em: ⟨http://dx.doi.org/10.1007/s11370-021-00366-7⟩.

ESPINOZA, N. Open Source Dataset, Door Detection Dataset. Roboflow, 2022. ⟨https://universe.roboflow.com/nathaly-espinoza/door-detection-zqt59⟩. Visited on 2024-08-19. Disponível em: ⟨https://universe.roboflow.com/nathaly-espinoza/door-detection-zqt59⟩.

RAMÔA, J. G. et al. Real-time 2D–3D door detection and state classification on a low-power device. SN Applied Sciences, Springer, v. 3, n. 5, p. 590, 2021. Disponível em: ⟨http://dx.doi.org/10.1007/s42452-021-04588-3⟩.

JOHNSON, J.; ALAHI, A.; FEI-FEI, L. Perceptual losses for real-time style transfer and super-resolution. In: SPRINGER. Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14. 2016. p. 694–711. Disponível em: ⟨https://arxiv.org/abs/1603.08155⟩.

DUMOULIN, V.; SHLENS, J.; KUDLUR, M. A learned representation for artistic style. arXiv preprint arXiv:1610.07629, 2016. Disponível em: ⟨http://arxiv.org/abs/1610.07629⟩.

GHIASI, G. et al. Exploring the structure of a real-time, arbitrary neural artistic stylization network. arXiv preprint arXiv:1705.06830, 2017. Disponível em: ⟨http://arxiv.org/abs/1705.06830⟩.

ZUIJLEN, M. J. V. et al. Materials in paintings (MIP): An interdisciplinary dataset for perception, art history, and computer vision. Plos One, Public Library of Science San Francisco, CA USA, v. 16, n. 8, p. e0255109, 2021. Disponível em: ⟨https://doi.org/10.48550/arXiv.2012.02996⟩.

KHAN, F. S. et al. Painting-91: A large scale database for computational painting categorization. Machine Vision and Applications, Springer, v. 25, p. 1385–1397, 2014. Disponível em: ⟨https://doi.org/10.1007/s00138-014-0621-6⟩.

JU, X. et al. Human-Art: A versatile human-centric dataset bridging natural and artificial scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. [s.n.], 2023. p. 618–629. Disponível em: ⟨https://doi.org/10.1109/CVPR52729.2023.00067⟩.