Multi-Fruit Tracking and 3-D Structure Recovery via CoTracker

Thiago Teixeira Santos

doi:10.22456/2175-2745.150960

Authors

Thiago Teixeira Santos Empresa Brasileira de Pesquisa Agropecuária (Embrapa) https://orcid.org/0000-0002-9272-3403

DOI:

https://doi.org/10.22456/2175-2745.150960

Keywords:

fruit detection, fruit tracking, agricultural robotics, digital agriculture

Abstract

In agricultural robotics and orchard automation, tasks such as fruit detection, tracking, and spatial localization are essential for applications like yield prediction and harvesting. However, these tasks are challenging due to the similar appearance of fruits, occlusions, and the inherent difficulties of field robotics, including uncontrolled lighting conditions and the variability in orchard environments. This work leverages CoTracker, a transformer-based model for point tracking using cross-track/cross-time attention, to simultaneously perform multiple fruit tracking, 3-D fruit localization, and camera pose estimation. The proposed approach demonstrates promising results in fruit counting, tracking, and scene reconstruction, highlighting its potential in agricultural automation.

Downloads

Download data is not yet available.

Author Biography

Thiago Teixeira Santos, Empresa Brasileira de Pesquisa Agropecuária (Embrapa)

Thiago T. Santos received his B.Sc. (2000), M.Sc. (2004)
and PhD (2009) degrees in Computer Science from the University of São Paulo. Since 2010, he is a researcher at Embrapa Agricultural Informatics, part of the Empresa Brasileira de Pesquisa Agropecuária, or the Brazilian Agricultural Research Corporation. His interests include 3D reconstruction for plant and animal phenotyping and applications of computer vision and machine learning in agriculture.

References

[1] HÄNI, N.; ROY, P.; ISLER, V. A comparative study of fruit detection and counting methods for yield mapping in apple orchards. Journal of Field Robotics, v. 37, n. 2, p. 263–282, 2020.

[2] SANTOS, T. T. et al. Multiple orange detection and tracking with 3-d fruit relocalization and neural-net based yield regression in commercial sweet orange orchards. Computers and Electronics in Agriculture, Elsevier BV, v. 224, p. 109199, set. 2024. ISSN 0168-1699. Disponível em: http://dx.doi.org/10.1016/j.compag.2024.109199.

[3] LI, T. et al. A multi-arm robot system for efficient apple harvesting: Perception, task plan and control. Computers and Electronics in Agriculture, Elsevier BV, v. 211, p. 107979, ago. 2023. ISSN 0168-1699.

[4] SHAMSHIRI, R. R. et al. Agricultural robotics to revolutionize farming: Requirements and challenges. In: Mobile Robots for Digital Farming. [S.l.]: CRC Press, 2025. p. 107–155.

[5] JONG, S. de et al. APPLE MOTS: Detection, Segmentation and Tracking of Homogeneous Objects Using MOTS. IEEE Robotics and Automation Letters, Institute of Electrical and Electronics Engineers (IEEE), v. 7, n. 4, p. 11418–11425, out. 2022. ISSN 2377-3774.

[6] VILLACRÉS, J. et al. Apple orchard production estimation using deep learning strategies: A comparison of tracking-by-detection algorithms. Computers and Electronics in Agriculture, Elsevier BV, v. 204, p. 107513, jan. 2023. ISSN 0168-1699.

[7] LIU, X. et al. Monocular camera based fruit counting and mapping with semantic data association. IEEE Robotics and Automation Letters, v. 4, n. 3, p. 2296–2303, 2019.

[8] MATOS, G. P. et al. Tracking and counting apples in orchards under intermittent occlusions and low frame rates. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. [S.l.: s.n.], 2024. p. 5413–5421.

[9] HARTLEY, R.; ZISSERMAN, A. Multiple View Geometry in Computer Vision. [S.l.]: Cambridge University Press, 2004. ISBN 9780511811685.

[10] SCHÖNBERGER, J. L.; FRAHM, J.-M. Structure-from-motion revisited. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). [S.l.: s.n.], 2016. p. 4104–4113.

[11] LOWE, D. G. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, Springer Science and Business Media LLC, v. 60, n. 2, p. 91–110, nov. 2004. ISSN 0920-5691.

[12] MUJA, M.; LOWE, D. G. Scalable nearest neighbor algorithms for high dimensional data. IEEE Transactions on Pattern Analysis and Machine Intelligence, v. 36, n. 11, p. 2227–2240, 2014.

[13] WOJKE, N.; BEWLEY, A.; PAULUS, D. Simple online and realtime tracking with a deep association metric. In: 2017 IEEE International Conference on Image Processing (ICIP). [S.l.]: IEEE, 2017.

[14] ZHANG, Y. et al. Bytetrack: Multi-object tracking by associating every detection box. In: Computer Vision – ECCV 2022. [S.l.]: Springer Nature Switzerland, 2022. p. 1–21. ISBN 9783031200472.

[15] KARAEV, N. et al. CoTracker: It Is Better to Track Together. In: LEONARDIS, A. et al. (Ed.). Computer Vision – ECCV 2024. Cham: Springer Nature Switzerland, 2025. p. 18–35. ISBN 978-3-031-73033-7.

[16] TERVEN, J.; CÓRDOVA-ESPARZA, D.-M.; ROMERO-GONZÁLEZ, J.-A. A Comprehensive Review of YOLO Architectures in Computer Vision: From YOLOv1 to YOLOv8 and YOLO-NAS. Machine Learning and Knowledge Extraction, v. 5, n. 4, p. 1680–1716, 2023. ISSN 2504-4990.

[17] SCARAMUZZA, D.; FRAUNDORFER, F. Visual odometry [tutorial]. IEEE Robotics and Automation Magazine, v. 18, n. 4, p. 80–92, 2011.

[18] DELLAERT, F. Factor graphs: Exploiting structure in robotics. Annual Review of Control, Robotics, and Autonomous Systems, Annual Reviews, v. 4, n. 1, p. 141–166, maio 2021. ISSN 2573-5144.

[19] SANTOS, T. T. et al. eContaFruto MOrangeT. Redape, 2024. Disponível em: https://www.redape.dados.embrapa.br/citation?persistentId=doi:10.48432/OI7BFG.

[20] MILAN, A. et al. MOT16: A Benchmark for Multi-Object Tracking. 2016. Disponível em: https://arxiv.org/abs/1603.00831.

[21] SANTOS, T. T. et al. eContaFruto OranDet: Base de dados para detecção de laranjas por imagem. Redape, 2024. Disponível em: https://www.redape.dados.embrapa.br/citation?persistentId=doi:10.48432/IG6VIQ.

[22] BERTASIUS, G.; WANG, H.; TORRESANI, L. Is Space-Time Attention All You Need for Video Understanding? [S.l.]: arXiv, 2021.

[23] NISTER, D. An efficient solution to the five-point relative pose problem. IEEE Transactions on Pattern Analysis and Machine Intelligence, Institute of Electrical and Electronics Engineers (IEEE), v. 26, n. 6, p. 756–770, jun. 2004. ISSN 0162-8828.

[24] LUITEN, J. et al. HOTA: A higher order metric for evaluating multi-object tracking. International Journal of Computer Vision, v. 129, p. 548–578, 2021.