Robotic retail surveying by deep learning visual and textual data

Autonomous systems for monitoring and surveying are increasingly used in retail stores, since they improve the overall performance of the store and reduce the manpower cost. Moreover, an automated system improves the accuracy of collected data by avoiding human-related factors. This paper presents ROCKy, a mobile robot for data collection and surveying in a retail store that autonomously navigates and monitors store shelves based on real-time store heat maps; ROCKy is designed to automatically detect Shelf Out of Stock (SOOS) and Promotional Activities (PA) based on Deep Convolutional Neural Networks (DCNNs). The deep learning approach evaluates visual and textual content of an image simultaneously to classify and map SOOS and PA events during working hours. The proposed approach was applied and tested on several real scenarios, presenting a new public dataset with more than 14.000 annotated shelves images. Experimental results confirmed the effectiveness of the approach, showing high accuracy (up to 87%) in comparison with the existing state of the art SOOS and PA monitoring solutions, and a signification reduction of retail surveying time (45%).