Deep learning technique - Based drone detection and tracking

eriod that drone detector is ran one time. 
2.4. Evaluation Dataset 
 To evaluate the performance of drone detector and drone detection and tracking system, the 
custom evaluation dataset is used. This dataset includes videos and images that were captured by 
smart phone camera. 
 Tab.1. Videos for algorithm evaluation. 
 No. Number of frame Resolution Usage 
 Video1.mp4 320 Single object detection and 
 Video2.mp4 300 tracking (SOT) 
 Video3.mp4 320 
 Video4.mp4 300 Multiple object detection 
 and tracking (MOT) 
 Video5.mp4 330 1080*1920 
 Test for different skip 
 Video6.mp4 300 
 frame value 
 Test Drone detector 
 Images 500 
 Intersection Over Union (IOU) [15] value are used to evaluate the accuracy of the drone 
detector and combine algorithm in the case of single object tracking (SOT). We compute the 
Intersection of the area of the predicted bounding box, and the area of the ground-truth bounding 
box, and divide by the Union of the two areas. The accuracy is then the average of IOU for all 
the frames. For multiple object tracking performance evaluate, the Multiple Object Tracking 
Accuracy (MOTA) [15] is used. 
 ()FN FP IDS
 MOTA 1 t t t t 
 GT
 t t
Where, FN (False negative) is the number of time that target is missed. FP (False positive) is the 
number of time that the tracking results are wrong. IDS are the number of time that target’s IDs 
are switched. GT are the number of ground-truth box in all frames. 
Tạp chí Nghiên cứu KH&CN quân sự, Số 73, 06 - 2021 13 
 Kỹ thuật máy bay & Thiết bị bay 
 3. EXPERIMENTAL RESULTS 
 The small drone that we used for creating the training and testing data sets for the drone 
detector and drone detection and tracking system is a mini quadcopter drone. This type of drone 
has a popular design and is widely used in amateur photography. 
 The testing process was run on Dell Inspiron with Intel(R) Core (TM) i5-3210 CPU; 8.00 GB 
RAM and Geforce GT 640M NVIDIA Graphic Card. Ubuntu 18.04 operation system is installed. 
The programming process used is Python 3.6 and OpenCV 4.0 version. We also test the 
algorithm on Intel(R) CEON E3-1231 v3 CPU; 8.00 GB RAM and ZOTAC-1060, 6GB Graphic 
Card which was installed using Ubuntu 18.04 operation with CUDA 10.1 to compare the 
processing speed and accuracy of algorithm in different hardware configuration. 
 A video with our experimental results can be found at the link: 
3.1. Drone Detection 
 The training results with different set of parameters are shown in tab.2. The fine-tuned model 
was test on 500 images which the model had not been trained before. The results showed that 
with the specific dataset, the value of batch_size and epoch number which controls the accuracy 
of the estimate of the error gradient when training neural networks are the 
important hyperparameters that influence the dynamics of the learning algorithm. 
 Tab.2. Drone detection training results with diffrent setting parameters. 
 The best result achieved was 90.8% of accurate detection when the hyperparameters were set 
as 8 for batch_size and 150,000 for training epochs. 
 Fig. 4. Drone detector evaluation. 
14 N. M. Quang, , T. X. Tung , “Deep learning technique- based drone detection and tracking.” 
Nghiên cứu khoa học công nghệ 
 We use the IOU (Intersection Over Union) value to evaluate the Drone detector with above 
setting confidence value. The fig.4 shows the Drone detection evaluation results, the aqua 
bounding box is ground truth box that is achieved by handcraft; the red bounding box is 
prediction box that is generated by drone detector with confidence value is set as 0.5. 
 Tab.3. The average of IOU. 
 Tab.3 shows the average of IOU value. Normally, if the IOU value is higher than 0.5 then the 
Detector is considered as a “good” Detector. 
 On the other hand, from fig.4, we can see that the confidence value and IOU value depended 
on the size of target when compare with the size of frames. We use the size_compare value to 
measure the relation between the size of drone and the size of frame in percentages. When the 
target is small compared with the frame size (about 1/16 the size of frame), the IOU and 
confidence value are lower. When the target size is larger enough compared with the frame size, 
those values are higher. 
 Fig. 5. Drone detector test result. 
 The Drone Detector was tested on images that were captured from previously unseen video 
footage. The detected result is shown in fig.5. It can be seen that the Drone Detector effectively 
recognized small drones in strong light fig.5.(a), complex background fig.5.(b), drone fly close 
the trees fig.5.(c) and drone fly close the buildings fig.5.(d). 
3.2. The combination of drone detection and tracking algorithm 
3.2.1. Algorithm testing with different value of skip frame 
 Tab.3. shows the object detection and tracking with different Skip_frame value, the results are 
Tạp chí Nghiên cứu KH&CN quân sự, Số 73, 06 - 2021 15 
 Kỹ thuật máy bay & Thiết bị bay 
achieved when algorithms are run on both CPU and GPU configuration. We can see that, with 
the CPU configuration, different values of skip frame directly affect the accuracy and running 
speed of the system. For the GPU configuration, we can see that, different value of Skip_frame 
affect the running speed of the system and provide the same multi object tracking accuracy. The 
selection pair of Skip_frame value and the confidence threshold are important for improving the 
running speed while maintaining the detection and tracking accuracy. 
 Tab.4. Drone detection and tracking with different Skip_frame value. 
 Confidence
 Input video
 Frames CPU GPU 
 Skip 
 FPS MOTA FPS MOTA 
 1 9.22 0.808 17.62 0.878 
 3 16.07 0.908 25.52 0.945 
 Video6.
 0.5 5 18.21 0.793 29.39 0.923 
 mp4 
 9 19.81 0.868 34.37 0.966 
 13 21.22 0.963 39.51 0.987 
 For the GPU configuration, we can see that, different values of Skip_frame affect the running 
speed of the system and provide the same multi object tracking accuracy. The selection pair of 
Skip_frame value and the confidence threshold are important for improving the running speed 
while maintaining the detection and tracking accuracy. 
 The drone detection and tracking system was also tested on video and managed to detect a 
small drone with the use of a smart phone camera. The results are shown in fig.7. 
 Fig. 6. Drone detection and tracking system test result. 
16 N. M. Quang, , T. X. Tung , “Deep learning technique- based drone detection and tracking.” 
Nghiên cứu khoa học công nghệ 
 The number of targets and red dots detected indicate a tracking result while the blue bounding 
boxes indicate the class probability. Fig.6(a) shows the tracking result without object detection, 
fig.6(b, c) show both the object detection and tracking results in different background condition, 
fig.6(d) shows the multiple object detection and tracking result. 
 From these results we can see that when combining detection and tracking algorithms in a 
system, we can achieve the system with the improvement of system perform in both running 
speed and tracking accuracy. 
3.2.2. Single object tracking and multiple object tracking 
 Tab.5. Single object tracking. 
 Skip Accuracy 
 Input videos Confidence FPS 
 Frames (IOU) 
 Run Video1.mp4 21.78 0.55 
 on 0.5 13 
 CPU Video2.mp4 22.44 0.58 
 Run Video1.mp4 41.61 0.74 
 on 0.5 13 
 GPU Video2.mp4 41.39 0.76 
 Tab.5 shows the single object tracking result when algorithms are run on both CPU and GPU 
configuration. We can see that, with the same input videos and setting parameters, the achieved 
of FPS value and tracking accuracy when the algorithm is ran on GPU configuration are higher 
than CPU configuration’s. 
 Tab.6. Multiple object tracking. 
 Input videos SkipFrame FPS MOTA 
 Video3.mp4 22.53 0.660 
 Run on CPU Video4.mp4 13 21.57 0.773 
 Video5.mp4 19.38 0.796 
 Video3.mp4 40.50 0.940 
 Run on GPU Video4.mp4 13 40.95 0.933 
 Video5.mp4 39.39 0.954 
 Tab.6 shows the multi object tracking result when algorithms are run on both CPU and GPU 
configuration. We can see that, with the same input videos and setting parameters, the achieved 
of FPS value and tracking accuracy when the algorithm is ran on GPU configuration are higher 
than CPU configuration’s. The result shows that the combination of object detection and object 
tracking algorithms provides an effective solution for real-time small drone detection and 
tracking. The system also performed good characteristic for handle multiple object tracking. 
 4. CONCLUSIONS 
 In this paper, we present a drone detection and tracking system based on deep learning 
algorithms. We can see that, by leveraging existed convolution neural network model and 
transfer learning technique as well as Google’s Colab application, we can develop the robust 
system for recognizing moving objects in input videos using a small custom dataset. The 
combination of object detection model and object tracking algorithm provides an effective 
solution for real-time small drone detection and tracking as well as handles multi-target 
Tạp chí Nghiên cứu KH&CN quân sự, Số 73, 06 - 2021 17 
 Kỹ thuật máy bay & Thiết bị bay 
tracking problem. 
 However, there are some problems in the proposed system that will take more research to 
improve. The first problem is rate of false detection (FP, FN) in some cases. This problem can 
lead to the bad effects for the performance of whole system. Secondly, the number of video in 
dataset for evaluate is small, it causes the evaluation results which just reflect a local meaning. 
 As further research, dealing with following problems and extension task will be focused on. 
Firstly, the quality of the synthetic dataset that directly affect the performance of the whole 
system is needed to improve. Base on this, the dataset for create a multi-type of drone detection 
and tracking system will be expanded. Secondly, the research should involve the problem of data 
fusion where the information of camera-based drone detection will be associated to the 
information from other detection method such as radar-based method, acoustic-based method or 
RF-based method. Additionally, the development of deployable application that applies to 
realizable Anti-drone system will be done complete. 
 REFERENCES 
[1]. Ulzhalgas Seidaliyeva, Daryn Akhmetov, Lyazzat Ilipbayeva, Eric T. Matson “Real-Time and 
 Accurate Drone Detection in a Video with a Static Background”, Sensors 2020, 20, 3856; 
 doi:10.3390/s20143856. 
[2]. Michael Jian, Zhenzhong Lu and Victor C. Chen, “Drone Detection and Tracking Based on Phase-
 Interferometric Doppler Radar”, 2018 IEEE Radar Conference. 
[3]. Dongkyu ’Roy’ Lee, Woong Gyu La, and Hwangnam Kim, “Drone Detection and Identification 
 System using Artificial Intelligence”, 2018 International Conference on Information and 
 Communication Technology Convergence (ICTC). 
[4]. J. Janousek, P. Marcon, J. Pokorny, and J. Mikulka, “Detection and Tracking of Moving UAVs”, 
 2019 Photonics Electromagnetics Research Symposium. 
[5]. A. Howard et al., "MobileNets: Efficient Convolutional Neural Networks for Mobile Vision 
 Applications", Computing Research Repositor, arXiv:1704.04861, 2017. 
[6]. Dlib C++ Library, (2018) "Correlation Tracker," [Online]. Available: 
[7]. Adrian Rosebrock, Simple object tracking with OpenCV, Available 
 at:https://www.pyimagesearch.com/2018/07/23/simple-object-tracking-with opencv. 
[8]. Yujie Du, Mingyu Gao, Yuxiang Yang, Jing Zhang2, Zhongfei Yu, “A Target Detection System for 
 Mobile Robot Based On Single Shot Multibox Detector Neural Network”, 2018 IEEE 4th 
 International Conference on Control Science and Systems Engineering. 
[9]. Hashir Ali, Mahrukh Khursheed, Syeda Kulsoom Fatima, “Object Recognition for Dental 
 Instruments Using SSD-MobileNet”, 2019 International Conference on Information Science and 
 Communication Technology (ICISCT). 
[10. Brad Dwyer, "How to Create a Synthetic Dataset for Computer Vision", https://blog.roboflow.com. 
[11]. Priya Dwivedi (2017). “Is Google Tensorflow Object Detection API the easiest way to implement 
 image recognition?”. Available at: https://towardsdatascience.com/is-google-tensorflow-object-
 detection-api-the-easiest-way-to-implementimage-recognition-a8bd1f500ea0. 
[12]. G. Gamage, I. Sudasingha, I. Perera, D. Meedeniya, “Reinstating Dlib Correlation Human Trackers 
 Under Occlusions in Human Detection based Tracking”, 2018 International Conference on 
 Advances in ICT for Emerging Regions (ICTer) : 092 – 098. 
[13. Lasitha Mekkayil, Hariharan Ramasangu, “Object Tracking with Correlation Filters using Selective 
 Single Background”, arXiv:1805.03453v1 [cs.CV] 9 May 2018. 
[14]. Adrian Rosebrock, OpenCV People Counter Available at : 
 https://www.pyimagesearch.com/https://www.pyimagesearch.com/2018/08/13/opencv-people-
 counter. 
[15]. B. Keni and S. Rainer, “Evaluating multiple object tracking performance: the clear mot metrics”, 
 EURASIP J. Image Video Process, Dec. 2008. 
18 N. M. Quang, , T. X. Tung , “Deep learning technique- based drone detection and tracking.” 
Nghiên cứu khoa học công nghệ 
 TÓM TẮT 
 HỆ THỐNG TỰ ĐỘNG PHÁT HIỆN VÀ THEO DÕI DRONE 
 SỬ DỤNG KỸ THUẬT HỌC SÂU TIÊN TIẾN 
 Cùng với sự phát triển của công nghiệp sản xuất, các loại thiết bị bay không người lái 
 kích thước nhỏ (còn được gọi là drone) ngày càng được sử dụng rộng rãi trong nhiều lĩnh 
 vực. Tuy nhiên, việc sử dụng drone một cách thiếu kiểm soát có thể mang đến những nguy 
 cơ tiềm ẩn như: sử dụng drone cho mục đích khủng bố, vận chuyển chất cấm, các hoạt 
 động trinh thám, xâm nhập khu vực cấm bay,... Xây dựng hệ thống tự động phát hiện và 
 theo dõi các thiết bị bay không người lái là một nhiệm vụ quan trọng trong bài toán giám 
 sát, bảo vệ an ninh trên không. Bài báo sử dụng kỹ thuật học chuyển tiếp (transfer 
 learning) để huấn luyện lại mạng nơ-ron học sâu SSD-MobileNet-v2 trên tập dữ liệu nhân 
 tạo, kết quả nhận dạng chính xác mục tiêu đạt được là 90.8%. Kết hợp thuật toán nhận 
 dạng drone với thuật toán bám đối tượng theo thuật toán bám tương quan và thuật toán 
 bám tâm đối tượng có thể nhận dạng và theo dõi hiệu quả đối tượng drone với kích thước 
 nhỏ trong các điều kiện khác nhau cũng như có khả năng phát hiện và theo dõi nhiều mục 
 tiêu cùng lúc. 
Từ khóa: UAV; Phát hiện và theo dõi drone; SSD-MobileNet-v2; Thuật toán bám tương quan; Bám tâm đối tượng. 
 Received April 07th 2021 
 Revised June 04th 2021 
 Published June 10th2021 
Author affiliations: 
 1Faculty of Control Engineering, Le Quy Don Technical University; 
 2East Asia University of Technology; 
 3Academy of Military Science and Technology. 
 *Corresponding author: xuantung.truong@gmail.com. 
Tạp chí Nghiên cứu KH&CN quân sự, Số 73, 06 - 2021 19