With high efficiency and efficacy, the trackers based on the discriminative correlation filter have experienced rapid development in the field of unmanned aerial vehicle (UAV) over the past decade. In literature, these trackers aim at solving a regression problem in which the circulated samples are mapped into a Gaussian label for online filter training. However, the fixed target label for regression makes trackers lose adaptivity in uncertain tracking scenarios. One of the typical failure cases is that the distractors, e.g., background clutter, camouflage, and similar object, are prone to confuse these trackers. In this work, an efficient approach to instantly monitor the local maximums of the response map for discovering distractors automatically is proposed. In addition, the regression target is accordingly learned, i.e., the location possessing local maximum indicates latent distractor and thus should be repressed by reducing its target response value in filter training. Qualitative and quantitative experiments performed on three challenging well-known benchmarks demonstrate that the presented method not only outperforms the state-of-the-art handcrafted feature-based trackers but also exhibits comparable performance compared to deep learning-based approaches. Specifically, the presented tracker has phenomenal practicability in real-time UAV applications with an average speed of ∼50 frames per second on an affordable CPU.