Correlation filter (CF) has proven its superb efficiency in visual tracking for unmanned aerial vehicle (UAV) applications. To enhance the temporal smoothness of the filter, many CF-based approaches introduce temporal regularization terms to penalize the variation of coefficients in an element-wise manner. However, this element-wise smoothness is stiff to the filter coefficients and can lead to poor adaptiveness in case of various challenges, e.g., fast motion and viewpoint changes, which frequently occur in the UAV tracking process. To tackle this issue, this work introduces a novel tracker with consistency pursed correlation filter, i.e., CPCF tracker. It is able to achieve flexible temporal smoothness by evaluating the similarity between two consecutive response maps with a correlation operation. By correlation operations, the consistency constraint allows for flexible variations in the response map without losing temporal smoothness. Besides, a dynamic label function is introduced to further increase adaptiveness in the training process. Considerable experiments on three challenging UAV tracking benchmarks verify that the presented tracker has surpassed the other 25 state-of-the-art trackers with satisfactory speed (~25 FPS) for real-time applications on a single CPU.