In order to analysis the abnormal vehicle behavior by trajectory fitting effectively, the whole process is divided into three steps: target detection and tracking, vehicle trajectory analysis, vehicle behavior detection. Firstly, a three-frame-differencing method is used to achieve initial target location and an improved tracking algorithm based on Kalman predictor is proposed; then, an adaptive segmented linear fitting algorithm is proposed to achieve vehicle trajectory fitting; finally, two parameters including the rate of velocity variation and the rate of direction variation are used to establish vehicle abnormal behavior detection model. Experiment result shows that the three high dangerous vehicle behaviors in road surveillance videos can be detected effectively: sharp brake, sharp turn, and sharp turn brake.