Intelligent video surveillance for elderly people living alone using Omni-directional Vision Sensor (ODVS) is an important application in the field of intelligent video surveillance. In this paper, an ODVS is utilized to provide a 360° panoramic image for obtaining the real-time situation for the elderly at home. Some algorithms such as motion object detection, motion object tracking, posture detection, behavior analysis are used to implement elderly monitoring. For motion detection and object tracking, a method based on MHoEI(Motion History or Energy Images) is proposed to obtain the trajectory and the minimum bounding rectangle information for the elderly. The posture of the elderly is judged by the aspect ratio of the minimum bounding rectangle. And there are the different aspect ratios in accordance with the different distance between the object and ODVS. In order to obtain activity rhythm and detect variously behavioral abnormality for the elderly, a detection method is proposed using time, space, environment, posture and action to describe, analyze and judge the various behaviors of the elderly in the paper. In addition, the relationship between the panoramic image coordinates and the ground positions is acquired by using ODVS calibration. The experiment result shows that the above algorithm can meet elderly surveillance demand and has a higher recognizable rate.