Abstract: This article briefly introduced the importance of natural interaction, especially hand gestures recognition, in augmented reality (AR) and proposed a simple AR based interaction system. And then, the main technologies used in the system were discussed, such as hand gesture image subtraction, image segmentations, finger gesture model, etc. Finally, these methods were verified with a virtual implementation of the control panel in a simulation system.