Abstract: An efficient spectrum allocation strategy is crucial to improving the spectrum utilization and Quality of Service (QoS) of users in femtocell networks. In this study, we formulate the downlink channel allocation problem in femtocell networks into a dynamic optimization problem. The formulation captures the stochastic nature of external packet arrivals as well as channel availability. Then, based on Q-learning techniques, we propose an Adaptive Spectrum Allocation Algorithm (ASAA), with a ε-greedy action exploration policy for the purposing of accelerating the rate of convergence. The algorithm can learn the statistics of both packet arriving processes and channel availability. In addition, it produces a near-optimal spectrum allocation strategy that aims at maximizing the system throughput. Simulation results demonstrate that the algorithm converges to a stationary spectrum allocation policy, which outperforms the classic Round Robin (RR) allocation policy.