Abstract: Proper coordination between the Directional Over-Current Relays (DOCRs) in sub-transmission and distribution systems is one of the important conditions for the system security. The problem of setting DOCRs and their coordination is a highly constrained optimization problem that has been stated and solved as a nonlinear programming problem. The integration of the Series Capacitor (SC) into the transmission line makes the coordination problem more complex. In this study, a new problem formulation is proposed to calculate the optimal relay settings of Directional Over-Current Relays (DOCRs) in series compensated systems. In this study, the Modified Adaptive Particle Swarm Optimization (MAPSO) is employed to calculate the optimal relay settings. The proposed method is tested on a case study and optimal results are compared between compensated and uncompensated systems.