Abstract: This study proposes a methodology to find the interest levels of two speakers in a conversation. The ANN-HMM approach-a hybrid method is adopted. The hybrid method uses language input as an additional parameter in addition to the acoustic features. The language input provides a measure of classification of the input speech utterance. A combined classifier is used to make a linear decision on the emotion of the uttered speech as an arousal or valence. When the decision is fed to the Generative Factor Analyzed Hidden Markov Model (GFA-HMM) it evidently substantiates to be a better method with good accuracy rate of classification of whether the speaker is entangled in the conversation or vice-versa. The proposed method produced highly satisfactory results for the Linguistic Data Consortium (LDC) emotional prosody dataset.