Shaykh Bin Baz was asked:

Question:

A man is doubtful in his prayer in any rak’ah. He does not know if he made one or two prostrations. What is required from him? Does he make the prostration of forgetfulness? And when is this prostration of forgetfulness performed in this situation: is it before or after the Tasleem? What do you advise the one who has a lot of doubts and satanic whispers in his prayer? May Allah reward you.

Answer:

“If his doubt arises after the prayer, that is after the Tasleem (salutations), this the doubt arises, then in this case his prayer is valid and there is no prostration of forgetfulness. As for if the doubt was in the prayer, then he does the prostration and builds on certainty. If he is doubtful whether he performed one or two prostrations, then he performs the second prostration, whether it be the first, second, third, or fourth rak’ah. He does the prostration of forgetfulness before the Tasleem. If he prostrates after the tasleem, then there is no harm, but prostrating before the tasleem is better.

As for if his doubt is after the prayer and after he finished, then the doubt arises, then this is from satanic whispers. Do not turn towards it. Doubt after prayer does not affect [it]”

[Reference]

Translated by

Faisal Ibn Abdul Qaadir Ibn Hassan
Abu Sulaymaan