Actor said he is still haunted by the performance