You can use who and whom anywhere and anytime you like; just use them correctly, especially around me, but I won’t say anything if you don’t because I don’t correct other people’s grammar in public, except here.
Who is generally used for the subject case; whom is typically used for the object case. To check your usage, invert the sentence/question:
SUBJECT CASE: I, you, he, she, it, we, you (plural), they
Who (subject case): Who had a wonderful time at the holiday party?
Who is the subject of the sentence. Invert the question: Mary (She) had a wonderful time at the party. Mary is the subject of the sentence and substitutes for who (subject case).
I didn’t know who was going with us.
Although who is the subject of a phrase that serves as the direct object for I don’t know, who is the subject of was going, so you would say “I didn’t know she was going with us.”
OBJECT CASE: me, you, him, her, it, us, you, them
Whom (object case): He gave the book to Jen (to her).
Invert the sentence: To whom (object case, which substitutes for her) did he give the book?
She is talking to John (him).
To whom (object case) is she talking? Also, to John is a prepositional phrase, and John is the object of the preposition to.
Who’s ready for whose?
My English Quarters
Hallie Belt, M.A. and B.A., English