You just be a normal person and continue about your day. If you want to take the time to try and explain it to him ok but he's not really gonna give a shit one way or the other. If a cop tries to use it against you as reasonable suspicion to detain and/or probable cause to search, that's a different story. Then you tell that cop in no uncertain terms that you do not consent to any search of your person or possessions. Nervousness and/or stuttering perceived as nervousness is not enough to give an officer reasonable suspicion to detain you or probable cause to search you. But if it's just a guy in the subway and he's not harassing you, why give it a second thought?