To put it briefly, if you don't speak, your already handicapped speaking ability will deteriorate even further. And the opposite situation if you speak regularly. Keep it up! This is not a big deal in my opinion