Thanks for visiting the Community.
Currently, there's no option to remove or turn off the quick response of the Google Home/Nest device. We're always looking for ways to improve and we'll take this as feedback. Please stay tuned on our social channels for updates and announcements.
I also suggest that you send feedback via voice command by saying, "Hey Google, send feedback," or by following the steps on this link.