Google Home and Google Nest speakers and displays don't have separate volume controls for Google Assistant and media playback unfortunately. As a workaround, you can try enabling the night mode feature which reduces the volume of assistant responses during set times.
