So.. I finally figured it out. The issue is Google home app. When selecting to use your own pictures, Google does pick randomized albums etc. however it does NOT display all albums available in your Google photos profile. So you unselect the albums you don't want.. but there are albums still selected which are unseen.
You need to unselect a the albums you don't want. Exit settings.. and save the options. Then re-enter and look for selected albums again that you do not want to show. Unselect them and keep doing it until only the photos you want appear in the preview.
What a pain. But until Google gets their act together.. this is how you fix it.
