Hey there Kevartje,
I'm sorry about the late response, thank you for being patient! Were you able to fix that Ambient Mode issue? If not, I'd be happy to look into it, since using Google Photos with Ambient Mode is a pretty key feature. The first thing I would recommend is disabling Photo curation on your personal photos. You can completely turn off curation for private and shared albums by changing the “Personal photo curation” setting to “Live albums only.” No curation at all is done on your Favorites.
I hope that helps, and if you have any other questions or concerns for me, please let me know!