Hello, I'm an end user as well. Nest currently has (3) different doorbells they are selling. Which one do you have? I don't believe any of them use a magnet to secure the doorbell.
The original Nest HELLO doorbell snaps in to place and requires a small tool to release it. The Nest Doorbell (battery) is not designed well and I have heard that even though you need a small tool to remove it, it can be pulled off if forced. The newly release Nest Battery (Wired) has a locking screw to hold it in place.