Are you mirroring your computer's screen? If so, that explains why your two experiences are so different.
For "normal casting", your computer tells the service (Netflix, Google Movies, etc) to push the content directly to your Chromecast device. Your computer doesn't handle any of the stream. It just acts as a remote control and gets occasional updates. There are times where you can actually turn OFF your computer and the streaming continues. Moving to a faster internet definitely helps this situation
But, when you mirror, your computer is heavily involved. It actually gets the content, displays it on your screen, records a copy of what it is displaying, and pushes that copy to the Chromecast device. That is a lot of work to do all at one time while trying to keep both your screen and your Chromecast display smooth. Upgrading your internet will not help this because the bottleneck is in your computer.
Try doing things that take any extra workload off of your computer; shut down programs you don't need, run a cleaning tool, make sure everything is up to date, add memory if you can.
Mirroring may never be totally smooth. But you may be able to make it better.
