That miter box is obviously a hunk of junk. And dowels are hard to work because they are round. It's hard to support them properly.
My only suggestion is to practice having your shoulder and elbow and hand and saw all in exactly the same plane so that when you move the saw it's going in a straight line and isn't being moved to the side at all. You really need to stand to the side of your cut line so that your shoulder lines up with the cut.
Also Jorgenson makes a
nice hand powered miter saw. It's a little expensive compared to that Xacto thing, but you will have an easier time with it.
Edit: If you get the Jorgensen saw, you'll need a clamp to hold it on the table.