Before you can install a new marine speaker, you will need to make sure it will fit in your installation cutout hole. Keep in mind that there are no standard installation sizes so it is very important to pay attention to the installation measurements. If your speaker is too big for your existing cutout, then you may need to cut a little to enlarge the hole. If your speaker is too small for the hole, you are pretty much out of luck and you should look for a larger speaker.
If you need to cut there are several different options. A hole saw would give you a perfectly symetrical hole, but that would only work if you were cutting a new hole because the center bit needs something to bite into. Since you already have a hole there is nothing but empty space there. A Dremel, RotoZip, or similar tool will work great for an enlargement. Those tools aren't great at cutting perfect curves, but it doesn't need to be exact. Your speaker grill will hide any imperfections in the hole as long as you are close.
Unless you are doing an exact replacement, it is very unlikely that the old mounting screw holes will line up perfectly where your old ones did. You should plan on drilling new holes. Diagrams are nice, but there is nothing like using the speaker to mark your hole locations. Hold the speaker in place and use a pencil to mark your holes. Then drill.
Before you mount the speaker in place with the screws, you will need to attach the speaker wires. These can be done with existing tabs or you may want to solder the end of the wire right to the terminals. Make sure that you get the positive and negative wires attached to the correct terminals. If you don't, your speakers will be out of phase and the sound quality will suffer. You might also want to use a dab of silicone caulking to protect the terminals.
Most marine speakers come with a rubber gasket to keep water from getting behind the mounting surface. If yours did not come with a gasket, it would be a good idea to run a bead of silicone around your cutout hole before you put the speaker in place to seal out any water.
Mount the speaker in place with your mounting screws and you are good to go!