Wear a dry suit which is big enough so you can layer up plenty of clothing under it, and which has dry socks, not ankle gaskets. Wear a couple of pairs of fluffy socks under the dry socks for insulation, and wear neoprene booties or other footwear over the dry socks which is nice and loose, so the footwear doesn’t restrict the circulation in your feet. Layer up enough clothing under the dry suit so you will be warm even if the wind comes up and the temperature dives while you're out on the water. I sometimes wear a layer of clothing on top of the dry suit so if the temperature increases during the day I can take it off while I’m out on the water. I have a warm hat which covers my ears, and I have a less warm hat which I can switch to if the weather warms up. With this kind of gear, you can be dry, cozy, and comfortable while fishing even in really lousy weather. More importantly, if you end up in the water, you'll stay warm and comfortable.
I find that it's easier to keep warm and dry in a sit-on-top pedal kayak than it is in a sit-on-top paddle kayak, because when paddling you get your hands wet, and water drips off your paddle onto your lap. It's also easier to keep warm in a sit-inside kayak than it is in a sit-on-top kayak, because a sit-inside kayak protects your lower body from cold weather and water. However, sit-on-top kayaks are much better for fishing than sit-inside kayaks.
Here’s a couple of pictures from a day of kayak fishing on Lindbergh Lake in Montana in May of 2014.
Here’s a couple of pictures from an early-May 5-day 116-mile unsupported kayak trip on the Middle Fork of the Salmon River in Idaho, starting on Marsh Creek. We floated through snow for the first day and a half of the trip. Here we are kayaking through a snow squall on Marsh Creek.
Here is a block of snow at the bottom of an avalanche gully on Marsh Creek.
Here is a lunch stop on Marsh Creek.
Here is a large pile of avalanche debris and snow on the second day of the trip.