Funny, I experience almost the opposite. When I strap down a heavier fish up front it's tougher for me to track straight. Cutting through the chop is an even wetter experience. I don't notice much of a difference in drift speed.
But then again I'm in a smaller kayak and take the minimalist approach when hauling gear. I'm about 250lbs total ( kayak, gear, myself ) in the water. Most of the weight is toward the middle. A 20#+ fish strapped on board does significantly effect front/rear weight distribution on my rig. I have to pray to the fish gods that I catch another so I can strap em to the other end and hopefully balance out.