Took it all apart after work today.
A bunch of the bearings looked like they have been rubbed with dry sandpaper but weren't bad enough to replace. Kind of worrying after just one use. The same issue was on all the surfaces the bearings touch. Not happy about that last bit, but wasn't really worth considering buffing smoother yet.
I decided to use a silicone lube from blaster. It has a similar consistency to wd40 but has Teflon and is supposed to have very good compression friction, so is recommended for bearings. It has just enough surface adhesion to keep the bearings in place if you have a second person keeping things still.
Anyway, back to smooth movements.
BTW, after putting the bearings carriers back together, it squeaked crazy and was hard to move one of the fins. Different issue from my original issue but I found out the problem was the reverse/forward lines. It was binding internally despite all efforts to keep that all organized. I wonder if the line is slipping between the bearings on some people's drives leading to squeaking when the drive is otherwise not making a grinding sound. Just pulling the levers fixed that issue though.
My bearings: