The same thing happened to me when installing the Boonedox Groovy Landing Gear on my PA14. I didn't have a hex bit allen wrench drill bit set so I used a manual allen wrench. It was difficult to turn the cap screw about half way through, and I think the side jerking motion caused the cross threading. Definitely use a drill, ideally an impact driver. Anyway, I was able to get the screw out, but the insert came with it (lots of prying under the spacer). Here's what it looks like:

I followed mmiller's advise and was able to use a tap to clean up the threads. It now threads fine. I didn't understand what I was dealing with, so hopefully the above image helps others. I may have been able to get the screw out holding the nut in place, had I tried.
Thoughts on what to do now? I was thinking about using some JB Weld PlasticWeld around the insert, and hammering it back down in. Then trying again once it dries, using an impact driver (straight down). Other ideas? Worst case I can go to plan B and use a backing nut and washer (as mmiller also mentioned).
Thanks in advance!