Not a gunsmith either and was wondering if you put a drop or two of break Free or lubricating oil in the pin hole yet before trying the punch.
Sounds like it's mushroomed on the other end.
100% correct, zero good reason. I will return it home and cross this bridge when it becomes necessary.Bingo. There is a technique to starting this pin out. You have to angle the punch so as to stay on the end of the pin and not push the punch into the end of the pin. This keeps it from mushrooming.