If you were a boxer, I'd say you are right. But then again, look at Boxers like Nikolai Valuev, that guy is basically a walking cupboard with arms, and yet he is a horrible boxer. Don't be fooled into thinking that building more muscle will you give you the upper hand in a fight, especially when weapons are involved. I am a rather tall guy at 6'23'' with a very large frame, so my whole structure can support more muscle, but bigger size means also slower movement. I can move incredibly fast for someone of my build, height and weight, yet it costs me alot more energy and coordination than someone of smaller frame. If you wanna be able to hold yourself in a fight, learn a martial art. All the muscle you need for that will build up during training. I practice full-contact Karate (Kyokushin), and I can tell you from first hand that having muscle alone brings squat. Look at the guys that do MMA. Sure, they are all buff and well-trained, but do you see one of them packing muscle like, say, Arnold Schwarzenegger during his early years? None. Keeping a muscular physique is a good thing, and if you can do 250 pushups in one set, that is already more than I can pack (allthough I do pushups on my knuckles or on fingers, I can rarely do more than a hundred).
The best idea if you don't wanna lose in a fight is: don't fight at all. To quote Mr. Miyagi from Karate Kid II: Best Defense against punch: no be there.
Buuuut, if you really wanna go there, here goes my exercise routine (all you need is a cheap set of dumbbells) which I do every other day, weight at 45 lbs. on each short dumbells.
200x repetitions hammer-curl in sets of 20.
150x repettions of bench press, in sets of 15
150x repetitions of kickback (triceps) in sets of 15
100x repetitions of wrist-curls (at half weight) in sets of 10
150x repetiotions of reclined shoulder press in sets of 15
dumbbell-less exercises:
200x crunches in sets of 20
100x pushups on knuckles, in sets of 50, sometimes the second set on 5 fingers, or the last ten on three fingers.
200x squats, in sets of 100
Start with low weight and many repetitions, and increase weight only gradually. Do ALOT of stretching, especially in the legs (positive side-effect for me is that I can do a split now).
Every other day in between I go running about 4 miles, but that is just me, as I still have to lose about 20 pounts of body-fat.