I believe that practical accuracy (which depends more on me) and sufficient penetration are the most important factors in a self-defense situation.
With handgun rounds, I'll happily take expansion if it occurs so long as said expansion doesn't inhibit penetration too much. At the end of the day with rounds of relatively low energy it is shot placement that trumps everything else.
Your question mentioned the claim that dum-dum rounds penetrate well with low recoil impulse. The low recoil impulse is simply due to them having to be light, low velocity loads due to the soft lead alloy -- to prevent them from unduly fouling the gun's barrel. If light-recoiling penetration is your goal, I would think any soft-lead load would work, whether it be round nose, wad-cutter (inverted or not) or some other profile since all of them have to be loaded light.
As for penetrating well, I am sure they can, but their performance will be more erratic than modern bonded jacketed hollow points particularly if they strike. bone. The soft lead tends to deform readily, sometimes fragmenting a bit and shedding weight. Playing devils advocate, if you want extreme penetration for a given cartridge, a hard-cast lead projectile with a large flat frontal area is probably the way to go since they don't deform easily at all.
Ammunition and firearms themselves are not so different in that we choose each of them to fill a certain role and we often look for the best combination of features / performance to do the job we have in mind.