I think that most will tell you it's best to have your pistol in a holster of some sort, as it keeps the gun in it's place, helps with concealment, and you can avoid what a lot of us have read about with people's pistols falling out of their pockets or down their pants leg at very inopportune times. (I actually can't think of an opportune time for that to happen...)
That said-- I've been waiting on an IWB holster for 5 months now (they originally said 5 weeks!!)

and I've carried my 3" Kimber in just about any form you could imagine. I've carried it Mexican style with a really tight belt, in the pocket of my hunting vest (not a big pocket!) in the outside pocket of my leather jacket, and have had no problems what-so-ever. It even stays in a readily accessible position if I put it in the pocket right.
You might be able to fit it in a cargo pants pocket, but it would do some awful sagging in those pants, I imagine. Do the holster. Search the forum, as there are quite a few threads on some quality holsters. Just make sure that your holster maker knows how to come through on his lead times!!