If the fruit is turning brown when small, or not developing at all, it could be insufficient light. This would also be my guess if you have a few fruits develop, but not many.
If the fruits seem to grow to a couple inches long or so, either green or red, but then shrivel, it may be a water issue. Could be not watering enough, or if you have more rain than needed, the roots could be rotting.
Finally, if no fruits ever develop, and the plant seems unhealthy, it could be a disease issue.
I wouldn't think it's a soil nutrient issue unless you never have tended to soil fertility in the past. Chili peppers are usually very forgiving of that.