Passing is not a secondary training for any skill - it should not slow down outside defense training. Handling or Driving could, but I'm not sure how much in case of the two mentioned players as they have only one of those skills lower than outside defense and not that terribly.
For the other training types, players will improve more quickly not only with additional playing time, but also a more skilled player will improve more quickly. For example, a great inside defender and rebounder will find it easier to improve his shot blocking than a poor inside defender and poor rebounder would. This encourages you to develop players with a combination of skills, but there are many combinations of skills which work well together -- you can develop players who will be a good fit for your team's needs.