The study of the micro-plastic behavior of rough surface contact is the critical link towards a fundamental understanding of contact, friction, adhesion, and surface failures at small length scales. In the companion paper (Yu et al., J. Mech. Phys. Solids 55, 489-516, 2007), we have studied the onset of surface yielding due to single-dislocation nucleation from a stepped surface under adhesive contact. Here we analyze the contact hardening behavior due to multiple dislocations in a two-dimensional dislocation model. Continuum micromechanical analyses are used to derive the configurational force on the dislocation, while a modified Rice-Thomson model is used to describe the dislocation nucleation. Dislocations nucleated from a surface source are stabilized and pile up as a result of the balance between the resolved driving force and the non-zero lattice resistance in the solid. The dislocation pileup will exert a strong back stress to prevent further dislocation nucleation and thus lead to the contact hardening behavior, the degree of which depends on the slip-plane orientation. Particularly, we find that the dislocation interactions between two slip planes can make the contact loading order-of-magnitude easy to nucleate multiple dislocations, which is thus named "latent softening". A mechanistic explanation shows that the latent softening is closely related to the mode mixity of the stress intensity at the surface step. Dislocation nucleation will modify the geometric characteristics of the surface step, so that the contact-induced stress state near the step, as described by the mode mixity, changes, which influences the subsequent dislocation nucleation. Our calculations show that the dislocation pileup on one slip plane can even cause spontaneous dislocation nucleation on the other slip plane without further increase of the contact load. Furthermore, it is found that the rough surface contact at small length scale can lead to the dislocation segregation and the formation of a surface tensile sub-layer. The discrete-dislocation model presented here provides novel insights in bridging the atomistic simulations and continuum plastic flow analysis of surface asperity contact. Implications of the theoretical predictions are also discussed.