Current diagnostic methods for skin cancers are based on some morphological characteristics of the pigmented skin lesions, including the geometry of their contour. The aim of this article is to model the early growth of melanoma accounting for the biomechanical characteristics of the tumor micro-environment, and evaluating their influence on the tumor morphology and its evolution. The spatial distribution of tumor cells and diffusing molecules are explicitly described in a three-dimensional multiphase model, which incorporates general cell-to-cell mechanical interactions, a dependence of cell proliferation on contact inhibition, as well as a local diffusion of nutrients and inhibiting molecules. A two-dimensional model is derived in a lubrication limit accounting for the thin geometry of the epidermis. First, the dynamical and spatial properties of planar and circular tumor fronts are studied, with both numerical and analytical techniques. A WKB method is then developed in order to analyze the solution of the governing partial differential equations and to derive the threshold conditions for a contour instability of the growing tumor. A control parameter and a critical wavelength are identified, showing that high cell proliferation, high cell adhesion, large tumor radius and slow tumor growth correlate with the occurrence of a contour instability. Finally, comparing the theoretical results with a large amount of clinical data we show that our predictions describe accurately both the morphology of melanoma observed in vivo and its variations with the tumor growth rate. This study represents a fundamental step to understand more complex microstructural patterns observed during skin tumor growth. Its results have important implications for the improvement of the diagnostic methods for melanoma, possibly driving progress towards a personalized screening.