PurposeWith suitable optimization criteria, hybrid models have proven to be efficient for preparing portfolios in capital markets of developed countries. This study adapts and investigates these methods for a developing country, thus providing a novel approach to the application of banking and finance. Our specific objectives are to employ a stochastic dominance criterion to evaluate the performances of over-the-counter (OTC) companies in a developing country and to analyze them with a hybrid model involving particle swarm optimization and artificial neural networks.Design/methodology/approachIn order to achieve these aims, the authors conduct a case study of OTC companies in Iran. Weekly and daily returns of 36 companies listed in this market are calculated for one year during 2014–2015. The hybrid model is particularly interesting, and the results of the study identify first-, second- and third-order stochastic dominances among these companies. The study’s chosen model uses the best performing combination of activation functions in our analysis, corresponding to TPT, where T represents hyperbolic tangent transfers and P represents linear transfers.FindingsOur portfolios are based on the shares of companies ranked with respect to the stochastic dominance criterion. Considering the minimum and maximum numbers of shares to be 2 and 10 for each portfolio, an eight-share portfolio is determined to be optimal. Compared with the index of Iran OTC during the research period of this study, our selected portfolio achieves a significantly better performance. Moreover, the methods used in this analysis are shown to be as efficient as they were in the capital markets of developed countries.Research limitations/implicationsThe problem of optimizing investment portfolios has to allow for correlations among returns from the financial maintenance period under consideration if an asymmetric distribution of returns exists (Babaei et al., 2015). Therefore, it is desirable to select an appropriate criterion in order to prepare an optimal portfolio and prioritize investment options. Although a back propagation technique is very popular in artificial neural (ANN) training, it is time-consuming to train a network in this way, and other methods such as particle swarm optimization (PSO) should be considered instead. In the hybrid combination of PSO and ANN, it is not the structure of a neural network that changes. Rather, the weighting method and the training technique chosen for the network are the important aspects, and these relate to PSO, so the only role ANN plays in this process is to reduce the errors.Practical implicationsThe hybrid model combining ANN and PSO is seen to be considerably successful for generating optimal results and appropriate activation functions. These results are consistent with the theoretical findings of Das et al. (2013) and an application of the simple PSO in a study conducted by Pederson and Chipperfield (2010). Our research results also confirm the efficiency of stochastic dominance criteria as noted in the studies conducted by Roman et al. (2013), ANN as in a study carried out by Kristijanpoller et al. (2014) and PSO as in studies conducted by Liu et al. (2015) and Deng et al. (2012). These studies were carried out in the capital markets of developed countries, whereas the authors’ analysis relates to a developing country.Originality/valueThe authors deduce that the tools and methods whose efficiency was proven in the capital markets of developed countries also apply to, and demonstrate efficiency in, two novel applications of portfolio optimization within developing countries. The first of these is gaining familiarity with the theory and practice of these research tools and the methods that enrich financial knowledge of investors in developing countries. The second of these is the application of tools and methods identified by investors in the capital markets of developing countries, which enables optimal allocation of financial resources and growth of the markets. The authors expect that these findings will contribute to improving the economies of developing countries and thus help with economic development and facilitation of improving trends.