The cohesive zone model (CZM) is a key technique for finite element (FE) simulation of fracture of quasi-brittle materials; yet its constitutive relationship is usually determined empirically from global measurements. A more convincing way to obtain the cohesive relation is to experimentally determine the relation between crack separation and crack surface traction. Recent developments in experimental mechanics such as photoelasticity and digital image correlation (DIC) enable accurate measurement of whole-field surface displacement. The cohesive stress at the crack surface cannot be measured directly, but may be determined indirectly through the displacement field near the crack surface. An inverse problem thereby is formulated in order to extract the cohesive relation by fully utilizing the measured displacement field. As the focus in this article is to develop a framework to solve the inverse problem effectively, synthetic displacement field data obtained from finite element analysis (FEA) are used. First, by assuming the cohesive relation with a few governing parameters, a direct problem is solved to obtain the complete synthetic displacement field at certain loading levels. The computed displacement field is then assumed known, while the cohesive relation is solved in the inverse problem through the unconstrained, derivative-free Nelder–Mead (N–M) optimization method. Linear and cubic splines with an arbitrary number of control points are used to represent the shape of the CZM. The unconstrained nature of N–M method and the physical validity of the CZM shape are guaranteed by introducing barrier terms. Comprehensive numerical tests are carried out to investigate the sensitivity of the inverse procedure to experimental errors. The results show that even at a high level of experimental error, the computed CZM is still well estimated, which demonstrates the practical usefulness of the proposed method. The technique introduced in this work can be generalized to compute other internal or boundary stresses from the whole displacement field using the FE method.