In the model where Primordial Black Holes (PBHs) form from large primordial perturbations, constraints on PBH abundance provide in principle constraints on the primordial curvature power spectrum. This connection however depends necessarily on the details of PBH formation mechanism. In this paper we provide, for the first time, constraints on the primordial curvature power spectrum from the latest limits on PBH abundance, taking into account all the steps from gravitational collapse in real space to PBH formation. In particular, we use results from numerical relativity simulations and peak theory to study the conditions for PBH formation for a range of perturbation shapes, including non-linearities, perturbation profile and a careful treatment of smoothing and filtering scales. We then obtain updated PBH formation conditions and translate that into primordial spectrum constraints for a wide range of shapes and abundances. These updated constraints cover a range of scales not probed by other cosmological observables. Our results show that the correct and accurate modelling of non-linearities, filtering and typical perturbation profile, is crucial for deriving meaningful cosmological implications.