According to common lore, massive elementary higher spin particles lead to
inconsistencies when coupled to gravity. However, this scenario was not
completely ruled out by previous arguments. In this paper, we show that in a
theory where the low energy dynamics of the gravitons are governed by the
Einstein-Hilbert action, any finite number of massive elementary particles with
spin more than two cannot interact with gravitons, even classically, in a way
that preserves causality. This is achieved in flat spacetime by studying
eikonal scattering of higher spin particles in more than three spacetime
dimensions. Our argument is insensitive to the physics above the effective
cut-off scale and closes certain loopholes in previous arguments. Furthermore,
it applies to higher spin particles even if they do not contribute to
tree-level graviton scattering as a consequence of being charged under a global
symmetry such as $\mathbb{Z}_2$. We derive analogous bounds in anti-de Sitter
spacetime from analyticity properties of correlators of the dual CFT in the
Regge limit. We also argue that an infinite tower of fine-tuned higher spin
particles can still be consistent with causality. However, they necessarily
affect the dynamics of gravitons at an energy scale comparable to the mass of
the lightest higher spin particle. Finally, we apply the bound in de Sitter to
impose restrictions on the structure of three-point functions in the squeezed
limit of the scalar curvature perturbation produced during inflation.