Measurements of the elliptic flow coefficient relative to the collision plane defined by the spectator neutrons v2 {SP} in collisions of Pb ions at center-of-mass energy per nucleon–nucleon pair √sNN = 2.76 TeV and Xe ions at √sNN = 5.44 TeV are reported. The results are presented for charged particles produced at midrapidity as a function of centrality and transverse momentum for the 5–70% and 0.2–6 GeV/c ranges, respectively. The ratio between v2 {SP} and the elliptic flow coefficient relative to the participant plane v2 {4}, estimated using four-particle correlations, deviates by up to 20% from unity depending on centrality. This observation differs strongly from the magnitude of the corresponding eccentricity ratios predicted by the TRENTo and the elliptic power models of initial state fluctuations that are tuned to describe the participant plane anisotropies. The differences can be interpreted as a decorrelation of the neutron spectator plane and the reaction plane because of fragmentation of the remnants from the colliding nuclei, which points to an incompleteness of current models describing the initial state fluctuations. A significant transverse momentum dependence of the ratio v2 {SP}/ v2 {4} is observed in all but the most central collisions, which may help to understand whether momentum anisotropies at low and intermediate transverse momentum have a common origin in initial state fluctuations. The ratios of v2 {SP} and v2 {4} to the corresponding initial state eccentricities for Xe–Xe and Pb–Pb collisions at similar initial entropy density show a difference of (7.0±0.9)% with an additional variation of +1.8% when including RHIC data in the TRENTo parameter extraction. These observations provide new experimental constraints for viscous effects in the hydrodynamic modeling of the expanding quark–gluon plasma produced in heavy-ion collisions at the LHC