We investigated the relationship between lipid binding glycoprotein apolipoprotein E (apoE; gene APOE) polymorphisms (ε4 allele carriers versus no carriers = ε4+/ε4-) and trace elements (TEs) (e.g., (methyl)mercury, arsenic, lead, cadmium, selenium, manganese, copper, and zinc) in mothers (N = 223) and their new-borns (N = 213) exposed to potentially toxic metal(loid)s from seafood consumption. The apoE isoform encoded by the ε4 allele is believed to have beneficial effects in early life but represents a risk factor for age-associated diseases. Under certain conditions ε4 carriers are more susceptible to oxidative stress and metal(loid) toxicity. DNA from Croatian pregnant women (N = 223, third trimester) and their new-borns (N = 176), was genotyped for APOE by TaqMan® SNP assay - rs429358 and rs7412. Seafood intake data and TE levels in maternal urine, milk, hair, peripheral venous blood, mixed cord blood, and new-borns' urine were available from previous studies. We compared TEs between ε4+ and ε4- carriers using Mann-Whitney U tests and applied multiple linear regression models to analyse the TE's dependence on the presence of allele ε4 (genotypes ε3/ε4, ε4/ε4) in combination with other explanatory variables. We identified 17% (n = 37) and 20% (n = 35) ε4 allele carriers in mothers and new-borns, respectively. The Mann-Whitney U test showed that mothers with the ε4 allele had significantly higher mean levels of (methyl)mercury in peripheral venous blood, cord blood, and hair; arsenic in urine and cord blood; and selenium in peripheral venous blood and plasma. However, taking confounders into account, only the maternal plasma selenium remained statistically significant in the linear regression models (ε4 carriers vs non-carriers: 62.6 vs 54.9 ng/mL, p < 0.001). Literature suggestions of possible ε4 allele impact on Hg levels were not observed, while superior selenium status observed in healthy pregnant women carrying allele ε4 could be linked to the proposed APOE ε4 beneficial effects early in life.