Neural machine translation (NMT), a relatively new language technology, has quickly taken over the place statistical machine translation models held as the most widely used machine translation method. But while it has many advantages in comparison to previously used methods, it also has its limitations, especially when it comes to translating long sentences. This paper examines machine translations in both directions of translation for the Croatian-English language pair, to find out whether source text sentence length significantly influences machine translation quality. The central hypothesis to be tested is that machine translation quality drops proportionally to rising sentence length. The second hypothesis is that there is a “breaking point” in the number of tokens in source sentences after which there is a considerable drop in the translation quality. Quality is defined as the ratio of the number of errors in translated sentences and the number of tokens in source sentences. The translations were generated using Google Translate in June 2019 and the manually compiled data set for human evaluation on which the study was conducted consists of three text types, one of each for each source language, as well as the translations of these texts. Overall, based on the limited data set, the first hypothesis is confirmed, while confirming the second hypothesis would require a significantly larger data set. Nevertheless, the results indicate a tendency for the error score to plateau beyond the 40-token mark.