prikaz prve stranice dokumenta Sustav za procjenu skalabilnosti platforme Apache Beam u obradi velikih količina prostorno vremenskih podataka
No public access
master's thesis
Sustav za procjenu skalabilnosti platforme Apache Beam u obradi velikih količina prostorno vremenskih podataka
Zagreb: University of Zagreb, Faculty of Electrical Engineering and Computing, 2022. urn:nbn:hr:168:549249
Novak, Aleksandar
University of Zagreb
Faculty of Electrical Engineering and Computing

Institutional repository: FER Repository

Cite this document

Novak, A. (2022). Sustav za procjenu skalabilnosti platforme Apache Beam u obradi velikih količina prostorno vremenskih podataka (Master's thesis). Zagreb: University of Zagreb, Faculty of Electrical Engineering and Computing. Retrieved from https://urn.nsk.hr/urn:nbn:hr:168:549249

Novak, Aleksandar. "Sustav za procjenu skalabilnosti platforme Apache Beam u obradi velikih količina prostorno vremenskih podataka." Master's thesis, University of Zagreb, Faculty of Electrical Engineering and Computing, 2022. https://urn.nsk.hr/urn:nbn:hr:168:549249

Novak, Aleksandar. "Sustav za procjenu skalabilnosti platforme Apache Beam u obradi velikih količina prostorno vremenskih podataka." Master's thesis, University of Zagreb, Faculty of Electrical Engineering and Computing, 2022. https://urn.nsk.hr/urn:nbn:hr:168:549249

Novak, A. (2022). 'Sustav za procjenu skalabilnosti platforme Apache Beam u obradi velikih količina prostorno vremenskih podataka', Master's thesis, University of Zagreb, Faculty of Electrical Engineering and Computing, accessed 04 November 2024, https://urn.nsk.hr/urn:nbn:hr:168:549249

Novak A. Sustav za procjenu skalabilnosti platforme Apache Beam u obradi velikih količina prostorno vremenskih podataka [Master's thesis]. Zagreb: University of Zagreb, Faculty of Electrical Engineering and Computing; 2022 [cited 2024 November 04] Available at: https://urn.nsk.hr/urn:nbn:hr:168:549249

A. Novak, "Sustav za procjenu skalabilnosti platforme Apache Beam u obradi velikih količina prostorno vremenskih podataka", Master's thesis, University of Zagreb, Faculty of Electrical Engineering and Computing, Zagreb, 2022. Available at: https://urn.nsk.hr/urn:nbn:hr:168:549249

Metadata
TitleSustav za procjenu skalabilnosti platforme Apache Beam u obradi velikih količina prostorno vremenskih podataka
Title (english)A System for Scalability Assessment of Apache Beam Platform in Processing Large Volumes of Spatio-Temporal Data
AuthorAleksandar Novak
MentorIgor Čavrak (mentor)
Committee memberIgor Čavrak (član povjerenstva)
Committee memberDanko Basch (član povjerenstva)
Committee memberIvana Bosnić (član povjerenstva)
GranterUniversity of Zagreb
Faculty of Electrical Engineering and Computing
Zagreb
Defense date and country2022-07-13, Croatia
Scientific / art field,
discipline and subdiscipline		TECHNICAL SCIENCES
Computing
Abstract
S povećanjem broja IoT uređaja i podacima koje dobivamo od njih, novi način obrade tih podataka je trebao biti stvoren. Ti alati koji procesiraju tokove podataka dali su ogromnu prednost nad tradicionalnim bazama podataka. Postoji velika lista alata koje nam pružaju rad s tokovima podataka, pa je pitanje koje od njih odabrati. Kako smo mi limitirani i ne može se očekivati da ćemo se naučiti sve te različite alate i načine kako pisati programe specifično za njih, solucija je došla u obliku platforme Apache Beam koja je dala unificirani model koji se može spojiti na veliki broj izvođača. S Apache Beamom jedina stvar o kojoj se moramo brinuti je kako ćemo napisati program, ne kako će se taj program izvoditi na nekom izvođaču u pozadini. Ovaj rad se fokusira na pokušaju pokretanja program na što više izvođača i testirati njihovu performansu. Sustav je bio testiran nad 3 izvođača Apache Flink, Apache Spark i Hazelcast Jet. Testiranje je pokazalo da postoji limitacija u pitanju koliko možemo povećati broj jezgri i čvorova kako bi dobili bolje performanse. Također je pokazalo limitaciju u Jetu, gdje ne može čitati više od limitiranog broja entiteta nad Kafka topicom s particijama.
Abstract (english)
With an increase of IoT devices and the data that they provide, an entirely new way of processing that data had to be created. These tools for processing data flows in real time, gave a huge advantage as opposed to traditional databases. There exists a large list of tools that can provide us with the processing capabilities, so it begs the question of which to choose. As we are limited with time and can't be expected to learn all the different tools and the ways in which we can create a program for them, Apache Beam came with a solution with a unified model that would connect a large set of runners. With Apache Beam, the only thing we need to care for is what we are implementing, not how it is going to run in the background on a runner. This theses focuses on trying to execute a program to test the throughput of the tools as well as the throughput of the system on as many runners as we can and test their performance. The system was tested with Apache Spark, Apache Flink and Hazelcast Jet. The tests showed that there was a limit to how much you can increase the number of nodes and their CPU core counts to increase their performance. It also showed a limitation of Jet whereby it cannot read more than a limited number of records from partitioned Kafka topics.
Keywords
Keywords (english)
Languagecroatian
URN:NBNurn:nbn:hr:168:549249
Study programmeTitle: Computing
Study programme type: university
Study level: graduate
Academic / professional title: magistar/magistra inženjer/inženjerka računarstva (magistar/magistra inženjer/inženjerka računarstva)
Type of resourceText
File originBorn digital
Access conditionsClosed access
Terms of useLicence In copyright icon
Public note
Created on2023-01-13 15:43:25