prikaz prve stranice dokumenta Filtriranje geoprostornog toka podataka korištenjem platforme Apache Spark
No public access
undergraduate thesis
Filtriranje geoprostornog toka podataka korištenjem platforme Apache Spark
Zagreb: University of Zagreb, Faculty of Electrical Engineering and Computing, 2019. urn:nbn:hr:168:185017
Britvec, Darko
University of Zagreb
Faculty of Electrical Engineering and Computing

Institutional repository: FER Repository

Cite this document

Britvec, D. (2019). Filtriranje geoprostornog toka podataka korištenjem platforme Apache Spark (Undergraduate thesis). Zagreb: University of Zagreb, Faculty of Electrical Engineering and Computing. Retrieved from https://urn.nsk.hr/urn:nbn:hr:168:185017

Britvec, Darko. "Filtriranje geoprostornog toka podataka korištenjem platforme Apache Spark." Undergraduate thesis, University of Zagreb, Faculty of Electrical Engineering and Computing, 2019. https://urn.nsk.hr/urn:nbn:hr:168:185017

Britvec, Darko. "Filtriranje geoprostornog toka podataka korištenjem platforme Apache Spark." Undergraduate thesis, University of Zagreb, Faculty of Electrical Engineering and Computing, 2019. https://urn.nsk.hr/urn:nbn:hr:168:185017

Britvec, D. (2019). 'Filtriranje geoprostornog toka podataka korištenjem platforme Apache Spark', Undergraduate thesis, University of Zagreb, Faculty of Electrical Engineering and Computing, accessed 24 June 2024, https://urn.nsk.hr/urn:nbn:hr:168:185017

Britvec D. Filtriranje geoprostornog toka podataka korištenjem platforme Apache Spark [Undergraduate thesis]. Zagreb: University of Zagreb, Faculty of Electrical Engineering and Computing; 2019 [cited 2024 June 24] Available at: https://urn.nsk.hr/urn:nbn:hr:168:185017

D. Britvec, "Filtriranje geoprostornog toka podataka korištenjem platforme Apache Spark", Undergraduate thesis, University of Zagreb, Faculty of Electrical Engineering and Computing, Zagreb, 2019. Available at: https://urn.nsk.hr/urn:nbn:hr:168:185017

Metadata
TitleFiltriranje geoprostornog toka podataka korištenjem platforme Apache Spark
Title (english)Filtering of Geospatial Data Streams using Apache Spark Platform
AuthorDarko Britvec
MentorKrešimir Pripužić (mentor)
Committee memberKrešimir Pripužić (predsjednik povjerenstva)
Committee memberLucija Petricioli (član povjerenstva)
Committee memberMatija Šulc (član povjerenstva)
GranterUniversity of Zagreb
Faculty of Electrical Engineering and Computing
Zagreb
Defense date and country2019-07-12, Croatia
Scientific / art field,
discipline and subdiscipline		TECHNICAL SCIENCES
Computing
Abstract
Cilj ovog rada je stvaranje učinkovitog raspodijeljenog sustava za filtriranje geoprostornog toka podataka korištenjem platforme Apache Spark. Arhitektura korištena u implementaciji programskog rješenja naziva se „objavi-pretplati“. Objave u sustavu predstavljaju poruke formata GeoJSON koje sadrže geoprostornu značajku (točku, liniju, poligon) te dodatne informacije. Pretplate u sustavu predstavljaju poligoni u prostoru uz koje vežemo dodatne značajke. Tok podataka implementiran je pomoću alata Apache Kafka. Za konzumaciju toka podataka koristi se Spark Streaming koji tok obrađuje u dijelovima koji se mogu primiti u određenom vremenskom intervalu (engl. micro-batch). Za filtriranje geoprostornih podataka korištena je programska knjižnica GeoSpark koja dodatno koristi knjižnice Java Topology Suite i Geotools. Učinkovito filtriranje prostornih podataka ostvareno je pomoću particioniranja skupa pretplata pomoću različitih strategija particioniranja. Izlaz sustava je poruka u formatu JSON koja sadrži identifikator objave i pripadne pretplate. U evaluaciji rješenja isprobane su strategije particioniranja pomoću Hilbertove krivulje, KDB-stabla, Q-stabla i R-stabla. Dodatno, za poboljšanje rezultata ispitano je i korištenje geoprostornih indeksa ostvarenih pomoću R-stabla i Q-stabla. U evaluaciji je također ispitan utjecaj korištenja pričuvne memorije za spremanje skupa pretplata.
Abstract (english)
The goal of this paper is to create an efficient distributed system for filtering of geospatial dana streams using the Apache Spark platform. Architecture used in the implementation is based on publish-subscribe pattern. Incoming messages in system carry a geospatial feature (eg. point, line or polygon on Earth) and some additional data formatted with GeoJSON specification. Geospatial subscriptions represent a set of polygons on Earth with some additional data. Geospatial data stream is implemented with Apache Kafka tool. Spark Streaming platform is used for consuming the incoming messages. It handles messages in portions (micro-batches) consumed within a time interval. GeoSpark library along with Java Topology Suite and GeoTools are used for processing the geospatial data. The effectiveness of geospatial filtering is improved with partitioning of the subscriptions dataset by using a different partitioning strategies. The system output consists of the identifiers of both messages corresponded subscriptions. Partitioning strategies used in the experiment are ones based on Hilbert curve, KDB-tree, Q-tree and R-tree. Additionally, evaluations examined the impact of using geospatial indexes based on Q-tree and R-tree and also the impact of using the cache functionality for persisting the subscription dataset in the memory of worker processes.
Keywords
Keywords (english)
Languagecroatian
URN:NBNurn:nbn:hr:168:185017
Study programmeTitle: Computing
Study programme type: university
Study level: undergraduate
Academic / professional title: sveučilišni/a prvostupnik/ prvostupnica (baccalaureus/baccalaurea) inženjer/inženjerka računarstva (sveučilišni/a prvostupnik/ prvostupnica (baccalaureus/baccalaurea) inženjer/inženjerka računarstva)
Type of resourceText
File originBorn digital
Access conditionsClosed access
Terms of useLicence In copyright icon
Public note
Created on2020-01-07 17:02:22