A Modular Software Architecture for Processing of Big Geospatial Data in the Cloud
Computers & Graphics, Vol.49 (2015), pp. 69-81
In this paper we propose a software architecture that allows for processing of large geospatial data sets in the cloud. Our system is modular and flexible and supports multiple algorithm design paradigms such as MapReduce, in-memory computing or agent-based programming. It contains a web-based user interface where domain experts (e.g. GIS analysts or urban planners) can define high-level processing workflows using a domain-specific language (DSL). The workflows are passed through a number of components including a parser, interpreter, and a service called job manager. These components use declarative and procedural knowledge encoded in rules to generate a processing chain specifying the execution of the workflows on a given cloud infrastructure according to the constraints defined by the user. The job manager evaluates this chain, spawns processing services in the cloud and monitors them. The services communicate with each other through a distributed file system that is scalable and fault-tolerant. Compared to previous work describing cloud infrastructures and architectures we focus on the processing of big heterogeneous geospatial data. In addition to that, we do not rely on only one specific programming model or a certain cloud infrastructure but support several ones. Combined with the possibility to control the processing through DSL-based workflows, this makes our architecture very flexible and configurable. We do not only see the cloud as a means to store and distribute large data sets but also as a way to harness the processing power of distributed computing environments for large-volume geospatial data sets. The proposed architecture design has been developed for the IQmulus research project funded by the European Commission. The paper concludes with the evaluation results from applying our solution to two example workflows from this project.
Rule-based Process Orchestration: An Expert System for the Dynamic and Infrastructure-Independent Generation of Geospatial Processing Chains
Campus Gießen, TH Mittelhessen, Master Thesis, 2015
The constant improvements of remote-sensing technologies along with the emergence of new technologies to gather geo-related information has led to an exponential growth of geospatial data along with the range of potential applications, during the last decades. New technologies are required to target the challenges which arise not only from the variety but more importantly from the volume of so called Big Data. This thesis presents a modular and flexible software architecture, which aims to address these issues through the rule-based generation of dynamic process chains in respect to properties such as type and volume of data and an efficient utilisation of the available processing infrastructure. The use of an abstract workflow model enables the definition of processing steps in a concise and comprehensible manner without the need to consider the efficient execution. After the exploration and evaluation of related solutions a precise analysis of the context is carried out. This way, all interacting systems and data sources are identified and considered during the following design of the architecture. Furthermore, the requirements and quality goals such as high-availability and scalability are pointed out. To proof the feasibility of the technology-independent architecture, a prototypical implementation is presented which can be also used as a groundwork for further developments. Finally, real-world scenarios are used to show how such a system behaves in conditions simulating the practical use, which allows a discussion about strengths and weaknesses of this approach.