H2020 IOStack

Welcome to IOStack

IOStack is an European research project funded by the H2020 initiative. The project is a consortium of several European industrial and research partners including University of Rovira i Virgili, IBM, MPSTOR, Eurocom and the BARCELONA SUPERCOMPUTING CENTER. A number of the partners will act as users of the system including IDIADA (Automotive), GRIDPOCKET (IOT) and ARCTUR (HPC).

IOStack is designed for deployments of data analytics in virtual environments. Virtual environments allow very flexible deployment of analytics frameworks but have less performance than bare metal deployments. IOStack will focus on how Software Defined Storage can use its knowledge of the cloud topology and the real time dynamic characteristics of the cloud to deploy analytics jobs that will run and complete within guaranteed SLAs and timescales.

The SDS knowledge of the static topology allows compute and storage locality to be optimized, understanding the dynamic load of the cloud allows a further optimization of which resources, paths and devices should be used in a given workload.

The main objective is to create IOStack: a Software Defined Storage toolkit for Big Data on top of the OpenStack platform. IOStack will enable efficient execution of virtualized analytics applications over virtualized storage resources thanks to flexible, automated, and low cost data management models based on software defined storage (SDS). Major challenges are:

• Storage and compute disaggregation and virtualization. Virtualizing data analytics to reduce costs implies disaggregation of existing hardware resources. This requires the creation of a virtual model for compute, storage and networking that allows orchestration tools to manage resources in an efficient manner. For the orchestration layer it is essential to provide policy-based provisioning tools so that the provisioning of virtual components for the analytics platform is made according to the set of QoS policies. 
• SDS Services for Analytics. The objective is to define, design, and build a stack of SDS data service enabling virtualized analytics with improved performance and usability. Among these services we include native object store analytics that will allow running analytics close to the data without taxing initial migration, data reduction services that will be optimized for the special requirements posed by virtualized analytics platforms, and specialized persistent caching mechanisms, advanced prefetching and data placement.
• Orchestration and deployment of big data analytics services. The objective is to design and build efficient deployment strategies for virtualized analytic-as-a-service instances (both ephemeral and permanent). In particular, the focus of this work is on data-intensive scalable computing (DISC) systems such as Apache Hadoop and Apache Spark, which enable users to define both batch and latency-snsitive analytics. This objective includes the design of scalable algorithms that strive at optimizing a service-wide objective function (e.g., optimize performance, minimize cost, etc...) under heterogeneous workloads.