The growth of the Green Village Data Platform: from 10,000 messages per year to 100 million per day

The Green Village is a loosely regulated open-air laboratory located on the campus of Delft University of Technology. Researchers live and work in this laboratory, surrounded by projects in the fields of sustainability, energy and climate adaptation. On the site, educational and research institutions experiment with new technologies, systems and products, and validate their sustainable innovations in the practical environment of Green Village.

From on-premise to public cloud

The Green Village Data Platform began in 2018 as a small-scale, open-source solution that ran locally (on-premise) on SURF's servers. This was sufficient in the beginning, but limited scalability. There were no IT systems available that would allow researchers to easily use real-time IoT data and that would give them the flexibility to customise the system to their own wishes. By cleverly combining various open-source tools, SURF developed a solution in the form of a customised platform.

Over time, the Green Village project grew and more and more researchers used the location for their research. This also increased the data flow to the platform: from less than one message per second to almost 1000 messages. This means a jump from 10,000 messages throughout 2019 to almost 100 million messages per day in 2024.

Graph: the growth of data flows towards The Green Village Data Platform between 2019 and 2024

Legend

Phase 1: Open source software
Phase 2: Paid managed services + appointment of own project manager
Phase 3: Adoption growth + appointment of own data manager

Upscaling with managed services

To accommodate the enormous increase in data traffic, SURF switched from an open-source solution to paid managed services in the public cloud in 2021. This made it possible to further professionalise the Green Village Data Platform and to easily scale it up. As a result, the speed and data volume on the platform increased more than a hundredfold.

The team behind Green Village also grew. In November 2020, a digitisation project manager joined, and two years later a data manager was added. Together they took care of the daily management of the data flows and the linking of new data flows to the platform. Thanks to their efforts, the growth of data once again took off enormously.

Collaboration as an accelerator

The real breakthrough came when SURF and Green Village jointly took responsibility for the data platform. This structural collaboration proved to be a game changer for the platform's development process. It enabled Green Village to develop further as an open-air laboratory for energy and energy management in the built environment.

The data collected on the Green Village Data Platform is growing by the day and is becoming increasingly valuable as a unique source of information for research into energy management. It is also playing an increasingly important role in accelerating the digital transition and the development of smart energy solutions.

Scaling up with SURF Cloud Research Consultancy

The SURF Cloud Research Consultancy service helps research institutes and companies set up and manage similar cloud solutions. In addition to our knowledge and experience in the field of data infrastructure and security, we are experts in building cloud solutions using public cloud services and integrating them with SURF services. We also provide support for the simple scaling up of projects and the efficient tackling of data challenges.