OATML

Orbit-to-ground framework to decode and predict biosignature patterns in terrestrial analogues

In the search for biosignatures on Mars, there is an abundance of data from orbiters and rovers to characterize global and regional habitability, but much less information is available at the scales and resolutions of microbial habitats and biosignatures. Understanding whether the distribution of terrestrial biosignatures is characterized by recognizable and predictable patterns could yield signposts to optimize search efforts for life on other terrestrial planets. We advance an adaptable framework that couples statistical ecology with deep learning to recognize and predict biosignature patterns at nested spatial scales in a polyextreme terrestrial environment. Drone flight imagery connected simulated HiRISE data to ground surveys, spectroscopy and biosignature mapping to reveal predictable distributions linked to environmental factors. Artificial intelligence–machine learning models successfully identified geologic features with high probabilities for containing biosignatures at s... [full abstract]

---

Kimberley Warren-Rhodes, Nathalie A. Cabrol, Michael Phillips, Cinthya Tebes-Cayo, Freddie Kalaitzis, Diego Ayma, Cecilia Demergasso, Guillermo Chong-Diaz, Kevin Lee, Nancy Hinman, Kevin L. Rhodes, Linda Ng Boyle, Janice L. Bishop, Michael H. Hofmann, Neil Hutchinson, Camila Javiera, Jeffrey Moersch, Claire Mondro, Nora Nofke, Victor Parro, Connie Rodriguez, Pablo Sobron, Philippe Sarazzin, David Wettergreen, the SETI Institute NAI Team
Nature Astronomy
[paper]

Multi-Spectral Multi-Image Super-Resolution of Sentinel-2 with Radiometric Consistency Losses and Its Effect on Building Delineation

High resolution remote sensing imagery is used in broad range of tasks, including detection and classification of objects. High-resolution imagery is however expensive, while lower resolution imagery is often freely available and can be used by the public for range of social good applications. To that end, we curate a multi-spectral multi-image super-resolution dataset, using PlanetScope imagery from the SpaceNet 7 challenge as the high resolution reference and multiple Sentinel-2 revisits of the same imagery as the low-resolution imagery. We present the first results of applying multi-image super-resolution (MISR) to multi-spectral remote sensing imagery. We, additionally, introduce a radiometric consistency module into MISR model the to preserve the high radiometric resolution of the Sentinel-2 sensor. We show that MISR is superior to single-image super-resolution and other baselines on a range of image fidelity metrics. Furthermore, we conduct the first assessment of the utility... [full abstract]

---

Muhammed Razzak, Gonzalo Mateo-Garcia, Gurvan Lecuyer, Luis Gomez-Chova, Yarin Gal, Freddie Kalaitzis
Journal of Photogrammetry and Remote Sensing (Jan 2023)
[Paper] [BibTex]

Open High-Resolution Satellite Imagery; The WorldStrat Dataset - With Application to Super-Resolution

Analyzing the planet at scale with satellite imagery and machine learning is a dream that has been constantly hindered by the cost of difficult-to-access highlyrepresentative high-resolution imagery. To remediate this, we introduce here the WorldStratified dataset. The largest and most varied such publicly available dataset, at Airbus SPOT 6/7 satellites’ high resolution of up to 1.5 m/pixel, empowered by European Space Agency’s Phi-Lab as part of the ESA-funded QueryPlanet project, we curate nearly 10,000 km² of unique locations to ensure stratified representation of all types of land-use across the world: from agriculture to ice caps, from forests to multiple urbanization densities. We also enrich those with locations typically under-represented in ML datasets: sites of humanitarian interest, illegal mining sites, and settlements of persons at risk. We temporally-match each highresolution image with multiple low-resolution images from the freely accessible lower-resolution Sentin... [full abstract]

---

Julien Cornebise, Ivan Oršolić, Freddie Kalaitzis
arXiv
[paper]

On Transfer Learning for Building Damage Assessment from Satellite Imagery in Emergency Contexts

When a natural disaster occurs, humanitarian organizations need to be prompt, effective, and efficient to support people whose security is threatened. Satellite imagery offers rich and reliable information to support expert decision-making, yet its annotation remains labour-intensive and tedious. In this work, we evaluate the applicability of convolutional neural networks (CNN) in supporting building damage assessment in an emergency context. Despite data scarcity, we develop a deep learning workflow to support humanitarians in time-constrained emergency situations. To expedite decision-making and take advantage of the inevitable delay to receive post-disaster satellite images, we decouple building localization and damage classification tasks into two isolated models. Our contribution is to show the complexity of the damage classification task and use established transfer learning techniques to fine-tune the model learning and estimate the minimal number of annotated samples requir... [full abstract]

---

Isabelle Bouchard, Marie-Ève Rancourt, Daniel Aloise, Freddie Kalaitzis
Remote Sensing
[paper]

Time-dependent Hillshades; Dispelling the Shadow Curse of Machine Learning Applications in Earth Observation

We show that machine learning models learn and perform better when they know where to expect shadows, through hillshades modeled to the time of imagery acquisition.Shadows are detrimental to all machine learning applications on satellite imagery. Prediction tasks like semantic / instance segmentation, object detection, counting of rivers, roads, buildings, trees, all rely on crisp edges and colour gradients that are confounded by the presence of shadows in passive optical imagery, which rely on the sun's illumination for reflectance values.Hillshading is a standard technique for enriching a mapped terrain with relief effects, which is done by emulating the shadow caused by steep terrain and/or tall vegetation. A hillshade that is modeled to the time of day and year can be easily derived through a basic form of ray tracing on a Digital Terrain Model (DTM) (also known as a bare-earth DEM) or Digital Surface Model (DSM) given the sun's altitude and azimuth angles. In this work, we use... [full abstract]

---

Freddie Kalaitzis, Gonzalo Mateo-Garcia, Kevin Dobbs, Dolores Garcia, Jason Stoker, Giovanni Marchisio
Earth Observation, EGU General Assembly 2022
[paper]

Physics informed deep learning to super-resolve and cross-calibrate solar magnetograms

Super-resolution techniques aim to increase the resolution of images by adding detail. Compared to upsampling techniques reliant on interpolation, deep learning-based approaches learn features and their relationships across the training data set to leverage prior knowledge on what low resolution patterns look like in higher resolution images. As an added benefit, deep neural networks can learn the systematic properties of the target images (ie texture), combining super-resolution with instrument cross-calibration. While the successful use of super-resolution algorithms for natural images is rooted in creating perceptually convincing results, super-resolution applied to scientific data requires careful quantitative evaluation of performances. In this work, we demonstrate that deep learning can increase the resolution and calibrate space-and ground-based imagers belonging to different instrumental generations. In addition, we establish a set of measurements to benchmark the performan... [full abstract]

---

Andres Munoz-Jaramillo, Anna Jungbluth, Xavier Gitiaux, Paul Wright, Carl Shneider, Shane Maloney, Atılım Güneş Baydin, Yarin Gal, Michel Deudon, Freddie Kalaitzis
Research Square
[Paper]

Cross-calibration, super-resolution, and uncertainty estimation of the conversion of MDI and GONG to HMI full-disk magnetograms using deep learning

Over the past 50 years, a variety of instruments have obtained images of the Sun's magnetic field (magnetograms) to study its origin and evolution. While improvements in instrumentation have led to breakthroughs in our understanding of physical phenomena, differences between subsequent instruments such as resolution, noise, and saturation levels all introduce inhomogeneities into long-term data sets. This has proven to be an insurmountable obstacle for research applications that require high-resolution and homogeneous data spanning time frames longer than the lifetime of a single instrument. Here we show that deep-learning-based super-resolution techniques can successfully up-sample and homogenize solar magnetic field images obtained both by space and ground-based instruments. In particular, we show the results of cross-calibrating and super-resolving MDI and GONG magnetograms to the characteristics of HMI. We also discuss the importance of agreeing on a standardized set of tra... [full abstract]

---

A Munoz-Jaramillo, A Jungbluth, X Gitiaux, P Wright, C Shneider, S Maloney, Freddie Kalaitzis, Atılım Güneş Baydin, Yarin Gal, M Deudon
Bulletin of the American Astronomical Society
[paper]

Water monitoring with Very High Resolution satellite imagery

The water volume on Earth's surface constantly varies with precipitation: an excess of water might lead to flooding, while its absence indicates upcoming droughts. We cannot afford in-situ monitoring devices on all rivers and streams worldwide, and free satellite imagery lacks the spatial and temporal resolution for continuous monitoring. This talk will provide several examples of water monitoring using PlanetScope daily imagery. The global daily coverage of Planet's data presents new opportunities for developing robust models of flood hazard, providing timely mapping in support of relief operations, and applying near real time predictive models for river flow estimation based on simultaneous measurements over entire river basins.

---

Freddie Kalaitzis, Gonzalo Mateo Garcia, Giovanni Marchisio
EGU General Assembly Conference Abstracts
[presentation]
[abstract]

RainBench: Towards Global Precipitation Forecasting from Satellite Imagery

Extreme precipitation events, such as violent rainfall and hail storms, routinely ravage economies and livelihoods around the developing world. Climate change further aggravates this issue. Data-driven deep learning approaches could widen the access to accurate multi-day forecasts, to mitigate against such events. However, there is currently no benchmark dataset dedicated to the study of global precipitation forecasts. In this paper, we introduce \textbf{RainBench}, a new multi-modal benchmark dataset for data-driven precipitation forecasting. It includes simulated satellite data, a selection of relevant meteorological data from the ERA5 reanalysis product, and IMERG precipitation data. We also release \textbf{PyRain}, a library to process large precipitation datasets efficiently. We present an extensive analysis of our novel dataset and establish baseline results for two benchmark medium-range precipitation forecasting tasks. Finally, we discuss existing data-driven weather foreca... [full abstract]

---

Christian Schroeder de Witt, Catherine Tong, Valentina Zantedeschi, Daniele De Martini, Freddie Kalaitzis, Matthew Chantry, Duncan Watson-Parris, Piotr Bilinski
AAAI, 2021
[arXiv]

Prediction of GNSS Phase Scintillations: A Machine Learning Approach

A Global Navigation Satellite System (GNSS) uses a constellation of satellites around the earth for accurate navigation, timing, and positioning. Natural phenomena like space weather introduce irregularities in the Earth's ionosphere, disrupting the propagation of the radio signals that GNSS relies upon. Such disruptions affect both the amplitude and the phase of the propagated waves. No physics-based model currently exists to predict the time and location of these disruptions with sufficient accuracy and at relevant scales. In this paper, we focus on predicting the phase fluctuations of GNSS radio waves, known as phase scintillations. We propose a novel architecture and loss function to predict 1 hour in advance the magnitude of phase scintillations within a time window of plus-minus 5 minutes with state-of-the-art performance.

---

Kara Lamb, Garima Malhotra, Athanasios Vlontzos, Edward Wagstaff, Atılım Güneş Baydin, Anahita Bhiwandiwalla, Yarin Gal, Freddie Kalaitzis, Anthony Reina, Asti Bhatt
Machine Learning and the Physical Sciences Workshop (ML4PS), NeurIPS 2019
[arXiv]

Correlation of Auroral Dynamics and GNSS Scintillation with an Autoencoder

High energy particles originating from solar activity travel along the the Earth's magnetic field and interact with the atmosphere around the higher latitudes. These interactions often manifest as aurora in the form of visible light in the Earth's ionosphere. These interactions also result in irregularities in the electron density, which cause disruptions in the amplitude and phase of the radio signals from the Global Navigation Satellite Systems (GNSS), known as 'scintillation'. In this paper we use a multi-scale residual autoencoder (Res-AE) to show the correlation between specific dynamic structures of the aurora and the magnitude of the GNSS phase scintillations (σϕ). Auroral images are encoded in a lower dimensional feature space using the Res-AE, which in turn are clustered with t-SNE and UMAP. Both methods produce similar clusters, and specific clusters demonstrate greater correlations with observed phase scintillations. Our results suggest that specific dynamic structures o... [full abstract]

---

Kara Lamb, Garima Malhotra, Athanasios Vlontzos, Edward Wagstaff, Atılım Güneş Baydin, Anahita Bhiwandiwalla, Yarin Gal, Freddie Kalaitzis, Anthony Reina, Asti Bhatt
Machine Learning and the Physical Sciences Workshop (ML4PS), NeurIPS 2019
[arXiv]

Single-Frame Super-Resolution of Solar Magnetograms; Investigating Physics-Based Metrics & Losses

Breakthroughs in our understanding of physical phenomena have traditionally followed improvements in instrumentation. Studies of the magnetic field of the Sun, and its influence on the solar dynamo and space weather events, have benefited from improvements in resolution and measurement frequency of new instruments. However, in order to fully understand the solar cycle, high-quality data across time-scales longer than the typical lifespan of a solar instrument are required. At the moment, discrepancies between measurement surveys prevent the combined use of all available data. In this work, we show that machine learning can help bridge the gap between measurement surveys by learning to super-resolve low-resolution magnetic field images and translate between characteristics of contemporary instruments in orbit. We also introduce the notion of physics-based metrics and losses for super-resolution to preserve underlying physics and constrain the solution space of possible super-resolut... [full abstract]

---

Anna Jungbluth, Xavier Gitiaux, Shane A.Maloney, Carl Shneider, Paul J. Wright, Freddie Kalaitzis, Michel Deudon, Atılım Güneş Baydin, Yarin Gal, Andrés Muñoz-Jaramillo
Machine Learning and the Physical Sciences Workshop (ML4PS), NeurIPS 2019
[arXiv]

Probabilistic Super-Resolution of Solar Magnetograms: Generating Many Explanations and Measuring Uncertainties

Machine learning techniques have been successfully applied to super-resolution tasks on natural images where visually pleasing results are sufficient. However in many scientific domains this is not adequate and estimations of errors and uncertainties are crucial. To address this issue we propose a Bayesian framework that decomposes uncertainties into epistemic and aleatoric uncertainties. We test the validity of our approach by super-resolving images of the Sun’s magnetic field and by generating maps measuring the range of possible high resolution explanations compatible with a given low resolution magnetogram.

---

Xavier Gitiaux, Shane Maloney, Anna Jungbluth, Carl Shneider, Atılım Güneş Baydin, Paul J. Wright, Yarin Gal, Michel Deudon, Freddie Kalaitzis, Andres Munoz-Jaramillo
Workshop on Bayesian Deep Learning, NeurIPS 2019
[Paper]

Freddie Kalaitzis publishes in Nature Astronomy

06 Mar 2023

Senior Research Fellow Freddie Kalaitzis, in collaboration with a team from NASA Ames Research and the SETI institute, published a study in Nature Astronomy on a new deep-learning based framework to accelerate the search for life on Mars and other extra-terrestial environments. Freddie was the key AI contributor in this study. You can read the paper here.

Freddie Kalaitzis publishes white paper on the State of AI for Earth Observations

02 Sep 2022

OATML Senior Researcher Freddie Kalaitzis, along with Cristian Rossi and Maral Bayaraa, have published a white paper on the State of AI for Earth Observations: an intro to sensors, ML and applications. Read their work here.

OATML Senior Researcher releases of WorldStrat dataset, in collaboration with WhyHow Ltd. and ESA Phi-Lab

20 Jul 2022

OATML Senior Researcher Freddie Kalaitzis, in collaboration with WhyHow Ltd. and ESA Phi-lab, has released the WorldStrat dataset: the largest dataset of open high resolution satellite imagery with a CC-NC license, stratified by global land-uses, with an application to Multi-Frame Super-Resolution. Find the paper and code. Congratulations to the whole team involved!

OATML Senior Research Fellow to lead €1M ESA project

11 May 2022

Freddie Kalaitzis will lead a €1M European Space Agency project to make Super Resolution technology for earth observations safer, more accountable, and open to everyone. Congratulations to the team!

Freddie Kalaitzis to give a spotlight talk at the AI for Earth Sciences workshop at NeurIPS 2020

05 Nov 2020

OATML Senior Research Fellow Freddie Kalaitzis will be giving a Spotlight Talk on Dynamic Hydrology Maps from Satellite-LiDAR Fusion at the AI for Earth Sciences workshop at NeurIPS 2020.

OATML Conference papers at NeurIPS 2022

OATML group members and collaborators are proud to present 8 papers at NeurIPS 2022 main conference, and 11 workshop papers. …

Full post...

---

Yarin Gal, Freddie Kalaitzis, Shreshth Malik, Lorenz Kuhn, Gunshi Gupta, Jannik Kossen, Pascal Notin, Andrew Jesson, Panagiotis Tigas, Tim G. J. Rudner, Sebastian Farquhar, Ilia Shumailov, 25 Nov 2022

OATML at ICLR 2022

OATML group members and collaborators are proud to present 4 papers at ICLR 2022 main conference. …

Full post...

---

Yarin Gal, Tuan Nguyen, Andrew Jesson, Pascal Notin, Atılım Güneş Baydin, Clare Lyle, Milad Alizadeh, Joost van Amersfoort, Sebastian Farquhar, Muhammed Razzak, Freddie Kalaitzis, 01 Feb 2022

22 OATML Conference and Workshop papers at NeurIPS 2020

OATML group members and collaborators are proud to be presenting 22 papers at NeurIPS 2020. Group members are also co-organising various events around NeurIPS, including workshops, the NeurIPS Meet-Up on Bayesian Deep Learning and socials. …

Full post...

---

Muhammed Razzak, Panagiotis Tigas, Angelos Filos, Atılım Güneş Baydin, Andrew Jesson, Andreas Kirsch, Clare Lyle, Freddie Kalaitzis, Jan Brauner, Jishnu Mukhoti, Lewis Smith, Lisa Schut, Mizu Nishikawa-Toomey, Oscar Key, Binxin (Robin) Ru, Sebastian Farquhar, Sören Mindermann, Tim G. J. Rudner, Yarin Gal, 04 Dec 2020