DENSER (DEeply uNderstanding Space weathER)

Architecture

A common architecture is shared for both products (ForMaL-SEP and ForMaL-Xrange), ie high-level components use the same code for both product.

The figure hereafter presents the DENSER Architecture:

DENSER Architecture

A design driver for this architecture was to make it generic when possible and to specialize sub-components only when needed.

All major building components of the architecture are dockerized and they are orchestrated with docker-compose. In the context of the DENSER project, the complete architecture was instantiated on a high-performance server at the premises of Space Applications Services.

• Data storage: Stores all data used and generated in the project, for both products.
  This includes:
  • input training/validation/testing data.
  • Input data used to generate predictions.
  • generated predictions and metadata about those predictions. The metadata contains the time needed to run the prediction and the status of the prediction (successful/failed)
• Data cleaning: Provide functionality to download, extract, pre-process and store pre-processed raw data into the central storage.
• REST API Provider: This component is central in the architecture as it provides the liaison between the various data producers and data consumers.
• Front end: End-user interface of both products.
• Reverse proxy: Provides access to both products on a single server.
• First line Reverse Proxy: Provides OpenAM agent, and HTTPS termination.
• Deployed ML models: Used for generating predictions.

Models

Four different models have been considered and implemented.

Finally the Recurrent Neural Network (RNN) based on LSTM (Long Short-Term Memory) architecture has been selected for the advanced validation.

The RNN Model is a recurrent model. It means that it can learn to use a long history of inputs, if it is relevant to the predictions the model is making. The model accumulates a series of timesteps, and makes a prediction of a series of future timesteps. This model can be parametrized with a number of lstm units. Adding too many lstm units can make the model overfit more quickly.

Training Environment

The available (pre-processed) input data is, for ML activities, used for three purposes:

• Training data: to learn the internal weights of an ML model
• Validation data: to perform an unbiased evaluation of the model while tuning the model by optimizing the models hyperparameters
• Test data: to perform an unbiased evaluation of the final model fit.

Model building and model training is performed in Jupyter notebooks, instantiated in GPU-enabled Docker containers (one per DENSER product) that are based on a single Docker image.

The figure hereafter presents the training environment for the DENSER models:

DENSER Training and Execution environments

This allows for quick development as blocks of python code can be (re)evaluated without needing to evaluate a larger codebase each time a change is made. This caters well to the highly iterative process of ML model development. In addition, Jupyter combines live code, visualizations and explanatory texts in a single entity, further inviting and facilitating ML- based experimentation and, afterwards, documentation.

Code that is shared between the various notebooks are put into separate python files and are imported in the various notebooks. Also the final trained models are handled this way. Given the pre-processed input data, initial feature engineering activities whereof it is suspected that these activities might be beneficial for all ML models, has been performed. Using the resulting set of features of data points, the ML model development process has started with implementation of the purged walk forward cross validation data splitting mechanism, allowing a model to draw from the resulting sets of training/validation and testing data.

Interface

The application provides a specific interface for each product that takes into consideration the specificity of the products.

orecasts can be browsed using the calendar. Then a viewport displays the prediction, and possibly the measurements in cast of a past prediction.

Markers provide hints such as the storm levels, or the different horizons.

The figures hereunder presents the DENSER interface for both ForMaL-SEP and ForMaL-Xrange products:

ForMaL-SEP interface

ForMaL-Xrange interface

Products performance

The validation of the two products has been an essential part of the DENSER project.

ForMaL-SEP

The main conclusion from the validation activities is that ForMaL-SEP has a good POD (Probability of detection) and AWT (Average warning time) compared with other models, but at the cost of a very high FAR (False alarm ratio). The model also shows a tendency to underpredict the expected maximal value on shorter time scales and overpredict for longer prediction horizons. While looking at the behaviour of the predictions during individual events, it was observed that for gradual events (flare location not magnetically connected to Earth) first a positive prediction is issued, then the predicted event length decreases since the eruption is not immediately followed by an increase in proton flux, but once they start rising the predicted event length increases again. The validation report that has been written for this task also lists several other issues at the model level. One of the main issues is the large class imbalance between the active and quiet periods. From the results provided by a first basic validation activity, some changes were applied to the model training in order to try to fix issues with respect to the background subtraction of the input data taken with different instruments, the response to flat, rising and decaying inputs. The number of solutions have been identified to improve the validation results. However, the output did not meet the expectations. Therefore, further work is needed in future to determine additional validation methods.

ForMaL-Xrange

The validation of the ForMaL-Xrange product used historical data collected during the years 1998-2018 and is based on a selection of 21 events of strong geomagnetic activity (Kp>8).

The main conclusion from the event-based validation is that during strong geomagnetic storms 1H forecasts by ForMaL-Xrange do not catch the peak values in magnetic field north component variations, but they show good performance in estimating the different activity levels (minor, moderate, strong, severe). This conclusion does not depend significantly on the location of the stations at the storm starting time. Comparison between the 1H forecasts by the AurorasNow! service and the ForMaL-Xrange forecasts show that the latter are in RMSE-sense better than the statistical forecast based on 5, 50 and 95 percentiles used in Auroras Now. Aurora enthusiasts today use the 95 percentiles by AurorasNow! in their decision-making. If that value is in the category of strong or severe activity, then the chances to get nice auroral photos is better than in average conditions. Our conclusion is that adding 1H forecasts by ForMaL-Xrange to the tables presented in AurorasNow! would improve the accuracy of the service.

Validation details are documented in the DENSER Validation Report.

Recommendations and roadmap

The overall DENSER project has led to define a list of recommendations and possible future developments:

ForMaL-SEP

ForMaL-Xrange