Plugins
Each research center has a different IT infrastructure and works with different frameworks. The aim of the model data explorer is to integrate these different systems into a user-friendly web-application. In order to be able to deploy the model data explorer at the different research centers, we aim for a modular structure where each component can be added or removed depending on the setup at the hosting institution.
This modular structure is implemented via plugins. Each plugin is a python package containing a django app that can be added or removed from the configuration. We distinguish between three plugin categories: core, framework and service plugins.
Creating plugins
To facilitate the creation of MDE-Apps, we have a cookiecutter-template that can be used to generate the basic structure of an MDE-App. The template is accessible at https://codebase.helmholtz.cloud/model-data-explorer/mde-app-template.
Core plugins
These plugins are essential to run the model data explorer as they define
central models that each plugin depend on. Currently we only aim for one
core plugin: the mde-core
plugin (see Models).
Framework plugins
These plugins define optionally but commonly used features for other plugins. These platforms should be independent of the IT infrastructure and run everywhere. We currently aim for three different framework plugins that can be used by other plugins:
mde-config
The configuration plugin defines a viewset-based structure (analog to the
django.admin
package) where service plugins can provide configuration interfaces for individual datasets.mde-config
additionally provides user configuration interfaces for themde-core
models.Features
Defines plugin-based configuration interface
Django-based template rendering using material design and a viewsets (like django admin)
provides interface to configure dataset - metadata - relations to datagroups - user roles - plugin structure for more
provides interface to configure datagroup - metadata - relations to datagroups - user roles - plugin structure for more
uses
mde-notifications
to send out notifications
mde-federation
The federation plugin to combine multiple MDE instances at different research centers. Admins can configure a federation between two instances to synchronize data groups and datasets. Furthermore, users can connect from one instance to the other using the
mde-oauth
plugin to synchronize the permissions between the two instances.mde-notifications
The notification plugin implements the functionalities to send out notifications to the users. Notifications are usually send out when we need confirmations by dataset or datagroup owners. The
mde-notifications
app additionally provides an interface to the user for configuring notifications.This plugin can also be used by service plugins to send out notifications.
Features
provides functionality to send notifications to the user
one websocket per user (for display of notifications in the browser)
email notifications
webpush notifications (e.g. to approve actions)
creates notifications when a user asks for permissions
provides interface to configure notifications on a dataset, datagroup and global level
optionally collates multiple notifications into single mail
mde-oauth
The OAuth plugin implements the possibility for the model data explorer to serve as a OAuth or SAML Provider. This gives service plugins the possibility to authenticate against the model data explorer and get the groups for the user.
Service plugins
Service plugins connect a dataset of the mde-core
to an external service.
This way, the model data explorer is configuring this external service. Service
plugins should be independent of other service plugins (in possible) but they
can depend on the above mentioned Framework plugins.
Service plugins can be based on a specific IT infrastructure. Currently we aim for the following setup at the Helmholtz-Zentrum Hereon.
These plugins can, however, be used at other research centers. The service plugins that we are developping are:
mde-viewer
A map-based viewer for datasets. This viewer provides a map-interface using ArcGIS4Javascript and static, angular-based frontend to display all metadata and services for an individual dataset. Services that are displayed in this viewer are configured through the mde-esri, mde-thredds, mde-analysis and mde-download plugins.
The layout for this viewer is designed in the FIGMA prototype accessible at https://www.figma.com/proto/FoCucLiWsUTfEQ6Fnob19s/Model-Data-Explorer-Portal.
Features
Angular and Material based Map frontend for datasets
provides selection of OGC/ESRI services for display
provides download selection (interface to mde-download)
provides analysis functionalities (interface to mde-analysis)
provides plugin-based design for additional panels
Todo
think more about how to implemenent plugin-based panel design in
mde-viewer
implements configuration for service catalog for datasets (plugin for mde-config)
WMS (THREDDS, ESRI)
WFS (includes unstructured grids)
ESRI ImageService
mde-thredds
A configuration interface for a THREDDS-Server. This app expects that the configuration files of a THREDDS-Server are mounted on the server that runs the model data explorer web frontend. The
mde-thredds
app provides a convenient web interface to write the XML-files for a THREDDS catalog and, and it can restart the THREDDS server when the configuration changed via a call to the TOMCAT manager.Features
Defines catalog per dataset
Defines catalog per datagroup
mounts MAMS-workspaces (
mde-mams
)Includes configuration interface for dataset catalog entries for all THREDDS-functionalities (plugin for
mde-config
)creates WMS for
mde-viewer
Creation of ISO-files via NcML-ISO-Mapping
Catalogs implement permissions based on groups with view permissions
User creates catalog entries via plugin for
mde-config
mde-csw
The metadata publisher for the model data explorer.The
mde-csw
plugin can combine multiple ISO-files (e.g. generated via THREDDS) and provides an OGC-compliant CSW endpoint via pycswTodo
Think more about
mde-csw
mde-mams
The data storage for the model data explorer. At Hereon, we host an instance of the
Media Assets Management System
, also called MAMS. MAMS is also deployed at other research centers, namely Geomar and AWI. Throughmde-mams
, users can request workspaces in MAMS to upload netCDF files. These workspaces are additionally mounted on the THREDDS-Server.Features
Creates workspaces on MAMS for each dataset via RestAPI
Mirrors permissions and group memberships of the user via RestAPI
User requests workspace via plugin for
mde-config
mde-analysis
The Analysis framework for the model data explorer. Based upon the Data Analytics Software Framework (DASF) this plugin provides the framework to compute aggregated analysis on datasets without the need to download the raw data.
Features
implements computation and download of aggregated statistics via DASF-backend modules
User registers backend modules (i.e. DASF BrokerTopics) via plugin for
mde-config
Permissions can be based on view permissions for the dataset or specified manually
provides Python-API to make computations via script (including authentication)
sends out a notification when the computation is done
sends notification to dataset owner when the backend module cannot be reached
mde-download
The download framework for the model data explorer. As
mde-analysis
based upon the DASF. This plugin provides the framework to download subsets of the data.Features
implements download of a subset of the original data via DASF-backend modules
The backend modules need to upload everything to file-exchange services such as SWIFT at DKRZ and provide a download link
User registers backend modules (i.e. DASF BrokerTopics) via plugin for
mde-config
Permissions can be based on view permissions for the dataset or specified manually
Provides Python-API to make download-requests via script
sends out a notification when the download is ready
sends notification to dataset owner when the backend module cannot be reached