Data Connectors
A data connector connects Geo Engine to an external data source.
It makes the data of that source available as layers and layer collections and as external datasets, without copying it into Geo Engine.
Source operators load external data with the same data parameter as internal datasets, using the ExternalDataId.
Configuring data connectors
Section titled “Configuring data connectors”A data connector is defined by a JSON definition.
Its type selects the kind of data connector, and its other fields configure it, e.g., the URL of the source, credentials, or a cache lifetime.
Every data connector has a name, a description and a priority that orders the data connectors in listings.
Some types take an id in their definition; the others, e.g., Gbif or Pangaea, always use the same fixed id.
Administrators manage data connector definitions with the /layerDb/providers endpoints or the manager app.
When the server creates its database schema, i.e., on the first start with an empty schema, it also loads the definitions in [dataprovider] provider_defs_path.
Files added there later are not loaded on a restart.
Access to a data connector is controlled by permissions, like access to datasets.
STAC credentials are stored in PostgreSQL.
Set [dataprovider] password_encryption_key (or GEOENGINE__DATAPROVIDER__PASSWORD_ENCRYPTION_KEY) to store them encrypted.
Available data connectors
Section titled “Available data connectors”| Type | Data source |
|---|---|
StacProviderDefinition |
A collection of a STAC API, split into datasets of bands with the same data type, resolution and projection |
SentinelS2L2ACogs |
Sentinel-2 Level-2A imagery as Cloud Optimized GeoTIFFs, found through a STAC API |
CopernicusDataspace |
Imagery of the Copernicus Data Space Ecosystem, via its STAC and S3 endpoints |
NetCdfCf |
NetCDF files that follow the CF conventions, on a server volume |
EbvPortal |
Essential Biodiversity Variables (EBV) cubes of the EBV Data Portal, stored as NetCDF |
Edr |
An OGC API – Environmental Data Retrieval service |
Gbif |
Species occurrences of GBIF, from a PostgreSQL copy of the GBIF data |
GfbioAbcd |
Biodiversity data in the ABCD standard of the GFBio collections, from PostgreSQL |
GfbioCollections |
Collections of the GFBio collection API, combined with ABCD and Pangaea data |
Pangaea |
Datasets of the PANGAEA data publisher |
WildLIVE! |
Camera trap data of the WildLIVE! portal |
DatasetLayerListing |
Not an external source: lists internal datasets as layers, grouped into collections by tags |
The fields of each type are part of the OpenAPI specification, under TypedDataProviderDefinition.
Using data from data connectors
Section titled “Using data from data connectors”You do not need to know which data connector a layer comes from to use it. Browse the layer collections of a data connector, then register the layer as a processing graph:
curl -X POST http://localhost:3030/api/layers/<data connector id>/<layer id>/processingGraphId \ -H "Authorization: Bearer $TOKEN"To keep a copy of external raster data in Geo Engine, POST /layers/{provider}/{layer}/dataset persists a layer as an internal dataset.