Custom Connector
Qatar Open Data Portal Connector
Qatar
The Qatar Open Data Connector enables seamless integration with the Qatar Open Data Portal, providing access to catalog and dataset data for analytics, visualization, and GIS. It acts as a proxy for dataset enumeration, record queries, data exports, and facet-based navigation.
Overview
Integration Overview
This document outlines each integration point, its purpose, configuration, and workflow support using the Qatar Open Data Connector (OpenDataSoft v2.1).
- getDatasets. Retrieves a list of all datasets in the catalog.
- listExportFormats / exportDatasets / exportCatalogCSV / exportCatalogDCAT. List and run catalog exports in a chosen format, CSV, or DCAT-AP.
- getDatasetsFacets. Retrieves global catalog-level facets.
- getDataset / getRecords / getRecord. Retrieve dataset metadata, its records, or a single record.
- listDatasetExportFormats / exportRecords / CSV / Parquet / GPX. List and run per-dataset exports in a chosen format.
- getRecordsFacets. Retrieves available facets for a dataset.
- getDatasetAttachments. Retrieves files or attachments related to a dataset.
Documentation
Detailed Integration Documentation
Catalog Datasets Retrieval
| Action | getDatasets |
|---|---|
| Purpose | Retrieves a comprehensive list of all datasets in the catalog — the primary entry point for exploring available data. |
| Configuration | Configure the connector with the appropriate environment variables for authentication and connectivity. |
| Parameters | Optional: select; where (ODSQL, e.g. publisher=“Planning and Statistics Authority”); order_by; limit (default 10, max 100); offset; refine; exclude; lang; timezone (e.g. Asia/Qatar); group_by; include_links; include_app_metas. |
| Output | Successful: JSON with total_count, _links, and a results array of dataset objects (dataset_id, fields, metas). Failure: invalid parameters. |
| Workflow example | Execute getDatasets to fetch datasets and identify one for further actions. |
Catalog Exports (list / format / CSV / DCAT)
| Action | listExportFormats / exportDatasets / exportCatalogCSV / exportCatalogDCAT |
|---|---|
| Purpose | Lists catalog export formats and exports the catalog in a chosen format, CSV (delimiter options), or RDF/XML (DCAT-AP). |
| Configuration | Configure the connector with the appropriate environment variables for authentication and connectivity. |
| Parameters | exportDatasets required: format (csv, json, xlsx). CSV optional: delimiter; list_separator; quote_all; with_bom. DCAT required: dcat_ap_format (e.g. dcat_ap_ch); optional include_exports, use_labels_in_exports. Plus catalog filters. |
| Output | Successful: a JSON list of export links or a file (CSV / RDF-XML). Failure: unsupported format / upstream API error. |
| Workflow example | Call listExportFormats, then exportCatalogCSV with delimiter=, for analysis. |
Catalog Facets Retrieval
| Action | getDatasetsFacets |
|---|---|
| Purpose | Retrieves catalog-level facet values (e.g. publisher) for guided navigation. |
| Configuration | Configure the connector with the appropriate environment variables for authentication and connectivity. |
| Parameters | Optional: facet (e.g. publisher); refine; exclude; where; timezone. |
| Output | Successful: JSON with links and facets arrays. Failure: invalid facet. |
| Workflow example | Execute getDatasetsFacets with facet=publisher to refine dataset queries. |
Dataset Metadata & Records
| Action | getDataset / getRecords / getRecord |
|---|---|
| Purpose | Retrieves metadata for a specific dataset; queries its records; and reads a single record by ID. |
| Configuration | Configure the connector with the appropriate environment variables for authentication and connectivity. |
| Parameters | Required: dataset_id (getRecord also needs record_id). Optional: select; where (ODSQL); group_by; order_by; limit (max 100); offset; refine; exclude; lang; timezone; include_links; include_app_metas. |
| Output | Successful: dataset metadata (dataset_id, fields, metas), a results array of records, or a single record. Failure: invalid dataset_id/record_id. |
| Workflow example | Use getDataset to select a dataset, getRecords with a where filter, then getRecord with a record_id. |
Dataset Exports (formats / CSV / Parquet / GPX)
| Action | listDatasetExportFormats / exportRecords / exportRecordsCSV / exportRecordsParquet / exportRecordsGPX |
|---|---|
| Purpose | Lists a dataset’s export formats and exports its records in a chosen format, CSV, Parquet (snappy/zstd), or GPX. |
| Configuration | Configure the connector with the appropriate environment variables for authentication and connectivity. |
| Parameters | Required: dataset_id (and format for exportRecords — e.g. csv, geojson). Optional: select, where, order_by, group_by, limit, refine, exclude, lang, timezone, use_labels, epsg; CSV delimiter options; parquet_compression; GPX name_field/description_field_list/use_extension. |
| Output | Successful: a file in the requested format. Failure: invalid format. |
| Workflow example | Execute exportRecordsParquet with a dataset_id, then ingest into BigQuery. |
Record Facets & Attachments
| Action | getRecordsFacets / getDatasetAttachments |
|---|---|
| Purpose | Retrieves facet values for a dataset’s records (guided navigation) and lists a dataset’s files/attachments. |
| Configuration | Configure the connector with the appropriate environment variables for authentication and connectivity. |
| Parameters | Required: dataset_id. Optional (facets): facet; where; refine; exclude; lang; timezone. |
| Output | Successful: facet enumerations (links + facets), or an attachments array (href, mime-type, title). Failure: invalid dataset_id/facet. |
| Workflow example | Use getRecordsFacets with facet=year to refine, and getDatasetAttachments to download supplementary files. |
Example Workflow: Exploring & Exporting Population Statistics
| Retrieve datasets | Use getDatasets to fetch datasets and identify a target (e.g. population-data). |
|---|---|
| Refine & query | Use getRecordsFacets with facet=year, then getRecords with where=year:2020. |
| Export | Use exportRecordsCSV / exportRecordsParquet / exportRecordsGPX (name_field=municipality) for the records. |
| Metadata & catalog | Fetch metadata via getDataset and attachments via getDatasetAttachments; export the catalog via exportCatalogCSV or exportCatalogDCAT. |
Use Case
The Qatar Open Data Connector enables seamless integration with the Qatar Open Data Portal, providing access to catalog and dataset-related data for analytics, visualization, and GIS applications. Acting as a proxy, it streamlines data retrieval — dataset enumeration, record queries, data exports (CSV, Parquet, GPX, DCAT), and facet-based navigation — for building data-driven applications on GCP.
Support
For technical support, contact custom-connectors-support@isolutions.sa.