Using the PDS API

Before an API call to a PDS can be made, the connection between the members PDS and the connecting service must be authenticated. The details of the authentication process can be found here.

Dataset Types 🔗︎ click to copy

A Mydex member's stored data forms are divided into two categories: Datasets and Transactional Datasets. The distinction between these datasets is detailed in the Data Schema.

Return Format 🔗︎ click to copy

The Connection API can return a response to an API request in either JSON or XML. The formats are interchangeable and are chosen by putting the desired file extension in the API request (eg. .json or .xml). In the following examples we use JSON.

Multiple Dataset Instances 🔗︎ click to copy

Occasionally, a member will have more than one dataset instance in their PDS (eg. multiple home addresses). In this situation, a Connection API request will return all dataset instances that the member has granted you permission to access. These datasets are identified by the key instance_N, where N is the N^th dataset instance, starting from N=0.

Filtering Datasets 🔗︎ click to copy

Returning a whole dataset might not be practical or desirable. For example, if you want to return only the latest values within a dataset where changes have occurred since the last time this dataset was queried. This can be overcome by filtering the results. Filtering results is achieved by passing two additional query parameters in the request: filters[{field_name}][operator] and &filters[{field_name}][value]. For example, if you wanted to get dataset rows greater than a specific ID value, you would add parameters like this: &filters[id][operator]>= and &filters[id][value]=21, where 21 is the ID value you are filtering against. For more details on filtering plus example code, please see under the heading Filtering Datasets at the bottom of this page.

Connection API Request Differences 🔗︎ click to copy

The key differences when making a Connection API request for each dataset type are summarised in the table below.

	Dataset	Transactional Dataset
About	Multiple datasets with potentially multiple dataset instances	Multiple datasets with potentially multiple dataset instances
Availability	On Production (https://api.mydex.org) & Sandbox (https://sbx-api.mydex.org) servers	On Production & Sandbox servers
URL	Uses the `api/pds/pds` path structure	Uses the `api/pds/transaction` path structure
When Retrieving Data	The `uid` (`345` in the following examples) is part of the path structure	The `uid` is part of the path structure
	The dataset `instance` is not included in the request	The dataset `instance` must be included as a parameter (eg. `&instance=0`)
	The `dataset` is passed as parameter	The `dataset` is passed as parameter
When Adding Data	The `uid` is part of the path structure	The `uid` is passed as a parameter (eg. `?uid=345`)
	The dataset `instance` must be included as a parameter	The dataset `instance` must be included as a parameter
	The `dataset` must not be included in the data	The `dataset` must be included in the data and as a parameter
	Updates are made using HTTP PUT	Updates are made using HTTP POST
When Filtering Datasets	Results can be filtered by using one of the following operators: `('=', '<', '<=', '>', '>=', 'LIKE', 'IN', 'BETWEEN');`	Results can be filtered by using one of the following operators: `('=', '<', '<=', '>', '>=', 'LIKE', 'IN', 'BETWEEN');`
Working with assets (files)	See link to dedicated documentation

In addition to the above, Connection API requests should also include the key, api_key and source_type=connection values as parameters.

Error responses 🔗︎ click to copy

Although Mydex strives to prevent such issues, there is always the chance that the PDX API may return an error when attempting to read or write to a PDS.

Typically, this occurs if there is a syntax error with your payload, and we recommend you double check the examples in this documentation to avoid such cases.

Common errors 🔗︎ click to copy

An invalid dataset in the request may return an error such as:

  {
    "error": "2: Table does not exist"
  }

Or, for example, an incorrect connection ID:

  {
    "error": "No connection access record."
  }

Or an incorrect PDS:

  {
    "error": "PDS does not exist."
  }

Trying to access a dataset that your dedicated connection or member has not granted permission for your connection to access:

  {
    "error": "Permission denied (The user has not granted you access to this dataset or instance)"
  }

If the error is due to some other exception when reading from or writing to the PDS, a general error may look like this:

  {
    "error": "Problem writing to the PDS, request_id: 24c6331b-1f70-47ee-92a0-a7deb847a21d"
  }

Locking and contention on a PDS 🔗︎ click to copy

A contention issue (multiple requests to the same PDS at the same time) or a backend high availability storage issue can, in rare cases, temporarily impede access to the PDS.

Most requests to a PDS - even large batch inserts - typically take less than 1 second to complete, so contention issues between simultaneous requests to the same PDS are rare.

Mydex's PDX API allows up to 40 seconds for a lock on the PDS to clear. If the lock is still not cleared after that time, the PDX API will back off and then retry (once only) the same request, up to another 40 seconds. This all happens as part of the one HTTP request your application has made. Please allow for a total of up to 180s for your request to time out at Mydex's side before attempting to re-send the request.

If the PDS is still locked after the retry that the PDX API itself performs as part of that same single request, the PDX API will return an error in the HTTP response like this:

  {
    "error": "The PDS was busy, please retry again later, request_id: 4c19129b-e135-4972-8809-76f88fed554a"
  }

Mydex recommends that subscribers only attempt to re-send the same payload if they were returned a 502/503 timeout error, or another HTTP response that contains one of the errors above. It is also recommended to wait and retry a few minutes after such an incident, to increase the likelihood of the issue having been addressed.

If you are not sure whether to re-send a request, consider first attempting a GET (retrieve) request to check the last row in the dataset. If the data is not that of your last request, it may be that you should re-send the request - however, be aware that other subscribers may have a data sharing agreement with the member and so may be the originators of the last row (in which case, your last row might be earlier in the list) - so this technique is not a guarantee that your data should be re-sent.

If you receive an error, are not sure what to do, and it contains a request_id in the error message, please feel free to contact Mydex and we can cross-reference the request ID in our logs to help you troubleshoot the issue.

Examples 🔗︎ click to copy

Example Connection API requests for each specific dataset, and details of dataset fields can be found in the Data Schema. General examples for each dataset type can be found below.

Datasets 🔗︎ click to copy

Retrieving Data 🔗︎ click to copy

Example API Request 🔗︎ click to copy

https://sbx-api.mydex.org/api/pds/pds/345.json
    ?key=YfMPI2iSFcBDQ5CdlJVQeXIbIUD156s0
    &api_key=5FXfidz9zw6YQaO2kWh0B154zz6wafrV
    &con_id=345-1545
    &source_type=connection
    &dataset=ds_home
    &instance=0

Pastable:

https://sbx-api.mydex.org/api/pds/pds/345.json?key=5FXfidz9zw6YQaO2kWh0B154zz6wafrV&api_key=jPssYUMVVNaLaYPIbY7P9ESGzdksCOlf&con_id=345-1545&source_type=connection&dataset=field_ds_home

Example Response 🔗︎ click to copy

"1": {
   "id": "1",
   "field_home_bedrooms": "3",
   "field_home_country": "United Kingdom",
   "field_home_county": "Fake County",
   "field_home_date_from": "1327398114",
   "field_home_date_to": "1611481314",
   "field_home_habitable_rooms": "2",
   "field_home_owner_street": "Fake Street",
   "field_home_owner_town": "Fake Town",
   "field_home_postcode": "G92 3KJ",
   "field_home_property_type": "House",
   "field_home_street": "18 Fake Street",
   "field_home_tenancy": "Lease Tenant",
},

Note: not all of the fields have been included in this example to improve readability.

Updating Data 🔗︎ click to copy

It is possible to update numerous fields in multiple dataset instances with your own data by issuing a PUT request to the Connection API.

Note: you can only update a record that exists. We recommend you first make a GET request to the dataset determine if the record exists. You will need the ID of the record you wish to update.

Example API Request 🔗︎ click to copy

As you saw in the previous example, the dummy member we created was given a dataset instance. We will now submit an API request to update select details in the dataset instance.

Many browsers don't natively support PUT, so the following example uses curl to make a PUT request. This example request has been restructured to improve readability.

curl
    -X PUT
    -H "Content-Type: application/json"
    -d '{
           "field_home_bedrooms": "2",
           "field_home_owner_name": "I. M. Landlord"
        }'
    'https://sbx-api.mydex.org/api/pds/pds/345.json?
        key=5FXfidz9zw6YQaO2kWh0B154zz6wafrV
        &api_key=jPssYUMVVNaLaYPIbY7P9ESGzdksCOlf
        &con_id=345-1545
        &dataset=ds_home
        &source_type=connection
        &instance=0
        &id=1'

This request changes the field_home_bedrooms field value and adds the field_home_owner_name field value. Tables of all available datasets and details of all their fields can be found in the Data Schema.

This request can also be made via a proxy PHP script. An example of how the script could be written is shown below.

<?php

$user_id = '345';

$instance[0] = 0;

$data = array();
$data['field_ds_home'][$instance[0]]['field_home_bedrooms'] = '2';
$data['field_ds_home'][$instance[0]]['field_home_owner_name'] = 'I. M. Landlord';
$data = (is_array($data)) ? http_build_query($data) : $data;

$arguments_array = array(
  'key=5FXfidz9zw6YQaO2kWh0B154zz6wafrV',
  'api_key=jPssYUMVVNaLaYPIbY7P9ESGzdksCOlf',
  'con_id=345-1545',
  'source_type=connection',
  'dataset=ds_home',
  'instance=0',
  'id=1'
);
$arguments = implode('&', $arguments_array);

$mydex_api_url = "https://sbx-api.mydex.org/api/pds/pds/" . $user_id . ".json?" . $arguments;

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, $mydex_api_url);
curl_setopt($ch, CURLOPT_HTTPHEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'PUT');
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);

$mydex_request = curl_exec($ch);
$mydex_request_decode = json_decode($mydex_request);

curl_close($ch);

?>

Example Response 🔗︎ click to copy

A successful update request via PUT will return a status code of 200 OK and echo the updated fields. For example, a response to the previous example will yield:

{
    "success": "1",
    "transaction": {
        "field_home_bedrooms": "2",
        "field_home_owner_name": "I. M. Landlord"
    }
}

If unsuccessful, an error will be returned with a suggestion on how to correct the API request.

Creating a new row 🔗︎ click to copy

If the PDS dataset has no records eg. returns an empty array when making a GET request to the API, you will need to create a row. This can be done using a POST request.

curl
    -X POST
    -H "Content-Type: application/json"
    -d '{
         "ds_home": {
            "field_home_bedrooms": "3",
            "field_home_country": "England",
            "field_home_county": "Essex",
            "field_home_date_from": "1327398114",
            "field_home_date_to": "1611481314",
            "field_home_num_residents": "4",
            "field_home_owner_country": "United Kingdom",
            "field_home_owner_county": "Anycounty",
            "field_home_owner_email_address": "landlord@landlord.com",
            "field_home_owner_name": "I. M. Landlord",
            "field_home_owner_postcode": "AB12 3CD"
         }
      }'
    'https://sbx-api.mydex.org/api/pds/pds/345.json?
        key=5FXfidz9zw6YQaO2kWh0B154zz6wafrV
        &api_key=jPssYUMVVNaLaYPIbY7P9ESGzdksCOlf
        &con_id=345-1545
        &dataset=ds_home
        &source_type=connection
        &instance=0

If successful, the API will return the following response:

{
    "success": "1",
    "last_insert_id": "1"
}

Note: you will need to use the value of 'last_insert_id' when updating the dataset. For Metadata datasets this will normally always be '1'.

Transactional Datasets 🔗︎ click to copy

Retrieving Data 🔗︎ click to copy

Example API Request 🔗︎ click to copy

https://sbx-api.mydex.org/api/pds/transaction/345.json
   ?key=5FXfidz9zw6YQaO2kWh0B154zz6wafrV
   &api_key=jPssYUMVVNaLaYPIbY7P9ESGzdksCOlf
   &con_id=345-1545
   &source_type=connection
   &dataset=ds_browsing_history
   &instance=0

Pastable:

https://sbx-api.mydex.org/api/pds/transaction/345.json?key=5FXfidz9zw6YQaO2kWh0B154zz6wafrV&api_key=jPssYUMVVNaLaYPIbY7P9ESGzdksCOlf&con_id=345-1545&source_type=connection&dataset=ds_browsing_history&instance=0

Example Response 🔗︎ click to copy

{
   "1":{
      "id":"1",
      "ds_id":"1",
      "browsing_history_url":"http://www.youtube.com",
      "browsing_history_title":"YouTube",
      "browsing_history_visit_date":"1397550508"
   },
   "2":{
      "id":"2",
      "ds_id":"2",
      "browsing_history_url":"http://www.youtube.com",
      "browsing_history_title":"YouTube",
      "browsing_history_visit_date":"1397550512"
   }
}

Adding Data 🔗︎ click to copy

The key changes when adding a transactional dataset are outlined in the table above.

Example API Request 🔗︎ click to copy

curl
   -X POST
   -H "Content-Type: application/json"
   -d '{
         "ds_browsing_history": {
            "browsing_history_url": "http://www.youtube.com",
            "browsing_history_title": "YouTube",
            "browsing_history_visit_date": "1397550510"
         }
      }'
   'https://sbx-api.mydex.org/api/pds/transaction.json
      ?uid=345
      &key=YfMPI2iSFcBDQ5CdlJVQeXIbIUD156s0
      &api_key=jPssYUMVVNaLaYPIbY7P9ESGzdksCOlf
      &con_id=345-1545
      &source_type=connection
      &dataset=ds_browsing_history
      &instance=0'

This request can also be made via a proxy PHP script. An example of how the script could be written is shown below.

<?php

$user_id = '345';

$instance[0] = 0;

$data = array();

$data['ds_browsing_history']['browsing_history_url'] = 'http://www.youtube.com';
$data['ds_browsing_history']['browsing_history_title'] = 'YouTube';
$data['ds_browsing_history']['browsing_history_visit_date'] = '1397550510';


$data = (is_array($data)) ? http_build_query($data) : $data;

$arguments_transactional_array = array(
'uid=' . $user_id,
'api_key=jPssYUMVVNaLaYPIbY7P9ESGzdksCOlf',
'source_type=connection',
'con_id=345-1545',
'key=5FXfidz9zw6YQaO2kWh0B154zz6wafrV',
'instance=' . $instance[0],
'dataset=ds_browsing_history',
);
$arguments_transactional = implode('&', $arguments_transactional_array);

$mydex_api_url = "https://sbx-api.mydex.org/api/pds/transaction.json?" . $arguments_transactional;

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, $mydex_api_url);
curl_setopt($ch, CURLOPT_HTTPHEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'POST');
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);

$mydex_request = curl_exec($ch);
$mydex_request_decode = json_decode($mydex_request);

curl_close($ch);

?>

Example Response 🔗︎ click to copy

A successful request will return a HTTP status code of 200 OK and the line {"success":1}. An unsuccessful request will return an HTTP error code and a message explaining the problem.

Adding Batch Data 🔗︎ click to copy

If you wish to batch-send multiple rows of data in a single request, then you can used a nested array of data in the request body. The raw JSON would look like this:

{
   "ds_energy_consumption": [
      {
         "created_timestamp": "1641909333",
         "field_energy_homeid": "1",
         "field_energy_reading_timestamp": "1648633780",
         "field_energy_reading_value": "1123.25",
         "field_energy_source_index_id": "1",
         "field_energy_source_name": "Kettle",
         "field_energy_source_type": "CAD",
         "updated_timestamp": "1641909333"
      },
      {
         "created_timestamp": "1641909333",
         "field_energy_homeid": "1",
         "field_energy_reading_timestamp": "1648633780",
         "field_energy_reading_value": "1153.50",
         "field_energy_source_index_id": "1",
         "field_energy_source_name": "Kettle",
         "field_energy_source_type": "CAD",
         "updated_timestamp": "1641909333"
      }
   ]
}

Example Batch API Request 🔗︎ click to copy

curl
   -X POST
   -H "Content-Type: application/json"
   -d '{
      "ds_energy_consumption": [
         {
            "created_timestamp": "1641909333",
            "field_energy_homeid": "1",
            "field_energy_reading_timestamp": "1648633780",
            "field_energy_reading_value": "1123.25",
            "field_energy_source_index_id": "1",
            "field_energy_source_name": "Kettle",
            "field_energy_source_type": "CAD",
            "updated_timestamp": "1641909333"
         },
         {
            "created_timestamp": "1641909333",
            "field_energy_homeid": "1",
            "field_energy_reading_timestamp": "1648633780",
            "field_energy_reading_value": "1153.50",
            "field_energy_source_index_id": "1",
            "field_energy_source_name": "Kettle",
            "field_energy_source_type": "CAD",
            "updated_timestamp": "1641909333"
         }
      ]
   }'
   'https://sbx-api.mydex.org/api/pds/transaction.json
      ?uid=345
      &key=YfMPI2iSFcBDQ5CdlJVQeXIbIUD156s0
      &api_key=jPssYUMVVNaLaYPIbY7P9ESGzdksCOlf
      &con_id=345-1545
      &source_type=connection
      &dataset=ds_browsing_history
      &instance=0'

Note: there is a limit of 1000 rows of data per request.

The request is exactly the same as the one above except we now have a nested array in the data payload. The parameters and endpoints are the same as adding data.

Updating Data 🔗︎ click to copy

Example API Request 🔗︎ click to copy

curl
   -X PUT
   -H "Content-Type: application/json"
   -d '{
         "browsing_history_url": "http://www.youtube.com",
         "browsing_history_title": "YouTube",
         "browsing_history_visit_date": "1397550510"
      }'
   'https://sbx-api.mydex.org/api/pds/transaction/345.json
      &key=YfMPI2iSFcBDQ5CdlJVQeXIbIUD156s0
      &api_key=jPssYUMVVNaLaYPIbY7P9ESGzdksCOlf
      &con_id=345-1545
      &source_type=connection
      &dataset=ds_browsing_history
      &instance=0
      &id=3'

This request can also be made via a proxy PHP script. An example of how the script could be written is shown below.

<?php

$user_id = '345';

$instance[0] = 0;

$data = array();

$data['browsing_history_url'] = 'http://www.youtube.com';
$data['browsing_history_title'] = 'YouTube';
$data['browsing_history_visit_date'] = '1397550510';


$data = (is_array($data)) ? http_build_query($data) : $data;

$arguments_transactional_array = array(
'uid=' . $user_id,
'api_key=jPssYUMVVNaLaYPIbY7P9ESGzdksCOlf',
'source_type=connection',
'con_id=345-1545',
'key=5FXfidz9zw6YQaO2kWh0B154zz6wafrV',
'instance=' . $instance[0],
'dataset=ds_browsing_history',
'id=3'
);
$arguments_transactional = implode('&', $arguments_transactional_array);

$mydex_api_url = "https://sbx-api.mydex.org/api/pds/transaction/345.json?" . $arguments_transactional;

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, $mydex_api_url);
curl_setopt($ch, CURLOPT_HTTPHEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'PUT');
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);

$mydex_request = curl_exec($ch);
$mydex_request_decode = json_decode($mydex_request);

curl_close($ch);

?>

For more information on how to structure the data for transactional data sets, see the Data Schema.

Example Response 🔗︎ click to copy

A successful update request will return a HTTP status code of 200 OK and the line {"success":1}. An unsuccessful request will return an HTTP error code and a message explaining the problem.

Filtering Datasets 🔗︎ click to copy

The following code shows an example READ request filtered by ID with a value >= 21. For example, this could be the the value of the record returned from the previous query.

https://sbx-api.mydex.org/api/pds/transaction/345.json
   ?key=YfMPI2iSFcBDQ5CdlJVQeXIbIUD156s0
   &api_key=5FXfidz9zw6YQaO2kWh0B154zz6wafrV
   &con_id=345-1545
   &source_type=connection
   &instance=0
   &dataset=ds_energy_consumption
   &filters[id][operator]>=
   &filters[id][value]=21 // eg.the previous id

Note the last two parameters &filters[id][operator]>= and &filters[id][value]=21.

A typical request would look like the following:

{{PDS_API_URL}}/api/pds/transaction/3490.json?key=xxxxxxxxxxxxxxxxxxxxxxxxxxxx&instance=0&api_key=xxxxxxxxxxxxxxxxxxxxxx&con_id=3490-40212&source_type=connection&dataset=ds_energy_consumption&filters[id][operator]>=&filters[id][value]=5

Any of the following operators can be used to filter results:

('=', '<', '<=', '>', '>=', 'LIKE', 'IN', 'BETWEEN');

When using operators like BETWEEN (where we have two values) we need two value parameters e.g. &filters[{field_name}][value][] and &filters[{field_name}][value][] (Note the empty square brackets to define the array).

For example, if we wanted to get rows between 3 and 5, we'd have the operator &filters[id][operator]=BETWEEN and the two values: &filters[id][value][]=3 and &filters[id][value][]=5.

The request would look like this:

{{PDS_API_URL}}/api/pds/transaction/3490.json?key=xxxxxxxxxxxxxxxxxxxxxxxxxxxx&instance=0&api_key=xxxxxxxxxxxxxxxxxxxxxxxxxxxx&con_id=3490-40212&source_type=connection&dataset=ds_energy_consumption&filters[id][operator]=BETWEEN&filters[id][value][]=3&filters[id][value][]=5