Master Schema Extension Good Practice Guide

Introduction🔗︎ click to copy

The Mydex Master Data Schema is a person-centric data model designed to be extensible to cover any and every aspect of a persons life over their whole lifetime. This is the data model used to support the Mydex Personal Data Store (PDS) platform.

The purpose of this good practice guide is to:

• explain the rationale behind the structure of the Master Data Schema;
• show how easily it can be extended to cover other use cases outside of the current system;
• facilitate community engagement with its ongoing extension and development;
• set out the recommended approach for extending the schema;
• give some worked examples of creating additional fields within the schema.

The critical point here is that it is person-centric - it does not take the standpoint of any particular Subscribing organisation, whether they be companies, charities or even standards organisations. If you look at the use of your personal data by Subscribing organisations you will soon see that there are bits of data scattered across all of the Subscribing organisations you have dealings with, with nobody having a complete picture - not even you. These Subscribing organisations may communicate amongst themselves or with you, but the complex arrangements can lead to inaccurate data and be hard to manage. This can be thought of as a ‘crown of thorns’, as shown in the diagram below.

The alternative approach is person-centric, putting the individual the centre of the web of data connections and firmly back in control. As the individual is now responsible for their own data and is able to share this with Subscriber organisations, the accuracy and completeness of the data improves and a greater degree of trust can be attained between the parties. This is shown in the ‘halo of trust’ diagram below.

The Mydex PDS provides the platform for individuals to create the halo of trust with the Subscribing organisations they deal with. This is explained in detail in the Mydex Charter and the Privacy Policy, but in summary the platform provides the individual with:

• A secure area for the storage and management of their personal data. This is unique to them and nobody else can gain access to this data, not even Mydex;
• The ability to securely connect to the Subscribing organisations they have dealings with in order to maintain an ongoing relationship;
• The tools to be able to consolidate, view, analyse and manage their data.

Schema Structure🔗︎ click to copy

Behind the PDS lies the data model which supports the storage and manipulation of all of the personal data fields. This is termed the Mydex Master Data Schema.

When we started designing the Mydex schema, there were a number of design constraints we imposed upon ourselves in order to make life simpler:

1. The schema, first and foremost, must be person-centric, i.e. to allow the individual to record all aspects of their digital life.
2. The design of the schema should incorporate best practice wherever possible. Where we find a publicly available schema for a particular use case, we may adopt the person-centric parts of the schema for inclusion within the schema.
3. The schema is not designed to provide support to any particular Subscribing organisations.
4. The schema should be open, easily extendible and simple to use.

The schema is structured as a series of flat ‘tables’ with each table corresponding to a dataset, which together form a very shallow hierarchy. Considering the example of a bank account where there is a top level dataset holding the metadata about the account (called ds_bank_account) and an associated transaction dataset (called ds_bank_account_transactions).

The Mydex Master Schema is published openly and accessible to all. Details of the datasets available on this documentation site, under Data Schema. Access to the data within the datasets is via API calls, which provide a simple mechanism for reading from and writing to the PDS. Again, details can be found on this site under Connection API.

Schema Policy🔗︎ click to copy

The approach adopted for the design and use of the schema has deliberately been kept as simple as possible in order for developers to concentrate on developing apps to work with the data rather than having to understand complex data structures associated with their data. All extensions to the existing schema will need to conform to this ‘keep it a simple as possible, but no simpler’ approach.

In addition, there are a number of ‘rules’ which are designed to ensure that all of the fields within the schema are consistent and easily identified.

Permitted Data🔗︎ click to copy

While there are many things which could be stored within the PDS, there are a few types of data which we do not store on the Mydex platform. For example, we do not wish to store blob files, such as videos, images, sounds or other multimedia files as:

These files are likely to be large and therefore will take up an increasing amount of storage and bandwidth. There are plenty of other Subscribing organisations which can provide storage for these types of objects and we feel it is better we support their use these storage services and focus on storing the meta-data around these types of objects and providing a centralised management of permissions and consent for access to them.

There are other considerations here for us as a community Interest Company a Trust Framework operator and a certified ISO27001and Fair Data company. These can be summarised as:

• We do not charge individuals for the services we provide to them and would not realistically be able to support an opened storage and network bandwidth commitment for high volumes of large objects. data itself is relatively small by comparison.
• Over time our members may choose to locate their core Personal Data Store wherever they wish and therefore support for federated data sources is fundamental to this vision as is specialised data and streaming services providers, not everything needs to be encrypted, it is more important for us to provide the framework for permission and consent management to wrap around this type of content.

The table below shows a number of different data types and list what can and cannot be stored within the PDS schema.

In addition, there are data on certain topics that we would not wish to be stored within the PDS schema. These are likely to be non personal or non personal-related data, such as reference data or general descriptions. As a general rule:

1. If the individual cannot be thought of as the owner/custodian/curator of the data then it should not be included within the PDS. Instead, a pointer should be included to the relevant source.
2. If the information is not needed as part of the developers application, then it should not be included within the PDS.
3. If, as part of a developer’s application, it is necessary to have access to information which may change over time, and there is no definitive online source to point to, then the data may be stored in the PDS. An example might be the Terms & Conditions for an insurance policy, which would vary from year to year.

Naming Convention🔗︎ click to copy

All new datasets and fields added to the schema will need to conform to the naming convention summarised below:

Master Data Schema Version 1 Naming convention🔗︎ click to copy

Datasets: ds_dataset_name

The value in red is constant throughout the Master Schema the value in blue represents the dataset name.

Fields within a dataset: field_[abbreviated dataset name]_field_name

The value in red represent the constant within the dataset itself, the value in blue is the field name itself within the dataset.

Dataset name:ds_employment

Description

Details of each employer

Fields in this dataset

Master Data Schema Version 2 Naming convention🔗︎ click to copy

Following review and feedback we have been developing a new simpler naming convention. We will maintain backward compatibility through use of a synonym table when it is implemented.

It is cleaner, it will make life easier for all developers and it will be more easily extensible as we provide API helper functions that deliver results back from requests for answers as opposed to pure data e.g. requests about status, age on a given day, entitlement etc. these types of enquiries will have a series of parameters and may perform complex enquiries across multiple datasets but be accessed via a simple API call.

Data Formats🔗︎ click to copy

In order for all data to be consistent data of a particular type should conform to specific formats, as detailed below. All data stored within the PDS uses the UTF-8 character set.

Reference Data🔗︎ click to copy

There are a number of areas where reference data has been included within the PDS schema. Examples of these occur in the system where there are drop-down menus to provide users with a limited set of options, such as the name of a country in a group of address fields. Wherever possible, we have used an existing standard, such as ISO 3166 for country names or ISO 4217 for currency codes, or failing this we have used best practice.

Developers should consider the use of this sort of reference data when requesting additional datasets and fields to be added to the schema, but will need to indicate the source of the reference data so that the list within the PDS can be maintained.

Making extensions to the schema🔗︎ click to copy

New datasets to cover new use-cases are constantly being added to the platform by Mydex. These start out on the Sandbox environment where they are thoroughly tested before graduating to the live platform.

As well as this, it is anticipated that others may also have in mind particular use cases which are not currently catered for within the PDS. These use cases are likely to require additions to the schema in order to support them, so we have made it easy for others to contribute to the extension of the schema, by proposing use-cases and associated fields of their own.

Preparatory Work🔗︎ click to copy

In order to decide what datasets and fields are to be add to the schema, a data modeller / developer will need to carry out some preparatory work. This is likely to be carried out as part of their analysis in any case, but is listed here for clarity.

The data modeller / developer will need to:

1. research within the domain area in order to establish:
   • are there any open standards which have been developed in the domain area?
   • if there are no open standards, are there any de facto standards?
   • are there any emerging trends?

   Many of the schemas that exist for various business areas have been developed from the standpoint of the Subscribing organisations that will use them. The PDS has a different view - that of the individual.

   As an example, vehicle insurance will include details of the correspondence address (i.e. the address of the individual), whereas the individual knows where they live and whilst it is important to make sure that the insurer has the correct address the individual is also interested in the contact details of the insurance company.

   In short, imagine that you are trying to catalog the data about yourself for your own personal use rather than for the benefit of an Subscribing organisation. This usually exposes additional datasets that are needed, the sorts of things that normally go on the letterhead or the template of documents posted, PDF’s generated or emails sent.

2. consider which of the datasets and attributes you will be delivering you are in a position to provide as verified data, something you are happy to confirm has been verified as part of your own business process. Verified data is valuable to the individual for use downstream. The express data as verified you need to be able to make the following statements about:
   • How the data was generated, what process was used within your Subscribing organisation, e.g. output from an online portal or CRM system, captured as part of a defined process by trained staff.
   • When the data was generated or captured, date and time
3. cross reference the range of datasets and sources into an overall view of the domain area.
4. reduce the aggregated whole down to those datasets and fields that are of relevance to the individual and potentially for onward sharing and use. The essential element here is person-centric, as has already been stated.
5. Segment the data attributes into three broad categories of meta, state and transactional data.
   • Meta data is the static data associated with the domain, such as a bank account sort code and account number or an insurance policy number, Universal Tax Reference number etc.
   • State data is a current value type data - e.g. bank balance, value of a pension policy, current preference settings or choices about a service, entitlement to something, number of points on a driving license, qualifications held, current tariffs for mobile phone, energy services etc.
   • Transactional data is usually held in separate dataset as it will be regularly added to typically these are activities and events throughout ones life, e.g. bank transactions, credit card transactions, itemised call history, browsing history, measurements taken at regular intervals, location data, billing and payment history, travel history anything that is an event that occurs over time.
6. Define the fields in terms of their key properties.

Making the changes🔗︎ click to copy

In reality there are only three cases for making changes to the existing Mydex Master schema:

• Cloning a dataset - Where the existing datasets are considered to be suitable for a particular use case, but one or more additional fields need to be added to the dataset(s).
• Adding a totally new dataset - Where a dataset (or datasets) does not currently exist within the PDS to cover the fields for a new use case or business area. Here the new dataset(s) and fields will need to be specified.
• Extending an existing dataset - Where the data covered by the specific dataset has grown as an industry or service matures and new values become part of the core records. This may include splitting out data historically concatenated that becomes available as distinct fields or it becomes recognised that parsing the data as it arrives is of benefit for onward use and analysis.

The process to be followed for all schema extensions is:

1. Check and double check that the fields do not already exist within the schema. Be aware that the set of fields you want may not be in one dataset but could potentially be spread over a number of datasets. Also, the names may not be as you initially imagined due to the Mydex naming convention, so it is wise to check the field descriptions.
   1. We have our own checking routines and analysis but it will only delay your request if you submit a request and it fails those tests.
2. Please use our online dataset request form. The form on our developer community site should be used to request any new datasets or fields which are required in the PDS schema. The can be found here.
   1. We plan to make it possible for developers to upload their requirements in JSON or XML format to speed up the process of extending the schema in due course.
3. Mydex will review the request for the additional dataset and fields.There are a number of things we will be checking for, such as:
   • That the proposed fields conform to the schema policy (see above).
   • That the proposed fields do not already exist within the PDS schema.
   • That there is not a better way of proportioning the proposed datasets and fields. For example, it may be that by splitting the proposed fields across a number of existing datasets rather than having them all in the same dataset as we seek to have a more general schema which is applicable to a much greater number of use cases.
   At this point there are a number of possibilities:
   • the required datasets and fields are accepted and implemented in the Mydex Master PDS schema.
   • it is found that some or all of the dataset(s) or field(s) already exist within the schema.
   • it is found that some or all of the dataset(s) or field(s) do not follow the naming convention.
   The originator of the request will be notified of the outcome of the review.
4. A formal quality assurance testing and assurance process will be conducted within our staging environment. Successful completion of the tests will result in the release into our Sandbox and Live environments.
5. The new datasets and fields are now ready to be used. Assuming that the Subscriber organisation is already contracted to use the Mydex service and any new use of the fields makes no difference to the terms for the connection and planned data sharing agreement, the Subscribing organisation can go ahead and use the API to access the new datasets and fields.
   • If the Subscriber organisation is new to Mydex or the use of the fields will change, the terms of the connection or the data sharing agreement currently in place the this will need to be reviewed and verified prior to the existing connection or proposed connection is approved for use on Live.
     • The request for this can be carried out in parallel to steps 3, 4 and 5 above. Details can be found under the Terms for Subscribers.

Top Tips 🔗︎ click to copy

From the work we have done to date of many years here are some top tips when modelling data for a personal data store.

Don’t concatenate datasets 🔗︎ click to copy

Concatenating information into a single field as is often done on typical reports and summaries but is totally unhelpful for data analysis and sharing via a personal data store.

A good example of such a case are URL’s. much of the value of analysis and insight from URL’s can be gained at the domain, subdomain and extension level. It is therefore better to store these as discrete elements within the PDS as it will speed up processing and insight.

Financial institutions often combine information in statement description fields from two or three data elements in their own systems for ease of presentation but it makes analysis difficult. It is easier to combine them later if need be. A good example is one field description containing the name of person paid, the account number paid into and the transaction reference, all data held separeately by the Subscribing organisation but concatenated for output purposes in things like statements.

Consider requesting parsing of data if you cannot do it yourself 🔗︎ click to copy

If you have data you want to send to a personal data store but cannot parse it yourself, we can potentially do this as it arrives if there is a consistent processing rule that can be applied to the dataset. We don’t like to do it as it is time consuming and also makes verification harder but if you do not have the ability to process the output data from your systems but can define the rule for parsing it we may be able to help so please ask.

Generalise wherever possible 🔗︎ click to copy

Where and whenever you can generalise. Don’t make your dataset organisation specific rather make it context or process specific.

An individuals life spans many years and they may collect data on the same subject from more than one place at the same time or over the years. The value is on the long term analysis as much as the short term usage during an existing relationship.

Adopt standards or defacto standards wherever you can. 🔗︎ click to copy

Vast time, money and effort has been invested in creating standards or working towards standards, we believe it is worth using this wherever possible. Our goal is interoperability and we have focused on making it possible to expose different formats and outputs to meet different needs through our Open API’s.

We recognise that different sectors, markets and countries may have different formats and datastandards. The more we do to generalise or adopt standards the easier it is for these Subscribing organisations and markets to work with us. Mydex seeks no competitive advantage or proprietary advantage in its schema it is for the world so trying to make it specific to one Subscribing organisations needs is not something that is helpful in the long term or valuable to anyone.

Additional Information 🔗︎ click to copy

• Mydex Community Platform - You can raise specific questions here about any aspect of Mydex CIC, its Trust Framework or Platform and in particular about our Master Schema.
• Mydex Master Schema - Our published Master Schema Datasets and Fields within them.
• Request New Data Set form - Email a request for a new PDS Dataset with your requirements after following this guide.
• Get support from Mydex CIC - use one of our support channels.
• Give us feedback on our documentation site