This is a collection of core data sources and curated data lists which may be useful for research and innovators in decarbonisation research.

Energy Sector

The following are data sources which are useful for energy research.

Energy Networks

In the UK due to the Presumed Open Data Principle within the data best practice guidence, many datasets have been made available on the network operators own data hubs. Some of the main hubs are listed below.

  • NESO Data Portal : The national system operator hub contains a large amount of data on balancing and ancillary including balancing mechanism units, forecasts, and costs.
  • Elexon BSC Insights Solutions : Contains data on balancing mechanism and settlement.
  • National Gas data portal, the gas transmission operator for GB, contains data on gas flows, system status and forecasts.
  • Low Carbon Contracts Company data portal contains data relevant to the CfD markets.

Each Transmission Network Operator has its own data portals of which there are three in the UK for different areas of the UK. These have data on the network infrastructure, fault levels, circuits etc.:

  • Scottish and Southern Electricity Networks– Transmission - covers North of Scotland: data portal
  • Scottish Power Transmission – covers South Scotland: data portal
  • National Grid Electricity Transmission – covers England and Wales

Each electricity distribution network operator (DNO) also has its own data hub focused on their own areas and also often includes telemetry data for demand and generation, data from innovation trials, and aggregated smart meter data:

Each of the four Gas Distribution Network Operator also has data portal containing information like pipe infrastructure, demand data, and linepack:

Curated Datasets

Some novel and valuable datasets are not available openly, and often are only available from innovation projects. Some of these are available on the energy companies portals, but finding specific data sets relevant to specific applications are not always available. There are several catalogues of data sets useful in energy, some are listed below:

  • Open Power Systems Data, an open data platform containing generation, demand and weather data.
  • Low Voltage Load Forecasting data: a curated list of time series data for use in load forecasting from household up to low voltage substation level.
  • IEEE PES Intelligence Systems Subcommittee open data sets aimed at research in power and energy areas, including data for EV’s, consumption, wind and weather etc.
  • The openmod initiative shares a curated list of energy data sets relevant for energy modelling.
  • Open Energy Data Initiative, a repository of high-value energy research datasets aggregated from US Department of Energy’s programs and Offices.
  • Monash Time Series Forecasting Repository curated datasets for time series forecast, including energy data.

Synthetic Data

Some data is difficult to share in large amounts, in these cases synthetic data can be valuable to help provide insights in lieu of real data. Below are a few tools for generating synthetic data.

  • OpenSynth, an LF Energy community for open sharing of synthetic energy data, includes Centre for Net Zero’s Faraday tool for generation synthetic smart meter data.
  • Renewables.ninja simulated PV and wind generation data
  • When2Heat simulated heating profiles is open synthetic building heat pump data for 28 European countries.

Tools for data extraction

Collating and aggregating data from multiple sources can be time consuming especially if the data uses different standards and formats, and has different access options. Below will be an list of some tools to support these extraction efforts.

  • Weave is a python tool from the Centre for AI and Climate and CEIMIA (Centre d’expertise internationale de Montréal pour l’avancement de l’intelligence artificielle) which helps to extract and join the aggregated half hourly smart meter data from various UK DNOs.

If you have relevant datasets you wish to add to the list please get in contact at advice@turing.ac.uk

Updated: