Cockpit: "Forecast Comparison"

Compare multiple forecasts of the same target. Discover conditions that correlate with model superiority.

Pro This cockpit is only available in Visplore Professional.

Overview

The starting point for this cockpit is one reference time series, as well as several forecasts of this reference as forecast time series. Common usecases are model selection for trained predictors, or deciding which provider to buy a forecast from, for example in case of weather forecast providers.

The cockpit shows which forecast is superior to the others, based on their deviations from the reference time series. On the one hand, it compares how often each forecast is most accurate, and under which circumstances. On the other hand, forecasts are compared by error metrics like RMSE or MAPE, for different time periods and categories. The knowledge gained can be implemented in external forecasting tools, for example, as a script in order to use the models correctly.

  1. Quantitative Forecast Overview: A list of error metrics for each forecast, considering all time stamps in Focus
  2. Best Fit Count: How often does each forecast come closest to the reference time series? (Overall, and per category)
  3. Error per Category: A selected error metric is plotted for different categories, to discover systematical errors.
  4. Time Series / Errors as Time Series: Time series plot of forecasts vs. their reference over time (first tab), and of the error time series (second tab).
  5. Drill Down: One selected error metric plotted per day, per categories, or per combinations of categories, with the option of selecting time ranges and categories for a detailed view by clicking on them.
  6. Table: A table-like view of single values that can be shown on demand.



Starting the cockpit - assigning time series roles


The following roles can be given to data attibutes in this cockpit. Use the icon in the toolbar to adjust them.


Quantitative Forecast Overview


In this list, the forecasts are compared by (global) error metrics. All records in focus are considered.

Sort the list by clicking the header of any error metric, and choose the set of displayed error metrics by clicking the view title, then "Choose displayed statistics" below.


Best Fit Count

This view counts which forecast came closest to the reference time series, overall (first tab) or per category (second tab). This means that the forecast with the smallest absolute distance from the reference time series is counted. The relative frequency of such "victories" is displayed for each forecast by the bar length. If several models have the (same) lowest forecast error at a time, this time stamp is counted as a separate category called "Ambiguous". The category "Missing" counts the number of time stamps where the value of the reference time series is missing.


Error per Category

This view shows how one error measure for the forecasts depends on categories or time intervals (hours, days, months,...). The plot only considers the data records in focus.

In the images above, you can see for example that forecast 3 (red) has a particularly bad RMSE in evenings, and months after September. In mornings, on the other hand, it has the smallest RMSE.

Click the x-axis label to change the categories to compare the errors by. Click the y-axis on the right side of the view to change the used error metric.


Drill Down

These views display one selected error metric per forecast using colored bars, for different (combinations) of categories. Each colored bar aggregates the deviations of one forecast's data that falls within the corresponding (combination of) categories.

The views can be used to perform selections of categories by clicking the category labels, which defines a focus for the other views. If a focus was already defined in other views, the Drill-Down views only consider the data records of the focus.

Using other error metrics than the RMSE: click the y-axis label on the right of the view.


Time Series


These plots show time series graphs of the forecasts and the reference. The "Time Series" view plots forecast and reference time series as such, while "Errors as time series" shows the residuals (i.e., the differences between forecast and reference) over time.

Key actions:

The diagram offers several options when clicking the view title "Time Series". The most important ones are:


Table ('Focus data records')


A table showing the single values. Only considers the data records in focus. Click the header of a column to sort the table by it. To change the set of displayed columns, click the header "Shown: 103 of 210 data attributes". Click single rows or drag a line with the left mouse button to select records, putting them into focus.

Exporting: A key use case of the table is exporting a selected subset of the data. To export the current state of the table, click on "Export" in the top right of this view.



License Statement for the Photovoltaic and Weather dataset used for Screenshots:
"Contains public sector information licensed under the Open Government Licence v3.0."
Source of Dataset (in its original form): https://data.london.gov.uk/dataset/photovoltaic--pv--solar-panel-energy-generation-data
License: UK Open Government Licence OGL 3: http://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/
Dataset was modified (e.g. columns renamed) for easier communication of Visplore USPs.