Matlab Connector (VisploreMatlab v1.4.0)
Overview and Use-Cases
VisploreMatlab is a Matlab toolbox to integrate Visplore cockpits into existing Matlab workflows. It offers a bidirectional link between the Matlab workspace and the interactive visual exploration features of Visplore. This supports multiple use-cases:
- Immediately start exploring data frames from any existing Matlab workspaces.
- Clean outliers interactively in Visplore and get the cleaned data frame back.
- Select and label data subsets like patterns, clusters, or training data in Visplore, and making this information available for Matlab.
- Visually debug Matlab code by visualizing results at any point during execution.
- Validate predictions of Matlab machine learning models visually in Visplore.
- Integrate datasets from multiple sources and importer packages in Matlab, and send the merged/joined data table to Visplore for analysis.
Installation
Requirements:
- 64bit Windows
- 64bit Matlab Version 2018a or newer
Installation:
VisploreMatlab comes as a toolbox, shipped with the Visplore installation. There are two ways of installing it:
- From a running Visplore instance: in the startup dialog of Visplore, choose the "Connect to Matlab" option, and select the "Install VisploreMatlab integration package" checkmark. When confirming the dialog, a code snippet is produced that, when executed in Matlab, installs the toolbox and immediately connects Matlab with that Visplore instance.
- Alternatively, you can install the toolbox via a double click in Matlab after navigating to the VisploreMatlab folder of your Visplore installation, or double clicking it in your file explorer and choosing the right Matlab version.
Note: After installing the toolbox, Matlab requires a restart for the toolbox to become operational.
Step by Step Guide
Once installed, this section describes an example workflow to demonstrate its basic usage.
You can open the demo file "visplore_demo.m" shipped with Visplore in the VisploreMatlab directory of your Visplore installation, and try out all the examples there.
Alternatively, you can type the commands in your Matlab environment as described in the following.
1) Connect Matlab and Visplore
There are two ways to connect Matlab and Visplore:
a) Start a new Visplore instance from Matlab and connect to it, by typing the following:
v = visplore()
This opens a new Visplore instance, and stores a handle to it in a variable "v". Any number of Visplore instances can be created and used at the same time. The distinct handles are used to communicate with any particular instance.
Note: This way of starting Visplore is ideal when Visplore is installed via Installer locally on the same machine as Matlab . In other cases, refer to option (b), described in the following.
b) Connecting to an already running Visplore instance (locally, or on another machine):
This alternative is to start a new Visplore instance by hand first (e.g. via Start menu or desktop icon). In the welcome dialog of Visplore, choose the option "Connect to Matlab". Confirming the dialog copies a code snippet to the clipboard, which you can paste and run in Matlab. It looks like this:
v = visplore('connect', true, 'ip', '127.0.0.1', 'port', 33113)
After pasting and running this snippet in Matlab, the variable "v" is a handle to the existing Visplore instance you connected to. The tools communicate via TCP network protocol, which is why IP and port are specified as parameters here . Please leave these parameters as they are produced by Visplore. The Visplore instance waits for incoming connections on that port, and the "visplore('connect',..)" call establishes the connection.
2) Prepare a dataframe in Matlab
A typical workflow starts with a data frame that is either already available in the Matlab workspace, or loaded, generated, or accessed at this point. In our example, we use a timetable, loaded from an open photovoltaic and weather dataset shipped with Visplore as csv file:
tt = visplore.load_demo_data('data', 'openpv_sampled');
3) Send dataframe to Visplore
We can now send this (or any other) data table to Visplore:
v.send_data(tt)
To see if your data successfully arrived at the Visplore instance, you can take a look at the statusbar at the bottom of your Visplore instance, and see if it shows any received records:
If something went wrong, it still shows:
4) Start a cockpit to explore the data
Once the data is available in Visplore, we can start a “Cockpit”, i.e., an entry point for a certain task or analysis question.
This can be done in one of two ways: either by a double click on the respective cockpit icon in Visplore, for example “Trends and Distributions”:
Alternatively, the cockpit can be started from Matlab via the following command:
v.start_cockpit("Trends and Distributions")
5) Interactively select data in Visplore, and make this information available in Matlab
A key feature in Visplore is the selection of data records using interactive selection mechanisms (selection rectangles, Lasso brushes, etc.). This allows the user to select interesting subsets, like outliers, patterns, clusters, or training data for modeling interactively in Visplore. These subsets can be made available in Matlab for downstream processing.
For example, you can drag an interval or rectangle with the left mouse in a “Time Series View” or “Histogram”, to select some data records, and then type the following command in Matlab:
boolean_mask = v.get_selection();
6) Label selected data in Visplore, and get labels in Matlab
After selecting records in Visplore like in the previous step, the selected records can be labeled by clicking "create" in the "Focus" bar then selecting "Label" (see Visplore documentation). This creates a named condition, that can be retrieved as a binary mask from Matlab using the following command. Say you selected outliers, and labeled them as a condition "Outliers", retrieve them with:
outliers = v.get_query("Outliers");
num_outliers = sum(outliers) %optionally print how many records are selected
plot(outliers) %optionally plot them
7) Clean data values in Visplore, and retrieve the cleaned data table as data frame
Visplore allows editing data records interactively, such as imputing outliers or missing values by interpolating neighbour values (see Visplore documentation).
In particular after peforming edits, but basically at any time, you can retrieve the current Visplore data table using:
new_table = v.get_data()
v.get_data('LimitToFocus', true).
8) Get data from a Visplore visualization as data frame
Many table-like visualizations support retrieving the displayed data as new pandas data frame. Examples are statistical tables, bar charts, heat maps, and raw value tables.
my_statistics = v.get_view_data("Statistics")
v.get_view_properties().
8) Get names of selected data attributes in Visplore, or set the selection of data attributes
When you interactively select data attributes in Visplore (e.g., in the "Statistics" view), retrieve the names of the selected attributes with:
v.get_selected_columns() #returns names as strings
set_selected_columns(), like this:
v.set_selected_columns({"A_Phase_Voltage_FlowerVillage_PV", "A_Phase_Voltage_Happyville_PV"})
9) Compute new data columns in Matlab, and make them available in Visplore
In this last example, a new data column is created in Matlab by multiplying an existing column that is selected in Visplore by 2. Use the send_data command with the "merge" parameter, to add the new column to the existing Visplore table:
selected_columns = v.get_selected_columns();
first_selected_column = selected_columns{1};
subtable = tt(:,first_selected_column); %new timetable that contains just the new column, and matching time keys
subtable.(first_selected_column) = subtable.(first_selected_column) * 2; %alter the values
subtable.Properties.VariableNames = strcat(subtable.Properties.VariableNames,'_times_two'); %rename the new column
v.send_data(subtable, 'mode', 'merge') % and send back to visplore (by merging)
But: When you update the values of an already displayed Visplore variable, e.g., running the snippet again with a different rolling window parameter, the values update immediately, allowing for efficient visual tweaking of parameters:
%% Tune parameters and immediately replace values in Visplore
new_varname = subtable.Properties.VariableNames{1}
subtable.(new_varname) = subtable.(new_varname) * 10; %alter the values
v.send_data(subtable, 'mode', 'merge') % and send back to visplore (by merging)
API Documentation
close(...)
Closes the connected instance of Visplore
Usage:
close()
Example:
% v is a visplore handle received by v = visplore():
%
v.close()
get_classification(...)
Returns the classification(= categorical column) for the given name, if existing, as data frame.
Usage:
get_classifcation(name)
Input parameters:
name - the name of the classification as string
Return type:
array
Example:
% v is a visplore handle received by v = visplore():
% Assumes, that a classification column called 'Classification Nr. 1' has been created in Visplore.
%
c = v.get_classification('Classification Nr. 1')
% returns pandas dataframe with the length of the Visplore table, and
% the classification as categorical column.
export_story(...)
Exports the visplore story to a pdf file.
Usage:
export_story_to_pdf(filepath)
Input parameters:
filepath (mandatory, string): full path of image to create, including the file extension
logopath (optional, string): full path of image to use as logo on each page (incl. extension)
Example:
% v is a visplore handle received by v = visplore():
% Assumes, that a story has been created in Visplore.
%
v.export_story_to_pdf("C:/Users/vs/Documents/Images/matlab.png")
get_columns(...)
Returns the names of the columns in the current visplore table
Usage:
get_columns()
Return type:
cell array
Example:
% v is a visplore handle received by v = visplore():
%
c = v.get_columns()
% returns for example {'A_Phase_Voltage_BrightCounty_PV', 'A_Phase_Voltage_Cloudington_PV', ...}
get_condition(...)
Returns a list of boolean values, stating for each record of the Visplore table if it is part of the named condition (true), or not (false).
Usage:
get_condition(name)
Input parameters:
name (mandatory, string): the name of a condition whose membership is returned
Return type:
logical array
Examples:
% v is a visplore handle received by v = visplore():
%
s = v.get_condition('Outliers')
% returns for example [false; true; false; true], stating which records belong to
% an existing named condition 'Outliers'
get_data(...)
Returns the data table of Visplore as pandas data frame.
Usage:
get_data()
Input parameters:
SelectionName(optional, string)
Uses the given selection for the basis of the returned data.
LimitToFocus(optional, boolean)
Determines whether only data records that are in focus should be returned.
ReplacementStyle(optional, char)
Determines how unsupported characters are replaced in
Matlab. options: 'hex', 'underscore', 'delete'
Default: 'underscore'
Return value:
table
Example:
% v is a visplore handle received by v = visplore() with data:
%
new_data_table = v.get_data()
%
% Get data limited to current focus
limited_data_table = v.get_data('LimitToFocus', false)
get_filter(...)
Returns a list of boolean values, stating for each record of the Visplore table if it passes the current Visplore filter (true) or not (false).
Usage:
get_filter()
Return type:
logical array
Example:
% v is a visplore handle received by v = visplore():
%
c = v.get_filter()
% returns for example [true; true; true; false; true; ... true;]
get_selected_columns(...)
Returns the names of the currently selected columns in the current visplore table
Usage:
get_selected_columns()
Input parameters:
fullNames (optional, boolean, default: false): if true, the full Visplore attribute hierarchy path is used
Example:
% v is a visplore handle received by v = visplore():
%
c = v.get_selected_columns()
% returns for example [{'A_Phase_Voltage_BrightCounty_PV'}, {'A_Phase_Voltage_Cloudington_PV'}, ...]
get_selection(...)
Returns a list of boolean values, stating for each record of the Visplore table if it is part of (1) the current Focus or (2) a named condition, if name is specified.
Usage:
get_selection()
Input parameters:
name (optional, string, default: None): name of a condition whose membership is returned.
If not specified, membership of the current Focus is returned. .
Is equivalent to calling get_condition(name).
Return type:
logical array
Examples:
% v is a visplore handle received by v = visplore():
%
s = v.get_selection()
% returns for example [true; false; false; false], stating that only the
% first record is currently in Focus.
%
s = v.get_selection('Outliers')
% returns for example [false; true; false; true], stating which records belong to
% an existing named condition 'Outliers'
get_view_data(...)
Returns the contents of a particular Visplore view as data frame.
Is not applicable for all views. Call get_view_properties() to see, which views of the current Cockpit support this. Supported examples : 'Statistics', 'Focus data records', 'Calendar', 'Histogram' Attention : at the moment, get_view_data is only possible for views that are visible in Visplore, not views hidden in tabs. Open them first.
Usage:
get_view_data(view_id)
Input parameters:
view_id(mandatory): the ID of the view whose contents are to be retrieved.
In most cases, this is the name displayed as view title in Visplore.
See get_view_properties() to see the View IDs, and if a view supports this.
Return type:
table
Example:
% v is a visplore handle received by v = visplore():
stats = v.get_view_data('Statistics')
% or
stats = v.get_view_data('Focus data records')
get_view_properties(...)
Returns a table stating for each view - the view ID needed for API calls to it - if the view is exportable via API calls
Usage:
get_view_properties()
Return type:
table
Example:
% v is a visplore handle received by v = visplore():
props = get_view_properties()
%returns for example
% DisplayedName Identifier SupportsDataExport SupportsImageExport
% ________________________ _____________________________________________ __________________ ___________________
%
% {'Statistics' } {'Statistics' } true true
% {'Heatmaps' } {'Temporal Overview' } true true
% {'Histograms' } {'Histograms' } true false
% {'Horizon Graph View' } {'Horizon Graphs' } false true
visplore.load_demo_data(...)
Loads an already prepared data set which can be used for Matlab testing.
Only works if Visplore has been installed using the installer.
Usage:
visplore.load_demo_data(data)
Input parameters:
data(optional, string) - The identifier of a packaged data set.
Possible sets are:
openpv - The full Open Photovoltaic
and Weather data set
openpv_sampled - A sampled subset of
the openpv data set
Examples:
% Load the sampled data set
data_sampled = visplore.load_demo_data();
% Load the full data set
data = visplore.load_demo_data('data', 'openpv');
load_session(...)
Loads a.visplore session with the given filename(full path)
Usage:
load_session(filename)
Input parameters:
filename(mandatory) : string specifying full name and path of visplore file
mode(optional, string): string specifying the mode on how data is restored when loading a session
support modes are: auto, askUser, fromEmbedded, fromLink, fromLinkWithDialog, no
Example:
% v is a visplore handle received by v = visplore():
%
v.load_session("C:/data/visplore_sessions/my_dashboard.visplore")
v.load_session("C:/data/visplore_sessions/my_dashboard.visplore",'mode','askUser')
save_image(...)
Saves a view as an image to the given location and filename
Usage:
save_image(viewID, filename)
Input parameters:
viewID(string, mandatory) - the ID of the view to export
fileName(string, mandatory) - the file name and location for the image
parameters(struct, optional) - in this optional struct, you can customize
the image export, using any of the
following parameters:
width, height (numbers): the total width/height of the image in pixels.
colwidth, rowheight (numbers): alternatives to total width and height, for
views that have rows or columns, for example:
Statistics, Heatmaps, Bar Chart, Heatmap,...
- colwidth defines the width per column in px,
- rowheight defines the height per row in px.
font (string): defines font type and size for all text in the image,
for example, "Arial 16", or "Tahoma 20"
xaxislabel (boolean or string): set to False to hide the x axis label,
or specify as string to show a custom x axis label.
yaxislabel (boolean or string): set to False to hide the y axis label,
or specify as string to show a custom y axis label.
labelfont (string): format for the label's font type and size, see below.
colorlegend (boolean or string): set to False to hide color legends,
or specify as string to show a custom legend label
visiblerangebars (boolean): set to False to hide scrollbars.
missingdatabar (boolean): set to False to hide info-bars above some plots.
context (boolean): set to False to show only data records that are in Focus.
selectionhandles (boolean): set to False to hide orange selection tool handles.
title (boolean or string): set to True to show the view name as title,
or specify as string to show a custom title.
titlefont (string): format for title's font type and size, see below.
description (boolean or string): set to True to show the Focus description in image,
or specify a string to show a custom description text.
descriptionfont (string): format string for description's font type/size, see below.
arealabel (boolean or string): Only available in some(aggregated) views.
set to False to hide the area label
or specify as string to show a custom area label.
rightyaxislabel (boolean or string): Only available in some(aggregated) views.
set to False to hide the right y axis label
or specify as string to show a right y axis label.
topxaxislabel (boolean or string): Only available in some(aggregated) views.
set to False to hide the top x axis label
or specify as string to show a top x axis label.
Example:
% v is a visplore handle received by v = visplore():
% exports the Time Series view with a hidden x-axis and a
% custom labeled y-axis
v.save_image("Time Series","C:/Users/vs/Documents/Images/matlab.png", struct('xaxis', false,'yaxis','Values'))
send_data(...)
Sends data to visplore, either replacing the current visplore table, or extending it.
Usage:
send_data(data)
Input parameters:
data(mandatory) - as a table.
supported column types are numerical, categorical, or timestamp attributes.
An additional type supported by some Visplore versions are 1D curves,
given as a column holding lists in the cells:
Each curve is a cell containing a double array like this:
Example: {[0,1,2; 100,-10,30]}
{[0,1,2; -20, -30, -40]} in line 2, etc.
mode(string, optional, default: 'auto')
a string that defines how the data is combined with a
possibly existing table in visplore.
Options : ('replace', 'merge', 'append', 'auto')
tableDescription(string, optional, default: None)
an optional table description that is shown in the title bar of Visplore.
If not specified, the data type or variable name is shown
Example:
% v is a visplore handle received by v = visplore():
%
v.send_data(df) % df is any table
set_selected_columns(...)
Selects the columns in Visplore that are specified by a list column names(strings).
Usage:
set_selected_columns(column_names)
Input paramters:
column_names: a list of strings specifying exact names of columns to select in Visplore.
In the same format, as retrieved by get_selected_columns()
Example:
% v is a visplore handle received by v = visplore():
%
li = [{'A_Phase_Voltage_BrightCounty_PV'}; {'A_Phase_Voltage_Cloudington_PV'}]
v.set_selected_columns(li)
set_selection(...)
Sets the current Focus of data records to those records marked in the provided sequence of boolean values. Optionally also stores these records as a named condition.
Usage:
set_selection(sequence)
Input parameters:
sequence(boolean, mandatory) - the records to select(where true). Must be length of the Visplore table.
description(optional, string) - a selection description, optional
name(optional, string) is the name of a named condition to create and set from these records.
Examples:
% v is a visplore handle received by v = visplore():
%
v.set_selection([true; false; false])
% sets the Focus to contain only the 1st record of a 3-record table.
%
v.set_selection([true; false; false], 'name', 'MyCondition')
% does the same, and creates a named condition called MyCondition that holds these records.
start_cockpit(...)
Starts a cockpit with the given name, if such a cockpit exists.
Usage:
start_cockpit(cockpit).
Input parameters:
cockpit(string, mandatory) - is the name of the cockpit that should be started
skipRolesDialog(bool, optional) - if True: signals to Visplore to attempt to automatically assign roles
The role assignment dialog is shown if no automatic role assignment is possible
Examples:
% v is a visplore handle received by v = visplore():
%
v.start_cockpit('Trends and Distributions')
visplore(...)
Starts a new instance of visplore and returns a handle to it.
The location of visplore.exe is assumed to be found in an environment variable, called VISPLORE_LOCATION, which is set by the Visplore installer, or by you. Alternatively, you can specify a different visplore location by a parameter. As a third option, the VisploreStarter application can be used (legacy support). Further optional parameters allow specify a cockpit to start, and passing data.
Usage:
visplore()
Input parameters:
local_visplore(optional, boolean) - whether a local Visplore installation is started.
Is possible if Visplore was installed by installer, or
when you set VISPLORE_LOCATION to path/visplore.exe manually.
Default: true. If false, the VisploreStarter app is used.
path(optional, char) - if specified, and the local_visplore parameter is set to true, a local
visplore.exe is started from this location instead of VISPLORE_LOCATION.
cockpit(optional, char) - specifies the name of a cockpit to start with visplore
data(optional) - initial data sent to the new visplore instance
(see send_data for a description of the expected format)
connect(optional, boolean) - Connects to an already running Visplore instance instead of starting a new one
Uses TCP IP connection. By specifying the IP + port pair, you can either connect to
a local Visplore, or to a Visplore running on another machine.
ip(optional, char) - ip address of the Visplore instance, e.g. '127.0.0.1'
port(optional, int) - the port that Visplore is listening to, e.g. 50123
you find the port in the 'Connect to Matlab' option in Visplore's welcome dialog.
Examples:
% Start a new visplore instance from a local installation
v = visplore() % Start a new visplore instance from an installer-less Visplore in a different directory
v = visplore('local_visplore',true, 'path','C:/temp/Visplore/Visplore/visplore.exe')