Python Connector (VisplorePy v1.5.1)
Overview and Use-Cases
Video available: The easiest way to get acquainted with the data connector is to watch the video tutorial in our Video AcademyVisplorePy is a Python 3 extension to integrate Visplore cockpits into existing Python workflows. It offers a bidirectional link between the Python workspace and the interactive visual exploration features of Visplore. This supports multiple use-cases:
- Immediately start exploring data frames from any existing Python workspaces, notebooks, etc.
- Clean outliers interactively in Visplore and get the cleaned data frame back.
- Select and label data subsets like patterns, clusters, or training data in Visplore, and making this information available for Python.
- Visually debug Python code by visualizing results at any point during execution.
- Validate predictions of Python machine learning models (e.g. scikit learn) visually in Visplore.
- Integrate datasets from multiple sources and import packages in Python, and send the merged/joined data table to Visplore for analysis.
- Go even further, and build an own importer GUI or starter application in Python, to extend the built-in import functionalities of Visplore even for non-python users.
Installation
Requirements:
- 64bit Windows
- 64bit Python 3.6 (or higher) environment, such as Anaconda.
Installation:
VisplorePy comes as a python wheel, shipped with the Visplore installation. There are two ways of installing it. Option 1 is straightforward if you use Jupyter, or the plain python console. If you use Spyder as environment, we recommend going with option 2.
- From a running Visplore instance: in the startup dialog of Visplore, choose the "Connect to Python" option, and select the "Install VisplorePy integration package" checkmark. When confirming the dialog, a code snippet is produced. When this is executed in your Python environment, it installs the Python wheel and immediately connects Python with that Visplore instance:
Note for Spyder-users: it is not recommended to run pip commands like the snippet above directly in Spyder (see here). Instead, we recommend option 2 for you.
- Alternative: using pip to install it in your terminal, e.g. Anaconda prompt (but outside of graphical shells like Spyder).
Note: You may need to adapt the path to match your Visplore installation and exact name of the wheel file. To find out where your Visplore is installed, click "Help" in a running Visplore, and "Where is my Visplore version installed?".pip install --upgrade --force-reinstall "C:/Program Files/Visplore/VisplorePy/visplorepy-0.0.1-py3-none-any.whl"
Step by Step Guide
Once installed, this section describes how to start VisplorePy, and sketches an example workflow to demonstrate its basic usage.
In case you use Jupyter notebook, you can load the accompanying demo notebook “Pandas_OpenPV_Demo.ipynb” from the VisplorePy subfolder of your Visplore installation, which contains several examples ready to try out.
Alternatively, you can type the example commands in any shell/console, as described in the following.
1) Connect Python and Visplore
There are two ways to connect Python and Visplore:a) Start a new Visplore instance from Python and connect to it, by typing the following:
import visplorepy
visplore = visplorepy.start_visplore()
This opens a new Visplore instance, and stores a handle to it in a variable "visplore". Any number of Visplore instances can be created and used at the same time. The distinct handles are used to communicate with any particular instance.
Note: This way of starting Visplore is ideal when Visplore is installed via Installer locally on the same machine as the Python environment (see start_visplore() in the API documentation for details). In other cases, refer to option (b), described in the following.
b) Connecting to an already running Visplore instance (locally, or on another machine):
The alternative is to start a new Visplore instance by hand first (e.g. via Start menu or desktop icon). In the welcome dialog of Visplore, choose the option "Connect to Python". Confirming the dialog copies a code snippet to the clipboard, which you can paste and run in your Python environment. It looks like this:
import visplorepy
visplore = visplorepy.connect("127.0.0.1",50200)
After pasting and running this snippet in Python, the variable "visplore" is a handle to the existing Visplore instance you connected to. The tools communicate via TCP network protocol, which is why IP and port are specified as parameters here (see API documentation for details). Please leave these parameters as they are produced by Visplore. The Visplore instance waits for incoming connections on that port, and the "connect()" call establishes the connection.
2) Prepare a dataframe in Python
A typical workflow starts with a data frame that is either already available in the Python workspace, or loaded, generated, or accessed at this point. In our example, we use a pandas data frame, loaded from an open photovoltaic and weather dataset shipped with Visplore, using this built-in command:
data = visplorepy.load_demo_data('openpv_sampled')
3) Send dataframe to Visplore
We can now send this (or any other) data table to Visplore:
visplore.send_data(data)
To see if your data successfully arrived at the Visplore instance, you can take a look at the statusbar at the bottom of your Visplore instance, and see if it shows any received records:
If something went wrong, it still shows:
4) Start a cockpit to explore the data
Once the data is available in Visplore, we can start a “Cockpit”, i.e., an entry point for a certain task or analysis question.
This can be done in one of two ways: either by a double click on the respective cockpit icon in Visplore, for example “Trends and Distributions”:
Alternatively, the cockpit can be started from Python via the following command:
visplore.start_cockpit("Trends and Distributions")
5) Interactively select data in Visplore, and make this information available in Python
A key feature in Visplore is the selection of data records using interactive selection mechanisms (selection rectangles, Lasso brushes, etc.). This allows the user to select interesting subsets, like outliers, patterns, clusters, or training data for modeling interactively in Visplore. These subsets can be made available in Python for downstream processing.
For example, you can drag an interval or rectangle with the left mouse in a “Time Series View” or “Histogram”, to select some data records, and then type the following command in Python:
selection_mask = visplore.get_selection()
6) Label selected data in Visplore, and get labels in Python
After selecting records in Visplore like in the previous step, the selected records can be labeled by clicking "create" in the "Focus" bar then selecting "Label" (see Visplore documentation). This creates a named condition, that can be retrieved as a binary mask from Python using the following command. Say you selected outliers, and labeled them as a condition "Outliers", retrieve them with:
outlier_mask = visplore.get_condition("Outliers")
7) Clean data values in Visplore, and retrieve the cleaned data table as data frame
Visplore allows editing data records interactively, such as imputing outliers or missing values by interpolating neighbour values (see Visplore documentation).
In particular after peforming edits, but basically at any time, you can retrieve the current Visplore data table using:
new_data_frame = visplore.get_data()
8) Get data from a Visplore visualization as data frame
Many table-like visualizations support retrieving the displayed data as new pandas data frame. Examples are statistical tables, bar charts, heat maps, and raw value tables.
my_statistics = visplore.get_view_data("Statistics")
visplore.get_view_properties() in the the API documentation.
9) Compute new data columns in Python, and make them available in Visplore
In this last example, a new data column is created in Python by computing a moving average from an existing data column. Select any data column in Visplore, then call the following snippet to compute the moving average for that selected column:
# compute smoothed variant of selected column and send to Visplore
first_selected_timeseries = visplore.get_selected_columns()[0]
new_column = first_selected_timeseries + "_smoothed"
data[new_column] = data[first_selected_timeseries].rolling(100, center=True, min_periods=1).mean()
visplore.send_data(data[['DateTime', new_column]], "merge")
visplore.get_selected_columns() returns the names of the currently selected variables in Visplore as a list. The next line concatenates the name of the first selected column with the word "smoothed", preparing the name for the new column. The next line computes the moving average using python functionality, by indexing into the original data frame using the name of the currently selected column. The result is appended as new column to the dataframe. Finally, the last line transfers a part of the extended data frame to Visplore. Specifically, it transfers the columns "DateTime" and the newly created column, while merging the records based on "DateTime" as a join index.
Note:If you run the snippet again with a different rolling window parameter, the values update immediately, allowing for efficient visual tweaking of parameters.
API Documentation
Use help(visplorepy) in Python to produce this documentation:
activate_story_page(...)
Activates the given visplore story. Usage: activate_story_page(storypage) Input parameters: storypage(mandatory, string): name (case insensitive) of the story page storypage(mandatory, number): number of the story page displayed top-left in story panel Example: # v is a visplore handle to a Visplore instance.: # Activate the story: v.activate_story_page("my story page") OR v.activate_story_page(6)close(...)
Closes the visplore application for this handle. Usage: close() Example: #v is a visplore handle received by v = visplorepy.start_visplore(): # v.close()connect(...)
Connects to an already running Visplore instance. Uses TCP IP connection. By specifying the IP + port pair, you can either connect to a local Visplore, or to a Visplore running on another machine. Usage: connect(ip, port) Input parameters: ip(mandatory, string) - ip address of the Visplore instance (e.g. "127.0.0.1") port(mandatory, int) - the port that Visplore is listening to (e.g. 50123) - you find the port in the 'Connect to Python' option in Visplore's welcome dialog. Example: visplore = visplorepy.connect("127.0.0.1", 70010)connect_local(...)
Connects to a local, already running Visplore instance. Usage: connect(port) Input parameters: port(optional, int) - the port that Visplore is listening to (e.g. 50123); If not given, tries to find the last started Visplore instance. - you find the port in the 'Connect to Python' option in Visplore's welcome dialog. Example: visplore = visplorepy.connect_local()export_story_to_pdf(...)
Exports the visplore story to a pdf file. Usage: export_story_to_pdf(filePath) Input parameters: filename(mandatory, string): full path of image to create, including the file extension logopath(optional, string): full path of image that is used as logo for each page Example: # v is a visplore handle to a Visplore instance.: # Export the story to a pdf: v.export_story_to_pdf("C:/temp/test.pdf")get_classification(...)
Returns the classification(= categorical column) for the given name, if existing, as data frame. Usage: get_classifcation(name) Input parameters: name - the name of the classification as string Return type: Pandas data frame Example: # v is a visplore handle received by v = visplorepy.start_visplore(): # Assumes, that a classification column called 'Classification Nr. 1' has been created in Visplore. # c = v.get_classification('Classification Nr. 1') # returns pandas dataframe with the length of the Visplore table, and # the classification as categorical column.get_columns(...)
Returns the names of the columns in the current visplore table (optionally limited to the selected ones) Usage: get_columns(selectedOnly=False, fullNames=False) Input parameters: selectedOnly: (optional, boolean, default: False) - if only the selected columns, or all columns are returned fullNames (optional, boolean, default: False): if True, the full Visplore attribute hierarchy path is used Return type: list Example: # v is a visplore handle received by v = visplorepy.start_visplore(): # c = v.get_columns(selectedOnly=True) # returns for example ['A_Phase_Voltage_BrightCounty_PV', 'A_Phase_Voltage_Cloudington_PV', ...]get_condition(...)
Returns a list of boolean values, stating for each record of the Visplore table if it is part of the named condition (True), or not (False). Usage: get_condition(name) Input parameters: name (mandatory, string): the name of a condition whose membership is returned Return type: list Examples: # v is a visplore handle received by v = visplorepy.start_visplore(): # s = v.get_condition('Outliers') # returns for example [False, True, False, True], stating which records belong to # an existing named condition 'Outliers'get_data(...)
Returns the data table of Visplore as pandas data frame. Usage: get_data(selectionName = "", limitToFocus=False) Input parameters: selectionName(optional, string) Uses the given selection for the basis of the returned data. limitToFocus(optional, boolean) Determines whether only data records that are in focus should be returned. Return value: a Pandas data frame Example: # v is a visplore handle received by v = visplorepy.start_visplore() with data: # new_data_table = v.get_data()get_filter(...)
Returns a list of boolean values, stating for each record of the Visplore table if it passes the current Visplore filter (True) or not (False). Usage: get_filter() Return type: list Example: # v is a visplore handle received by v = visplorepy.start_visplore(): # c = v.get_filter() # returns for example [True, True, True, False, True, ..., True]get_image(...)
Retrieves the view image data and returns it to Python. Usage: get_image(viewID, parameters) Input parameters: viewID(mandatory, string): the ID of the view to export as image. In most cases, this is the name displayed as view title in Visplore. See get_view_properties() to see the view IDs, and if a view supports this. parameters(optional, dictionary): in this optional dictionary, you can customize the image export, using any of the following parameters: - width, height (numbers): the total width/height of the image in pixels. - colwidth, rowheight (numbers): alternatives to total width and height, for views that have rows or columns, for example: Statistics, Heatmaps, Bar Chart, Heatmap,... - colwidth defines the width per column in px, - rowheight defines the height per row in px. - font (string): defines font type and size for all text in the image, for example, "Arial 16", or "Tahoma 20" - xaxislabel (boolean or string): set to False to hide the x axis label or specify as string to show a custom x axis label. - yaxislabel (boolean or string): set to False to hide the y axis label or specify as string to show a custom y axis label. - labelfont (string): format for the label's font type and size, see below. - colorlegend (boolean or string): set to False to hide color legends or specify as string to show a custom legend label - visiblerangebars (boolean): set to False to hide scrollbars. - missingdatabar (boolean): set to False to hide info-bars above some plots. - context (boolean): set to False to show only data records that are in Focus. - selectionhandles (boolean): set to False to hide orange selection tool handles. - title (boolean or string): set to True to show the view name as title, or specify as string to show a custom title. - titlefont (string): format for title's font type and size, see below. - description (boolean or string): set to True to show the Focus description in image, or specify a string to show a custom description text. - descriptionfont (string): format string for description's font type/size, see below. - arealabel (boolean or string): Only available in some(aggregated) views. set to False to hide the area label or specify as string to show a custom area label. - rightyaxislabel (boolean or string): Only available in some(aggregated) views. set to False to hide the right y axis label or specify as string to show a right y axis label. - topxaxislabel (boolean or string): Only available in some(aggregated) views. set to False to hide the top x axis label or specify as string to show a top x axis label. Example: # v is a visplore handle to a Visplore instance.: # Export a view in default size and settings: v.get_image("Time Series") # Export in specific size: v.get_image("Time Series", {"width": 1000, "height": 400}) # Export with specific font: v.get_image("Time Series", {"font": "Arial 16", "title": "My Plot"})get_selected_columns(...)
Returns the names of the currently selected columns in the current visplore table Usage: get_selected_columns() Input parameters: fullNames (optional, boolean, default: False): if True, the full Visplore attribute hierarchy path is used Example: # v is a visplore handle received by v = visplorepy.start_visplore(): # c = v.get_selected_columns() # returns for example ['A_Phase_Voltage_BrightCounty_PV', 'A_Phase_Voltage_Cloudington_PV', ...]get_selection(...)
Returns a list of boolean values, stating for each record of the Visplore table if it is part of (1) the current Focus or (2) a named condition, if name is specified. Usage: get_selection(name = None) Input parameters: name (optional, string, default: None): name of a condition whose membership is returned. If not specified, membership of the current Focus is returned. . Is equivalent to calling get_condition(name). Return type: list Examples: # v is a visplore handle received by v = visplorepy.start_visplore(): # s = v.get_selection() # returns for example [True, False, False, False], stating that only the # first record is currently in Focus. # s = v.get_selection('Outliers') # returns for example [False, True, False, True], stating which records belong to # an existing named condition 'Outliers'get_view_data(...)
Returns the contents of a particular Visplore view as data frame. Is not applicable for all views. Call get_view_properties() to see, which views of the current Cockpit support this. Supported examples : 'Statistics', 'Focus data records', 'Calendar', 'Histogram' Attention : at the moment, get_view_data is only possible for views that are visible in Visplore, not views hidden in tabs. Open them first. Usage: get_view_data(view_id) Input parameters: view_id(mandatory): the ID of the view whose contents are to be retrieved. In most cases, this is the name displayed as view title in Visplore. See get_view_properties() to see the View IDs, and if a view supports this. Return type: pandas data frame Example: # v is a visplore handle received by v = visplorepy.start_visplore(): stats = v.get_view_data('Statistics') # or stats = v.get_view_data('Focus data records')get_view_properties(...)
Returns a table stating for each view - the view ID needed for API calls to it - if the view is exportable via API calls Usage: get_view_properties() Return type: pandas data frame Example: # v is a visplore handle received by v = visplorepy.start_visplore(): props = v.get_view_properties() #returns for example # Displayed Name Identifier Supports Data Export Supports Image Export # 0 Statistics Statistics True True # 1 Heatmaps Heatmaps True True # 2 Histograms Histograms True Falseload_demo_data(...)
Loads demo data and returns it as a pandas.DataFrame. Usage: load_demo_data(dataset) Input parameters: dataset(optional, string) - the identifier of the demo set. Currently packaged data sets are: 'openpv_sampled' - A sampled version of the Open Photovoltaic and Weather data set. (DEFAULT) 'openpv' - The full Open Photovoltaic and Weather data set. Example: #loads the openpv_sampled data pvData = visplorepy.load_demo_data()load_session(...)
Loads a.visplore session with the given filename(full path) Usage: load_session(filename) Input parameters: filename(mandatory): string specifying full name and path of.visplore file restoreData(optional, string): string specifying the mode on how data is restored when loading a session. supported modes are: - "auto": - "askUser": - "fromEmbedded": - "fromLink": - "fromLinkWithDialog": - "no": Example: # v is a visplore handle received by v = visplorepy.start_visplore(): # v.load_session('C:/data/visplore_sessions/my_dashboard.visplore')save_image(...)
Exports a view as image, with optional parameters, and saves it as PNG file to a location. Usage: save_image(viewID, filepath, {parameters}) Input parameters: viewID(mandatory, string): the ID of the view to export as image. In most cases, this is the name displayed as view title in Visplore. See get_view_properties() to see the view IDs, and if a view supports this. filename(mandatory, string): full path of image to create, including the file extension parameters(optional, dictionary): in this optional dictionary, you can customize the image export, using any of the following parameters: - width, height (numbers): the total width/height of the image in pixels. - colwidth, rowheight (numbers): alternatives to total width and height, for views that have rows or columns, for example: Statistics, Heatmaps, Bar Chart, Heatmap,... - colwidth defines the width per column in px, - rowheight defines the height per row in px. - font (string): defines font type and size for all text in the image, for example, "Arial 16", or "Tahoma 20" - xaxislabel (boolean or string): set to False to hide the x axis label or specify as string to show a custom x axis label. - yaxislabel (boolean or string): set to False to hide the y axis label or specify as string to show a custom y axis label. - labelfont (string): format for the label's font type and size, see below. - colorlegend (boolean or string): set to False to hide color legends or specify as string to show a custom legend label - visiblerangebars (boolean): set to False to hide scrollbars. - missingdatabar (boolean): set to False to hide info-bars above some plots. - context (boolean): set to False to show only data records that are in Focus. - selectionhandles (boolean): set to False to hide orange selection tool handles. - title (boolean or string): set to True to show the view name as title, or specify as string to show a custom title. - titlefont (string): format for title's font type and size, see below. - description (boolean or string): set to True to show the Focus description in image, or specify a string to show a custom description text. - descriptionfont (string): format string for description's font type/size, see below. - arealabel (boolean or string): Only available in some(aggregated) views. set to False to hide the area label or specify as string to show a custom area label. - rightyaxislabel (boolean or string): Only available in some(aggregated) views. set to False to hide the right y axis label or specify as string to show a right y axis label. - topxaxislabel (boolean or string): Only available in some(aggregated) views. set to False to hide the top x axis label or specify as string to show a top x axis label. Examples: # v is a visplore handle to a Visplore instance.: # Export a view in default size and settings: v.save_image("Time Series", "C:/temp/image.png") # Export in specific size: v.save_image("Time Series", "C:/temp/image.png", {"width": 1000, "height": 400}) # Export with specific font: v.save_image("Time Series", "C:/temp/image.png", {"font": "Arial 16", "title": "My Plot"})send_data(...)
Sends data to visplore, either replacing the current visplore table, or extending it. Usage: send_data(data, mode) Input parameters: data(mandatory) - in PANDAS dataframe format. supported column types are numerical, categorical, or timestamp attributes. An additional type supported by some Visplore versions are 1D curves, given as a column holding lists in the cells: Each curve is a list of two lists, (1) the axis values and (2) curve values Example: [[0,1,2],[100,-10,30]] in line 1, [[0,1,2],[-20, -30, -40]] in line 2, etc. mode(string, optional, default: 'auto') a string that defines how the data is combined with a possibly existing table in visplore. Options : ('replace', 'merge', 'append', 'auto') tableDescription(string, optional, default: None) an optional table description that is shown in the title bar of Visplore. If not specified, the data type or variable name is shown Example: # v is a visplore handle received by v = visplorepy.start_visplore(): # v.send_data(df) # df is any dataframeset_categorical_filter(...)
Defines a categorical filter for the given channel. Usage: set_categorical_filter(channelName, values) Input parameters: channelName(mandatory, string): channel name of the channel in Visplore values(optional, string): sequence of strings that define the filter mode(string, optional, default: 'Set') a string that defines how the filter is manipulated Options : ('Set', 'SelectAll', 'Delete') Example: # v is a visplore handle to a Visplore instance.: # Define a filter: v.set_categorical_filter("Wind_Directions", ["North","South"]) # Select all categories of 'Plant' in a filter: v.set_categorical_filter("Plant", mode="selectAll") # Remove a filter: v.set_categorical_filter("Wind_Directions", mode="Delete")set_selected_columns(...)
Selects the columns in Visplore that are specified by a list column names(strings). Usage: set_selected_columns(column_names) Input paramters: column_names: a list of strings specifying exact names of columns to select in Visplore. In the same format, as retrieved by get_selected_columns() Example: # v is a visplore handle received by v = visplorepy.start_visplore(): # li = ['A_Phase_Voltage_BrightCounty_PV', 'A_Phase_Voltage_Cloudington_PV'] v.set_selected_columns(li)set_selection(...)
Sets the current Focus of data records to those records marked in the provided sequence of boolean values. Optionally also stores these records as a named condition. Usage: set_selection(sequence, description = '', name = '') Input parameters: sequence(boolean, mandatory) - the records to select(where True). Must be length of the Visplore table. description(optional, string) - a selection description, optional name(optional, string) is the name of a named condition to create and set from these records. Examples: # v is a visplore handle received by v = visplorepy.start_visplore(): # v.set_selection([True, False, False]) # sets the Focus to contain only the 1st record of a 3-record table. # v.set_selection([True, False, False], name = 'MyCondition') # does the same, and creates a named condition called MyCondition that holds these records.start_cockpit(...)
Starts a cockpit with the given name, if such a cockpit exists. Usage: start_cockpit(cockpit). Input parameters: cockpit(string, mandatory) - is the name of the cockpit that should be started skipRolesDialog(bool, optional) - if True: signals to Visplore to attempt to automatically assign roles The role assignment dialog is shown if no automatic role assignment is possible Examples: # v is a visplore handle received by v = visplorepy.start_visplore(): # v.start_cockpit('Trends and Distributions')start_visplore(...)
Starts a new instance of visplore and returns a handle to it. The location of visplore.exe is assumed to be found in an environment variable, called VISPLORE_LOCATION, which is set by the Visplore installer, or by you. Alternatively, you can specify a different visplore location by a parameter. As a third option, the VisploreStarter application can be used (legacy support). Further optional parameters allow specify a cockpit to start, and passing data. Usage: start_visplore(local_visplore=True, path=None, cockpit=None, data=None) Input parameters: local_visplore(optional, boolean) - whether a local Visplore installation is started. Is possible if Visplore was installed by installer, or when you set VISPLORE_LOCATION to path/visplore.exe manually. Default: True. If False, the VisploreStarter app is used. path(optional, string) - if specified, and the local_visplore parameter is set to True, a local visplore.exe is started from this location instead of VISPLORE_LOCATION. cockpit(optional, string) - specifies the name of a cockpit to start with visplore data(optional) - initial data sent to the new visplore instance (see send_data for a description of the expected format) Examples: # Start a new visplore instance from a local installation # v = visplorepy.start_visplore() # Start a new visplore instance from an installer-less Visplore in a different directory # v = visplorepy.start_visplore(local_visplore=True, path="C:/temp/Visplore/Visplore/visplore.exe")