===== Python Packages and Indigo =====

This page describes how **Indigo 2022.2.0** works with Python, specifically with respect to packages/libraries.

Indigo 2022.2 shipped with [[api_release_notes:3.2|API version 3.2]] and is the first version to contain support for Python version 3 (check the [[:API Version Chart]] to see what version of the API is available in which Indigo versions). With this version of the API, Python 2 and Python 3 are now automatically installed as part of the Indigo installer. For Indigo 2022.2, we are including both Python 3.10.2 and Python 2.7.18 (for backwards compatibility - it is an x86 install and will require Rosetta to run on Apple Silicon Macs). The Python 2.7.18 install was specifically added because macOS 12.3 removed its Python install and is a transitional step only. Python 2 and support for APIs < 3.0 will be removed in Indigo 2022.2 later this year.

You may know that Indigo plugins run in a process called the IndigoPluginHost (IPH). With this release, there are now 2 versions of the IPH process: IndigoPluginHost (IPH2 for clarity), which runs the legacy Python 2 API, and IndigoPluginHost3 (IPH3) which runs the new Python 3 based API. We determine which version to use when starting a plugin (by which API the plugin specifies). API 3.0 and later will run the IPH3 process, anything earlier will run the older IPH2 process.

For this release, **all embedded and external Python scripts will be run in IPH3 (and therefore Python 3)** - so you will need to update your scripts if they aren't working (we can't automatically tell if they will work or not unfortunately). We believe that simple scripts will most likely work without change, but if not, you can walk through [[https://github.com/IndigoDomotics/IndigoSDK/blob/main/Updating%20to%20API%20version%203.0%20(Python%203).md|the document]] mentioned above and perhaps spot the changes you need. You can also post your scripts on the [[https://forums.indigodomo.com/viewforum.php?f=364|Help Converting to Python 3]] forum to get help with any conversion issues.

The Indigo 2022.2 installer includes the Python 2.7.18 installer (which is Intel only) and the Python 3.10.2 installer (includes Intel and Apple Silicon support) from [[https://www.python.org]] - the official Mac installers for the Python language. As a result, any installs of other versions of Python from that website may cause unexpected results in Indigo and potentially cause it to break. We've taken all precautions we can to try and avoid conflicts, but there is only so much we can do. So please use caution when installing other Python versions to your server machine. If you have any questions about other Python versions, [[https://forums.indigodomo.com/viewforum.php?f=106|post your questions on our forums]].
=== Installing Additional Python Packages with Indigo 2022.2 === 

As noted above, Indigo 2022.2 ships with two versions of Python. Each version contains several modules in addition to Python's standard core modules. The specific modules installed with each version of Python are listed below. However, in some instances, you may need to manually install one or more additional Python packages to support a third-party plugin or your own custom scripts. The manner in which this is done, depends on the Indigo API version that the plugin supports. 

<color orange>**We were able to extend support for Python 2 as a convenience for our users. As a result, Indigo 2022.2 will continue to support Python 2 and Python 3. Users should plan for Python 2 support to be dropped in Indigo 2023.1.**</color>

For plugins that use Python 3, use ''pip3 install [package]'' commands to install packages:

<code>
Last login: Tue jan 1 12:34:56 on console
user@my mac ~ % pip3 install tensorflow
...
</code>
 
For plugins that use Python 2, use ''pip2 install [package]'' commands to install packages:

<code>
Last login: Tue jan 1 12:34:56 on console
user@my mac ~ % pip2 install pytorch
...
</code>

To see which packages are installed, use the ''list'' command:

<code>
Last login: Tue jan 1 12:34:56 on ttys000
user@my mac ~ % pip3 list
Package            Version
------------------ ---------
astroid            2.9.3
certifi            2021.10.8
charset-normalizer 2.0.11
cycler             0.11.0
fonttools          4.29.1
...
user@my mac ~ %
</code>
**<color blue>Note</color>**: Some Python packages -- and specifically those that are included in IPH3 as [[#python_3102_packages_installed_by_indigo|described below]] -- will not appear in the ''pip3 list'' output.

To see more detailed information about an individual package, use the ''show'' command.

<code>
Last login: Sun Mar 27 10:20:04 on ttys000
user@my mac ~ % pip3 show requests
Name: requests
Version: 2.27.1
Summary: Python HTTP for Humans.
Home-page: https://requests.readthedocs.io
Author: Kenneth Reitz
Author-email: me@kennethreitz.org
License: Apache 2.0
Location: /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages
Requires: charset-normalizer, certifi, idna, urllib3
Required-by: 
user@my mac ~ % 
</code>
Particularly helpful in the ''show'' output is the dependencies information -- both what the package requires and what requires the package. Note that the dependencies information may not include information about packages that are included with the IPH.

<color red>Installing additional Python versions on the Indigo 2022.2 server machine is __**NOT**__ recommended (see above). Installing additional Python modules should only be done when necessary.</color> 
=== Python Packages for Plugin Developers ===

Because the packages included with Indigo should not be modified (to ensure a smoothly running server), plugin developers must include any other desired version within the plugin package. Plugin developers should note that some Python packages will not integrate universally across all installations as they rely on code compiled for specific hardware configurations (for example, Intel vs. Apple silicon). The IPH processes use the following [[https://docs.python.org/3/tutorial/modules.html#the-module-search-path|Module Search Paths]] (Python interpreters use ''PYTHONPATH'' to describe the order of directories to search for installed packages, the first matching package will be used):

The ''PYTHONPATH'' for IPH3 is:
  - ''Contents/Server Plugin'' (inside the plugin bundle)
  - ''Contents/Packages'' (inside the plugin bundle)
  - IndigoPluginHost3 (inside the app itself)
  - ''/Library/Application Support/Perceptive Automation/Python3-includes''
  - ''/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages''

and the ''PYTHONPATH'' for IPH2 is:
  - ''Contents/Server Plugin'' (inside the plugin bundle)
  - IndigoPluginHost (inside the app itself)
  - ''/Library/Application Support/Perceptive Automation/Python2-includes''
  - ''/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages''

Packages are installed in three ways:

  - Some packages are included within the IPH itself. These are "protected" and can not be modified ([[#python_3102_packages_installed_by_indigo|listed below]]).
  - You may add modules/packages that you develop yourself to the ''Python#-includes'' folders above, and those will be shared across all Indigo Plugins and Scripts.
  - Other packages are installed using a ''pip3'' (or ''pip2'') process. **<color orange>Warning</color>**: Indigo will preinstall several packages ([[#python_3102_packages_installed_by_indigo|listed below]]) at install time. You should avoid modifying those unless absolutely necessary as other 3rd party plugins may rely on them.

=== Access to IPH3 Libraries ===
There are some important implications that result from having some libraries included in the IPH3.  Namely, those packages will be available to plugins, Indigo scripts (embedded and linked), the Indigo python shell (from the menu item or by doing ''indigo-host'' in a terminal window), but IPH3 libraries will __**NOT**__ be available if you start the python interpreter  directly (by doing python2 or python3 in a terminal window).  Refer to this chart to determine accessibility:

^  Method  ^  Access to\\ IPH3 Libraries  ^
|  plugins  |  yes  |
|  embedded scripts  |  yes  |
|  linked scripts  |  yes  |
|  Indigo python shell using Indigo menu  |  yes  |
|  Indigo python shell using Terminal ''indigo-host''  |  yes  |
|  Mac Terminal using ''python3'' command  |  no  |

=== Python Packages for Scripters ===

In this release, all Python scripts (either embedded or linked) are executed using the IPH3 process using Python 3. Therefore, the server will be able to locate any packages your scripts use in the same way as described above:

  - IndigoPluginHost3
  - /Library/Application Support/Perceptive Automation/Python3-includes
  - /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages

If you need a module/package for your plugin, you can install it via ''pip3'' as shown above, which will install it into the Python ''site-packages'' folder. We recommend (but don't require) the following folders be used for custom scripts and modules:

  * **/Library/Application Support/Perceptive Automation/Scripts** -- any linked scripts you use for Indigo Schedules and Actions using the "Execute Script (Script and File Actions" command should be stored in this folder.
  * **/Library/Application Support/Perceptive Automation/Python3-includes** -- any modules you have developed (a module is a file that contains classes, functions, constants, etc.) that you want to share across multiple scripts, should be stored in the ''/Library/Application Support/Perceptive Automation/Python3-includes'' folder. (Indigo versions prior to 2022 should use the ''/Library/Application Support/Perceptive Automation/Python2-includes'' folder).

**<color orange>Warning 1</color>**: as mentioned above, Indigo will preinstall several packages ([[#python_3102_packages_installed_by_indigo|listed below]]) at install time. You should avoid modifying those unless absolutely necessary as other 3rd party plugins may rely on them.

**<color orange>Warning 2</color>**: you should use caution when naming your custom scripts to ensure they don't supplant other Python packages. For example, if you install the **holidays** Python package using ''pip3 install holidays'', it will be installed in the **site-packages** folder. If you also have a custom script in the ''Python3-includes'' folder named **holidays.py**, the IPH will import your script instead of the **site-packages** version (because the Python3-includes folder is searched before the site-packages folder).
=== If Something Goes Wrong ===

To provide users with a measure of safety regarding the Python packages that Indigo installs, the Indigo installer also includes the capacity to repair the Python installation if something should become damaged. For example, if a user were to accidentally upgrade one of the Python packages -- and as a result cause the Indigo server to stop functioning properly -- re-running the Indigo installer may be able to bring things back under control. There's [[indigo_2022.2_documentation:python_version_conflicts|a few things you can try]] should you run into conflicts between Python versions.

Re-running the Indigo installer on an existing server shouldn't affect user-installed libraries (existing user-installed libraries should be unaffected).

=== Moving Indigo to a New Machine ===
Depending on the method used, any Python modules installed via **pip3** may need to be re-installed on the new machine. For more information, consult [[https://wiki.indigodomo.com/doku.php?id=indigo_2022.2_documentation:getting_started#moving_an_indigo_installation_to_another_mac|Moving an Indigo Installation to Another Mac]].
=== Python 3.10.2 Packages Installed by Indigo ===

Indigo 2022.2 ships with support for the following Python 3 packages in addition to the packages installed with the standard Python framework. Generally, if earlier versions of the listed packages are installed at the location Indigo uses, they will be upgraded to the version numbers below.
 
^  Included inside IPH 3\\ Unmodifiable  ^^  PIP installed by Indigo\\ PIP Modifiable (but not recommended)  ^^
^  Python Package    ^  Version  ^    Python Package     ^    Version     ^
| py-applescript     |     1.0.3 | cycler                |         0.11.0 |
| certifi            | 021.10.08 | cryptography          |         39.0.2 |
| charset_normalizer |    2.0.12 | - cffi                |         1.15.1 |
| python-dateutil    |     2.8.2 | - pycparser           |           2.21 |
| idna               |       3.3 | dictdiffer            |          0.9.0 |
| jedi               |    0.18.1 | fonttools             |         4.31.2 |
| oauthlib           |     3.2.0 | future                |         0.18.2 |
| packaging          |      21.3 | httpx                 |         0.23.3 |
| parso              |     0.8.3 | janus                 |          1.0.0 |
| pudb               |  2022.1.0 | - typing_extensions   |          4.4.0 |
| pyaes              |     1.3.0 | Jinja2                |          3.1.2 |
| pygments           |    2.11.2 | kiwisolver            |          1.4.0 |
| pyparsing          |     3.0.7 | MarkupSafe            |          2.1.1 |
| pyserial           |       3.5 | matplotlib            |          3.5.1 |
| requests           |    2.27.1 | numpy                 |         1.22.3 |
| requests-oauthlib  |     1.3.1 | packaging             |           21.3 |
| urllib3            |    1.26.8 | Pillow                |          9.0.1 |
| urwid              |     2.1.2 | pyparsing             |          3.0.7 |
| xmljson            |     0.2.1 | python-dateutil       |          2.8.2 |
|                    |           | pyobjc                |          8.4.1 |
|                    |           | sanic                 |         22.6.1 |
|                    |           | - aiofiles            |         22.1.0 |
|                    |           | - httptools           |          0.5.0 |
|                    |           | - multidict           |          6.0.4 |
|                    |           | - sanic-routing       |         22.3.0 |
|                    |           | - sanic-session       |          0.8.0 |
|                    |           | - ujson               |          5.7.0 |
|                    |           | - uvloop              |         0.17.0 |
|                    |           | - websockets          |           10.4 |
|                    |           | scipy                 |          1.8.0 |
|                    |           | six                   |         1.16.0 |

=== Python 2.7.18 Packages Installed by Indigo ===

Indigo 2022.1 ships with the following Python 2 packages in addition to the packages installed with the standard Python framework.
 
^  Included inside IPH 2\\ Unmodifiable  ^^  PIP installed by Indigo\\ PIP Modifiable (but not recommended)  ^^
^  Python Package   ^  Version  ^       Python Package         ^   Version    ^
| enum              |     1.1.6 |backports.functools-lru-cache |        1.6.4 |
| oauthlib          |     2.0.2 |cycler                        |       0.10.0 |
| requests          |    2.10.0 |future                        |       0.18.2 |
| requests-oauthlib |     0.8.0 |kiwisolver                    |        1.1.0 |
| pudb              |    2016.1 |matplotlib                    |        2.2.5 |
| py-applescript    |     1.0.3 |numpy                         |       1.16.6 |
| pyaes             |     1.3.0 |pyparsing                     |        2.4.7 |
| pygments          |     2.1.3 |python-dateutil               |        2.8.2 |
| pyserial          |     3.1.1 |pytz                          |       2022.1 |
| python-dateutil   |     2.8.2 |scipy                         |        1.2.3 |
| six               |    1.16.0 |six                           |       1.16.0 |
| urwid             |     1.3.1 |subprocess32                  |        3.5.4 |
| xmljson           |     0.1.7 |                              |              |