In our last article on my journey into Python, we wrote our first useful application, a nifty little graphical password generator. If you managed to plough your way through that post, my hat off to you.
To refresh your memory our previous articles, you can read them at the links shown below:
- Moving up the stack – time to learn Python – part 1
- Moving up the stack – time to learn Python – part 2
- Moving up the stack – time to learn Python – part 3
- Moving up the stack – time to learn Python – part 4
- Moving up the stack – time to learn Python – part 5
- Moving up the stack – time to learn Python – part 6
- Moving up the stack – time to learn Python – part 7
- Moving up the stack – time to learn Python – part 8
In our last article I left you with a completed application, however, to use it on a target machine, you would still need a full Python implementation to run it, if you remember one of the requirements was that the application needed to run on a target host without needing a Python interpreter being installed. So today we are going to turn our nifty little script into a standalone application.
Preparing the environment.
Even though this is a very simple program, turning it into an application is quite a complex process. However, do not be worried we will run through it all on this article.
Project Prerequisites:
The first thing we need to do is confirm that we have the necessary modules installed to undertake the task, you are looking for “pysinstaller”. Another thing of note here is that program compiling is a platform specific process, this makes sense, as you cannot run a Mac program on a Windows desktop or visa-versa.
So how do we verify what modules we currently have installed?
At the command prompt issue the following command: python -c “help(‘modules’)”
You should receive something similar to the following output:
As you can see we do not currently have ““PyInstaller”” deployed in our dev environment, so lets fix that little problem.
At the command line issue the following command:
pip install “PyInstaller”
If this is successful you should receive an output very similar to what is shown below:
The “pip” installer is quite powerful it will as is the case with the majority of installers calculate the necessary dependencies needed for the package to run and install them too. So now lets verify that we do indeed have ““PyInstaller”” installed. We will now reissue our previous command:
python -c “help(‘modules’)”
if everything goes to plan you should be able to see “PyInstaller” in your list of modules:
And right there you can see “PyInstaller”, that command is a little over the top for installation verification, fortunately, we can just use ““PyInstaller” –version” to verify:
That’s better and a lot more informative as we can now also see that have installed the latest and greatest (at the time of writing) version of ““PyInstaller”, version 5.13.2”.
What is more interesting is that the module count went up by so much. This is quite simple. “PyInstaller” is actually a package. Remember those? They package up a number of modules into a working application, if you look closely at the two images you can see that there are a significant number of recently installed modules between the first and second image. You should see things like ““PyInstaller”, atexit, getopt, rlcompleter, runpy and sched”
There is another verification method too as we used pip to install “PyInstaller” if we issue the below command:
pip list
We will receive a response similar to what is shown below, you can clearly see that we have installed ““PyInstaller”” and the version is indeed 5.13.2:
What is “PyInstaller”?
“PyInstaller” is a powerful tool that allows developers to package Python applications into standalone executables for distribution. One of the key features of “PyInstaller” is its ability to operate in various modes, each with its own set of benefits and drawbacks. In this article, we will explore the different operating modes of “PyInstaller” and how they can be used to optimize the packaging process.
“PyInstaller” Operating Modes
“PyInstaller” operates in three different modes: “onefile”, “OneDirectory”, and “SpecFile”. Each mode has its own advantages and disadvantages, and choosing the right mode depends on the specific needs of the application being packaged.
OneFile Mode
“onefile” mode packages the entire application into a single executable file. This mode is useful for distributing applications that require no external dependencies, as everything is contained within the executable. “onefile” mode also makes it easy to distribute and install the application, as users only need to download and run a single file.
However, “onefile” mode has some limitations. The resulting executable can be quite large, especially if the application includes large libraries or resources. Additionally, “onefile” mode can be slower to start up than other modes, as the entire application needs to be extracted from the executable before it can be executed.
OneDirectory Mode
“OneDirectory” mode packages the application into a directory containing all of its dependencies. This mode is useful for applications that require external dependencies, as everything is contained within a single directory. “OneDirectory” mode also allows for more efficient caching of dependencies, as they can be reused across multiple executions of the application.
However, “OneDirectory” mode can be more difficult to distribute and install than “onefile” mode. Users need to download and extract the entire directory, which can take longer than simply downloading and running a single file. Additionally, “OneDirectory” mode can result in a larger overall package size than “onefile” mode, as each dependency needs to be included in the package.
SpecFile Mode
“SpecFile” mode generates a Python script that can be used to build the application package. This mode is useful for customizing the packaging process, as developers can modify the generated script to include or exclude specific files or directories. “SpecFile” mode also allows for more fine-grained control over the packaging process, as developers can specify exactly how they want the application to be packaged.
However, “SpecFile” mode requires more manual intervention than “onefile” or “OneDirectory” mode. Developers need to modify the generated script to include all necessary dependencies, which can be time-consuming and error prone. Additionally, “SpecFile” mode can result in a larger overall package size than “onefile” or “OneDirectory” mode, as developers may include unnecessary files or directories in the package.
Choosing the Right Mode
Choosing the right operating mode depends on several factors, including the size and complexity of the application being packaged, the number and type of external dependencies required, and the desired distribution method. For small applications with few external dependencies, “onefile” mode may be the best choice. For larger applications with many dependencies, “OneDirectory” or “SpecFile” mode may be more appropriate.
Creating our standalone Python application
The simplicity of the application effectively choses the” “PyInstaller”” mode, for this our first attempt at compiling we will use the basic “onefile” method.
Project File Structure:
If you remember, when we started writing the password generator we did not configure any directory file structure other than the root package directory; this was because the program was quite simplistic. We only have two functions defined. This makes the creation of the standalone application easier.
Compiling our application
As we have already stated ““PyInstaller”” is an excellent tool for compiling Python scripts into standalone executables; due to the simplicity of our script we will use the “onefile” mode, to package the entire application into a single executable file.
Using “PyInstaller”‘s “onefile” mode is easy. Simply run the following command:
pyInstaller --onefile your_script.py
press enter and you are off.
A quick “ls” or “dir” in the source directory will show several changes. What at first may have appeared to be a very simple undertaking, has made some significant changes in our source directory. Well to be fair this was expected. Lets dive a little deeper into the process that has just happened with our python script
The final act of the process will create a single executable file in the newly created “dist” directory. This file will be named after our script, with the appropriate extension for your operating system (e.g. .exe on Windows).
One of the main benefits of using “onefile” mode is that it makes distribution of your application much simpler. Instead of having to distribute multiple files (e.g. your script, any required libraries, etc.), you can simply distribute a single file. This can make it much easier for users to install and run your application. Another benefit of “onefile” mode is that it can help protect your source code. When you distribute your application as a single executable file, it is much more difficult for users to reverse-engineer your code. However, there are a few things to keep in mind when using “onefile” mode. First, the resulting executable file may be larger than if you were to distribute your application as multiple files; our simple script produced a 10mb file. This is because all of the required libraries and resources are included in the executable file.
This all seems a little simplistic. And you know me I need to know the how and why; not just see the result.
Diving in and seeing what happened
We already saw one change the creation of the resultant exe file in the “dist” folder. However there was also a build folder created. Looking in that folder we are presented with another folder
That has the same name as our python file; so let’s look in there.
Now that looks interesting; there are lots of created files here. If take a look at the output of the “PyInstaller” command we can actually see these files being created. However looking at the output of the “PyInstaller” command the first file to be created is the “password_genterator.spec” file. This file contains information about the application’s dependencies and how to build the executable. The “.spec” file is a Python script that can be edited to customize the build process.
The “.spec” file is split into sections that define the application’s properties, such as the name, version, and application icon. It also includes a list of the script files that should be included in the build, as well as any additional data files that need to be packaged with the application. The file also lists the application’s dependencies, which can be automatically detected by ““PyInstaller”” or manually specified by the developer. Finally, the .spec file includes options for how to build the executable, such as whether to use compression or add debugging information.
PS C:\Users\tomho\python\NewPassword> “PyInstaller” --onefile .\python_password_generator.py 1119 INFO: “PyInstaller”: 5.13.2 1119 INFO: Python: 3.11.4 1136 INFO: Platform: Windows-10-10.0.23430-SP0 1136 INFO: wrote C:\Users\tomho\python\NewPassword\python_password_generator.spec
The next message indicate that the process is extending the PYTHONPATH environment variable with additional paths. The PYTHONPATH environment variable is used by Python to search for imported modules, along with other standard and third-party library directories listed in Python’s “sys.path”.
1155 INFO: Extending PYTHONPATH with paths ['C:\\Users\\tomho\\python\\NewPassword']
The next stage of the installation process is analysing the program to be compiled to determine which modules are required by the script and how they should be packaged into the final executable.
The “Building Analysis because Analysis-00.toc is non-existent” message indicates that “PyInstaller” is building the Analysis because it does not exist yet. The completed Analysis is stored in this file.
“The Initializing module dependency graph…” message shows that ““PyInstaller”” is initializing a graph of module dependencies. This determines if any modules require other modules to run and how they should be packaged into the final executable.
“The Caching module graph hooks…” message indicates that ““PyInstaller”” is caching hooks for modules. A Hook is a script that provides additional information about how to package certain modules into the final executable.
1714 INFO: checking Analysis 1715 INFO: Building Analysis because Analysis-00.toc is non existent 1715 INFO: Initializing module dependency graph... 1719 INFO: Caching module graph hooks...
This log message it indicates that “PyInstaller” is analyzing the “base_library.zip” file and loading module hooks for heapq, encodings, and pickle. The Caching module dependency graph… message indicates that “PyInstaller” is caching the module dependency graph, which as we know is to determine which modules are required by other modules and how they should be packaged into the final executable.
1804 INFO: Analyzing base_library.zip ... 5234 INFO: Loading module hook 'hook-heapq.py' from 'C:\\Users\\tomho\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\”PyInstaller”\\hooks'... 5439 INFO: Loading module hook 'hook-encodings.py' from 'C:\\Users\\tomho\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\”PyInstaller”\\hooks'... 8886 INFO: Loading module hook 'hook-pickle.py' from 'C:\\Users\\tomho\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\”PyInstaller”\\hooks'... 14012 INFO: Caching module dependency graph...
The next section indicates that “PyInstaller” is running the Analysis process and adding Microsoft.Windows.Common-Controls to the dependent assemblies of the final executable. These dependent assemblies are required by the python executable.
The Analysis process is performed by “PyInstaller” to determine which modules are required by the script and how they should be packaged into the final executable and used Analysis-00.toc file which contains the relevant information needed.
14245 INFO: running Analysis Analysis-00.toc 14286 INFO: Adding Microsoft.Windows.Common-Controls to dependent assemblies of final executable required by C:\Users\tomho\AppData\Local\Programs\Python\Python311\python.exe
This next section indicates that “PyInstaller” is analysing the python_password_generator.py file and processing the module hooks. The “Building Tree because Tree-00.toc is non existent” message indicates that “PyInstaller” is building the Tree because it does not exist yet. A Tree is a list of files and directories that are copied to the distribution directory.
The “Looking for ctypes DLLs” message indicates that “PyInstaller” is searching for ctypes DLLs ctypes is a foreign function library for Python that provides C compatible data types and allows calling functions in DLLs or shared libraries
15092 INFO: Analyzing C:\Users\tomho\python\NewPassword\python_password_generator.py 15391 INFO: Processing module hooks... 15407 INFO: Loading module hook 'hook-_tkinter.py' from 'C:\\Users\\tomho\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\”PyInstaller”\\hooks'... 15409 INFO: checking Tree 15409 INFO: Building Tree because Tree-00.toc is non existent 15409 INFO: Building Tree Tree-00.toc 15595 INFO: checking Tree 15595 INFO: Building Tree because Tree-01.toc is non existent 15595 INFO: Building Tree Tree-01.toc 15622 INFO: checking Tree 15622 INFO: Building Tree because Tree-02.toc is non existent 15622 INFO: Building Tree Tree-02.toc 15667 INFO: Looking for ctypes DLLs
The next sections log messages provide information about the analysis, runtime hooks, and dynamic libraries used during the conversion process. The “Analyzing run-time hooks …” message shows that “PyInstaller” is analysing the runtime hooks required for the executable these are things like the MS Visual C++ Runtimes. Next two messages relate to adding the necessary hooks for the inspect module (pyi_rth_inspect.py ) and tkinter (pyi_rth_tkinter.py), finally, you can see that we are now searching though any necessary search paths for other dependencies.
15700 INFO: Analyzing run-time hooks ... 15700 INFO: Including run-time hook 'C:\\Users\\tomho\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\”PyInstaller”\\hooks\\rthooks\\pyi_rth_inspect.py' 15717 INFO: Including run-time hook 'C:\\Users\\tomho\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\”PyInstaller”\\hooks\\rthooks\\pyi_rth__tkinter.py' 15783 INFO: Looking for dynamic libraries 598 INFO: Extra DLL search directories (AddDllDirectory): [] 598 INFO: Extra DLL search directories (PATH): “system path removed”
This is a cute one that caught my eye. This message it indicates that “PyInstaller” is searching for Python eggs. These are a distribution format for Python packages that contain pre-built code and metadata about the package.
If a script imports a module from an egg, which ours does not, “PyInstaller” will add the egg and any dependencies to the set of needed files. This ensures that the final executable has access to all the required modules and libraries at runtime.
17097 INFO: Looking for eggs
The next set of log messages provide information about the analysis, runtime hooks, and dynamic libraries performed by “PyInstaller” during the conversion process. Here we can see that we are using the common python library located in Appdata, we have found no “Found binding redirects: []: “PyInstaller” did not find any binding redirects”. We are now also creating our warnings file which is stored in the build\filename directory as “warning_password_generator.text”. finally we create a html file that stores our graph cross-reference analyse, again in the build\filename directory this time as “xref_python_password_generator.html”.
These log messages help track the progress of “PyInstaller”’s conversion process and ensure that all necessary components are included in the final executable.
17115 INFO: Using Python library C:\Users\tomho\AppData\Local\Programs\Python\Python311\python311.dll 17115 INFO: Found binding redirects: [] 17117 INFO: Warnings written to C:\Users\tomho\python\NewPassword\build\python_password_generator\warn-python_password_generator.txt 17164 INFO: Graph cross-reference written to C:\Users\tomho\python\NewPassword\build\python_password_generator\xref-python_password_generator.html
The next set of log messages provide further information about the analysis, runtime hooks, and dynamic libraries performed by “PyInstaller” during the conversion process this time about the PYZ file; which is a compressed archive of the source python script and its dependencies. The final message related to Bootloader which is a component installed during the installation of “PyInstaller” that bootstraps a python programme. It is a self-contained executable that is generated once and used for all subsequent conversions. The bootloader is responsible for initializing the Python interpreter, loading the necessary modules, and executing your Python code.
The pre-compiled bootloaders provided with “PyInstaller” are static executables that impose no restrictions on the version of Python being used. However, if you want to modify the bootloader source or build a custom bootloader, you can do so by following the instructions provided in the “PyInstaller” documentation
17304 INFO: checking PYZ 17305 INFO: Building PYZ because PYZ-00.toc is non existent 17305 INFO: Building PYZ (ZlibArchive) C:\Users\tomho\python\NewPassword\build\python_password_generator\PYZ-00.pyz 17754 INFO: Building PYZ (ZlibArchive) C:\Users\tomho\python\NewPassword\build\python_password_generator\PYZ-00.pyz completed successfully. 17833 INFO: checking PKG 17833 INFO: Building PKG because PKG-00.toc is non existent 17833 INFO: Building PKG (CArchive) python_password_generator.pkg 27894 INFO: Building PKG (CArchive) python_password_generator.pkg completed successfully. 27946 INFO: Bootloader C:\Users\tomho\AppData\Local\Programs\Python\Python311\Lib\site-packages\PyInstaller\bootloader\Windows-64bit-intel\run.exe
This final block of messages contain the meat and two veg of the compile process. The message “checking EXE” tells us that “Pyinstaller” is checking the EXE file, which is the final executable that will be created from your Python script. Now due to it not being available the next message “Building EXE because EXE-00.toc is non existent” tells us that “Pyinstaller” is building the EXE file because it does not exist yet. The message “Building EXE from EXE-00.toc” informs us that “Pyinstaller” is using the information stored in the EXE-00.toc file to build the EXE file. Next “Pyinstaller” copies the bootloader executable into a temporary files called “python_password_generator.exe.notanexecutable” which is stored in the “dist” folder. The next several info messages are informing us about icons, as we do not actually have one for this application we can safely ignore them. Next we look at the manifest file. This is an XML formatted file that contains all our information and can be used to recreate a build by adding the “–manifest myapp.exe.manifest” option to the “Pyinstaller” command. The message “Updating resource type 24 name 1 language 0” shows us that we have now embedded the manifest file in the psudeo executable file, this is followed by “Pyinstaller” “appending PKG archive to EXE” and “Fixing EXE headers”; we are finally informed that we have built the final “EXE from EXE-00.toc [and that it has]
27946 INFO: checking EXE 27947 INFO: Building EXE because EXE-00.toc is non existent 27949 INFO: Building EXE from EXE-00.toc 27949 INFO: Copying bootloader EXE to C:\Users\tomho\python\NewPassword\dist\python_password_generator.exe.notanexecutable 28053 INFO: Copying icon to EXE 28063 INFO: Copying icons from ['C:\\Users\\tomho\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\”PyInstaller”\\bootloader\\images\\icon-console.ico'] 28126 INFO: Writing RT_GROUP_ICON 0 resource with 104 bytes 28126 INFO: Writing RT_ICON 1 resource with 3752 bytes … 28126 INFO: Writing RT_ICON 7 resource with 1128 bytes 28147 INFO: Copying 0 resources to EXE 28149 INFO: Embedding manifest in EXE 28149 INFO: Updating manifest in C:\Users\tomho\python\NewPassword\dist\python_password_generator.exe.notanexecutable 28211 INFO: Updating resource type 24 name 1 language 0 28211 INFO: Appending PKG archive to EXE 28228 INFO: Fixing EXE headers 31324 INFO: Building EXE from EXE-00.toc completed successfully.
Summary
In summary, this has been a long post, but I felt that it was important to understand the process that was being carried out. Creating an executable using “PyInstaller”‘s “onefile” mode is a fairly simple action but the underlying process are complex. it is a powerful feature that can simplify distribution of your Python applications. By packaging everything into a single executable file, we can make it easier for users to install and run our applications, while also helping to protect our source code. However it is important to be aware of the potential downsides, such as larger executable file sizes and the need to include any required external files.
