ESP-IDF, the software development framework for the ESP32 series of chips, is in active development since 2016. It was initially designed with a command-line-interface (CLI) usage in mind. Over the past few years, we have received many requests from our customers to simplify the ESP-IDF installation process and usage. This article describes a few recent improvements in this area.

The Build Systems

When ESP-IDF was first released, it had a build system based on GNU Make at its core. The build system was quite flexible and allowed for the rapid use of the ESP-IDF codebase. It worked quite well on Linux and macOS, which were the operating systems used by most Espressif developers at the time. The build system did suffer from several issues, though. The main one was that, on Windows, GNU Make required an emulation layer like MSYS or Cygwin. This meant much slower project builds on Windows, compared to those on Linux with the same hardware. The other issue was that adding GNU Make-based projects to integrated development environments (IDEs) was difficult to be done reliably. For a while, ESP-IDF documentation provided instructions on using Eclipse CDT with its GNU Make build system. This approach worked in many cases but, when it didn't (mostly on Windows), troubleshooting was complex.

To address these issues, in 2018 the ESP-IDF project started moving to a new build system which was based on CMake. While we tried to keep the overall “look and feel” of ESP-IDF similar between Make and CMake, this was still a significant change for many users who were working on already existing projects. For this reason, we decided to keep supporting the Make-based build system in parallel with developing the new CMake-based one. However, in ESP-IDF v4.0, released in February 2020, the CMake-based build system became the default one, and the Make-based one was demoted to “legacy” status. Since then, new features, such as support for ESP32-S2, ESP32-C3 and ESP32-S3 chips, were added to the CMake-based build system only.

The CMake build system was key to better integrations with IDEs. In 2019, we started developing plugins for two IDEs: Visual Studio (VS) Code and Eclipse. These plugins made features that were specific to ESP-IDF available directly in the IDEs, thus simplifying the process of getting started with ESP-IDF.

Tools and Installation

ESP-IDF relies on a growing number of tools, such as cross-compiler toolchains and debugging tools. These tools are provided by Espressif as binary distributions for each of the supported operating systems. Before ESP-IDF v4.0, users had to download the correct toolchain, extract it somewhere on their computer, and then add it to the `PATH` environment variable. This basic approach let the users set up their environment exactly the way they wanted. It was, unfortunately, quite error-prone especially when users had to switch between different ESP-IDF versions. In ESP-IDF v4.0, new scripts were added to the SDK, allowing users to set up the correct tools automatically. The installation approach was chosen such that multiple ESP-IDF versions could be installed side by side on the computer, and each could use the correct toolchain without any conflicts.

The installation process for Windows users was further improved this year, as we overhauled our ESP-IDF Tools Installer. We analyzed the issue reports on the ESP-IDF installation process, and addressed the main causes of errors in this update. Firstly, the installer can now set up standalone copies of the Python interpreter and the Git version control system, so that ESP-IDF no longer depends on the ones that may be installed on the computer. Secondly, the installer uses a fallback source of Python packages (“wheels”), in case some of the packages are not available from official sources. Thirdly, the installer runs additional checks for prerequisites, which helps detect potential issues with antivirus software, installation directories and other common problems. The installer is now available in two variants: “online”, a minimal installer which downloads required software during the installation process, and “offline”, which is a monolithic package that includes everything necessary.

We continue listening to developers' feedback about ESP-IDF tools and installation, and we will keep working on improvements in this area.

Integrated Development Environments

In addition to the command line interface, Espressif provides two development environments for ESP-IDF users: the Visual Studio (VS) Code extension for ESP-IDF and the ESP-IDF Eclipse Plugin. Both environments have been in active development since 2019. This year we focused our IDE development efforts on three areas: installation and onboarding process, core development experience (such as creating and configuring projects, indexing, flashing, monitoring), and JTAG-related improvements.

Based on the lessons we have learned from the IDF Tools Installer for Windows, both IDEs have been improved in terms of their onboarding process. Windows users can now also get the Eclipse and IDF plugin set up from the offline version of the installer. The onboarding process in our VS Code extension has been reworked and can now be completed in just a few steps. We also implemented an integration of our VS Code extension with esp-idf.json, and we embedded python (using Espressif’s pip index) and git in the VS Code extension, in order to automate the onboarding process as much as possible in the Windows environment.

As our newest ESP32-S3 and ESP32-C3 chips contain embedded USB-JTAG interfaces, several improvements have been made to JTAG-related features of the IDEs. In addition to flashing over the serial port, we have added support for flashing over JTAG. With the IDF Eclipse plugin 2.2.0 and the OpenOCD plugin (v0.10.0-esp32-20201125 and above), the user can run applications via JTAG conveniently. To do this, the user simply needs to choose the JTAG flashing option in the launching configuration and select the appropriate board. Also, the possibility of forgetting to flash the application before entering the debugging session is now eliminated, since flashing the application via JTAG has become the default option. In case the user has not installed the required version of OpenOCD, we will alert them with the appropriate information, while also prompting them to install the latest version.

Both of the IDEs come with heap-tracing functionality, which allows the user to generate a svdat dump file via debugging (by adding start/stop breakpoints in the code), and to do the heap dump analysis subsequently. This is a very useful feature for analysing heap events and their memory allocations, as well as for identifying possible memory leaks.

Future Plans

Our work on improving ESP-IDF developers’ experience does not end here. Aside from improvements to our existing tools, we are researching new solutions, such as containerization, online coding, and prototyping workspaces. We welcome your feedback as it helps us guide the development direction we will choose in the future. You can send us your comments via either the ESP32 forum or specific GitHub repos, such as ESP-IDF, IDF-Eclipse-plugin and VSCode-ESP-IDF-extension.