Mi Casa su Botnet?

Learning the Internet of Things with
WaterElf, unPhone and the ESP32

Hamish Cunningham

Iteration 4 (Spring 2021, 0.914.2021-03-06; PDF version).
Copyright © Hamish Cunningham. Licence: CC-BY-SA-NC 4.0.

1 Hope, Technology and Heath Robinson

1

This is a work in progress!
Early chapters are complete; later ones will evolve
further during the course. Incomplete chapters start
with a warning; use them at your peril!

These notes describe COM3505 (the Internet of Things) iteration 4, running in Spring 2021. Each chapter begins with general discussion, history or theory, then finishes with instructions for a week’s worth of practical work. Although later material depends to varying degrees on preceding chapters, we have tried to make the order of the practical work as flexible as possible so that you can learn in whatever sequence is convenient for you (partly because we are expecting less access to laboratory time this year due to the pandemic, and some students will be remote working).

The practical work in this chapter and next, though, covers enrolling on the course and setting up your development environment, so please prioritise its early completion.

So, what is the IoT and where does this course fit in? Read on…2

1.1 Welcome to the Toy Shop!

We’re six month late, but you made it! You’ve worked hard for years to get this far, and now you get to play :)

As computer scientists and software engineers we spend much of our time with machines whose secrets are hidden beneath multiple abstraction layers: BIOS, bootloader, operating system, programming language, library API, protocol definition, SDK, etc. etc. These layers allow us to build systems of incredible power and sophistication, but they also (intentionally) obscure the operation of the underlying hardware engine. This course is a chance to rip the top off the box of tricks, hotwire the starter motor and delve deep into the grungey secrets of the electronic wizardry that makes our discpline possible.

The course is also a way to understand how the latest great wave of technological transformation is streaming into our lives, enabled by the combination of tiny connected microcontrollers, cyber-physical systems and ubiquitous networking. This transformation is driven by the same logic as the explosive consequences of TCP/IP and HTML, but if the last 20 years were about the web, then the next 20 will be about making. Just as always-on connectivity and decentralised production in the virtual world made revolutions in creating, sharing and consuming on-line, now the same changes are starting in the world of manufacturing, and the consequences are likely to be massive. (There’s perhaps an 80-20 ratio between economic activity devoted to atoms, or physical processes, in comparison to bits, or informational processes.) In the IoT there’s still a space to get in on the ground floor, and learn how to make with the foundational hardware of the next wave.

Hang on to your hat; the future is a whirlwind!

1.1.1 What’s the Catch?

Actually there are two :(

First, you’re going to be programming a tiny little device that has, in comparison to that shiney laptop on your desk or that smartphone in your pocket, close to zero computational resource. There’s no paging, for example, so in the sense of time sharing (first developed in the 1960s) our devices don’t even run an operating system! This can be a challenge. Be prepared.

Second, you’re going to need to install an SDK of one sort or another on your own machine. Partly this is because the pandemic has forced us to reduce the amount of on-campus time you will have to work with University-supported machines; partly it is because you need to be able to spend concentrated hours playing with the beasts, and there’s nowhere better than your own computer for experimenting.

To help you with these challenges we will:

If you don’t have a machine that you can use for this and need something cheap, you might try a Raspberry Pi 4 or a Pi 400 (preferably the 8GB models), both of which are very capable of running the basic IoT SDK that we work with.

If you’ve already got a machine running MacOS or Windoze (or OS/360) don’t panic:

And: look on the bright side, the more you do, the more you learn :)

1.1.2 Catch #3: ‘Click Here to Kill Everybody?’

Actually I lied: there are three catches:

As the perpetual beta style of service-oriented computing moves into physical spaces, we’re all at the mercy of suppliers’ willingness to use us as guinnea pigs – only this time our homes and our clothes and our handbags are the venue of choice.3 Will the future of the IoT be dominated by botnets and fraud? Or will we use the new machines to help face up to the massive survival challenges that a finite world poses to an expansionist system? I think you’re part of the answer.

1.1.3 IoT: From the General to the Specific

As humanity’s latest pandemic swirls across the world, wreaking havoc on the poor, the old and the unlucky (and giving the powerful their latest excuse for moulding us all into yet more profitable shapes), some small comfort may be found in the contemplation of the intricate. The history of machines is part of what distinguishes our species: cooperating to create the preconditions of our existence using ever more sophisticated engines.

Computational engines have brought a new level of generality to the picture: Turing machines exemplified by von Neumann architectures and running on some millions or billions of transistors have become a universal mechanism for information processing and automatic control, and collectively we are deploying more and more compute power, memory and storage at a truly astonishing rate. The extreme expression of this tendency is cloud computing, to the degree that it is now hard to imagine a compute problem which would exceed the combined power of the modern cloud.

At the opposite extreme from the general purpose computer are those engines that are tailored (ever more precisely) to a specific purpose. These machines, in the limit, form minimal solutions to complex problems. There are few fields that exhibit this minimalism as purely as what is currently often called the Internet of Things4. As computation has permeated almost all corners of technology a particular class of problems has become prominent, where we seek not generality but to consume the smallest amount of resource that can possibly succeed. The use cases for a general purpose machine are always expanding and we can always, potentially at least, justify the devotion of more resource to their construction. The uses case for the Internet of Things are inextricably tied to objects whose sizes, costs and operational environments present a constant resource challenge, a constant downwards pressure on compute cycles and power consumption.

The solutions that we build in the IoT are beautiful in their intricate simplicity, minimal expressions of the vast power of our universal machines, and their “…perfection is attained not when there is nothing more to add, but when there is nothing more to remove”5.

Of course we don’t always achieve minimalist elegance on our first tries. Heath Robinson captured the feeling I have when I look at some of my previous attempts at IoT devices:

6

This course is about transforming our computational selves from high priests of the general purpose to paragons of minimalism (probably via the joyfully messy intermediary of Heath Robinson).

Enjoy the ride!

1.1.4 Hope

What if we wish not simply for the comfort offered by contemplating (and, later, controlling) the miraculous intricacies of task-specific computational engines, but also for hope?

Take a minute. Switch off your phone. Look at the objects around you, remember the food on your plate at your last meal, feel the warmth of your clothing, the security of the building you are sitting in. The economic system that creates all of these things, all of these preconditions for your existence, is driven by a single imperative that equates profit with value. When that equation is combined with the vast and impersonal forces of transnational corporate competition the results are well known. We use our atmosphere as a sink for the carbon remains of an aeon of decayed flora and fauna. We use each other as expendable “human resources.” We crowd sick animals together in zoonotic melting pots and express surprise at the apocalyptic consequences.

In this somewhat gloomy context there are nevertheless several glimmers of hope connected to the IoT… but allow me, dear reader, a little suspense in my narrative: we’ll come back to how to save the world after we’ve done some of the spade work. Watch this space :)

1.2 How the Course Works

The course has two sides, practical and theoretical. The practical work is split in two halves:

  1. up to week 7 we program the basic functions of an IoT device, step by step
  2. from week 8 we choose a project from a range of options and work on it intensively until week 12

(The project, and hardware kit, is yours to keep at the end of the course.)

We provide information via three main channels:

(There is also a small amount of information on the Blackboard page for the course, and later in the term there may be hands-on sessions available in the Diamond Electronics Lab (DIA 2.02) on Monday afternoons, though due to the pandemic these are restricted in number.)

There is a notes section for each week of the course8. Each chapter ends with a “week N course notes” section describing our objectives for this part, and specifiying practical work that we want you to complete. By and large you can read the first parts of the chapters in lean-back mode: they are discursive, and intend to give a flavour of the field and its historical and technical contexts. The latter parts of the chapters are more directive, and you’ll need to sit at a keyboard and follow instructions; the intent is to lead you to practical outcomes and running code.

There is example code in the exercises tree of the course materials repository. This code covers many of the tasks that we ask you to complete week by week. You can, of course, copy the answers without trying to come up with your own solution, but you will learn little by doing so. The recommended method is to implement your own answer to the exercises, then go to the relevant example code and compare it with yours.9

This is a 10 credit (level 3) module; this means about 100 hours work. It is taught over one semester, so it averages out at about 7-8 hours per week. (Depending on your existing skills and previous experience you may need more or less time.)

1.2.1 Main Changes from Iteration 3

We needed to change quite a lot from last time the course ran! The 2021 version relies on 3 Ts: Toys@home, lots of Text, and Tech support :)

1.2.2 Assessment

Threshold and grading works by separating the process of passing the course from the process of attaining a good grade. The threshold assessments check that you have a basic understanding of all the material; the grading assessment assesses the quality of that understanding in depth.

The course is assessed by two practicals (lab1, lab2) and a (multiple choice) exam.12 Lab1 happens at week 7 and lab2 at week 12 (and the exam during the standard exam period at the end of the semester). (A mock exam during term will allow you to practice the material.)

You must pass lab1 and the exam in order to pass the course. This will give you a score of 40%. In order to score more than 40% you must also complete lab2, the project.

Lab1 and the exam are assessed as pass/fail only (and no score is given for lab1). You will have a week to complete lab1 and 24 hours within which to sit the exam. A pass in both contributes 40 marks to your final result.

Lab2 is a project which students spend several weeks completing at the end of the course. The assessment is open-ended and qualitative in nature. It is marked out of 100 and contributes a maximum of 60 marks to your final score.

This is an advanced course and the project assessment in particular is intentionally open-ended. There is no single right answer! If you’re unsure about how much effort to expend the rule of thumb is to put in as many hours as is reasonable for a 10 credit module (see above; perhaps 10 hours per week).

We’ll give more detailed guidance in relation to each of the assessments as they occur; the things to remember to begin with are that:

1.3 COM3505 Week 00: Preliminaries

This section describes stuff to do before you start the course:

1. We will communicate with you using Git, both to deliver course materials and assignments to you, and for you to submit coursework to us. Therefore you are not fully enrolled for the course until you set up an appropriate git repository and give us the details. Do this ASAP!

2. Spend some time getting to know git and the other tools we’ll be using. (The Linux command line is powerful, elegant and, to newcomers, challenging. Using git from the command line adds another set of challenges. A bit of preparation will help!)

3. Refresh your memory of the Diamond Electronics lab safety protocols. You cannot participate in lab sessions without having completed the safety work.

Details follow.

1.3.1 Setting up your Git Repository

First register an account on GitLab.com if you don’t have one already. Use your Sheffield email address (ending in @sheffield.ac.uk) to register.

Second, create a private repository (project) there called com3505-student-2021:

(Make sure to select the Initialize repository with a README option.)

Third, go to the repository members settings…

…and add hamishcunningham

…as a project maintainer:

Lastly submit your GitLab user name to our server (see next section).

1.3.2 Tell us your Account User Name on GitLab

Your user name (or ID for our purposes) is the name you log in with, and appears in the URL of the site when you’re logged in. For example, Hamish’s user name / ID is hamishcunningham and his public GitLab projects appear at gitlab.com/hamishcunningham (and GitHub ones here , and SourceForge here…).

We need to know your username and match it up with your student record so that we can pull the coursework that you commit there. Tell us your username by making an HTTP GET request to the server that lives at https://HOSTNAME.shef.ac.uk:9194/ (where HOSTNAME is given on your MOLE/Blackboard page for the course). This server expects GET requests at the path com3505-2021 with parameters that include email and gitid. (Email addresses should end in @sheffield.ac.uk.)

Note that the server is only visible inside the University firewall so you’ll need to use the VPN if you’re working on a different network. Also, it uses a self-signed TLS certificate, so you’ll need to turn off certificate checking for this host if accessing from a browser. (The server is secure, but doesn’t embed a certificate from a known authority, so browsers give a warning.)

To send us your username, make a GET request something like the following (substituting MyGitUname for your gitlab account username and MyName for your email):

https://HOSTNAME.shef.ac.uk:9194/com3505-2021?gitid=MyGitUname&
  email=MyName@sheffield.ac.uk

You can do this via a web browser, or using wget or curl or any programming language or other tool that talks HTTP. If you feel like a challenge and you manage to figure out how do it with an ESP32 then:

  1. be careful not to flood the server with multiple requests!
  2. pat yourself on the back: you’re ahead of the game :)

For example, to use curl:

curl -k 'https://HOSTNAME.shef.ac.uk:9194/com3505-2021?gitid=MyGitUname&
  email=MyName@sheffield.ac.uk'

The response should look like this:

Received:  gitid=MyGitUname;  email=MyName@sheffield.ac.uk; 

Note: don’t send your numeric ID from GitLab, but the alphanumeric user name / path that appears in your project URLs.

(Later in the course we’ll use the same technique to gather data from the IoT devices you’ll be building, but that’s another story!)

1.3.3 Good Tools to Learn

We’ll be using most or all of the following tools in the course. It is a good idea to get to know them before starting (though you should be able to pick them up as we go along if needed).

In addition we will need a code editor or Integrated Development Environment (IDE) for writing IoT device firmware in C and C++, e.g.:

We’ll cover how to set up these tools next week. We’ll cover how to set up these tools next week.

1.3.4 STAYING SAFE in the Lab

The electronics lab in the Diamond (DIA 2.02) is a fantastic learning environment, and we give you as much time in there as we can during the course. It is also a potentially dangerous environment. Before coming to any lab session you must do the following:

Soldering is optional this year (but definitely a useful skill for IoT prototyping); if you want to do soldering in the lab, please also watch this tutorial video.

1.4 COM3505 Week 01 Course Notes

Have you done the preliminaries?? If not, now would be a good time :)

1.4.1 The Electronics Lab

Parts of the practical work for the course are best done in the electronics laboratory (DIA 2.02), but this laboratory is running with a reduced capacity to allow social distancing. That means each student will have at most two or three sessions in the lab during the 2021 course. You are also entitled to use the facilities in the iForge (the Diamond 1.01 project space) but again capacity is drastically reduced at present.

We have reduced the lab-based components of the work; please fit it in whenever you get lab access.

1.4.2 Learning Objectives

Our objectives this week are to:

1.4.3 Assignments

1.4.4 Working with your Git Repository

The assessment of your practical work on the course is delivered by checking into your git repository and pushing to the “origin remote” (on gitlab.com). If you don’t push anything, we won’t be able to give you feedback or to mark your work, so it is important to get used to this system early on.

Do this the first time you start work on a new machine:

check out your repository from gitlab into your own file space

Other useful commands:

Note: git (and GitLab / GitHub) have become industry standards in recent years, so it is important for you to get to know at least the basics. However, an independent survey of experts recently estimated that of 2,153 git command options fully 213% were either contradictory, confusing or error-inducing (or all three). [I may have made that bit up.] It is depressingly easy to get git into a mess! [I didn’t make that bit up.] What to do? Here’s one way to escape from git hell

Let’s say you have conflicts in your com3505-student-2021 repository and the process of resolving them is proving difficult. To re-create a clean version of the repository (in your home directory):

cd
mv com3505-student-2021 saved-repo
git clone https://gitlab.com/YourGitLabUname/com3505-student-2021.git

You’ve now got a fresh copy to work with; if you have changes in the saved version you can copy them over to the new, then commit and push from there.

Yes, it was called ‘git’ for a reason…15

1.4.5 First Lab Checklist

(There may be no of few lab sessions in 2021. Don’t panic! We will post out kits to you and help you work with them on-line.)

This section describes what to do when you get your first lab session:

If you finish early you might also want to learn about measuring simple values (using the multimeters and/or oscilloscope), but this is optional.

The hardware kits give you an ESP32 net-connected microcontroller, sensors and actuators, and the means to prototype experimental circuits: the foundations of IoT hardware.

For reference each kit should contain:

1.5 Further Reading

2 Definitions, and a Burning Question

Low cost networked computers are adding eyes and ears (or sensors) and arms, legs and voices (or actuators) to the Internet. These new devices are being connected to on-line brains (big data and deep learning in the cloud). This new field is the IoT, of course. Will the result be a ‘world robot?’ (Schneier 2017) This chapter will start to cover some of the context and history of the IoT. Chapters that follow will cover the hardware that makes it possible, the communications protocols and security systems it relies on, and the cloud-side analytics that make sense of the data it produces.

The practical work this week is to getting to know the SDKs and IDEs17 that allow us to develop software (or firmware) for the net-connected microcontrollers that are the foundations of IoT devices.

We’ll begin with a couple of definitions, then look at the genesis of the software ecosystem that we’ll be using to program our IoT devices, before moving on to the practical question of development tools.

2.1 Defining the IoT

There are many definitions of the Internet of Things (IoT); one of the most exciting is given by Bruce Schneier in his provocative piece Click Here to Kill Everyone:

…the Internet of Things has three parts. There are the sensors that collect data about us and our environment… Then there are the “smarts” that figure out what the data means and what to do about it… And finally, there are the actuators that affect our environment. … You can think of the sensors as the eyes and ears of the internet. You can think of the actuators as the hands and feet of the internet. And you can think of the stuff in the middle as the brain. We are building an internet that senses, thinks, and acts. This is the classic definition of a robot. We’re building a world-size robot, and we don’t even realize it. (Schneier 2017)

At the other extreme the IoT is about what becomes possible when networked microcontrollers become cheap enough to embed in very many everyday contexts, from central heating thermostats to garage doors. These devices face tight constraints of power usage and cost, and concomitant challenges to their security and functionality. They also quickly become extremely numerous, driving work on big data analytics and cloud computing.

The technology of networked devices goes back perhaps 50 years or more. The coining of the term itself is often credited to Kevin Ashton in 1999, while working at the Auto ID Center at MIT (Ashton 2011), of which more later.

2.2 Revolutionary Code: from MIT Printers to the Arduino

If we understand the past we stand a better chance of seeing into the future. Where did the IoT come from? Where did the systems we’re going to be coding come from? Later on we’ll learn about an embedded electronics ecosystem that brought the needs of Italian artists together with the origins of the operating system that drives more of the internet than any other and the compiler suite that has been ported to more architectures than any other and spawned the Arduino.

But first: do you own a phone?

You are very probably carrying a phone, but do you own it?! Call me an old stickler, but I think if I own something then:

Do those things apply to your phone? How about your laptop? Your tablet? In the 2020s we often buy electronics which the people who sell them claim we own, but if we take the trouble to read the licencing documents that accompany them we often find that we have few rights over them, and that repair, for example, is expensive or difficult or voids the manufacturer’s warranty. Much of the electronic and computational ecosystems we’ll spend most of our time with in this course arose from a similar realisation some 50 years ago.

2.2.1 Return with me to Boston in the 1970s…

Back when I was a wee small snotty thing, and many of you were only minute folds in the quantum potential of possible future universes, the PDP 10 was a cool computer:

18

It was the first machine to make time sharing (multitasking) common, and it had a huge maximum memory of… a whole megabyte!19

The operating system code (assembler) on the PDP 10 and its immediate cousins was routinely shared and improved by a community of programmers (or “hackers”20) at MIT, including one Richard Stallman.

2.2.2 Whaddya Mean, I can’t Fix It?!

As computing companies became bigger and more profitable in the 1980s practices began to change: there started to be no more access to source code, and users would have to sign an NDA21 even to get access to a binary. Stallman suffered the consequences of an aversion to NDAs when he was refused access to the source for a printer control program, even though his intention was to improve that program. He says: ’This meant that the first step in using a computer was to promise not to help your neighbor. A cooperating community was forbidden. The rule made by the owners of proprietary software was, “If you share with your neighbor, you are a pirate. If you want any changes, beg us to make them.”" [Ibid.]

2.2.3 What To Do?

’So I looked for a way that a programmer could do something for the good. I asked myself, was there a program or programs that I could write, so as to make a community possible once again?

The answer was clear: what was needed first was an operating system. That is the crucial software for starting to use a computer. With an operating system, you can do many things; without one, you cannot run the computer at all. With a free operating system, we could again have a community of cooperating hackers—and invite anyone to join. And anyone would be able to use a computer without starting out by conspiring to deprive his or her friends.’ (Ibid.)

And this is what lead to the kernel code that runs your Android phone, the GNU/Linux operating system that runs the majority of servers in the cloud, and the compiler code you’ll use to create firmware to run on the ESP32 in this course…

Why does this matter?

And, perhaps most important of all, we have been describing the genesis of the whole free and open source software movement, which has been responsible for huge portions of the code running our world, and which in combination probably constitute the most complex machine ever constructed. Where will it go? Where will humanity go, locked into a triple crisis of pandemic, environment and economy? The two questions are (perhaps uncomfortably) tightly linked.

Ok, enough context for now. We’ll come back to IoT history next week; now for something practical :)

2.3 Coding Support Tools: IDEs, SDKs, Libraries

When we write code to run on a microprocessor (e.g. on your laptop, phone, or that Cray you have in the garden shed) we use all the facilities of the operating system, a modern programming language and its library ecosystem to insulate ourselves from the underlying hardware and to provide a set of abstractions, APIs and components that can make us productive at high speed. In recent times we have started adding on-line APIs the mix, making our potential deployments of computational resource truly vast.

On a microcontroller like the ESP32 in your kits, we have a lot less support: programming is typically with low level languages (C, C++, occasionally bits of assembler), the ‘operating system’ is more like a task management library, and the development tools are often cranky and basic. Firmware usually has to be uploaded to the device over a serial line or JTAG connection, and the tooling to perform this task is different for every hardware family.

In some cases a new world of Javascript or Python programming and drag-and-drop upload (e.g. over USB OTG) is starting to become available for the IoT, but:

  1. this is not yet widespread
  2. it still uses the manufacturer’s C libraries under the hood

The second of these points in particular means that projects that need to use the hardware efficiently and to exploit all of the available facilities will most often write in C/C++.

This section begins by looking at the basic (toolchain, SDK and library) building blocks of development on the ESP32, then goes on to describe available development environments. Later on (section 2.4.3) will talk you through installing your own development environment and taking your first steps in programming an IoT device.

2.3.1 Toolchains: In the Beginning, There Was The C Compiler…

In any novel embedded system the foundational step is to port a C compiler to the new chip. The compiler, linker and the tools that take a binary image and burn it into the memory space of the device (in a form that can then be sucked up by a bootloader and turned into a running process) are called collectively the toolchain. The compiler is generally an unusual beast, as it has to run on one platform (your development machine, x86_64 perhaps) but produce binaries for a different architecture (Xtensa being the one on the ESP32 in your parts kit); i.e. we need a cross compiler. So when installing a development environment for the ESP32 we need to pick up the Xtensa port of GCC (the GNU Compiler Collection, aka GNU C Compiler).

There’s good news and bad news about this step:

There are snakes in the long grass, wear your wellingtons.

2.3.2 ESP-IDF, FreeRTOS and the ESP32 Arduino Core

A compiler, when ported to an IoT microcontroller, will open up the magical gates of productivity and allow you to do… not very much at all! The differentiators between the (many) different chips competing for IoT oxygen relate to their hardware facilities, and to access these we almost always need more than is provided by the standard C library. (On the smaller, more resource-constrained devices, the standard library may not even be present due to lack of space; in the case of the C++ library this is actually quite likely.) In order to sell chips, then, the microcontroller manufacturer needs to supply a set of libraries that expose the hardware functionality of its wares in as friendly and powerful way as possible.

2.3.2.1 ESP-IDF

Espressif, maker of the ESP32, provides an SDK and library set called ESP-IDF, or Espressif IoT Development Framework, whose development is hosted here on GitHub.

ESP-IDF has grown to be a large and successful open source project that incorporates ports of many other libraries to ESP hardware. For example, MbedTLS (source code) is a very popular small footprint cryptographics library (supporting certificate-based security and the TLS protocol that underlies security on the web via HTTPS). Espressif support a port of MbedTLS (documented here, source here) that exploits the ESP32’s hardware acceleration facilities for tasks like random number generation or hashing. This hides the peculiarities of the ESP’s underlying hardware from the programmer, who can instead use familiar and well-documented abstractions as surfaced by the MbedTLS library.

Other libraries that are supported for the ESP32 via ESP-IDF include:

In each case IDF adapts the library to particularities of the underlying hardware, making it easier for programmers to exploit the chip to its full potential without learning about the grungey details of which register does what under what conditions.

Whenever we write code for the ESP32 we will be using ESP-IDF APIs, either directly or indirectly. To begin with, the easiest way is to use them indirectly, letting the Arduino compatibility layer take the strain; see below.

As of early 2021 there are three main versions of IDF that we will come across:

From version 4 IDF changed its principal build system from GNU Make to CMake, and this has made supporting both versions quite complex. There are three scripts provided by the distribution that set up the environment needed for the build systems to work:

Each of these has changed and evolved over different versions of IDF, and they present a difficult moving target to IDEs and automation scripts like Eclipse, PlatformIO and VSCode. (It is also a rapidly changing system integrating many 3rd party libraries.) This means that it can be quite challenging to get a 4.x IDF setup working with the Arduino layer, so to begin with using 3.3.x is probably the best option. (In Feb 2021 it now looks much easier to do 4.2 with PlatformIO; see below.)

An additional complexity is that IDF provides a configuration tool, menuconfig (based on the KConfig language), originally developed for the Linux Kernel. This exposes many of the optional features of IDF and of the libraries it incorporates. To use it you have to be doing a native IDF build of one type or another (which means you can’t easily use it in the Arduino IDE).

2.3.2.2 FreeRTOS

Another important facility that we can access via the Espressif SDK is FreeRTOS, an open source real time ‘operating system’. FreeRTOS provides an abstraction for task initialisation, spawning and management, including timer interrupts and the ability to achieve a form of multiprocessing using priority-based scheduling. The ESP32 is a dual core chip (although the memory space is shared across both cores, i.e. it provides symmetric multiprocessing, or SMP). FreeRTOS allows us to exploit situations where several tasks can run simultaneously, or where several tasks can be interleaved on a single core to emulate multithreading.

This doesn’t necessarily make life simpler! Most microcontroller code, at least in the hobbyist and maker spaces, is written to target single core chips, and the potential for shooting yourself in the foot rises very quickly when trying to adapt to a world where multiple things may be happening (or at least seeming to happen) all at once. Who owns this piece of memory? What will happen when two tasks both try and talk over this bus? Why does my code behave differently if I pause a task for a few microseconds? To begin with at least, it may be easier to limit yourself to a single task that runs an infinite loop, and worry about FreeRTOS later on. (This is what happens, in fact, when we delegate our main program entry point to the Arduino “core” for ESP32: the setup and loop procedures which characterise Arduino programming are implemented for us using a FreeRTOS task, but we don’t have to worry about the details.)

Later on, interrupts, tasks, event queues, mutexes and semaphores will all become objects of interest, and if you’ve a mind to dive into the mysteries of FreeRTOS that’s where you’ll find them. Enjoy!

2.3.2.3 The Arduino Core for ESP32

Fairly early in the lifetime of the ESP32’s predecessor chip, the ESP826625, a Bulgarian developer going by the somewhat cryptic moniker of me-no-dev decided that what was needed was integration with the Arduino ecosystem. He set out to develop a compatibility layer between Arduino libraries and development tools and the ESP8266. The work rapidly captured the imagination of a large number of developers, and has become a mainstay of the ESP community: nowadays there are commits from more than 500 people in the github repos, and perhaps 10,000 commits overall. Espressif could see that this was a good thing: they hired me-no-dev and he now works full time on the “Arduino core” for the ESP32. The project can be found here.

What’s an Arduino core when it’s at home? I’m not sure where the term originated, but it basically means all the code that interfaces a device and its toolchain, compiler settings, and libraries to a) other Arduino library code and b) the Arduino IDE. This piece of kit is incredibly useful, because the Arduino ecosystem is massive. Pick up any electronic component (sensor, actuator, dog brush or kitchen sink) and the likelihood is that someone has published an Arduino-compatible library for it. This makes programming from the Arduino APIs hugely more productive than otherwise, because we’re almost always working from widely used running code. Win win win!

Motherhood, apple pie, happy ever after. Except: VERSION HELL!!!

2.3.2.4 Version Hell!!!

When we program the ESP32 we’re typically using C code from ESP-IDF, C code from the libraries that have been ported to the ESP chip family and included as part of ESP-IDF, C++ code that translates between ESP-IDF and the Arduino APIs, and C++ code that exposes sensor and actuator hardware within the Arduino ecosystem. (All of these are in active development, and all have their own release cycles.) We are then compiling this morass with a toolchain that has its own version trail, controlled by a build system that comes in a whole bunch of different and subtly incompatible flavours and, finally, uploading it to the IoT device using a Python script written in … erm, Python (possibly one of the least stable programming languages ever devised26).

Hmmm.

The types of things that tend to go wrong are:

So: if things go wrong, don’t worry. You’re not alone. Section 2.4.3 tries to map out a path of relatively easy and reliable options. But first, let’s round of this discussion with a look at the available build system CLIs and IDEs.

2.3.3 Developer Tools: CLIs and IDEs

The game, lest we forget amidst all this gratuitous verbiage, is to (cross-)compile our code against 1001 IoT and embedded systems libraries and burn the resultant firmware images to an ESP32. Along the way it might be nice to throw in a bit of:

We have two main families of options for doing (some subset of) these things:

Examples of IDEs include:

Examples of CLI build systems include:

I was talking to someone recently who works at one of the big semiconductor companies (the one that designed the chip in your phone) about the difference between microcontrollers and microprocessors and remembering all the weird challenges that the former tend to throw at you. “Welcome to embedded,” she smiled, where getting your toolchain installed can be a week’s work :)

The rest of this section will describe two of the most straightfoward options for getting started, and also provide a few notes on two options for using VSCode.

Note: you can try out the recipes here to get started on programming the device, or just read them and then follow the instructions in sec. 2.4 below.

2.3.3.1 Using the Arduino IDE (ArdIDE)

One of the simplest and most reliable ways of programming the ESP32 is using the Arduino IDE. This IDE is a little like me: quite robust and reliable, but a little antiquated. It is a good place to get started, but possibly not where you want to remain for the whole course. Your choice though: if you like it, go with it. 

To pick up the compiler toolchain and other specifics we need to install an IDE plugin known as a core. The easiest way to install the IDE and the ESP32 core is to follow the instructions on the core github site. You can choose the development release version; install core version 1.0.5 (rc4 or above), which should give you IDF release 3.3.4.33

For example:

Congratulations, you have successfully burnt your first ESP32 firmware!

Another useful example to try at this point is GetChipID:

If you got that lot to work, you have:

Finally, a note on versions and restrictions: as discussed in section 2.3.2.1 the rapid pace of change in IDF has meant that the Arduino core lags behind a little, and the instructions above will result in an installation based on IDF 3.3.4. Using the board manager installation method also means that the configuration utility for IDF (menuconfig) can’t be used (because the Arduino layer installs a pre-compiled version of IDF).

Probably time for a brew!

2.3.3.2 CLI on a Raspberry Pi

This section describes setting up an ESP-IDF CLI build. The instructions are for a Raspberry Pi running the Debian Buster port (which used to be known as Raspbian but now seems to have mutated into RaspiOS34). The instructions should work pretty much unchanged on Ubuntu 20.04. (If you are on other platforms you’ll need to adapt the instructions or use a VM or docker – see next section.)

If you need a cheap machine to develop for the IoT, Raspberry Pi models 4 or 400 are both capable of doing the job. They won’t be the fastest environment, but for around £/$/€100 you can get a capable computer that will plug into your TV or HDMI monitor and do everything you need to for the course. (A good choice would be this kit, for example, or this one.)

Below I describe how to get up and running with ESP32 development on the Pi. Note that the setup process took several hours (on a fast network), so don’t leave it until 10 minutes before your deadline :) (The complete setup uses around 2.5GB of the flash disk, depending on configuration.)

Here’s some more timing information, for install and build (using the firmware and scripts in the HelloWorld example, which will be introduced later):

Note that although setup and clean compiles are slow on the Pi, the compile/burn/test cycle is reasonably quick at under 30 seconds. This means that you can work productively on a Raspberry Pi, so long as you allow an hour for the initial setup. (This is from the command line, of course; running a big IDE like VS Code or Eclipse would no doubt be slower!)

To set up the operating system:

To get the ESP32 development environments running:

(Note: some things don’t work in the old versions, but they’re more stable. The version that the Arduino IDE install route detailed above pulls down is IDF 3.3.4 / Arduino core 1.0.5, and this currently is the most reliable combination.)

For example, using a pre-release IDF 4.3 in December 2020:

To try burning an example:

Well done! Please give yourself a pat on the back. If you’re feeling brave: VSCode for the Pi is available here :)

2.3.3.3 CLI Using Docker

If you don’t have an Ubuntu or RasbiOS environment available, you can try using docker (although you won’t be able to burn firmware directly unless you’re on another Linux platform).

If you’ve cloned the course repo (the-internet-of-things) and cd’d into it, and connected an ESP32 to USB, you should now be able to do the following:

(Command-line syntax for docker on non-Linux platforms may also be a little different; see the docker docs.)

If you’re on Linux, and you have the correct device specified (/dev/ttyUSB0, for example), then you should also be able to burn the firmware, e.g. by just doing ./magic.sh. This won’t work on MacOS or Windoze because no easy way to map the serial port driver model currenly exists In this case you’ll need to copy the the-internet-of-things/exercises/HelloWorld/build/HelloWorld.bin file out of your container and burn the firmware using a local install of ESP-IDF (which somewhat defeats the object!). If you want to try this, the command that is needed to burn the firmware can be copied from the output of the docker build, e.g.:

...
Generated /home/ubuntu/the-internet-of-things/exercises/HelloWorld/build/HelloWorld.bin
[1081/1082] cd /root/esp/esp-idf/components/esptool_py && /usr/bin/cmake -D IDF_PATH="/root/esp/esp-idf" -D ESPTOOLPY="/root/.espressif/python_en...D WORKING_DIRECTORY="/home/ubuntu/the-internet-of-things/exercises/HelloWorld/build" -P /root/esp/esp-idf/components/esptool_py/run_esptool.cmake
esptool.py --chip esp32 -p /dev/ttyUSB0 -b 921600 --before=default_reset --after=hard_reset write_flash --flash_mode dio --flash_freq 40m --flash_size 4MB 0x8000 partition_table/partition-table.bin 0x16000 ota_data_initial.bin 0x1000 bootloader/bootloader.bin 0x20000 HelloWorld.bin
esptool.py v3.0-dev
Serial port /dev/ttyUSB0
Connecting......

2.3.3.4 VSCode IDF Extension

A new kid on the block for ESP32 development support is the ESP-IDF VSCode plugin: see the installation guide for details. The plugin is still pretty new, and the process of getting it to work with a clone of both the IDF and the Arduino core is pretty involved. When it works, though, it looks like a nice environment to use :)

I got it to work like this (on Ubuntu 20.04):

See also: Quick User Guide for the ESP-IDF VS Code Extension.

2.3.3.5 Using VSCode with the Arduino Extension36

This section is about using VSCode in a more basic, but still useful mode.

Some more tweaks may need to be done for full IntelliSense compatibility, so you may need to hide squigglies until we figure out the additional paths you need to add. You can still use these instructions to do things like uploading your code to the ESP32, though.

First install VSCode itself; on Ubuntu you can do that like this on recent versions:

snap install code --classic

You can then run the beast from the command line with code, or from the launcher by vs....

Now install the C++ and Arduino extensions:

E.g.:

"arduino.path": "/.../arduino-PR-beta1.9-BUILD-115",
"C_Cpp.default.includePath": ["/.../arduino-PR-beta1.9-BUILD-115/libraries/"]

Make sure your JSON is still valid, and save it. You might also need to close and reopen Visual Studio Code for your include path to be recognised.

Doing all of the above will give you access to many of the Arduino IDE’s functions from within the Command Palette (usualy Ctrl+Shift+P) within VSCode under the same names.

2.3.4 FAQ

Ok, that’s the end of Chapter 2’s general material. The rest of the chapter gives specific tasks for Week 2.

2.4 COM3505 Week 02 Notes

2.4.1 Learning Objectives

Our objectives this week are to:

2.4.2 Assignments, Set Up, Exercises 1 & 2 (Ex01, Ex02)

Notes:

Exercises:

As noted in chapter 1’s “how the course works”, there are example solutions in the course materials tree. Have a go at doing your own version before looking at these!

2.4.3 Set up your Programming Environment

As discussed above there are lots of options, and lots of potential pitfalls, so give yourself plenty of time to do this task, and don’t be surprised if you need to come back to it multiple times to refine your toolset.

I recommend that you start with the Arduino IDE, then a command-line (CLI) build, e.g. pio run using PlatformIO. Once you have these working, if you wish you might then try one of the VSCode options. The ambitious amongst you might try Eclipse (if you’ve used it before and liked it), or Arduino Create, or PlatformIO/VSCode. The important thing is just to get one environment running (and if you’re happy with it feel free to stop there!).

Follow these steps:

2.4.3.1 Using PlatformIO and the Firmware Template

From Feb 10th 2021 the PlatformIO Espressif 32 development platform now supports ESP IDF v4.2 and a recent Arduino core (“2.0.0”). To run this with the firmware template (in HelloWorld or Thing) do this:

2.4.3.2 Using magic.sh and the Firmware Template

NOTE: these examples are designed for Ubuntu 20.04 and recent versions of RaspiOS. You’ll need a VM or emulation or etc. if you want to try them on other platforms.40

In our the-internet-of-things course materials git repository you can find a tree called exercises/HelloWorld. This contains example firmware code (functionally similar to the Blink example supplied with the Arduino IDE) and a script called magic.sh that (if it works) could simplify your install and build work.

For example, to install and setup IDF and try a CLI build you can do this:

# after cloning:
cd the-internet-of-things/exercises/HelloWorld

# get a list of options:
./magic.sh -h

# clone and check out IDF and the Arduino core: 
./magic.sh setup

# attempt an IDF build, flash and monitor:
./magic.sh

There’s also some support for the Arduino IDE:

# download and install:
./magic.sh setup-arduino-ide

# run:
./magic.sh arduino-ide

If you want to access other idf.py commands (see the IDF documentation for details), try

./magic.sh idf-py your-favourite-command

To take a copy of one of the examples, use the copy-me-to command from inside the example. E.g. if you have your gitlab repo checked out in your home directory, this would create a new tree there:

./magic.sh copy-me-to ~/com3505-student-2021/MyNewExample

For more details take a peak at the README or magic.sh.

2.4.3.3 Using the University Machines

You can (just about) do the programming for the course on both Windows and Linux on the University desktop machines. (The latter is rather out-of-date and slow, unfortunately, and although there is a development process running to improve the Linux installation it looks unlikely to bear fruit in time for the 2021 course.)

On both platforms you’ll probably need to use only the Arduino IDE; VSCode is also present but installing plugins for PlatformIO or ESP-IDF hasn’t worked so far.

To use University Linux, boot the PC (e.g. in DIA 2.02) into “Diskless Linux” (LTSP); more details and available locations here. ArdIDE should work, though slowly.

2.4.4 Hardware 2: Sensor/Actuator Board

We are using a breadboard (also known as proto-board) to assemble our circuits in the labs. Breadboards allow components and wires to be pushed into holes and connected without soldering. (In Hardware 1 you use one to experiment with voltages, resistors, and measuring a circuit’s performance with a multimeter. These are important skills for IoT device prototyping, and will be useful for your project work later in the term.)

To recap, a breadboard has some connections between holes already made – shown in this diagram:

Diagram of breadboard connections41

2.4.4.1 Using a Breadboard to Make a Sensor/Actuator Circuit

Take your ESP32 feather device and carefully lay it in place on the breadboard over the holes indicated:

Feather in breadboard

Now push gently on the edges of the feather to insert the pins into the holes – you are aiming to keep the device parallel to the breadboard as it goes into the holes, rather than push one side down and leave the other up. Keep moving around pushing in different places if it seems stuck – often new breadboards are a bit stiff at first. If the pins aren’t going in easily then check that they are all lined up above the holes correctly – if not you can bend them gently with a pair of fine pliers. You will need to use some force to push the device down onto the breadboard. To get an idea of how much force might be needed, take a jumper wire and try inserting that into a hole, then multiply that by 28. Try not to bend the header pins – if you do then use the pliers to return them to their correct orientation.

Once the feather device is inserted, add a LED, 120Ω (or thereabouts) resistor and push-button switch from your kit as shown:

Blinky components in breadboard

Pay attention to the orientation of the LED – as it is a (light emitting) diode it will only work one way round. The longer lead of the LED is the anode and it connects to the ESP32 output pin – the shorter lead is the cathode and it connects to the negative or ground connection. Add jumper leads to complete the circuit as shown – colour codes help to make the circuit more readable:

Blinky components with connections in breadboard

The circuit pictured above can be represented as a schematic – this is a more abstract representation of the components and connections:

Blinky schematic

The trick here is that the LED is connected to pin 32 and the switch to pin 14, so we might usefully define these in our C code as int pushButton = 14, externalLED = 32;.

The code for this sketch should make use of the internal pull-ups inside the ESP32. Simply pass the “input pullup” macro to the pinMode command: pinMode(pushButton, INPUT_PULLUP) These are optional resistors that connect to +v and the input pin. These make the inputs high, unless they are connected to ground. See this diagram:

Diagram of internal pull-up

Below are some pictures of an example board. Note that this one is using the ESP32 with “stacking headers” (additional sockets on top of the board) so we can use both the breadboard sockets or the stacking sockets to connect to.

You’re now ready to work with the LED and switch in firmware to answer Exercise 2 as above.

2.5 Further Reading

3 History; Blinking Things; WiFi

Where have we got to? By now you should have:

In this chapter we do two things:

There’s lots to do: crack on!

3.1 The Multiple Personalities of the Arduino Project

When we teach computing to beginners, we teach how to build something from the ground up. A new programming language: hello world. A machine learning method: the pseudo code for its algorithmic expression. This is necessary and valuable, but it hides a crucial fact about software development in the 2020s: we almost never build anything significant without starting from the work of tens of thousands of other people. These people form communities, and communities adopt and evolve tooling and workflows to create ecosystems. Choosing which ecosystems from the work of our predecessors we try to hitch a ride on is often one of the most influential decisions in any development project.

I think it fairly safe to say that few people would have predicted, around the turn of the millenium, that one of the most significant advances in embedded electronics and physical computing would be driven by “the development of contemporary art and design as it relates to the contexts of electronic media” (Barragán 2004). Starting in the early naughties, the Arduino project (Monk 2013; Banzi and Shiloh 2014b; Arduino 2017) had by the mid tennies come to provide the standard toolkit for experimentation in low power circuits. Today the ecosystem that resulted is first port of call for code to drive the sensors and actuators that give the IoT its interface to the physical world. And as we saw in the previous chapter, the Arduino IDE also provides us with one of the easiest ways to get started programming the ESP32.

We’ll start this chapter by looking in a little more detail at the various contributions that the project has made, and have a bit of a sniff around its development environment and the type of (C++) code it supports.

The Arduino Project

Arduino can refer to any or all of:

Arduino C++ refers to preprocessing, macros and libraries that are available via the IDE. (C++ is a medium-level language layered on the C (low level) systems programming language.) Over the last decade or so there has been a quite massive community of open source developers contributing to the ecosystem of code and documentation and forum posts and github repos and etc., which has made Arduino C++ a very productive environment for IoT development.

Remember that the ESP32 is not an Arduino (or even an AVR-based device), but there is a compatibility layer that interfaces it to the Arduino IDE. This makes lots of code for sensors and actuators and etc. magically available for the ESP.

An Arduino board

Having started out as a service project for arts and design students in Ivrea, Italy, in the early 2000s (who were using microcontrollers to create interactive exhibits) it brought cheaper hardware than the alternatives (based on Atmel’s AVR chips), and added an IDE derived from work on Processing and Wiring. Some millions (or possibly 10s of millions) of official and unofficial boards are now in existence, and it was for a long time the most popular platform for embedded electronics experimentation.

The Arduino IDE is a Java Swing desktop app that preprocesses your code (or “sketches”). Arduino’s programming language is C++ with lots of added libraries and a bit of code rewriting. It runs the GNU C++ toolchain (gcc) as a cross-compiler to convert the Arduino C++ dialect into executable binaries. The IDE includes a compilation handler that converts sketch firmware .ino files into a conventional .cpp file (poke around in /tmp or equivalent to see what the results look like).

Binaries are then uploaded (“burned”) to the board using various other tools (in our case a Python tool from Espressif). The IDE then allows monitoring of serial comms from the board, and provides access to libraries and example code. If you ask it nicely it will plot graphs for you, e.g. of the output of a microphone:

Plotting sound waves

As we noted in chapter 2 the IDE is pretty basic, and you may well want to move on to more sophisticated tools later on. It is well worth getting to know it to start with, however, as it is typically the fastest way to get started with new hardware, and the fastest way to find a working example from which to develop new functionality.

3.2 A Crossover Point

In sec. 2.1 we saw two ways very different to define the IoT: as a nascent world robot, or as the simple consequence of increases in computational power and network connectivity in microcontroller hardware. This section gives a bit more context to the origins of the field on the one hand, and the hardware space of IoT devices on the other.

When we look back at the antecedents of the IoT, it becomes clear that the field represents a crossover point between many earlier (and ongoing) areas of computer science and software engineering. Related and predecessor fields include:

If you understand roughly what each of these terms means then you have a good basis for understanding what the IoT means (and of being able to distinguish the marketing speak from the actuality of possible technology options).

3.2.1 The Early History of the IoT42

The term Internet of Things was coined by Kevin Ashton in 1999 for a presentation at Procter and Gamble. Evolving from his promotion of RFID tags to manage corporate supply chains, Ashton’s vision of the Internet of Things is refreshingly simple: “If we had computers that knew everything there was to know about things — using data they gathered without any help from us — we would be able to track and count everything, and greatly reduce waste, loss and cost. We would know when things needed replacing, repairing or recalling, and whether they were fresh or past their best.” (Ashton 2011)

Of course, devices had been ‘on the internet’ for several years before this, from at least 1982 in the case of a drink vending machine at Carnegie Mellon University in Pittsburg (Machine, n.d.). Using a serial port converter, the status lights of the front panel of the machine were connected to the Computer Science departmental computer, a DEC PDP-10 (for which today’s equivalent cost would be around $2 million!). The Unix finger utility was modified to allow it to report the level of coke bottles and whether they were properly chilled. Internet users anywhere could type “finger coke@cmua” and read the status of the machine. (It is notable that the world’s first IoT device was enabled by openly available Unix source code.)

A camera pointed at a coffee-pot in Cambridge’s computer science department was video-streamed on the internet from 1991, and when control from the web to the camera was established in 1993 the first webcam was born (Fraser 1995). Presaging very contemporary anxieties, a toaster had been connected to the internet in 1990 at the INTEROP exhibition (Romkey 2017), and nearly caused a strike as preparing food was an activity allocated to unionised labour. However it wasn’t until 2001 that a toaster became an IoT device in a modern sense, able to dynamically query a webservice for the current weather forecast, and then burn the appropriate pictogram onto a piece of toast (Ward 2001).

So we can see that from the earliest days of IoT (when it was often called pervasive or ubiquitous computing) our current concerns around open source, security (Denning and Denning 1977) and human obsolescence were already recognised.

3.2.2 The Current State of IoT Hardware43

“It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity…” (Dickens 1877)

We do indeed live in an epoch of belief and incredulity, with feverish hype of IoT widespread across the electronics and computing industries. Want a £400 wifi connected juicer that locks you into proprietary ingredient pouches anyone? Meanwhile ‘policy bloggers’ burble excitedly about techno-utopias such as smart cities that eliminate traffic — “Imagine a city with no traffic at all” (O’Muircheartaigh 2013). Security researchers are desperately trying to warn us of the dangers of letting our personal data leak out from our devices (Sarthak Grover and Feamster 2016). After FitBit users’ personal exercise data was exposed publicly (Loftus 2011) the company solemnly announced “We have also updated our default settings for new users for activity sharing to ‘private.’”

Current hardware is very diverse, with major companies such as Intel, Texas Instruments, Broadcom etc. competing to have their chips chosen by manufacturers. However the arrival of Espressif’s ESP8266 (and now the ESP32 (Community 2017; Kolban 2017) has shaken up the rest of industry by charging $2 per chip instead of $20. This has attracted a lot of attention and stimulated the creation of community-driven knowledge, documentation, libraries and tools; so much so that it is significantly quicker and easier to develop for this platform than most others.

IoT hardware can be classified according to its connectivity technology; currently the useful ones are ethernet, wifi, bluetooth, zigbee, z-wave and cellular. Whilst wired connections are still relevant for some applications, it seems that most recent developments have concentrated on wireless devices. If we are to buy hundreds of IoT devices each in the next few years, we certainly won’t be plugging them all into ethernet cables. Cellular technology remains stubbornly expensive both to buy and to run; fine for the now life-critical mobile phone, but not for those hundreds of devices.

Of the remaining mainstream wireless technologies, bluetooth’s USPs are it’s ultra low-power short-range attributes and that every phone has it. For devices with small batteries such as fitness trackers, this allows them to last a few days between charges. Wifi has major issues with power use and connection negotiation speed but it has emerged as a major IoT connectivity choice because of it’s ubiquity. Then there are the Z’s – Z-wave is proprietary but popular with blue chips like Honeywell and GE, Zigbee is an open standard also popular with big corporations – the Phillips Hue light bulbs use it as does the Nest thermostat.

Several ‘hubs’ have been launched by manufacturers such as Google, Amazon and Samsung that aim to bring all these devices together under one control, rather than having to manage dozens apps of physical remotes. For example, Samsung’s SmartThings hub has 4 wireless radios covering z-wave, zigbee, bluetooth and wifi plus an ethernet port.

3.3 COM3505 Week 03 Notes

3.3.1 Learning Objectives

Internet? What internet?! Up to now we’ve been programming the ESP as a standalone device. In the next period we’ll create and connect to networks, and figure out how to configure connections for devices without UIs.

Our objectives this week are to:

Practical work will include:

(There are a lot of exercises this week; you’ll benefit from completing all of them, but if you don’t have time don’t worry, just make sure to study the model solutions in exercises/Thing.)

3.3.2 Assignments

Exercises:

If you get the access point to work and then join it (e.g. from a phone or laptop), when you load http://192.168.4.1 in a browser, you should see something like this:

Now you’ve made a thing which is (almost) on the internet :)  Do a little jig, dance around the room, make a cup of tea.

3.3.3 Notes on the Model Code from Week 2

(You’ll find model versions of the exercises in exercises/Thing.)

3.3.3.1 Recap: Connecting to the ESP32

The ESP32 board we are using has a micro-usb socket to provide power and also allow communications between the microcontroller and your computer. Start by connecting the two together using the supplied cable. Start the Arduino IDE (e.g. by magic.sh arduino-ide). Ensure that the Tools>Board selected is the “Adafruit ESP32 Feather.” If you’re running the IDE locally (e.g. on your own machine or on the LTSP desktops) look in the Tools>Port menu to check that the serial connection has been established.

3.3.3.2 Various Arduino Functions

Note: in the Arduino IDE certain words such as OUTPUT, HIGH, TRUE etc. are pre-defined and shown in blue. Similarly functions such as pinMode, Serial.begin are coloured in orange – this can help you catch syntax errors.

Serial.begin(115200);           // initialise the serial line

Serial communication (sending or receiving text characters one by one) has to be initiated with a call to the begin function before it can be used. The serial communications between the ESP32 and computer can operate at various speeds or baud rates – we use 115200 baud. If you aren’t getting any response, or gibberish characters on the serial port monitor, then check you’ve got the correct speed set.

pinMode(BUILTIN_LED, OUTPUT);   // set up GPIO pin for built-in LED

pinMode is an Arduino command that tells the microcontroller to set up a single pin (first parameter) using a certain mode (second parameter). The two basic modes are INPUT and OUTPUT; INPUT_PULLUP connects an internal resistor between the pin and 3.3V (good for listening for pull down events; we’ll hear more about these later in the course).

delay(1000);                  // ...and pause

The delay function in Arduino takes milliseconds as it’s parameter – so a delay(1000); command pauses for 1 second. Note: this is a blocking method! Nothing else can happen on the core that is being paused for the duration of the call!

uint64_t mac = ESP.getEfuseMac(); // ...to string (high 2, low 4):

The ESP32 Arduino layer includes some helpful functions like this one to allow us to get (read) the status of the electronic fuses. After the silicon for the ESP32 is manufactured using a common mask, each one is programmed to give it a unique identity – including things like MAC addresses. These are one-time electronic ‘fuses’ that burn the MAC address into the chip so that it cannot be reprogrammed.

3.3.3.3 Reading from Switches

pinMode(14, INPUT_PULLUP); // pin 14: digital input, built-in pullup resistor

This pinMode call enables built-in pullup resistors connected between the pin and the positive supply voltage (3.3V). These prevent the input ‘floating’ when it isn’t connected to anything and instead make the input go high.

  if(digitalRead(14) == LOW) { // switch pressed

The digitalRead function returns the binary state of the input pin given as a parameter. Because we are using a pullup resistor and connecting the switch to 0V – the logic that digitalRead returns is reversed. Therefore, when the switch is pressed, the function returns LOW.

3.3.4 Exercise 03 Notes

3.3.5 Extension to Blinky (exercise 02)

Assemble the components shown in this schematic on your breadboard:

Blinky Traffic Lights schematic

You should then have a breadboard that looks like this:

Blinky Traffic Lights breadboard

Here’s a picture:

3.3.6 A Final Breadboard Prototype: 9 LEDs

Prototyping IoT devices has become much easier and cheaper in recent years. This makes development cycles much faster: from idea to prototype to product is now a cycle measured in weeks and months instead of years. We’ll finish our quick tour of prototyping skills (that began in Hardware 144 with soldering, breadboarding and using the multimeter and signal generator etc.) by building a more (physically) complicated circuit. This week we’ll build it on a breadboard, then when you’re in the lab you have the option to solder it onto matrixboard (which is a typical cycle in early prototyping; the next step would be to design a PCB and start manufacturing test boards).

In the second half of term you’ll have the opportunity to build projects that involve more complex hardware, e.g. to add GPS, or MP3 playing, or motor drivers, or ultrasonics, or etc. These may require soldering, and when they don’t work first time the multimeter and oscilloscope are our first ports of call to discover why.

(Hate soldering? No lab access? Don’t worry, there are also projects that only use the ESP32 itself.) The circuit is electrically very simple, being Week02’s Blinky with lots more LEDs, each protected by its own resistor:

9 LEDs breadboard 20

The cathode (short leg) of each LED is connected in series with a 180Ω resistor that connects in turn to ground. The anode is connected to one of the GPIO (general purpose input-output) pins of the ESP32.

3.3.6.1 Pinouts

Because we’re using lots of pins, it becomes a bit tricky to fit all the connections in, and we need to check the various references to see where is good to connect, e.g.:

Pinouts on the ESP32 Huzzah

An additional complexity is that (in common with other modules) the ESP has several names for many of its pins, e.g.:

ESP32 pinout schematic

There’s a good description of pinout detail here; and a good general discussion on which ESP32 pins to use here.

In this case we’ll use these pins:

// LEDs
uint8_t ledPins[] = {
  GPIO_NUM_26,
  GPIO_NUM_25,
  GPIO_NUM_21,
  GPIO_NUM_4,
  GPIO_NUM_27,
  GPIO_NUM_33,
  GPIO_NUM_15,
  GPIO_NUM_32,
  GPIO_NUM_14,
};

The course repo has example code called 9LEDs firmware.

3.4 Further Reading

4 Country of the Blind: Networking Devices Without UIs

This chapter covers two key tasks that the vast majority of all IoT devices must implement: provisioning and update. Provisioning is about how devices are supplied with network credentials, and update is about how devices are brought up-to-date with new versions of their firmware.

Over the next two weeks we’ll study these in some detail, and implement first provisioning and then update.

4.1 Provisioning and Update

When you first power up a Chromecast or a Firestick and plug it into your TV, for example, it is useless. These devices, sharing many typical characteristics of the IoT, are special purpose stream-and-decode machines that are fed a URL by their controller (your smartphone) and then act as an intermediary between the cloud and the television. Without a network connection, nothing can happen.

Whenever we supply IoT devices to third parties, we face the same problem; to solve it either:

  1. the device has to ship with network credentials, or
  2. support end-user configuration

Chromecast leverages Google’s Home phone app to achieve the second of these.

Secondly, all IoT devices must be considered vulnerable to attacks of one sort or another, just like any networked computer. This course is fairly new, but has already seen two major vulnerabilities in the ecosystems we rely on:

The first of these (WPA) was fixed fairly rapidly by patches to the relevant libraries. The second was impossible to fix in firmware, and as a result Espressif had to develop version 3 silicon (which is only becoming available now in 2021, e.g. as the ESP32-S3). The attack is quite difficult and restricted (being a. physical, b. complex and c. per-device), but no software patch is possible to fix the vulnerable existing hardware: all ESP32s in the field prior to this point will continue to be susceptible to this attack.

Quoting (Schneier 2017) (Click Here to Kill Everybody: Security and Survival in a Hyper-connected World):

“Everything is a computer. Ovens are computers that make things hot; refrigerators are computers that keep things cold. These computers—from home thermostats to chemical plants—are all online. The Internet, once a virtual abstraction, can now sense and touch the physical world. As we open our lives to this future, often called the Internet of Things, we are beginning to see its enormous potential in ideas like driverless cars, smart cities, and personal agents equipped with their own behavioral algorithms. But every knife cuts two ways. All computers can be hacked. And Internet-connected computers are the most vulnerable. Forget data theft: cutting-edge digital attackers can now crash your car, your pacemaker, and the nation’s power grid.”

Clearly, we need the ability to update our devices in the field; in the worst case we might even need to brick45 the hardware and send a replacement! How do we manage this process? Read on…

4.1.1 WiFi-based Provisioning

The most obvious way to provision (or provide network credentials to) a network-connected microcontroller like the ESP32 is to connect over WiFi and send the credentials over the radio. There have been several options developed to do this, including:

The SmartConfig control flow looks like this:

A more general solution, which is possible on all devices (like the ESP32) that support operating as a WiFi access point (AP), is to allow client connections over e.g. WPA + HTTP(S) (which is as secure as the rest of the web!) and then serve HTML to the client that allows triggering of a scan for available networks and entering of network credentials (which the device can then use to try and join itself, as a client). This is called WiFi-based provisioning, and is a robust and common approach. (The exercises this week are to implement this technique, using the webserver from last week. Flow of control is a little complex, involving the ESP32, your smartphone or laptop, and wireless router. Give yourself plenty of time to think it through, and try to develop your own solution before looking at how mine works.)

4.1.2 Over-the-Air Updates (OTA)

How can we ensure that IoT devices get patched promptly when security holes appear?

The answer is OTA, or over-the-air updates: when your microcontroller is connected via a fast network (e.g. over WiFi), it can download new firmware versions whenever they’re available. There are upsides, but also downsides:

Several partitions in flash are used for update staging, e.g.:

46

Here we see three partitions in use:

The sequence of an update then goes something like this:

Diagrammatically:

47

See below for an exercise to implement this style of OTA.

Note that the Arduino IDE also provides a form of OTA, accessible via the Tools>Port menu option when running an OTA capable sketch from a machine connected to the same network as the ESP. See File>Examples>ArduinoOTA>BasicOTA for an example sketch. (This only works via the IDE; more details on this video.)

We’ll move on to implementing OTA (on top of WiFi provisioning) next week.

4.1.3 WiFi Provisioning + OTA = ???

What happens when we combine both approaches under a web front end that provides management of the update binaries, device registration and the like? An early attempt at this type of system (for the ESP8266 and later ESP32) was built by Andreas Spiess and colleagues, called IoTAppStory:

This project provides a library for devices to poll a firmware repository and recognise when updates are available, then triggering OTA update. It follows an “app store” model as described in this video.

4.1.4 RainMaker: Official Provisioning & OTA

When we’re programming the ESP32, we can take advantage of RainMaker, which, from IDF 4.0 and above, is Espressif’s official solution to the provisioning and OTA tasks. (It also provides a simple device control and configuration mechanism via library calls and a phone app.)

We call the library from app_main, specifying any parameters we would like to expose on the device. We also upload firmware to a repository to support OTA. We can now manage the device from a phone app.

The advantages:

Disadvantages:

For example, if we have three devices integrated (or “claimed”) on our RainMaker account, the android app displays:

If we then open a device that exposes three controls, we see something like this:

All this is so neat, Espressif drew a pretty picture to go with it :)

Give it a try?

4.2 COM3505 Week 04 Notes

4.2.1 Learning Objectives

Our objectives this week are to:

4.2.2 Assignments

To turn things into internet things we need to connect them to a network. One way to connect a device without a user interface to a wifi network is to make it a wifi access point and serve a website that displays what network access points are available. We can then allow the user to choose an access point and enter the key.

4.2.2.1 Coding Hints

Ex07 is about templating: basic string manipulation to create HTML from component parts; the model solution uses an array of strings, but there are, as usual, many different valid solutions (inc. reusing 3rd party libraries).

Ex08 needs to use WiFiClientSecure, and talk to the server on the specified port from Blackboard, and get the URL and paths correct; under those circumstances it will return a line of text starting “Received….” Note that in some versions of the Arduino core you need a patch: see below.

Ex09 is about running a web server (and wifi access point to connect to it) which lists all the other access points the ESP is in range of, and allows you to connect to one of your choice.

We can use the solutions from Ex06 (create an access point and web server) along with Ex07 (utilities for creating web pages) to get us started here. (See the arrangements for running multiple setups and loops in main.cpp for an example of chaining the various exercises together.) Then for Ex08 and Ex09 we can use the WiFi and WiFiClientSecure classes, with calls to these methods (this isn’t complete!):

WiFi.begin(SSID, PSK);
WiFi.status();
WiFi.localIP();
webServer.handleClient();
myWiFiClient.connect(com3505Addr, com3505Port);
myWiFiClient.print(
  String("GET ") + url + " HTTP/1.1\r\n" +
  "Host: " + com3505Addr + "\r\n" + "Connection: close\r\n\r\n"
);
myWiFiClient.available();
myWiFiClient.stop();
myWiFiClient.readStringUntil('\r');

For Ex09 my code additionally uses:

WiFi.scanNetworks();
WiFi.SSID(i);
WiFi.RSSI(i);

Dive in!

4.2.2.2 Patch the core for Ex08

If you hit a problem getting your ESP to connect to the course server (using the WiFiClientSecure.connect() method) then it may be due to a mismatch between the way the server is secured and the version of the Arduino core library you’re using. (This library layers on top of a port of mbedTLS in ESP IDF to implement secure, socket-based web communication between clients and servers over HTTPS.) Make sure that your problem isn’t something simpler first (e.g. connection to the wrong wifi network!) but if you’re sure that your ESP can see the server but is refusing to connect, try the following.

The HTTPS server that we use to collect data from your ESP devices is hosted inside the University firewall, and has no external visibility. This means that we can’t use e.g. LetsEncrypt or similar to generate a signed X509 certificate, and have to use a self-signed certificate instead. What happens when we try to make a connection? A web browser will give the user a warning and allow them to make an exception if required, but in the IoT we typically have no user interface (and often no user present). So we need to tell the library code (in this case WiFiClientSecure) not to check the certificate authority (CA) chain.

In early versions of the Arduino core it is easy: just don’t present a certificate and it will connect without problems. In very recent versions it is also easy: supply a client certificate (self-signed) and call a setInsecure method and it will connect without checking the CA chain. Versions in the middle present a problem as neither option is available, and this includes the IDF 4.3 beta version that our default setup uses. To get over this problem, we must import (or “cherry pick”) a single commit that implements the setInsecure method. Assuming you’ve used the setup method magic.sh setup then you can do this as follows:

cd arduino-esp32
git pull
git describe
git cherry-pick ef99cd7fe7778719c92d6f8df0f10d3f0f7aa35e

Now edit libraries/WiFiClientSecure/src/ssl_client.cpp to resolve conflicts (search on “====” and delete that and the “>>>>” and “<<<<” parts). To tell git that the repository isn’t in conflict any more you can do:

git add `libraries/WiFiClientSecure/src/ssl_client.cpp` to register resolution

4.2.2.3 Which WiFi Network? What if it Doesn’t Connect?

It is possible to get an ESP to connect to Eduroam, but the Enterprise WPA security protocol can be tricky to navigate, so:

You may find it convenient to connect your ESP to a portable hotspot created via your phone or other device. On campus you can use the “other devices” network; see Blackboard for details.

If your ESP doesn’t connect, don’t panic. Try printing some diagnostics, phoning a friend or standing on your head and singing the Congolese national anthem backwards in Latin a couple of times. Then try some more diagnostics.

You may also hit problems getting other devices to connect to the ESP32’s internal access point. Different operating systems try to guess the characteristics of the networks they are asked to join in various ways, and this sometimes interacts badly with the model code. Android in particular may (or may not!) refuse to join the ESP’s access point, or join and then fail to deliver HTTP requests to the server we set up. Things to try if this happens:

When you’ve managed to get the ESP onto a wifi network, it will (on serial) report its IP address; you can then use that IP to connect to the web server instead of 192.168.4.1, which tends to be more reliable.

Finally, you probably don’t want to use the University’s WifiGuest network, as this requires sign-in.

4.2.2.4 Details of Our Cloud Server (for Ex08)

The internal cloud server for the course accepts simple GET requests with parameters that can be used to pass a little data to our persistent store.

The data format is key=value, e.g.: email=hamish@gate.ac.uk, and it requires a .ac.uk email address. A request to store the device’s MAC address would look like this: https://HOSTNAME.shef.ac.uk:9194/com3505-2021?mac=ThisESPsMAC&email=MyName@sheffield.ac.uk

(As before HOSTNAME is given on your Blackboard page for the course, and note that the server is inside the University firewall.)

4.2.3 Moving 9 LEDs to Matrixboard

Note: this requires soldering, which you may well not have done due to restricted lab access. No worries, just have a read through and (optionally) do it later on if you get in the lab.

Last week we built a 9 LED breadboard. This week the task is to transfer the circuit to a piece of matrixboard and solder it up into a more permanent form. (Breadboards are great for short-lived experiments, but some of the projects we’ll build later on are intended to be more robust, so we need to practice our soldering and circuitry skills a little more.)

You’ll need your breadboard, and some more parts from your kit:

Try folding the resistors in a group:

Place the resistors and LEDs first, ready for soldering:

Soldering this circuit is fiddly; remember to make sure both surfaces are hot before applying the solder.

Use bent wire from some of the components to form a ground rail:

When you’ve done the LEDs and resistors, add a ground jumper, and then move the other jumpers (connecting the GPIOs to the LEDs’ anodes) over:

Fold the wires together to make connections between jumpers, LEDs and resistors, and solder them:

Having an extra pair of hands helps!

Now burn the firmware in the (e.g. using the 9LEDs code).

4.3 Further Reading

Finally, (The Economist 2019)’s Chips with everything (from The Economist Sept 2019 Special Issue on the IoT) presented an interesting summary of the IoT gold-rush in general, and smart homes work in particular, which we excerpt below:

One way to think of it is as the second phase of the internet. This will carry with it the business models that have come to dominate the first phase—all-conquering “platform” monopolies, for instance, or the data-driven approach that critics call “surveillance capitalism.” Ever more companies will become tech companies; the internet will become all-pervasive. As a result, a series of unresolved arguments about ownership, data, surveillance, competition and security will spill over from the virtual world into the real one.

Start with ownership. As Mr Musk showed, the internet gives firms the ability to stay connected to their products even after they have been sold, transforming them into something closer to services than goods. That has already blurred traditional ideas of ownership. When Microsoft closed its ebook store in July, for instance, its customers lost the ability to read titles they had bought

Virtual business models will jar in the physical world. Tech firms are generally happy to move fast and break things. But you cannot release the beta version of a fridge.

In the virtual world, arguments about what should be tracked, and who owns the resulting data, can seem airy and theoretical. In the real one, they will feel more urgent.

The need for standards, and for iot devices to talk to each other, will add to the leaders’ advantages—as will consumer fears, some of them justified, over the vulnerability of internet-connected cars, medical implants and other devices to hacking.

The trick with the iot, as with anything, will be to maximise the benefits while minimising the harms. [p. 13]

Attracted by the lure of new business, and fearful of missing out, firms are piling in. Computing giants such as Microsoft, Dell, Intel and Huawei promise to help industries computerise by supplying the infrastructure to smarten up their factories, the sensors to gather data and the computing power to analyse what they collect. They are competing and co-operating with older industrial firms: Siemens, a German industrial giant, has been on an iot acquisition spree, buying up companies specialising in everything from sensors to office automation. Consumer brands are scrambling, too: Whirlpool, the world’s biggest maker of home appliances, already offers smart dishwashers that can be controlled remotely by a smartphone app that also scans food barcodes and conveys cooking instructions to an oven.

A world of ubiquitous sensors is a world of ubiquitous surveillance. Consumer gadgets stream usage data back to their corporate makers. Smart buildings—from airports to office blocks—can already track the people who move through them in real time. Thirty years of hacks and cyber-attacks have proved that computers are insecure machines. As they spread, so will that insecurity. Miscreants will be able to exploit it remotely and at a huge scale. [p. TQ 4]

Consumers can buy smart light bulbs, such as Hue from Philips, a Dutch electronics giant, which can be switched on or off by phone or voice and can generate thousands of tones and shades. Viewers of “12 Monkeys,” an American science-fiction tv series released in 2015, can download an app that will sync with their light bulbs, automatically changing their colour and brightness to match the mood of an episode moment by moment.

On the difficulties of hooking Smart Home systems together (Ben Wood):

“It’s a very Heath Robinson kind of patchwork, a jigsaw puzzle of connectivity.”

products from one manufacturer often fail to work well with those from another. Standards do exist: Zigbee and z-wave are wireless networking protocols designed for the type of low-power radios found in smart-home gadgetry. But many firms either use proprietary standards or implement existing standards in ways that prevent their products working with those from other companies.

Many companies are involved. Tim Hatt at gsma Intelligence says that telecoms firms are keen to find new, higher-margin businesses rather than simply acting as “bit pipes,” so they have built smart-home offerings as well. Vodafone, a telecoms company, advertises the v-Home hub as a central control point for smart-home devices. sk Telecom, a South Korean firm, has the Nugu. at&t, an American company, offers its Smart Home Manager. Others are startups, such as Wink, which launched with backing from General Electric. In Britain, even British Gas, a former state-owned energy monopoly, has got in on the act. It launched Hive, a smart-home ecosystem in 2013.

That fragmentation means risks

Until fairly recently, says Mr Wood, the assumption was that smart homes would be controlled from phones. But, he says, the reality is different. “Pulling out your phone, unlocking it, tapping an app, then using it to turn the lights on, is much more complicated and annoying than simply walking across the room and pushing a button.” Voice, he says, is by far the most convenient user-interface.

Amazon’s Alexa and Google Home, the two firms’ smart-speaker products, already have greater market penetration than rival smart-home hubs. [pp. TQ 5-6]

5 Sensing and Responding

We’ve looked at what the IoT is and where it came from. We’ve delved into the build and burn tools that we need to update the firmware on an IoT device. We’ve blinked (or actuated) LEDs and read (or sensed) from switches. This chapter will look in more detail at the various ways in which the microcontrollers in IoT devices (like the ESP32) talk to sensors and actuators, or, to put it another way, at the local protocols available on the device. We’ll also look in a bit more detail at what types of sensor and actuator we will typically encounter in the IoT.50

The practical side of the chapter will then continue from last week’s work on provisioning with an exercise on over-the-air (OTA) update.

5.1 Analog and Digital Sensors

A sensor is an electronic component designed to mesh the physical and digital worlds51 by converting a physical phenomenon into a signal which can be measured electrically. Sensors can be broadly classified as either analog or digital, and either active or passive. An analog sensor produces a continuous output signal whereas a digital sensor produces a discrete (usually binary coded) output signal. An active sensor requires an external power supply to generate an output reading whereas a passive sensor does not.

When we want to turn an analog signal into a digital one we need to think about both:

In other words, we have to consider how often we will sample the signal, and at what accuracy each of those samples will be taken. The consequences of these decisions dictate how much information loss the digitisation process will result in. Visually:

52

The X axis in these graphs represents time (“t”) and the Y axis is the strength of the analog signal (or amplitude, “A”). If the signal is varying very rapidly then we need to sample very often in order to capture the changes, and if the signal is varying between very large and very small amounts then we need a large number to store all its different possible levels. For sound waves, sampled from the output of a microphone, it is typical to take more than 40,000 samples per second and to use a 16 bit number to distinguish the loudness level at each of those points in time. (This is the standard which CD audio uses, for example.)

5.1.1 Two Ways to Sense Light Levels

As an example of analog vs. digital sensing, consider these two light sensors, both of which convert the amount of light incident upon them into electrical signals:

Many hundreds of different sensors are available, measuring everything you can think of and a few others besides. They range in price from a few pennies to many thousands.

In common with many Swiss people, Andreas Spiess lives in a building with an atomic shelter. Check out this video for his description of building a Gieger Müller Tube sensor (or Geiger Counter).

5.2 Reading from Analog Sensors

In general when designing sensing equipment in our IoT devices we prefer to find digital equipment that talks to the microcontroller on one of the standard buses that our SDK has library code support for (e.g. SPI, I2S or I2C; see below). In cases where we can’t find a digital version of the sensor, however, we can fairly easily perform direcy measurement of analog signals on the ESP32 as follows.

To begin with we need to check that the voltages produced by our sensor fall into an appropriate size range for the microcontroller to measure. If the values are too high we will need to reduce them; they are too small then we’ll need an amplifier. The former is easy, the latter not so :(

Let’s say that we’re using a range of between zero and one volt (or 0–1.1V as used in this example from Espressif). We can easily expand the range to say 0-5v using a potential divider, in which two resistors allow us to tap off a reduced signal like this:

The signal at “Vout” is found according to this formula:

Online tools can help with the maths, choosing the best fit of resistor values and so on.

We have expanded the range of signals we can deal with by reducing the signal that presents at the microcontroller (as “Vout”). In the other case (where we have signals too small to measure), reducing the input range (e.g. to 0-0.1v) requires active amplification of the signal, which we might typically address by adding a dedicated circuit board for the purpose. The amplification problem can be quite hard to solve reliably, as when we increase the signal we are likely to increase the amount of noise we’re picking up at the same time. Specialist sensing devices for very small analog signals are often quite expensive as a result!

After reading from an analog sensor, we perform the conversion to digital by means of an Analog to Digital Converter (ADC). An ADC samples the analog voltage and converts it into a digital number. The ADC has a voltage range and a number of bits; in our case on the ESP32 these are 0-1V and 12bit respectively. (Other ranges are available but aren’t very linear.) If we use 12 bits this means the digital number reported is between 0-4095 (2^12 = 4096). (Note that this limits the resolution of our measurements to ≅ 0.25mV.)

Here’s the code to take a reading from pin A0 (which connects to the ESP’s analog-to-digital converter two, ADC2):

const int sensorPin = A0; // select the input pin for the analog reading
short sensorValue = 0;    // 16 bit integer for value from the sensor

void setup() {
   Serial.begin(115200);  // initialise the serial monitor at 115200 baud
   pinMode(sensorPin, INPUT);  // set the sensor pin to be an input
}

void loop() {
   sensorValue = analogRead(sensorPin); // take analog reading and store
   Serial.println(sensorValue);    // print the value on the serial monitor
   delay(1000);    // wait for 1000ms (= 1 second)
}

Does it feel strange to see an integer (short sensorValue above) used for an analog voltage? As noted earlier, when we digitise an analog signal we’re shifting it from the continuous world (which might at first sight seem more naturally represented as floating point) to the discrete. As long as our integer is wide enough to store enough amplitude values for the purposes we are reading the sensor for, then it is perfectly appropriate.

In the exercises section below we’ll look at the ESP32’s (capacitative) touch sensing capabilities, which also read an analog signal and produce a digital output.

5.3 Digital Sensors

In some cases the signal produced by a sensor is binary to begin with – for example, switches may be either on or off, and we can read that binary state directly from the microcontroller’s general purpose input/output (GPIO) pins.

Example code:

const int switchPin = 12; // select the input pin for the switch
bool switchState = 1;     // for the value coming from the sensor

void setup() {
  Serial.begin(115200);  // serial monitor at 115200 baud
  pinMode(switchPin, INPUT_PULLUP);     // pullup, see below
}

void loop() {
  switchState = digitalRead(switchPin); // read digital input
  if (switchState == false) {
    // do stuff when switch pressed
    // (note reverse logic i.e. false means pressed)
  }
}

It is also now common for sensors to be pre-packaged with dedicated circuitry to do the analog-to-digital transformaion before the signal ever leaves the sensor housing. In this latter case, it is usual for the output from the sensor to use one of a handful of dedicated communications protocols intended; the next section describes several of these protocols.

5.3.1 Avoid Floating Voters

When using digital reads we need to be careful that our input pins aren’t left “floating” in order to minimise ghost signals that may otherwise be triggered by noise picked up on the circuitry we’re connected to. In other word, inputs don’t like to be left unconnected or “floating” – this results in fluctuating signals and can cause other, intermittent and difficult to diagnose problems.

It is common therefore to add a resistor that bridges the input to a stable value, either high or low. When connected to the positive supply voltage, it is a pull-up resistor. When the resistor is connected to ground the resistor is called a pull-down resistor. So long as the resistor is a moderately high value it will stop the input floating, but doesn’t interfere with normal operations (5-10Kohms is typical).

An example schematic of a pullup resistor:

pullup resistor

And a pulldown:

pulldown resistor

5.3.2 Vcc by any Other Name Would Smell as Sweet

While we’re talking about circuit schematics, let’s clear up some terminological issues. Confusingly there are several different terms for the positive and negative voltages that power electronics:

Clear? You’re a genius! Give yourself a huge slap on the back and treat yourself to a chocolate bar this instant.

5.4 Local Protocols: UART, SPI, I2C, 1-Wire…

IoT devices have more protocols associated with them than the average farm-yard dog has fleas. WiFi, LoRaWAN, Bluetooth, NB-IoT, Sigfox… although these vary in range from a few meters (Bluetooth) to several tens of kilometers (LoRaWAN etc.) they are all, relatively speaking, long-range protocols. In this section we look at short-range, or local protocols.

Local protocols define communications mechanisms for microcontrollers to talk to sensors and actuators using buses of various types. For example, the I2C (Inter-Integrated Circuit) protocol toggles the values of a Serial Data Line (SDA) and a Serial Clock Line (SCL) to transmit data up to a few megabits per second. On an oscilloscope an I2C transmission might look like this:

Different protocols use different transmission strategies, and their data rate, viable circuit length, reliability and ease of use all vary depending on how these strategies are organised. One thing that they all do is to provide an easier data transfer method than bit banging: toggling GPIO directly with firmware to send signals to or collect signals from peripheral sensors or actuators. Except for the very simplest of cases (like those exemplified above where we read a switch state from an input pin), local communications protocols are a more robust and simpler to implement option.

We’ll look at a small handful of common protocols below.

5.4.1 UART

The Universal Asynchronous Reception and Transmission (UART) protocol (or serial port) is at the simple end of the food chain.

As in other cases, digital highs and lows are the basis for communications for the UART protocol. The most common setup is to have a single start bit (LOW), 8 data bits and then a stop bit (HIGH). Each group of 8 data bits is a byte that represents a character in ASCII.

53

Seeedstudio provide a useful comparison of UART, SPI and I2C:

UART I2C SPI
complexity: simple easy to chain many devices complex as devices increase
speed: slowest faster than UART fastest
# devices: up to 2 devices up to 127 but may get complex as devices increase many, but there are practical limits and may get complicated
# wires (plus ground): 1 2 4
duplex: full duplex half duplex full duplex
# masters / slaves: single only multiple slaves and masters only 1 master but can have multiple slaves

There’s a code example for UART below.

5.4.2 SPI

The Serial Peripheral Interface (SPI) is a bit more complex, using synchronous signaling with a clock line in addition to the communications lines. This means that communications can be much faster than using a simple serial protocol like UART. In addition to these lines, each device has a chip select line that is used to select the device that you want to respond to the data. This means that several devices can use the same bus, only talking on the bus when their chip select line is pulled low.

54

Counting ground, this SPI needs four bus lines: MOSI (Master Out Slave In); MISO (Master In Slave Out); SCLK (Serial Clock). In addition each device needs its own chip select line.

5.4.3 I2C

The Inter-Integrated-Circuit (I2C, or I²C if we’re being posh) protocol is more complex still, using its two wire bus to potentially link up to 127 devices.

55

Slave devices are addressed by the master device using signals on the bus itself instead of using separate select lines, and each device listens to the bus and only responds when it is addressed directly.

I2C is alone in the buses we’re covering here in providing a delivery guarantee, with confirmation that the listener device received the payload. Note that this sometimes results in bus lock-ups!

There’s a code example for I2C below.

5.4.4 1-WIRE

Yet more complex (and less common in the circuits we have worked with) is 1-Wire, a protocol that is somewhat similar to I2C but, as its name suggests, only uses a single data line (plus, as usual, ground).

Complex protocol, you should be aware of the details and where to find them if you need to, but happily, a well tested, documented library is available for use with minimum effort. Like I2C 1-Wire allows multiple devices. It supports a lower data rate than I2C but longer cable length (up to 50m+ vs. 1-2m).

Notably, 1-wire is used by Apple MagSafe power supplies (and by Dell and others) to exchange signals between the computer being powered and the circuitry controlling the PSU. No doubt there is an advantage to this arrangement somewhere, but as Gareth says, “Now my laptop won’t charge unless I have an approved charger – tor my own comfort and safety.” :(

5.4.5 Other Local Protocols

Other protocols we may come across include:

5.4.6 Talking the Talk: Local Protocol Examples

As with all but the most basic of communication protocols, the actual gorey details how data is signalled, how noise and errors are minimised, and how devices keep out of each other’s way when sharing buses are complex. Luckily, in the Arduino ecosystem other people have done the hard work of encapsulating all these details in libary code. To put it another way, open source allows us to see further by “standing on the shoulders of giants” :)57

So you (mostly) don’t need to worry about the details of the implementation, with start and stop bits, ack(nowledgement)s, bit order, etc. (At least, that is, until stuff starts going wrong!)

Here’s an example of using the Arduino versions of UART and I2C (the second of which is known confusingly as Wire). You have already used the built-in UART library to print to the serial monitor:

Serial.begin(115200);    // initialise the serial monitor at 115200 baud
Serial.print(“Hello world”);    // simple printing characters on the serial port

SPI and I2C are as easy to use (I2C example shown here):

#include <Wire.h>           // include the I2C library
Wire.begin();               // initialise the port;  pins 22 & 23 on ESP32
Wire.beginTransmission(44); // transmit to device #44 (0x2c)
Wire.write(byte(0x20));     // write the value 20 in hex (32 in decimal)
Wire.endTransmission();     // stop transmitting

This pattern (of using library code to do the heavy lifting is typical of our microcontroller/sensor/actuator ecosystem – or, to quote Gareth again, “the giants are stacked!”

Most of the time, you can use a pre-written library that deals with all of the communications details for you. These provide functions such as readSensor() – this typically requests a reading from the device, gets the response and formats it for you. For example the Adafruit library for the TSL2561 light sensor provides the functions setGain() and getLuminosity(); a getDustDensity() is provided by the dust sensor library we have used for air quality monitoring; and so on.

As usual, our good friend Andreas Spiess has a nice video that’s relevant, this one showing how to “connect many devices (sensors and displays) to one Arduino using the I2C bus. It starts with a simple hommade bus, shows how to find out the addresses of the different devices and ends with a demo of a system with three devices connected to an Arduino.” Recommended.

Ok, that completes our brief tour of local protocols. Now let’s finish off this discussion with a look at actuators, or the “arms and legs” of our “world robot,” to echo (Schneier 2017).

5.5 Actuators

Broadly speaking these devices change the physical world in response to electrical signals. These signals might be the amplitude of a sound pressure wave (to drive a loudspeaker), or the direction and speed of travel of a motor or solenoid.

Outputs can also be information, of course – such as a tweet, an SMS, a record in a database, or etc. – and the change that we want to create can just be a signal of some type – lighting an LED, perhaps, or running the vibration motor in a smartphone or smartwatch.

As with our protocol-based communications above, or our digital sensor readings before that, we’re generally in the game of picking up an existing library for whatever actuator hardware we’re trying to control. More giants :)

The ESP32 has several kinds of outputs that can drive actuators:

None of these outputs can supply much current – 12mA at maximum. (By Ohm’s law this 12mA limit at 3.3V implies we need a resistance of at least 275Ω in the circuit we’re driving.)

In practive we often need an amplifier or driver to supply the power the actuator needs, and we may also wish to isolate our low-voltage circuitry from the equipment we’re controlling – this is especially true of mains voltages, which can kill! (Never work with mains circuits if you’re not a qualified electrician!) Below we’ll talk about two ways to deal with higher power or higher voltage actuators.

5.5.1 High Power Actuators with Relays

If we want to switch a device which requires a current larger than a few miliamps we can use a mechanical device called a relay. (We might also commonly use transistors, especially MOSFETs, though these are a little more complex to wire up.)

In relays we have a control circuit, and a switching circuit. A small amount of power applied to the control circuit operates an electromagnet. The magnetic force causes the switch contacts to open and/or close, thus controlling the switch circuit. The switch is completely separate from the electromagnetic coil:

When the coil power is removed, the magnetic field collapses very quickly, inducing a massive voltage. We can protect against this voltage spike with a ‘snubber’ diode. Other devices that operate with a coil such as solenoids and motors also need snubbing.

5.5.2 High Voltage Actuators with Radio Control

How do you switch mains without a risk assessment?! We encourage our students to be adventurous in their learning… but please don’t touch any circuits that use mains electricity!

There’s an easier way, which is also very safe: use a remote control power socket that responds to radio control signals to switch mains electricity. We don’t need to touch the dangerous stuff. Instead, we send commands on the radio frequency (433MHz) that the sockets are tuned to. The transmitter is a low-voltage actuator component which is quite harmless:

The sockets are an off-the-shelf consumer item (which we never open or modify in any way!):

Again, a pre-written library does all the hard work – e.g. mySocket.send(4281651, 24);. More on this in Chapter 8 for our home automation projects.

5.5.3 Electric Blankets, Fish Farming and Liverpuddlians

What do cold nights, Liverpuddlian urban agriculture and the IoT have in common?!

This is a screenshot of a web page served from an ESP32 used to control

Now you know.

5.6 COM3505 Week 05 Notes

On the practical side of the course we’ve also now covered quite a lot of ground: from blinking LEDs and reading from switches to getting our devices to talk to WiFi and push data into the cloud, and last week we worked on provisioning. Let’s tie this together with a working demo of Over-the-Air update, and we will be fairly close to calling our first tour of IoT programming complete.

5.6.1 Learning Objectives

Our objectives this week are to:

5.6.2 Assignments

Exercises:

Lots of notes below!

5.6.2.1 ProUpdThing: Provisioning and Firmware Update

The course repository contains exercises/Thing/main/Ex10.cpp and also exercises/ProUpdThing/, which both implement firmware for WiFi-based provisioning and Over-the-Air (OTA) updates. The former uses a local HTTP server to make the .bin available for download by the ESP; the latter uses GitLab to host the .bin and does an HTTPS GET from there. For a third, more production-ready approach, have a look at Espressif’s RainMaker API, which hosts updates on AWS and provides an oven-ready C API for driving OTA on the ESP (via a webapp).

While working, think about possible future enhancements:

5.6.2.2 Configuring ProUpdThing

The ProUpdThing build assumes that a file called private.h exists in your home directory ($HOME). To allow the firmware to download from your (private) GitLab repo, you must create a Personal Access Token (PAT) on gitlab.com and add it to private.h:

Screenshots (with my token hidden!):

You also need to define your project ID (which is a unique identifier for your gitlab repository; it appears next to the project title on the top-level page). Copy it and define it in your private.h file:

#define _GITLAB_PROJ_ID "1234567"

Check that you have everything in place by building, flashing and performing updates as below.

The firmware does two jobs, the first being to allow wifi provisioning, the second to support OTA updates.

5.6.2.3 Provisioning

When the firmware starts up, it calls a library function (joinmeManageWiFi) to check if WiFi has previously been configured and if so try and join the network. If that fails, it starts up its own WiFi access point on the ESP32 and runs a webserver on port 80 (with DNS capture, which works most of the time; if not, try 192.168.4.1). If you join the access point from a phone or laptop, the webserver provides a list of other available access points, and allows you to enter the network key. It then shuts down its own access point and joins the network you specified.

A diagrammatic view:

5.6.2.4 OTA

When WiFi has been successfully configured and the ESP has an internet connection, another library function (joinmeOTAUpdate) checks to see if new firmware is available on gitlab. This is arranged via files pushed to a subdirectory called firmware/ as follows.

When a .bin has been exported from the IDE or built via IDF it is pushed to gitlab in a file called firmware/<<version>>.bin (e.g. firmware/6.bin for version 6). The highest version number available is stored in a file called firmware/version. In the code (see main.cpp) an int variable called firmwareVersion is set to the new version. We can then trigger an OTA update by pushing new versions of the .bin and version files, and restarting the ESP.

The project magic.sh script includes a command push-firmware that automatically takes the new version and pushes it to gitlab with an updated version file.

Experiment with the operation of the firmware, until you understand this process.

For example:

cd ProUpdThing
./magic.sh

Now we’re watching what is happening on the device on serial. In a different terminal edit main/main.cpp, make a trivial change (e.g. Serial.printf("now running v%d\n", firmwareVersion");), and increment the version number. Then:

In another terminal:

./magic.sh push-firmware

You should now see the device check gitlab, recognise the presence of new firmware, and update.

5.6.2.5 Hints

5.6.3 The ESP’s Sense of Touch

The ESP32 provides touch sensing capabilities via 10 of its GPIO pins. Each of these pins can measure capacitative variance incident on electrodes embedded in touch pads. (Or, if we’re doing things in a hurry, a bit of handy wire connected to the relevant pin!)

The ESP’s Arduino core defines shortcuts for the GPIO pins that are touch-capable (and accessible on your boards) as follows (in file .../arduino-esp32/variants/feather_esp32/pins_arduino.h):

static const uint8_t T0 = 4;
static const uint8_t T1 = 0;
static const uint8_t T2 = 2;
static const uint8_t T3 = 15;
static const uint8_t T4 = 13;
static const uint8_t T5 = 12;
static const uint8_t T6 = 14;
static const uint8_t T7 = 27;
static const uint8_t T8 = 33;
static const uint8_t T9 = 32;

As usual, these capabilities can be accessed in firmware via both an Arduino API (which is fairly basic) or an ESP IDF API (which is more powerful but more complex).

In Arduino-land, the following will read a value from GPIO 14 and store the current value in tval:

int tval = touchRead(T6);

(What is GPIO 14 and how do I find it?! See the discussion on pinouts in sec. 3.3.6.1.)

The values returned from touchRead (or the IDF equivalent touch_pad_read) are not binary (unlike digitalRead, for example), but represent an analog measure of capacitance. To control an OTA process, we probably only need a binary output (“is the user touching the magic pad or not?”), so your code will need a method for translating the analog signal to a binary yes/no decision. There is also noise in the signal, so you may need to discard outlying values or perform averaging of values.

Below some example output from the Arduino IDE serial plotter, with a jumper cable attached to GPIO pin 14, and Serial.printf("%d\n", touchRead(T6)); in loop(). The high values (around 75) are where there is no touch happening; the first low range (around 60) is where I’m touching the outside of the cable (which is covered in plastic); the second low range (around 15) is where I’m touching the metal end of the cable. In all cases there are random spikes of noise!

Serial monitor touch

(The serial monitor is accessible via the IDE’s Tools menu.)

The IDF API for touch sensing is detailed here. Using this API in filtered mode (with a filter period of 100mS) results in a graph like this (for similar test conditions as above):

Serial monitor touch, IDF API

There’s considerably less noise in this version (though note the relatively slow descent to the low state for a pin touch).

6 Machine Learning and Analytics in the Cloud

* OOOPS!!! *
This chapter isn’t finished, enter at your peril!

It is time to step back a little from the device and its local protocols for talking to sensors and actuators. Below we’ll discuss about processing your data to produce efficient prediction models that can run at the edge. But first, a little digression about AI.

6.1 Is AI about Intelligence?

I was talking to a friend of mine recently who consults for UK universities on e-learning tech and is writing a book about critical reasoning in the age of algorithmic intelligence. In the 1980s we studied cognitive science and artificial intelligence (AI) together, and have watched with interest the recent resurgence in the latter. When we started out AI was first and foremost a project to understand human intelligence by modelling it on computer. By contrast, in the twenty twenties AI is about using applied statistics58 to estimate the probability of a translation fragment, or steer a self-driving car, or recommend a product to someone who just surfed to the page of a similar item. It is also much better funded and much more noise is made about how transformative it may be or become.59 How has this happenned? Were there specific breakthroughs? Changes in social and economic context?

My friend and I ended up swapping emails on the following question:

To what extent – if at all – does the claim survive that AI replicates, reproduces or tells us something useful about human intelligence? Is there an acknowledged end to that narrative (and if so, when and why?)

I think that what happened is that it became vastly profitable to use applied statistics (aka Machine Learning) to tailor ecommerce websites to recommend products to customers that other customers have previously bought when shopping for similar items. This is the foundation of the business models of Google and Facebook, for example. In the case of the former, their initial rise to prominence (and ability to sell advertising) was, of course, based on web search, and this led by various paths to them becoming key defenders of an open web (in the sense invented by Berners-Lee). If Google can’t index it, you can’t search it, and someone else (e.g. Facebook in their partly walled garden, or Apple in theirs) gets to sell the advertising instead. This polarisation of digital corporate conflict encouraged Google in providing all sorts of Really Useful Infrastructure, from satnav and maps to calendars and gmail, and to making these offerings all work as well as possible, mostly for free – or in exchange for your data, of which they amassed a volume for which the term humungous rapidly became quite inadequate….

And it turns out that if you have a really huge number of examples of human beings behaving in particular ways (especially if those ways involve a finite set of decisions: which book to buy, or who to include in an email cc. list, or which route to drive around a traffic jam), then you can get computers to do Really Useful Stuff in supporting human intelligence. (Not least: get speech recognition to work well enough that lots of people now want to use it in their homes, on Alexa or Home or Siri.)

Is this AI, in any sense that the original researchers of that field would have recognised? Probably not, but by now few people remember or care. The amounts of money being made are vast, and they drive the research agenda and reward the acolytes who, of course, invent new rafts of terminology to repel the casual boarder and valourise their knowledge. AI became almost synonymous with Machine Learning (counting lots of occurrences to inform probabilistic choice); Deep Learning (using perceptrons and other network representation models, and stacking them) became one of the chief mysteries; “Big Data”(and data analytics) came to be touted as the saviour of high tech UK60; and so on.

The project of understanding human intelligence, then, is now rather eclipsed (at least in my neck of the woods), by the “let’s get some data and count features and do prediction” school. Will this trend connect up again with the original project? Will progress be more substantial now that we have increased our data set sizes by orders of magnitude? Or is it, as one of the first web browser engineers61 once said, the fate of our generation to spend our best years working on advertising?!

You have more chance of finding out than I do. Enjoy the chase!

6.2 IoT, Big Data Analytics, and Deep Learning

6.2.1 Machine Learning at the Edge62

Machine Learning has gone through some tremendous growth during the last decade, mostly driven by progress in training deep neural networks. Their benefits are impacting more and more applications, used in image classification, object detection, speech recognition, translations and many more. Wherever data is being generated, it is likely that machine learning can be used to automate some tasks. That is why it is important to study machine learning in the context of data generated by IoT devices.

Currently, machine learning operates under a common computing paradigm. Data is moved to the cloud (or a centralised server) for data processing. There, data is labeled and sanitised for quality control, and can be used to train machine learning models. How these models are used can differ: either continue to upload data to the cloud for run-time inference (e.g., voice recognition with Alexa, or Siri), or move the model to the edge to perform inferences close to the data source. Due to privacy concerns the latter is likely to grow in adoption over the coming year. Here, we are going to explore the process of training a neural network model and how this can be made available and used at the edge (on your IoT device).

6.2.1.1 Motivation

Edge computing devices are usually constrained by resources (battery, memory, computing power, etc.). The amount of computation they do is thus limited, and so should the machine learning models running at the edge.

If you have ever used the Google personal assistant on your phone, you would know that this can be triggered by using a keyword “Okay Google.” This relies on a voice recognition model running continuously on the phone to detect when you are calling it. Because the CPU is often turned off in idle mode to save battery, on most devices the keyword spotting task runs on a DSP (Digital Signal Processing) unit. The DSP consumes just milliWatts (mW) of power, but it is limited in its computing resources, having just kilobytes (KB) of memory available. So the keyword spotting model needs to be really small. For instance, the Google assistant voice wake-up model is just 14 KB. The same resource constraints we find on microcontrollers, having just KB or memory. We will see in this section how we can build such small machine learning models and how these can be deployed on small devices.

6.2.1.2 Introduction to Machine Learning

Although Machine Learning may see daunting at first, involving a lot of maths and statistics, some basic concepts can be applied out of the box, without going too much into details of how these work and how you can develop the optimal training process for your task. The latter is still an active research area, and even machine learning specialists cannot agree on the optimal methods. But getting satisfactory results from your machine learning models can be achieved with default options and some parameter tuning as we will learn in this section.

We use Machine Learning when the patterns in data are not immediately obvious to us for how to programme about them. For instance, we know for sure that water boils at 100°C, so if we have a reliable thermometer we can programme with a hard threshold on that value. That is how the common thermostats in boilers and refrigerators are built. But if we work with noisy data, or the data exhibits patterns that are too complex for us to spot at a quick glance, it is better to use machine learning for those tasks. Voice recognition is one example, where direct thresholds are hard to impose. That is because each time we utter a word or phrase there is some slight deviation in the tone, pronunciation, opening of the mouth, angle to the microphone, background noise and many other altering conditions. But machine learning can extract information from a lot of data examples to automatically focus on the more meaningful attributes during training. Once such a model is trained, we can use it in our programmes to make inferences in the real-world without understanding the complexity of the data ourselves.

There are many different approaches to machine learning. One of the most popular is deep learning, which is based on a simplified idea of how the human brain might work. In deep learning, an artificial neural network is trained to model the relationships between various inputs and outputs. We call it deep because modern network architectures have many stacked layers of parameters or weights (resembling the connections between brain neurons). Each architecture is designed for specific tasks and a lot of effort has been going in research over the last few years to determine the best architectures for each task. For instance, image classification works well with Convolutional Neural Networks, whereas text translations benefit from Recurrent Neural Networks.

In the following sections we are going to have a crash course on the most essential part of deep learning to get you started with training and deploying a machine learning model. It is by no means complete of what you can do in machine learning, but we hope this will get you curious and excited about this topic so you will study it in greater details in other courses.

6.2.1.3 Training Deep Neural Networks

Training is the process by which a model learns to produce the correct output for a given set of inputs. It involves feeding training data through a model and making small adjustments to it until it makes the most accurate predictions possible.

Training a neural network will require a sizable dataset of plausible examples. It is important that your model is trained on a distribution of data which is as close as possible to the real data. Assuming you have a dataset of labeled instances (annotated with the ground-truth class, for example the uttered word in keyword spotting), you will assign a portion of that data for training, a smaller fraction for validation (to evaluate the choices of parameters you make during training) and finally a test set that you never touch until the very end when you are happy with your model’s performance on the validation set and you want to determine its final performance (inference accuracy) on the test set. A common split is 60%:20%:20%, with the largest set always for training.

During the training process, internal parameters of the network (weights and biases) are adjusted so that prediction errors on your training set are slowly diminished. In the forward-pass through the network, the input is transferred at each layer in the network, all the way to the final layer which produces a data format that is easily interpretable. The most common output format of a classification neural network is a vector of activations where each position is associated with a different class. The vector position with the highest value indicates the winning class. For instance, if we want to detect the spoken words between the following classes “Yes,” “No” and “Unknown” (in that order in the output vector), and our network produces the following output vector: [0.9, 0.02, 0.08], we will associate the estimation to the class “Yes.” The network is trained by adjusting the weights of the network such that the output will get as close as possible to [1, 0, 0] if “Yes” is indeed the ground-truth for the spoken word. This is done by training on many more examples at once, each contributing to adjusting the value of the weights. We stop the training when the model’s performance stops improving.

During training we look at two metrics to assess the quality of the training: accuracy and loss, in particular on samples from the validation set. The loss metric gives us a numerical indication of how far the model is from the expected answer, and the accuracy indicates the percentage of times the model chooses the correct class. The target is to have high accuracy (close to 100%) and low loss value (close to 0). Of course, these are task dependent and for some tasks getting to 70% accuracy is considered a very good achievement.

The figure above shows a common training behaviour pattern. Over a number of epochs (iterations over the training set), the accuracy increases and stabilizes at a certain value – at which point we say that the model has covered; and the loss decreases towards its convergence point.

Training neural networks is usually based on try and error in choosing the right training parameters (or hyperparameters because they are empirically determined). After each adjustment, the network training is repeated and metrics visualised until we get to an acceptable performance. The common hyperparameters adjusted during training are: learning rate, which indicates how much the values of weights should be updated at each epoch. A common value to start with is 0.001, network structure – number of layers and the number of neurons per layer, learning optimisation parameters (such as momentum, etc.), activation functions (sigmoid, tanh, ReLU), number of training epochs.

Once we are happy with the model produced during training, we test it on the test set to assess its final accuracy. We are then ready for deployment to the device where inferences will be produced. But there are some more optimisations that can be performed to make the model run more efficiently on the device.

6.2.1.4 Neural Network Quantization

When deep neural networks first emerged, the hard challenge was getting them to work at all. Naturally, researchers paid more attention to training methods and model recognition accuracy, and little on inference performance on hardware. It was much easier for researchers to work with floating-point numbers, and little attention was given to other numerical formats.

But as these models become more relevant for many applications, more focus is now going into making the inference more efficient on the target device. While the cost of training a model is dominated by the cost of the data engineer time, the operational cost of running inefficient models on millions of devices dramatically outbalance the development cost. This is why companies are now so interested in designing and training the best machine learning models that can run efficiently on edge computing devices.

Quantization is one popular solution to make neural networks smaller. Instead of storing the weights of neural networks in the traditional 32-bit floating-point numerical representation, quantization allows us to reduce the number of bits required to store weights and their activations. Here we will look at quantization, reducing weights to 8-bit fixed-point representation.

Definition: Quantization is an optimisation that works by reducing the precision of numbers used to represent the model’s parameters, which by default are 32-bit floating-point numbers. This results in smaller model size, better portability and faster computations.

Floating-point representations are essential during the training process for very small nudges of the weight values based on gradient descent. While that data representation makes a lot of sense for infinitely small adjustment of the weights in several passes over the training set, during the inference time an approximated result for the estimation will likely still determine the same class (although with less intensity on the last layer of the network). This is possible because deep neural networks are generally robust to noise in the data, and quantized values could be interpreted as such.

The effect of applying quantization is a little bit of loss in accuracy (the approximated result will not always match that produced by the trained model in 32-bit floating-point representation). But the gains in terms of performance on latency (inference time) and memory bandwidth (on device and in network communication) are justifiable.

6.2.1.5 The Quantization Method

Neural networks can take a lot of space. The original AlexNet, one of the first deep neural networks for computer vision, needed about 200 MB in storage space for its weights.

To take an example, let’s say we want to quantize in 8-bit fixed-point (using integer values between 0 and 255), the weights of a neural network layer that has values between -2.5 and 6.5 in 32-bit floating-point representation. We associate the smallest weight value with the start of the quantized interval (-2.5 will be represented as 0) and the largest with the maximum value (6.5 will become 255). Anything between the min and max will be associated with the closest integer value in a linear mapping of [-2.5, 6.5] to [0, 255]. So with this mapping, 2.4912 will be 127 because it falls approximately at the middle of the interval.

scale = (max(weights) - min(weights)) / 256

x_code = quant(x) = (x - min(weights)) / scale

x_reconstruct = dequant(x_code) = min(weights) + x_code * scale

The first benefit is that the storage of the model is reduced by 75% (from 32 bits down to 8 bits representation of each weight). This can give substantial benefits if only just for communication cost and flash storage and loading. In the simplest form of device implementation, the numbers can be converted back to 32-bit representation and using the same computation approach in 32-bit precision to perform the inference. But with newer computation frameworks (e.g., TensorFlow Light) the weights can be used directly in 8-bit representation for integer operations (inputs are also quantized to 8-bit precision). This will also take advantage of the hardware support for accelerating to perform the operations in parallel in a single instruction with SIMD (single instruction multiple data) configurations. With this, 4 times more 8-bit values are loaded from flash in the same instruction and more values are stored in a single register and in the cache. If implemented efficiently, theoretically this can speed up the execution by 4x (performing 4 different 8-bit operations in the same clock time needed for one 32-bit operation). Not to mention that integer operations are generally more efficient in hardware, so they consume less energy compared to floating-point computations.

The advantages of quantization:

In summary, quantizations are great, because if you reduce the model representation from 32-bit floating-point representations to 8-bit fixed-point representations you save storage space, memory bandwidth on loads, and even speed up computations and energy consumption where hardware permits it. And for some devices (such as 8-bit microcontroller) using quantization may be the only option to run inference models on them.

6.2.1.6 Keyword spotting exercise

Has a lot of applications to enable the triggering of speech recognition. For instance okay google, or Alexa are keywords that trigger the device to start recording your commands. Your voiced commands are then sent through the internet to the cloud for speech recognition, to interpret your commands into computer instructions. However, knowing when to start recording your commands is important. It would be too costly to stream continuous recording to the cloud, not to mention privacy invasive. But running the keyword spotting algorithm locally, on your device in the home is safe and cost effective. The audio is streamed to the microcontroller which analyses with small and efficient algorithms to spot when the enabling word is used.

Step 1 Data collection

For Keyword Spotting we need a dataset aligned to individual words that includes thousands of examples which are representative of real world audio (e.g., including background noise).

Step 2: Data Preprocessing

For efficient inference we need to extract features from the audio signal and classify them using a NN. To do this we convert analog audio signals collected from microphones into digital signals that we then convert into spectrograms which you can think of as images of sounds.

Step 3: Model Design

In order to deploy a model onto our microcontroller we need it to be very small. We explore the tradeoffs of such models and just how small they need to be (hint: it’s tiny)!

Step 4: Training

We will train our model using standard training techniques explored in Course 1 and will add new power ways of analyzing your training results, confusion matrices. You will get to train your own keyword spotting model to recognize your choice of words from our dataset. You will get to explore just how accurate (or not accurate) your final model can be!

Step 5: Evaluation

We will then explore what it means to have an accurate model and running efficient on your device.

6.3 COM3505 Week 06 Notes

This week we publish “Lab” Asssessment 1, which you will have one week to complete. More details nearer the time!

6.4 Further Reading

Pete Warden and Daniel Situnayake, TinyML: Machine Learning with TensorFlow Lite on Arduino and Ultra-Low-Power Microcontrollers, O’Reilly Media, 2019 (Warden and Situnayake 2019).

7 Clouds; Networks; IoT Device Gestation

* OOOPS!!! *
This chapter isn’t finished, enter at your peril!

TODO

7.1 COM3505 Week 07 Notes

7.1.1 Learning Objectives

Our objectives this week are to:

7.1.2 Assignments

7.1.2.1 Coding Hints

7.1.2.2 Setting up an IFTTT Applet

TODO check this still works

IFTTT stands for “if this, then that.” We can think of the “this” as an incoming notification (or source), and the “that” as a triggered action (or sink). IFTTT “applets” accept notifications from diverse sources and trigger actions on diverse sinks. For example, we can set up an HTTP-based trigger (using their “webhooks” service) that causes Twitter to tweet a message.

First create an account on ifttt.com and/or download the app for your phone. (You’ll need a Twitter account too if you want to tweet.) In IFTTT navigate your way to new applet creation:

This will bring up If +__This__ Then That. Run through these steps:

This brings up If (hook logo) Then +__That__. Now:

We’ve now set up most of what we need on IFTTT, but we need to copy a token to allow our ESP to authenticate against the server. To do this either

This will give you a page saying something like:

Your key is: my-long-key-string
Back to service
To trigger an Event
Make a POST or GET web request to:

https://maker.ifttt.com/trigger/{event}/with/key/my-long-key-string
...
You can also try it with curl from a command line.

curl -X POST https://maker.ifttt.com/trigger/{event}/with/key/my-long-key-string

Replace {event} with your event name, e.g. my-first-ifttt-trigger and click Test it, or copy the curl statement and try it from the command line.

NOTE: IFTTT services have quite a high latency, and this seems particularly true for Twitter. You can expect to wait 10 minutes or more for the tweet to appear in public! (You should be able to test the service without waiting for that to happen though, as the POST request will return a Congratulations! You've fired the {event} event message if it succeeds. Also check the Webhook>Settings>Activity page to see if Twitter accepted the trigger or if some other problem may have occured. YMMV!

Make a note of your key, and the URL that the service lives at. (Following the style of previous weeks’ exercises, you could put the key in your private.h file, e.g.: #define _IFTTT_KEY "j7sdkfsdfkjsdflk77sss".

7.1.2.3 Accessing the IFTTT Applet from Firmware

Now that we have a working service, we just need to get the ESP32 to call it over HTTP(S). This example from the ESP32 Arduino core can be adapted to talk to IFTTT with a little work. If we define a doPOST method (based on the example) that takes a URL and a JSON fragment in IFTTT format, we can do the service trigger job like this:

void doPOST(String url, String body) {
// ...
}

void iftttTweet(String messageDetails) {
  String url(
    "https://maker.ifttt.com/trigger/waving-at-twitter/with/key/" _IFTTT_KEY
  );
  doPOST(url, messageDetails);
}

(You need to define the _IFTTT_KEY in your private.h file in the normal way.)

Good luck!

The course GitLab repository contains exercises/IFTTTThing, a model solution to the exercise. (Note: this assumes you have an ultrasonic sensor attached to pins A0 and A2 like this.)

7.2 Futher Reading

8 Applications

* OOOPS!!! *
This chapter isn’t finished, enter at your peril!

Welcome to the end of the beginning: from now on we’re in project territory. The most important part of the rest of the course is to design, build, test and document an IoT device. Most often this involves both hardware and firmware (and sometimes cloud-based software), but it is also possible to do a good project that is only in firmware.

The first half of this chapter does three things:

The rest of the chapter then details project options.63

8.1 Beep my Earing Whenever I Start Sounding Like a Donkey

Applications of IoT technology are many and varied. Existing products include:

DIY projects are numerous and diverse. For example:

8.2 Projects: Design, Build, Document

The project is both a great opportunity to learn about the entirety of an IoT device and the main way that we assess the depth of your learning. (See chapter 1 for more details of the threshold and grading assessment method in use from 2021.) The latter point means that you should pay a good deal of attention to documentation and presentation of your work: functionality is important, but the best projects will be those that combine great functionality with great documentation, both of the device itself and of the process of its creation.

You now need to start:

As always you should keep checking in and pushing to your repository as you iterate through design, development, testing and documentation phases.

8.2.1 Possible Projects

There are many possible projects, some using additional hardware, some just using the ESP32 on its own. Here are some suggestions.

We supply a variety of add-on hardware for you to use in project work, but if you prefer it is also possible to do a project using just the ESP32 itself, for example:

Projects using additional hardware:

Below we detail hardware build issues, relevant libraries and example code etc., after a look at the LiPo batteries used by some project options.

8.3 LiPo Safety

8.3.1 What are Lithium Polymer Batteries?

Lithium Polymer (or LiPo) cells are one of the most effective commercially available rechargeable batteries, having high energy and power density. These batteries are used in all manner of mobile applications and are particularly popular with remote control (RC) hobbyists.

If needed you will be given a LiPo battery as part of the project hardware for COM3505. The battery is potentially dangerous if damaged, or the electronics connected to the battery are damaged, or the battery is connected to inappropriate hardware. If in doubt, stop work and ask for help!

8.3.2 What are the Dangers?

Although these cells are very useful, they can be dangerous when mistreated or misused. There are three main causes of cell failure: puncture, over-heating and over-charging. If the battery fails as a result of any of these, hazardous electrolyte leakage, fire or explosions may occur.

The rate of transfer of energy within the battery is strongly linked to its internal heat; when over-heated the cell’s long term performance will degrade, more energy is released and the battery gets hotter, creating a dangerous feedback loop. If this continues the electrolyte will vaporise and cause the battery to expand, which may lead to fire or even an explosion.

This same effect can be caused by over-charging the battery, or in some cases even just leaving it charged in the wrong circumstances. Henry (one of our previous teaching assistants on the course) used to fly an RC helicopter that ran off a multi-cell LiPo pack. Having forgotten to discharge it, it was left in a container in his shed. Many months later the cell exploded in a ball of flame nearly burning down the shed!

The sensitive chemistry of the batteries can also lead to fire if the battery gets punctured and vents into the air.

8.3.3 Avoiding Problems

ALWAYS take the following precautions when using LiPos in COM3505:

8.4 Build and Development Notes

8.4.1 A Simple Robot Car

The arrival of robot overlords has been much anticipated. Get ahead of your fellow humans by building a simple robot car! The one we’ll be making is from Adafruit:

robot

When following the build instructions, read through them first before jumping off to the linked details and starting assembly – there are some steps in the earlier instructions that you are later told to modify.

Follow Adafruit’s build instructions to construct the robot chassis and then attach the breadboard on top. (Ignore the “Temporarily unable to load embedded content” message.)

(See also the other Adabox002 videos. but again, remember that we’ll be using the ESP32 feather as our processing board rather than the bluetooth board supplied in AdaBox002.)

You will need a motor driver board, and use a small pozidrive screwdriver and a small pair of pliers (available in the electronics lab or project space – DIA 2.01 or 1.01) to put the headers into the stepper motor connections as shown:

motor driver with headers

Now we can connect the motors to the header pins:

robocar front

Then screw the battery box wires into the connector at the side of the motor driver – make sure to put the red lead in the + connection and the black one into the – connection:

robocar side

You can follow the next section here: basic code for robot to test out the robot.

We are using the aREST library to provide a RESTful interface – one that responds to HTTP calls to URI’s like //esp32/forward.

The code is on github here and the code is explained on this Adafruit page.

Modify the sketch to use your local wifi access point (phones on tether work well as they report the IP of connected devices). Also change the first two lines to adapt this code to run on the ESP32; they should read:

    #include <WiFi.h>
    #include <aREST.h>

Once you have programmed your ESP32 you can discover it’s IP address either on the serial port or via your hotspot. Using a laptop, join the same network as the ESP32. Edit the script.js file inside the interface directory to replace the IP address with that for your ESP32. Now you can run the demo.html file in a browser for wifi control of the robot!

8.4.1.1 Robot Car: Kit List

8.4.2 Binary Diff for Incremental OTA

(This project doesn’t require any additional hardware, using just a bare ESP32, so is a good choice if you’re not interested in the electronics prototyping side of things.)

As we discussed earlier in the course, the resource-constrained nature of IoT devices makes keeping them up-to-date with security patches and new features a significant challenge, and in situations where network connectivity may be interrupted it is common to allocate two or three times the requisite amount of flash memory to store firmware: one for the current version and one to download the next (updated) version into (and sometimes another one to hold a “factory reset” version). This adds a significant cost to the device.

This project is to port an approach to incremental updates that is used to minimise the size of updates in several contexts including Ubuntu’s snap package manager and Google Chrome’s courgette updates process. Incremental updates work by analysing the difference (or delta) between a new version of (in our case) firmware and the previous version, then sending only the list of changes. A client-side routine is then required to check that the change bundle has come down the pipe successfully and if so apply the change list to the firmware image.

This isn’t a new approach; binary diff for software update is quite common. This blog post has a good summary; systems often use bsdiff or xdelta. The disadvantage is that when differencing object files small changes in source can result in huge changes in the binary: “compiled code is full of internal references where some instruction or datum contains the address (or offset) of another instruction or datum.” The Chrome courgette system is particularly interesting however, because it performs a limited decompile of the object code and then analyses the changes between versions with the address changes separated out from instruction-level changes. This can result in a much smaller diff. (Recent versions of Firefox have taken up the same system, I believe.) Chromium (the open source version of Chrome), for example, sometimes achieves a huge reduction in update size using courgette:

(Posted here.)

To do this type of update on an ESP32 we would need to:

This project is not an easy option, but if successful would be a real contribution to the community :)

8.4.2.1 Advanced Topic: Drag&Drop Update

As an advanced topic, it would be interesting to consider the possibility of closing the gap between the old world of C++ firmware burn tooling and the new one of drag-and-drop Python or JavaScript script update that has started to be a common development mode for microcontrollers such as the BBC’s Micro:bit or boards supporting CirciutPython or the Espruino JavaScript port. Can we compress binary firmware updates sufficiently to rival the new approaches?

Another issue to consider here is board support for USB OTG: if we can treat the board as a mass storage device then drag-and-drop becomes trivial in most operating systems. This is one of the key differences between the ESP32 and the ESP32S2, for example.

8.4.3 TV Remote, TV-B-Gone: IR-Remote Projects

Using an IR LED, start by reviewing the TV-B-Gone codes and code inspiration, then read on for how to get started. Make sure you have collected an infra-red reciever: IR reciever that demodulates the IR transmissions and produces digital signals straight into the ESP32. (You can use these for testing and also to demo the operation of your project in the documentation.)

If we attach infra-red (IR) LED’s to our ESP, we can use it to send remote control commands – for example to turn off TV’s, or perhaps issue a series of commands such as turn on the TV, turn on a satellite box, change the input source, etc.

The mechanics of using the timers is complex and also different on the arduino uno and the ESP32. Luckily the well established IRremote Arduino Library by z3t0 handles this for us. The latest version lists support for ESP32 receiving IR only!

Good old Andreas Spiess has implemented the missing send functionality from the IR remote library.

The libary should be modified to reflect the IR LED pin used on your board – so if you’re installing this yourself then modify the IRremote.h file – line 262 – change: byte timerPwmPin=3; to read: byte timerPwmPin=12; (assuming you’re connecting on pin 12).

In order to know the codes that are used in a certain device (there are hundreds of different propriatary formats!) you can search on the internet – noting that there are at least three ways to write the binary formats and several other ways to express the codes even in a particular protocol.

This page gives good if dense info – combined with this page listing Sony TV device codes.

Ok – so, from the code page, we see that the first table lists basic codes, such as code 21 for power. The Sony:1 at the head of the table tells us that of the various device codes used by Sony TVs, these codes are part of device 1.

So the righto page 12 bit worked example gives us a template for how to proceed:

Our command is code 21 – convert to the 7 bit binary value 0010101 and reverse it to get 1010100. My device code 1 gets expressed as a 5 bit binary value 00001 and reverse it to get 10000. Put these together to get 101010010000, which is A90 hex. Whew! The 12-bit nature of the codes explains the second parameter in the call.

And looking at the example sketches included with the library, IRsendDemo does indeed use code 0xA90 – this provides confidence that maybe I’ve got my sums right – and lo and behold – it turns my TV off!

It is often easier to connect an IR receiver device: IR receiver device and read the codes that are produced by an existing remote control. In order to do this we use a TSOP4838 photoreceiver device that has a sensor, amplifier, demodulator and signal conditioning circuitry all built in. It connects directly to the ESP32 as illustrated:

sensor in socket

Pin 1 is Data Out, pin 2 is Ground and pin 3 is Vcc – 3.3V in our case. This means you can just put the sensor into the expander as shown above, and the data comes into pin A0 (on this image). (However it isn’t very robust!)

NOTE: on the latest (2019) unPhone, pin A0 won’t work for this purpose; use A1 (next along) instead:

sensor in socket

For a more reliable connection, solder the sensor directly into the board (though not on A0!):

sensor in holes

When you have a device that sends IR codes and a receiver that decodes them, you can put them together for testing like this:

two devices

8.4.4 Light Sensor

We are using the TSL2591 light sensor in a handy breakout from Adafruit.

You will need to solder some jumper wires to four points on the sensor - these are the same whether you have the square or round board:

Light sensor

Then it’s a simple matter of plugging the wires into the expander like so:

Light sensor expander

8.4.5 Remote Control Power Sockets for Home Automation

Control the world!!! Or at least, things that plug into a mains socket… This project adds remote mains power switching to your ESP32 using a 433 MHz radio transmitter. This type of transmitter is a common mechanism for remote controls, e.g. for your central heating. The actuator we are using is a radio transmitter circuit that uses the 433MHz frequency:

433MHz transmitter

This is part of the ISM band of frequencies that consumer electronic devices can use without license – in our case we are using electric sockets that switch on and off in response to codes issued on that frequency:

433MHz mains socket

By using a radio-controlled mains socket we cunningly avoid having to interface directly with the mains. Being able to switch a mains powered device such as a light, fan, heater etc. opens up the possibilities of control greatly – hopefully it’s already giving you some ideas…

Power Socket circuit schematic

As you can see from the schematic we can just connect the transmitter without needing any additional components. This transmitter is designed to operate at 5V – but it does work at 3.3V with a lower power consumption but reduced range. If you want to use this transmitter on batteries, you might like to experiment with running it on 5V vs 3.3V and see whether the trade-off is desirable for your project needs.

Soldering the transmitter into the expander should look something like this:

Expander and 433MHz transmitter

Here I’ve soldered wires into the holes just in front of the ones for the transmitter. On the expander the prototype holes aren’t connected to each other, so on the back I made sure the wires connected to the pins of the transmitter:

Expander bottom showing 433MHz connections

The sockets come in packs of three and with a single push button remote control. Eight different sets of codes are used to signal to the sockets so that you can use more than three sockets in one location. The sockets are programmed to match the remote (which can’t be changed), but can also be re-programmed to change the code they respond to. For approx. 3 seconds when power is applied, the socket is in learning mode; if a control code is broadcast during this period the socket will learn this code and begin responding. If no code is received during the first 3 seconds, the socket goes into normal mode and responds to the last code learned. Each socket code has an ‘on’ command and an ‘off’ command – there seems to be no other commands and no feedback – so no ability to read the status of the socket for example.

You can initialise the handy RC Switch library like thus:

#include <RCSwitch.h>
RCSwitch mySwitch = RCSwitch();

Then a couple of setup configurations:

// Transmitter is connected to esp32 Pin #12
mySwitch.enableTransmit();

// We need to set pulse length as it's different from default
mySwitch.setPulseLength(175);

Now we can send a command to the switch:

mySwitch.send(4281651, 24);

I’ve mapped out the codes for all 8 sets of sockets here:

Without an antenna the range of the transmitter is only a meter or so; but a simple aerial can be made with a piece of wire. Using this aerial range can often reach 20-30m and usually can work through walls and ceilings. Instructions can be found here (credit: Ben Schueler).

Aerial

Once you’ve made your aerial you should solder it so it connects with the pin on the right of the transmitter – marked ‘ANT’ so it will look something like this:

Expander with aerial

8.4.5.1 Home Automation: Kit List

8.4.6 Sound Input

We used to use Adafruit’s I2S microphone breakout board. Figuring out how to wire and drive it is a little tricky but not impossible :)

More recently we’ve been experimenting with Chris Greening’s ICS-43434 mic board. There’s documentation and code from his github here.

8.4.6.1 CMG ICS-43434

Chris Greening’s ESP32-based Alexa-alike uses a custom microphone board:

I wired this up to the Huzzah ESP32 like this:

There’s code to drive it and collect the audio here. Note that the Feather Huzzah pins to connect the mic to are:

(Config field .data_out_num is for outward communication on I2S and should be set to I2S_PIN_NO_CHANGE as we’re only listening to the mic in this case.)

To (re-)create the lib/tfmicro tree in MicML follow his instructions here.

8.4.6.2 Adafruit SPH0645LM4H

Try this:

Then you can try this to test:

cd ...the-internet-of-things/exercises/MicML
./magic.sh arduino-ide

And open Tools>SerialPlotter.

You should see something like this:

The hardware setup will look something like this (with a white wire for ground, instead of black):

If you hit problems, there’s useful info in this post, and Espressif’s I2S documentation here.

8.4.7 MP3 Player

The ESP32 feather can be attached to a featherwing with a VS1053 MP3 decoder chip which can also do a nice job of synthesising MIDI input. Store your .mp3 files on an SD card (after formatting it to FAT), and see the example code for how to play via the headphone jack.

Note that streaming audio is a blocking operation for at least one of the ESP32’s cores, so your UI will have to find clever ways to squeeze into available processing power!

8.4.8 NeoPixels; Dawn Simulator Alarm

“Show me someone who is bored of LED’s and I will show you someone who is bored of life.” Ahem. This project uses beautiful and useful NeoPixel LEDs to simulate dawn:

Adafruit’s NeoPixels are bright multicolour LEDs controlled via a single signal wire (plus two for power). Read up on them here - we’re using the NeoPixel stick and the NeoPixel Fether.

There used to be a bug with the ESP32 having difficulty with the timing see here, but hopegfully it was fixed here.

To get them working, first solder on three connections for power and data:

Neopixel wired.

Then connect these to the USB 5V, GND and pin A0 as shown:

Neopixel on expander

8.4.8.1 Dawn Simulator: Kit List

NeoPixel array featherwing

8.4.9 Peer-to-Peer Voting Systems

The challenge with this project is to provide mechanisms to prevent vote stuffing, interception and ensure privacy of votes.

The first thing to do is to use the code for Ex09 so that you have a way to configure your ESP on new networks:

void setup09() { // wifi UI
  ...
}

void loop09() {
  dnsServer.processNextRequest(); // handle DNS  requests
  webServer.handleClient();       // handle HTTP requests
  ...
}

Secondly, find the location of your board using Google’s geolocation API. In order to make the things we’re voting on localised, we need a way to find our location and report this.

This could be provided by a GPS chip, however these consume a moderate amount of power and take a fair amount of time to initialise and get a ‘first fix.’ In addition they don’t work well in urban canyons (between buildings) or indoors.

Alternatively we can use a cunning technique that relies on the relative strength of wireless signals. By comparing the signal strength seen by a device with a map of access points the location of the device can be inferred through triangulation.

Google has already done the tedious task of collecting the location of millions of wireless access points and makes an API available (to developers) that takes a list of wireless signals in the location, and returns a latitude and longitude co-ordinate.

Github user @gmag11 has contributed code for the ESP32 that uses the google API to return location data - Google location code

You will need to get a google API key from Google Geolocation API

Note: in order to Get a list of available votes, you could use a Twitter account to store and access votes, using Twitter’s location function, or implement an IFTTT notification mechanism.

8.4.10 Panic Button

This project uses Google’s geolocation API and/or the GPS featherwing to figure out where you are when you press the button.

The idea is to provide a way to signal a “panic” (or need for urgent assistance) to others, e.g. security staff. In order to be a useful panic button, we need a way to find our location and report this.

This could be provided by a GPS chip, however these consume a moderate amount of power and take a fair amount of time to initialise and get a ‘first fix.’ In addition they don’t work well in urban canyons (between buildings) or indoors.

Alternatively we can use a cunning technique that relies on the relative strength of wireless signals. By comparing the signal strength seen by a device with a map of access points the location of the device can be inferred through triangulation.

Google has already done the tedious task of collecting the location of millions of wireless access points and makes an API available (to developers) that takes a list of wireless signals in the location, and returns a latitude and longitude co-ordinate.

Github user @gmag11 has contributed code for the ESP32 that uses the google API to return location data - Google location code.

You will need to get a google API key from Google Geolocation API.

8.4.11 Ultrasonic sensors

The HC-SR04 ultrasonic sensor is a cheap and cheerful way of sensing distance to solid objects like a wall. The device is well described here: tutorial on ultrasonic sensor.

We can connect it to an ESP32 without needing any additional components:

ultrasonic sensor on breadboard.

Following the schematic here:

ultrasonic sensor on breadboard.

You can use this sensor without a library, as its operation is so simple. The sensor is triggered by a 10ms pulse to the TRIG pin, from GPIO pin 25. The sensor sends an ultrasonic pulse and waits for the response, triggering the ECHO pin to GPIO pin 34. The time between the trigger and the echo is directly proportional to the distance between the sensor and the object. The sensor doesn’t work properly with small objects and complex fields of view - it’s really designed for detecting obstacles like walls. In code, once the trigger pulse has been set, the arduino function pulseIn is useful to time the response:

int duration = pulseIn(echoPin, HIGH);
int distance = (duration/2) / 29.1;
Serial.print(distance);
Serial.println(" cm");

However, this sensor is not very well behaved! It occasionally gives wrong values, sometimes the pulse never returns to the sensor correctly, and sometimes a spurious signal will be reported. Simple functions to average readings or exclude results that are more than 50% larger or smaller than the average of the last few results can help clean up the data.

8.4.12 Predictive Text UI

There’s a simple predictive text library (see predictor.cpp) in your exercises tree (which was originally written by Mark Hepple – thanks Mark! – and then ported to the ESP32). Currently the UI, using a touchscreen featherwing, is very basic – can you improve it? How many words can you support on the ESP without running out of memory?

8.4.13 Musical Instrument

Using the VS1053’s MIDI synthesis capability and e.g. the ESP’s touch sensors as control mechanisms many musical instruments become possible. Pick up a music player featherwing if you want to build one of these.

8.4.14 Bedtime Tracker

Eyes drooping? C programming not seeming quite as exciting as usual? Perhaps you should be getting more sleep :) Can you track your screen-off shut-eye time using a light sensor, and present the data back in an easy to interpret form?

8.4.15 Battleships Game

(This requires a pair of ESPs, both with touchscreens, so check that these are available to you before chosing!)

This is a popular game with simple UI requirements, perfect for a connected microcontroller like the ESP32.

8.4.16 DIY Alexa

Chris Greening’s ESP32-based Alexa-alike using a custom microphone board….

Links:

8.4.17 Smart Watches

The Lilygo T-Watch-2020 is a smart watch development kit based on the ESP32….

8.4.18 A Note on UIs

These libaries might be worth looking at if you want to do sophisticated UI stuff:

8.4.19 Air Quality Monitor

NOTE: we’re retiring this project, partly because it is quite challenging, especially without access to the lab, and partly because there are newer and better solutions that we haven’t had time to integrate into the course as yet. If you’re super confident and really want to give this a go, please get in touch, but if not please choose another project.

The project used three sensors to measure various aspects of Air Quality:

Air Quality assembled

This project is complex to wire up!

The old instructions, for reference:

We’ll be using two low-cost sensors to give an indication of air quality, a dust sensor (Sharp GYP2Y1010AU0F) and a Volatile Organic Compounds sensor (MQ-135). In addition we’re using a digital temperature and humidity sensor, the AM2302, a clone of the popular DHT22.

Both air quality sensors are analog sensors that are powered by 5V. They return a voltage that is between 0V-5V; the level of the voltage is proportional to the level of dust or VOC’s sensed.

Because our ADC (Analog to Digital Converter) in the ESP32 only has a limited input voltage range (0-1.5V with other ranges available) we need to scale the sensor output voltage with a potential divider.

A potential divider is made of two resistors that divide the voltage according to the formula:

    Vout = Vin ( R2 / R1+R2 )

We need to divide the Vout by 5 to scale it correctly, giving Vout approx 1.3 and Vin=5; working out the equation to fit into the available resistor values is a nice excercise for the reader.

If the reader does not feel like exercising then online calculators, e.g. this one will select resistors that minimise the error.

For further reading about the voltage divider see Sparkfun’s tutorial here.

The output voltages from these dividers go to the ESP32 – it has multiple ADC channels – we are using A2 and A3.

In our circuit, resistors R4 & R5 form a potential divider for the VOC sensor, reducing it’s range from the 0-5V the sensor outputs to 0-1.25V. The ADC has a switchable attenuator and so we can set it for 1.5V range with analogSetPinAttenuation(A0, ADC_2_5db). We’re using this range as it is more linear than the 11db range, but still gives a convenient voltage divider, and we avoid going to the end of the range where the conversion looses accuracy.

For the dust sensor, we can also use the same voltage divider to reduce the 5V output down to a measurable 1.25V, resistors R2 & R3 form this potential divider.

The AM2302 sensor uses a proprietary digital protocol but luckily a library hides that from us. We add a 10K resistor R6 to act as a pull-up on the data signals from this sensor to the microcontroller.

The other components in our circuit, R1 and C1, are needed to make the dust sensor operate correctly. For more information on the dust sensor, read the application note. Note – the capacitor C1 is an electrolytic type that has polarity – make sure you wire it in the correct way round!

a schematic

For construction, I recommend using breadboard at first, and then once the circuit is working you can convert it to soldered connections. Stick the breadboard and dust sensors down to the assembled expander board using sticky pads:

Expander with bb dust

The idea is to place the dust sensor so that the hole in the centre of the sensor is below the board, so that air (and dust) can be measured:

Dust sensor

Using the schematic as a guide, connect your component together on the breadboard. You may find the wires from the dust sensor difficult to insert into the breadboard – try twisting the strands of wire together. You may need to use a soldering iron to melt some solder onto the twisted wire – this is called ‘tinning’ the wire:

Tinned wires64

Then I suggest starting by connecting the power wires – there are three voltages in this circuit, two potential dividers and three signals going into the ESP32! With all those connections it really helps to use colour codes to help keep them all in the right places. I’ve used:

After all the sensors, wires and passive components have been placed on the breadboard it might look something like this:

Breadboard with sensors

Seriously – once you have got the circuit working – you are strongly advised to solder the wires neatly in place otherwise you will be spending a lot of time debugging loose wires!

8.4.19.1 Air Quality: Kit List

8.5 COM3505 Week 08 Notes

8.5.1 WARNINGS!!!

We’re now using LiPo batteries – please review the safety instructions above and be sure to follow them. If in doubt ask a member of staff.

8.5.2 Learning Objectives

Our objectives this week are to:

8.5.3 Assignments and Assessment

8.6 Further Reading

Following on from the work we did in the last couple of weeks (on provisioning and over-the-air updates), have a look at these:

9 Learning in the Fog – AI on the Edge

* OOOPS!!! *
This chapter isn’t finished, enter at your peril!

If there is a subject with a greater reliance on impenentrable jargon than computing, I have yet to find it. Fog? Edge? (Mist, even?) Since “cloud” has become the accepted term for distributed computing, computation that happens locally has started to acquire its own terminology building on the same metaphor. A smart phone or an IoT device is an edge device (a droplet perhaps, or edge node). The set of edge devices on a network make a fog or mist.65

We start this chapter by putting machine learning and AI in the context of edge devices in the fog. After motivation, the first issue to address is the relatively low power of computational resource available outside of the datacenter, which leads to a discussion of federated learning.

9.1 Why Learn at the Edge?66

The vast majority of current applications are designed for data processing to be performed predominantly in the cloud. There are several advantages for this model, such as:

But more recently, privacy is becoming a growing concern in the population. Data has long been seen as an important asset by large companies, but only now individuals are starting to recognise this fact about their own data. Streaming data to the cloud may not be the model of future IoT applications.

So what can we do instead to protect data privacy? One answer is to train Machine Learning models at the edge. The method with the most traction in current research is Federated Learning.

9.2 Federated Learning

Definition:

Federated learning is a machine learning setting where multiple entities (clients) collaborate in solving a machine learning problem, under the coordination of a central server or service provider. Each client’s raw data is stored locally and not exchanged or transferred; instead, focused updates intended for immediate aggregation are used to achieve the learning objective. Focused updates are updates narrowly scoped to contain the minimum information necessary for the specific learning task at hand; aggregation is performed as earlier [sic.] as possible in the service of data minimization … (Kairouz et al. 2019)

In essence, instead of sending your data to the cloud to train an ML model, with Federated Learning the ML model is sent from the server to the edge (on the client). Using the local data of the client, the model is updated to reflect a better performance on the local data. This local update of the model is then sent to the server for aggregation with updates from many other clients. The local data is discarded after the model update and never leaves the device in its raw form.

Federated Learning enhances the following principles:

Federated Learning has been receiving substantial interest recently, both from research and industry, so it’s worth exploring it in the context of IoT data. The underlying conditions for this method to work in the IoT space are that sensing devices (or nearby computing hosted on the local network, e.g. gateways or desktops) have sufficient computing resources to store a fraction of the data and perform the model updates.

In 2016, McMahan et al. introduced FL: “We term our approach Federated Learning, since the learning task is solved by a loose federation of participating devices (which we refer to as clients) which are coordinated by a central server.” (Brendan McMahan et al. 2016) An unbalanced and non-IID (identically and independently distributed) data partitioning across a massive number of unreliable devices with limited communication bandwidth are the best defining set of challenges.

9.2.1 How does FL work?

Developing a FL framework:

  1. Problem identification: The model engineer identifies a problem to be solved with FL.
  2. Client instrumentation: If needed, the clients (e.g. an app running on mobile phones) are instrumented to store locally (with limits on time and quantity) the necessary training data. In many cases, the app already will have stored this data (e.g. a text messaging app must store text messages, a photo management app already stores photos). However, in some cases additional data or metadata might need to be maintained, e.g. user interaction data to provide labels for a supervised learning task.
  3. Simulation prototyping (optional): The model engineer may prototype model architectures and test learning hyperparameters in an FL simulation using a proxy dataset.
  4. Federated model training: Multiple federated training tasks are started to train different variations of the model, or use different optimization hyperparameters.
  5. (Federated) model evaluation: After the tasks have trained sufficiently (typically a few days), the models are analyzed and good candidates selected. Analysis may include metrics computed on standard datasets in the datacenter, or federated evaluation wherein the models are pushed to held-out clients for evaluation on local client data.
  6. Deployment: Finally, once a good model is selected, it goes through a standard model launch process, including manual quality assurance, live A/B testing (usually by using the new model on some devices and the previous generation model on other devices to compare their in-vivo performance), and a staged rollout (so that poor behavior can be discovered and rolled back before affecting too many users). The specific launch process for a model is set by the owner of the application and is usually independent of how the model is trained. In other words, this step would apply equally to a model trained with federated learning or with a traditional datacenter approach.

The steps in training processes:

  1. Client selection: The server samples from a set of clients meeting eligibility requirements. For example, mobile phones might only check in to the server if they are plugged in, on an unmetered wi-fi connection, and idle, in order to avoid impacting the user of the device.
  2. Broadcast: The selected clients download the current model weights and a training program (e.g. a TensorFlow graph) from the server.
  3. Client computation: Each selected device locally computes an update to the model by executing the training program, which might for example run SGD on the local data (as in Federated Averaging).
  4. Aggregation: The server collects an aggregate of the device updates. For efficiency, stragglers might be dropped at this point once a sufficient number of devices have reported results. This stage is also the integration point for many other techniques which will be discussed later, possibly including: secure aggregation for added privacy, lossy compression of aggregates for communication efficiency, and noise addition and update clipping for differential privacy.
  5. Model update: The server locally updates the shared model based on the aggregated update computed from the clients that participated in the current round.

Typical numbers found in the deployments used at Google’s scale are presented in the table below.

Scale
Total population size 106 – 1010 devices
Devices selected for one round of training 50 – 5000
Total devices that participate in training one model 105 – 107
Number of rounds for model convergence 500 – 10000
Wall-clock training time 1 – 10 days

9.2.2 Applications of FL

Google makes extensive use of FL in the Gboard mobile keyboard and in Android Messages. Each time you type on the Gboard and it predicts the word you intend to type next based on the text context, that is used as signal to strengthen the model being used for word prediction. This model is then pulled from your phone and from other users’ phones to update a large global model that is capable of predicting the next word more robustly. Note here, not all the words you type are considered for sharing in the global model, such as usernames or passwords. But anything that falls in the general sense of using the language makes it into the global model.

Apple is using cross-device FL in iOS 13 for applications like the QuickType keyboard and the vocal classifier for “Hey Siri.” The keyboard strategy is very similar to that of Google’s. For the vocal keyword it stores samples that have a lower confidence and did not activate the device. If these are followed by a stronger example that wakes up the device, the immediately previous samples are considered for training and updating the local model.

A growing application of FL is now taking place in medicine and banking, where data is highly confidential, but actors can allow computing to happen in their system closer to the data.

9.2.3 Research challenges

Non-Identical and Independently Distributed (non-IID) phemomena in FL is usually explored in two directions – in hardware and in data:

Two bottlenecks appear in practice:

These bottlenecks can be addressed by compression using quantisation or model reduction:

9.2.4 Summary

Federated Learning is a distributed learning setting. This is built on many clients contributing updates to a shared model. A centralised server chooses a set of clients to update the model. These clients update the global model with their own local data and push that back to the server. The server aggregates the updates from the contributing clients and pushes the updated model for many more clients to use (even if they don’t participate in the update).

9.2.5 Hands on experience

This course has no lab associated to the theoretical concepts. However, there are many resources online if you want to start playing with FL. The more popular repository is FedML. This has a set of examples and benchmarks you can explore independently. An dedicated implementation for IoT devices is proposed here https://github.com/FedML-AI/FedML-IoT.

9.3 COM3505 Week 09 Notes

TODO Shift forwards weeks 8/9/10/11/12

9.3.1 Learning Objectives

Our objectives this week are to:

TODO

TODO - incoming, for advanced projects? - WolfSSL / SSH server - LittleVGL

9.3.2 Assignments and Assessment

Your GitLab repository will now contain:

Your tasks for this week are:

9.3.3 Coding Hints / Configuration Instructions

9.3.4 The IBM LMIC (LoraMAC-in-C) Library

The unPhone LoRaWAN module requires a modified version of the LMIC library, which is now installed on the lab machines. If you’re using your own machine, you need to copy it into your Arduino/libraries directory. I’ve pushed a copy of the LMIC tree to lmic in your repositories for this purpose.

9.3.5 Talking to TTN, Triggering IFTTT

TODO some of this material (the IFTTT) is now in week 7

Create an account on https://ifttt.com and download the app for your phone, log into your new IFTTT account on the app. (If you don’t have a smart phone or prefer to use a computer, this will all work on a web browser instead, though you will then need to use twitter or email integration to send notifications.)

TODO the below IFTTT is a bit out of date

Create a New Applet in IFTTT, click on the +this and choose webhooks. Click on the blue box Receive webhook and enter an event name like unphone. Click on the create trigger button.

Now click on the +that part of the screen to configure the action. Search for notifications and click the bell icon (if you’re using the smart phone app; if not, click on e.g. gmail or twitter). Choose the Send a notification from the IFTTT app option, and click on the Create action button. Click the finish button to create the applet.

Now click on the webhooks word where it is to the right of My Applets – or you can navigate to the page https://ifttt.com/maker_webhooks. Then click on the documentation button to show your personal key – leave this page open as you will need this to connect to the things network.

Now go to https://www.thethingsnetwork.org/ and sign up for an account. Once that is created and you have logged in, click on your account in the top right of the screen and go to console. Click on applications and then add application.

Give your application a name like myapp or something more imaginative if you like. You can add a description to help you remember what the application does. The EUI is created automatically and the handler should be set to ttn-handler-eu. Hit the add application button.

Now you should be looking at the overview page for your new application – go to the devices tab and click on register device. Give your device a Device ID like myunphone or better, and click on the icon in the Device EUI field to have it generated for you. The other fields will also be generated for us. Click on the register button to create your device.

Now we need to set our device to use the ABP activation method – so go into the settings and choose ABP as the Activation Method and then click Save.

Now we need to copy the three keys from the ttn device overview into the firmware for the device (using LoRaWANThing.ino in your repository). Next to Network Session Key click on the <> button to see the key in C format. Copy this array of numbers into the LoRaWANThing.ino sketch where the NWKSKEY is defined, replacing the default array of zeros. It looks like this:

static const PROGMEM u1_t NWKSKEY[16] =
  { 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0 };

Repeat this for the App Session Key, copying this into the APPSKEY definition. Finally do the same for the Device Address – except this time just copy it without clicking on the <> button. For example:

static const u4_t DEVADDR = 0x00000000 ;

You should see that the keys you are replacing have the same formats as the keys you have created. With these three keys copied into your sketch it should now be able to send a message using LoRaWAN.

The last step is to join the application receiving data on the things network with the IFTTT applet. Click on the name of your application (where it appears near the top of the page in the navigation breadcrumbs) to get to the application overview page on the things network console webpage. Click on integrations, then add integration, then choose IFTTT maker. For process ID you can give any name. For event name give the exact name you used to create the applet in IFTTT – for example unphone. Now you need to copy the key from the IFTTT page we opened earlier into your TTN integration Key field.

Once you click Add integration (and when you’ve burned LoRaWANThing to your unPhone) – your phone’s IFTTT app (or email or twitter according to what you chose) should notify you every minute or so as messages are sent from your code via LoRaWAN to a nearby gateway, then to a ttn application, via an integration to the IFTTT service, and from their to the TTN app on your phone.

Simple, right?

10 WaterElves, Gripples and Fish Poo: IoT Case Studies

* OOOPS!!! *
This chapter isn’t finished, enter at your peril!

TODO

Nature food paper: (Edmondson et al. 2020)

10.1 COM3505 Week 10 Notes

10.1.1 Learning Objectives

Our objective this week are to get cracking on the projects, continuing the course theme of learning by doing. Three more weeks and you’ll have had a real taste of creating a new IoT device, from breadboard to prototype, and be all set to become a practising IoT engineer :)

10.1.2 Assignments and Assessment

11 IoT 2021 Review

* OOOPS!!! *
This chapter isn’t finished, enter at your peril!

11.1 Themes

The themes the module covers include:

11.2 COM3505 Week 11 Notes

11.2.1 Learning Objectives

Our objectives this week are to:

11.2.2 Assignments and Assessment

Projects: two more weeks left! We recommend that you:

11.3 Further Reading

On application-layer communication protocols:

Cautionary tales:

12 Gateway to the Future

* OOOPS!!! *
This chapter isn’t finished, enter at your peril!

Pandoc

Wikipedia: Markdown

12.1 COM3505 Week 12 Notes

12.1.1 Learning Objectives

Our objectives this week are to:

12.1.2 Assignments and Assessment

You’ve almost finished!

NOTE: NC PC is the North Campus PC lab; enter from Broad Lane, turn right through the gates and it is on the ground floor of the big grey portacabin on your right. (Allow lots of travel time!)

12.2 Colophon

Bibliography

Adelantado, F, X Vilajosana, P Tuset-Peiro, B Martinez, J Melia-Segui, and T Watteyne. 2017. Understanding the Limits of LoRaWAN.” IEEE Commun. Mag. 55 (9): 34–40.
Arduino. 2017. Arduino IDE Guide.” https://www.arduino.cc/en/Guide/Environment. https://www.arduino.cc/en/Guide/Environment.
Ashton, Kevin. 2011. That ‘internet of things’ thing.” RFiD Journal 22 (7).
Banzi, Massimo, and Michael Shiloh. 2014b. Getting Started with Arduino: The Open Source Electronics Prototyping Platform. Maker Media, Inc.
———. 2014a. Getting Started with Arduino: The Open Source Electronics Prototyping Platform. Maker Media, Inc. https://market.android.com/details?id=book-Xd3SBQAAQBAJ.
Barragán, H. 2004. Wiring: Prototyping physical interaction design.” Interaction Design Institute, Ivrea, Italy. https://scholar.google.ca/scholar?cluster=4073615779562206947&hl=en&as_sdt=0,5&sciodt=0,5.
———. 2016. The untold history of Arduino.” Luettavissa: Https://Arduinohistory. Github. Io/Luettu. https://scholar.google.ca/scholar?cluster=15872095708386818356&hl=en&as_sdt=0,5&sciodt=0,5.
Bassi, Alessandro, Martin Bauer, Martin Fiedler, Thorsten Kramp, Rob Van Kranenburg, Sebastian Lange, and Stefan Meissner. 2013. Enabling things to talk.” Designing IoT Solutions with the IoT Architectural Reference Model, 163–211.
Blenn, Norbert, and Fernando Kuipers. 2017. LoRaWAN in the Wild: Measurements from The Things Network.” arXiv [Cs.NI], June. http://arxiv.org/abs/1706.03086.
Brendan McMahan, H, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. 2016. Communication-Efficient Learning of Deep Networks from Decentralized Data,” February. http://arxiv.org/abs/1602.05629.
Community, Esp32. 2017. Esp32 Forum.” https://esp32.com/. https://esp32.com/.
Cunningham, H. 2012. Agile Research.” ArXiv e-Prints http://arxiv.org/abs/1202.0652v1 (February). http://adsabs.harvard.edu/abs/2012arXiv1202.0652C.
Denning, Dorothy E, and Peter J Denning. 1977. Certification of programs for secure information flow.” Commun. ACM 20 (7): 504–13. https://doi.org/10.1145/359636.359712.
Dhanjani, Nitesh. 2015. Abusing the Internet of Things: Blackouts, Freakouts, and Stakeouts. “O’Reilly Media, Inc.”
Dickens, Charles. 1877. A Tale of Two Cities: And, Great Expectations. Lee; Shepard. https://play.google.com/store/books/details?id=opBBAQAAMAAJ.
Dizdarević, Jasenka, Francisco Carpio, Admela Jukan, and Xavi Masip-Bruin. 2019. A Survey of Communication Protocols for Internet of Things and Related Challenges of Fog and Cloud Computing Integration.” ACM Comput. Surv. 51 (6): 1–29. https://doi.org/10.1145/3292674.
Doctorow, Cory. 2012. Pirate cinema. Macmillan. https://freekidsbooks.org/wp-content/uploads/2020/01/FKB-Stories-Cory_Doctorow_-_Pirate_Cinema.pdf.
———. 2019. Unauthorized Bread.” https://craphound.com/category/unauthorizedbread/. https://craphound.com/category/unauthorizedbread/.
Doukas, Charalampos. 2012. Building Internet of Things with the Arduino. USA: CreateSpace Independent Publishing Platform.
Edmondson, Jill L, Hamish Cunningham, Daniele O Densley Tingley, Miriam C Dobson, Darren R Grafius, Jonathan R Leake, Nicola McHugh, et al. 2020. The hidden potential of urban horticulture.” Nature Food 1 (3): 155–59.
Fraser, Quentin Stafford-. 1995. Trojan Room Coffee Pot Biography.” https://www.cl.cam.ac.uk/coffee/qsf/coffee.html. https://www.cl.cam.ac.uk/coffee/qsf/coffee.html.
Greenfield, Adam. 2017. Rise of the machines: who is the ‘Internet of things’ good for? Guardian, June. https://goo.gl/uIUCrD.
Grover, Sarthak, and Nick Feamster. 2016. The internet of unpatched things.” Proc. FTC PrivacyCon. https://www.ftc.gov/system/files/documents/public_comments/2015/10/00071-98118.pdf.
Grover, Siddharth. 2017. The Internet of Things (2016).” International Journal of Computer Science and Engineering. https://doi.org/10.14445/23488387/ijcse-v4i8p101.
Hunn, Nick. 2018. British Smart Meters cost £28 million EACH.” https://www.nickhunn.com/british-smart-meters-cost-28-million-each/. https://www.nickhunn.com/british-smart-meters-cost-28-million-each/.
Kairouz, Peter, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, et al. 2019. Advances and Open Problems in Federated Learning,” December. http://arxiv.org/abs/1912.04977.
Kolban, Neil. 2017. Kolban’s Book on ESP32.
Kranenburg, Rob van, and Alex Bassi. 2012. IoT Challenges.” Proc. Int. Wirel. Commun. Mob. Comput. Conf. 1 (1): 9. https://doi.org/10.1186/2192-1121-1-9.
Kurniawan, Agus. 2016. Smart Internet of Things Projects. Packt.
Li, Y, and H Cunningham. 2008. Geometric and Quantum Methods for Information Retrieval.” SIGIR Forum 42 (2): 22–32. http://www.sigir.org/forum/2008D-TOC.html.
Loftus, Jack. 2011. Dear Fitbit Users, Kudos On the 30 Minutes of ‘Vigorous Sexual Activity’ Last Night.” Gizmodo.
MacDermott, A, T Baker, and Q Shi. 2018. Iot Forensics: Challenges for the Ioa Era.” In 2018 9th IFIP International Conference on New Technologies, Mobility and Security (NTMS), 1–5.
Machine, Coke. n.d. The "Only" Coke Machine on the Internet.” https://www.cs.cmu.edu/~coke/history_long.txt. https://www.cs.cmu.edu/~coke/history_long.txt.
Margolis, Michael. 2011. Arduino Cookbook: Recipes to Begin, Expand, and Enhance Your Projects. “O’Reilly Media, Inc.”
McEwen, Adrian, and Hakim Cassimally. 2013. Designing the Internet of Things. John Wiley & Sons.
Monk, Simon. 2013. Programming Arduino Next Steps: Going Further with Sketches. McGraw Hill Professional.
Naik, N. 2017. Choice of effective messaging protocols for IoT systems: MQTT, CoAP, AMQP and HTTP.” In 2017 IEEE International Systems Engineering Symposium (ISSE), 1–7. https://doi.org/10.1109/SysEng.2017.8088251.
NERC. 2016. NERC Guidance on the Safe Use of Lithium Batteries.” http://www.nerc.ac.uk/about/policy/safety/procedures/guidance-lithium-batteries/. http://www.nerc.ac.uk/about/policy/safety/procedures/guidance-lithium-batteries/.
Nold, Christian, and Rob van Kranenburg. 2011. The Internet of People for a Post-oil World. Lulu.com.
Nordrum, Amy. 2016. The internet of fewer things [News].” IEEE Spectrum. https://doi.org/10.1109/mspec.2016.7572524.
O’Muircheartaigh, Fionan. 2013. The Smart City: Using Technology to Reduce Congestion in London.” https://www.sustainablebusinesstoolkit.com/the-smart-city-using-technology-to-reduce-congestion-in-london/. https://www.sustainablebusinesstoolkit.com/the-smart-city-using-technology-to-reduce-congestion-in-london/.
O’Reilly, Tim, and Cory Doctorow. 2015. Opportunities and Challenges in the IoT. O’Reilly Media, Inc. https://www.oreilly.com/library/view/opportunities-and-challenges/9781492048220/.
Perzanowski, Aaron, and Jason Schultz. 2016. The End of Ownership.” https://mitpress.mit.edu/books/end-ownership. https://mitpress.mit.edu/books/end-ownership.
Pfister, Cuno. 2011. Getting Started with the Internet of Things: Connecting Sensors and Microcontrollers to the Cloud. “O’Reilly Media, Inc.”
Romkey, J. 2017. Toast of the IoT: The 1990 Interop Internet Toaster.” IEEE Consumer Electronics Magazine 6 (1): 116–19. https://doi.org/10.1109/MCE.2016.2614740.
Rubell, Brent. 2018. Welcome to Adafruit IO.” https://learn.adafruit.com/welcome-to-adafruit-io; Adafruit. https://learn.adafruit.com/welcome-to-adafruit-io.
Schneier, Bruce. 2017. Click Here to Kill Everyone.” NY Mag, January. http://nymag.com/selectall/2017/01/the-internet-of-things-dangerous-future-bruce-schneier.html.
Schwartz, Marco. 2016a. Home Automation with the ESP8266: Build Home Automation Systems Using the Powerful and Cheap ESP8266 Wifi Chip. CreateSpace Independent Publishing Platform.
———. 2016b. Internet of Things with ESP8266. Packt Publishing Ltd.
Sivaraman, V, H H Gharakheili, A Vishwanath, R Boreli, and O Mehani. 2015. Network-level security and privacy control for smart-home IoT devices.” In 2015 IEEE 11th International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob), 163–67.
Slama, Dirk, Frank Puhlmann, Jim Morrish, and Rishi M Bhatnagar. 2015. Enterprise IoT: Strategies and Best Practices for Connected Products and Services. “O’Reilly Media, Inc.”
Stallman, Richard. 2002. Free Software, Free Society: Selected Essays of Richard M. Stallman. Lulu.com.
Sterling, Bruce. 2014. The Epic Struggle of the Internet of Things. Strelka Press. https://www.amazon.co.uk/Epic-Struggle-Internet-Things-ebook/dp/B00N7EKIJ4.
Tablan, V, K Bontcheva, I Roberts, and H Cunningham. 2014. Mı́mir: an Open-Source Semantic Search Framework for Interactive Information Seeking and Discovery.” Journal of Web Semantics. https://doi.org/10.1016/j.websem.2014.10.002.
Tablan, V, I Roberts, H Cunningham, and K Bontcheva. 2013. GATECloud.net: a Platform for Large-Scale, Open-Source Text Processing on the Cloud.” Philos. Trans. R. Soc. Lond. A 371 (1983).
Thakur, Manoj. 2016. Zero to Hero ESP8266. Circuits4you.com.
The Economist. 2019. How the world will change as computers spread into everyday objects.” https://www.economist.com/leaders/2019/09/12/how-the-world-will-change-as-computers-spread-into-everyday-objects; The Economist. https://www.economist.com/leaders/2019/09/12/how-the-world-will-change-as-computers-spread-into-everyday-objects.
The Things Network. 2018. The Things Network Manifesto.” https://github.com/TheThingsNetwork/Manifest. https://github.com/TheThingsNetwork/Manifest.
Upton, Eben, and Gareth Halfacree. 2014. Raspberry Pi user guide. John Wiley & Sons.
Ward, Mark. 2001. BBC News.” BBC. http://news.bbc.co.uk/1/hi/sci/tech/1264205.stm.
Warden, Pete, and Daniel Situnayake. 2019. TinyML: Machine Learning with TensorFlow Lite on Arduino and Ultra-Low-Power Microcontrollers. “O’Reilly Media, Inc.” https://play.google.com/store/books/details?id=tH3EDwAAQBAJ.

  1. Wilgengebroed, via Wikimedia.↩︎

  2. Reading this on GitLab.com? There’s also a PDF version or a GitLab Pages version.↩︎

  3. In February 2021, for example, The Register reported that Amazon-owned “smart” doorbell maker Ring “is suffering a major outage with many of its video doorbells effectively dead, turning smart homes into very dumb ones.”↩︎

  4. There are many other terms we can correctly apply to this field, and we’ll look at these later on as part of putting our work into its historical, social and economic contexts.↩︎

  5. Airman’s Odyssey, Antione de St. Exupery.↩︎

  6. From Socks Studio.↩︎

  7. See an error, omission or something that could be improved? Please raise an issue or make a pull request.↩︎

  8. Guess which chapter this is. Did you say “one?” Pat yourself on the back, you’re getting the hang of this IoT stuff already!↩︎

  9. Your answer better than ours? Well done! Send me a pull request if you have time.↩︎

  10. For this iteration pretty much everything is now open and freely accessible in its up-to-the minute form (apart from the exam questions!).↩︎

  11. All statements in this document are true to the best of our knowledge, except those that aren’t.↩︎

  12. In 2021 all of these are on-line and asynchronous.↩︎

  13. Or an equivalent emulation environment like Cygwin, WSL or a VM or a container.↩︎

  14. Emacs? What’s that?↩︎

  15. Linus Torvalds quipped “I’m an egotistical bastard, and I name all my projects after myself. First ‘Linux,’ now ‘git.’”↩︎

  16. If you’re reading this as a GitLab ‘.mkd’ file (e.g. on gitlab.com or a checked-out repository), citations don’t link anywhere. Try the PDF or the GitLab Pages version to find the references.↩︎

  17. SDK: Software Development Kit; IDE: Integrated Development Environment.↩︎

  18. Wikimedia.↩︎

  19. Recently the sucessor computer to the PDP10, the PDP11, has been completely emulated on an ESP32!↩︎

  20. ‘The use of “hacker” to mean “security breaker” is a confusion on the part of the mass media. We hackers refuse to recognize that meaning, and continue using the word to mean someone who loves to program, someone who enjoys playful cleverness, or the combination of the two.’ (Stallman 2002)↩︎

  21. NDA: Non-Disclosure Agreement. If you’re considering signing one of these read it VERY carefully!↩︎

  22. Python’s release policy: if it isn’t incompatible with all previous and future versions, add some library dependencies to abandoned projects that fail to compile on all target architectures, a new virtualisation environment to deal with different Python versions and a new language feature that gets activated at random depending on what day of the week it is and what flavour of tea you’re drinking.↩︎

  23. This will be called something different (e.g. export.bat) on non-unix platforms.↩︎

  24. When this happens, deleting the relevant tree from ~/.espressif/python_env and re-running esp-idf/install.sh can help.↩︎

  25. During our tale Espressif’s love of easily understandable product names and narrative coherence will often be in evidence; a recent example is an apparent intention to name an ESP32 successor that uses a completely different instruction set and architecture – RISC V – the ESP32-C3. Good to see them sticking to their guns and being just as thoroughly mystifying for new stuff as they were for old.↩︎

  26. Python undoubtedly has many fine qualities. In fact I wish I knew it better, and used it more. It also, unfortunately, has its own particular VERSION HELL, which does tend to complicate life a little here and there.↩︎

  27. No! I’m not coming! You’ll never take me a….↩︎

  28. Subsequent lectures will be provided by our backup lecturer.↩︎

  29. Don’t worry, they’re quite sane.↩︎

  30. So far.↩︎

  31. If you try Arduino Create (or Arduino CLI) and have spare time, why not fork this repository on GitLab, add a section about how it went and make a pull request?↩︎

  32. This was true in late 2020 at least. As the release cycles of the Arduino core and ESP IDF progress PlatformIO support will likely stabilise again, so it is probably worth a try, especially if you like VSCode, with which it integrates well. (Stop press: Feb 2021 4.2 release looks good!)↩︎

  33. This is quite an old release but is the best supported by the Arduino core (as of December 2020).↩︎

  34. Actually the names seem to relate to some change in responsibilities; delve here if you care.↩︎

  35. Why the wierd checkout numbers? These are git commit hashes, which identify unique commits in the version history. We need them because of version hell! (Less flippantly, there are only a subset of versions where IDF and Arduino core match up, especially if we want to use recent developments.)↩︎

  36. This section contributed by Simon Fish.↩︎

  37. Guy says you’re still going to need to wear the silly hat though. At least I think that’s what he said. I always listen very carefully.↩︎

  38. The magic.sh script supports an argument copy-me-to which copies and renames a firmware tree. E.g. cd HelloWorld; ./magic.sh copy-me-to ~/NewExample.↩︎

  39. Hardware 1 is the electronics lab intro sheet – see last week’s notes. Because lab access is restricted you’re likely to be doing this one (“hardware 2”) before hardware 1. Don’t worry, but do have a read of Hardware 1 first if you get time.↩︎

  40. Or feel free to port them! And send me a pull request!↩︎

  41. From Kornakproto.↩︎

  42. This section contributed by Gareth Coleman.↩︎

  43. This section contributed by Gareth Coleman.↩︎

  44. You may not have done hardware 1 yet of course; don’t worry, we’ll get to it later, with luck!↩︎

  45. If you damage your hardware beyond repair, it turns from an interesting electronic gadget into an inanimate lump: it becomes, in other words, pretty much like a brick. Hence the colloquial verb “to brick,” which bears an intimate relationship with the term “FUBAR” in the engineer’s vocabulary. It is recommended to avoid exploring these in practice as far as you can manage.↩︎

  46. From Luca Dentalla.↩︎

  47. From Andreas Schweizer.↩︎

  48. If you’re using the magic.sh script, this can set up environment variables for you and call idf.py to erase flash like this: magic.sh idf-py erase_flash.↩︎

  49. Hitting it won’t help of course, but might make you feel better. Temporarily: when Andy asks you to explain why you need a new one the happy feeling will likely evapourate. You could try blaming it on the silly hat that Guy made you wear?↩︎

  50. This material is derived from a lecture by Gareth Coleman. Thanks G!↩︎

  51. In fact we’re simplifying history a bit if we say that sensors were originally designed to mesh the physical and digital, as the original electrical sensors pre-date the use of numerical information processing in digital computing machines. Early examples included thermistors and microphones, in the 1800s. In modern usage, however, we’re almost always interested in turning whatever electrical signal is generated by our sensor into a digital representation of that signal, in order to manipulate it in the digital realm that underlies almost all contemporary computational processes.↩︎

  52. From rpi.edu/dept/phys.↩︎

  53. From electronics.stackexchange.com.↩︎

  54. From Sparkfun.↩︎

  55. From: Sparkfun.↩︎

  56. Thinking about the vunerability of car control systems reminds me of a particularly hair-raising landing in Athens a couple of decades ago, in an Airbus A320, one of the first planes to use a “fly by wire” digital control system. The old Athens airport was beside the sea and used to suffer from swirling cross-winds when a storm blew up, and on this occasion we went round three times, aborting each of the first two runs due to massive turbulence. The passenger beside me had to make use of the vomit bag, and none of us felt very confident that we were going to end up safe and sound on the ground any time soon. To reassure us one of the crew came on the intercom and said “Don’t worry, we’re quite safe, the plane is controlled by computer!” At which point the anxiety levels of the computer scientists on the flight went up around a thousand percent :) When we had eventually landed and were waiting for baggage, I caught sight of the pilot. (It was an old airport, and the crew picked up their bags from the same place as the passengers.) The pilot’s hair was plastered to his forhead by sweat, and he was dragging heavily on a cigarette. Luckily the control systems did their job that evening, but it was still a pretty hairy time.↩︎

  57. Wikipedia: in English the most familiar expression of this concept is by Isaac Newton in 1675: “If I have seen further it is by standing on the shoulders of Giants.”↩︎

  58. Aren’t deep learning algorithms (and the neural network models that underlie them) more than just “applied statistics?” Maybe. It is true that neural net architectures are inspired by human brain biology, but also true that in their typical current form they have little biological plausibility: the neurons in our heads are more complex, more analogue, more connected. Our brains are also at the center of hugely complex organisms with vast input ranges and rich feedback loops – and wants, desires, needs… The neural nets that are used for deep learning are good at generalising over a large amount of data, and perhaps they are better at doing that than other more obviously statistical methods. I suspect, however, that the key to the success of their modern applications are the truly enormous datasets that have been created (by human beings) as part of our rush to move all types of information persistence and sharing onto the web. They are learning count-based abstractions over human authored data, rather than modelling the processes by which humans originate that data in the first place.↩︎

  59. Until around half a dozen years ago my day job was computational infrastructure for the extraction of information from human language text, which, depending on which funding agency or research lab was paying the bills, was either down-to-earth software engineering or flight-of-fancy pioneering AI. There is an argument that the crossover space between literary and technical competence is what allows the EU to gain competitive advantage and still avoid falling foul of World Trade Organisation anti-subsidy rules: people like me write the science fiction wrapper that allows states to fund “research” which might otherwise just be “development.” One unfortunate downside of this arrangement is that those who have only the literary (or marketing) competence and no real technical skill or commitment can also sometimes slip in to join the party, with the result that the core pre-requisites of science and engineering (evidence, repeatability, transparency) sometimes become lip-service window dressing, and the corporatisation of university research takes another step towards the inconsequential mirroring of advertising fantasy. But I digress!↩︎

  60. A decade ago I was luck enough to spend a short year working as ANR Chaire d’Excellence at the Internet Memory Foundation in Paris with Julien Masanès and colleagues. They used to run the UK national archives web provision, so for several years if you went to a UK government webpage that no longer existed you would be forwarded to an archival copy served out of a datacenter in Amsterdam managed by French sysadmins, partly paid for by EU high tech research budgets. They worked on early big data infrastructure and analytics using map reduce over HDFS in Hadoop, sucked down from the web by their custom crawler. Ahead of their time!↩︎

  61. Jamie Zawinski’s website is so beautiful! I want one!↩︎

  62. This section contributed by Valentin Radu.↩︎

  63. Much of the project circuitry and their descriptions were developed by Gareth Coleman.↩︎

  64. Image from thesolderblog.blogspot.co.uk.↩︎

  65. It works for me. I live in a country where rain is a way of life.↩︎

  66. These sections on machine learning contributed by Valentin Radu.↩︎