Mi Casa su Botnet?

Learning the Internet of Things with
WaterElf, unPhone and the ESP32

Hamish Cunningham
(with Gareth Coleman and Valentin Radu)

Iteration 4 (Spring 2021, 0.1037.2021-07-07; PDF version).
Copyright © Hamish Cunningham. Licence: CC-BY-SA-NC 4.0.

1 Hope, Technology and Heath Robinson

1

These notes describe COM3505 (the Internet of Things) iteration 4, running in Spring 2021. Each chapter begins with general discussion, history or theory, then finishes with instructions for a week’s worth of practical work. Although later material depends to varying degrees on preceding chapters, we have tried to make the order of the practical work as flexible as possible so that you can learn in whatever sequence is convenient for you (partly because we are expecting less access to laboratory time this year due to the pandemic, and some students will be remote working).

The practical work in this chapter and next, though, covers enrolling on the course and setting up your development environment, so please prioritise its early completion.

So, what is the IoT and where does this course fit in? Read on…2

1.1 Welcome to the Toy Shop!

We’re six month late, but you made it! You’ve worked hard for years to get this far, and now you get to play :)

As computer scientists and software engineers we spend much of our time with machines whose secrets are hidden beneath multiple abstraction layers: BIOS, bootloader, operating system, programming language, library API, protocol definition, SDK, etc. etc. These layers allow us to build systems of incredible power and sophistication, but they also (intentionally) obscure the operation of the underlying hardware engine. This course is a chance to rip the top off the box of tricks, hotwire the starter motor and delve deep into the grungey secrets of the electronic wizardry that makes our discpline possible.

The course is also a way to understand how the latest great wave of technological transformation is streaming into our lives, enabled by the combination of tiny connected microcontrollers, cyber-physical systems and ubiquitous networking. This transformation is driven by the same logic as the explosive consequences of TCP/IP and HTML, but if the last 20 years were about the web, then the next 20 will be about making. Just as always-on connectivity and decentralised production in the virtual world made revolutions in creating, sharing and consuming on-line, now the same changes are starting in the world of manufacturing, and the consequences are likely to be massive. (There’s perhaps an 80-20 ratio between economic activity devoted to atoms, or physical processes, in comparison to bits, or informational processes.) In the IoT there’s still a space to get in on the ground floor, and learn how to make with the foundational hardware of the next wave.

Hang on to your hat; the future is a whirlwind!

1.1.1 What’s the Catch?

Actually there are two :(

First, you’re going to be programming a tiny little device that has, in comparison to that shiney laptop on your desk or that smartphone in your pocket, close to zero computational resource. There’s no paging, for example, so in the sense of time sharing (first developed in the 1960s) our devices don’t even run an operating system! This can be a challenge. Be prepared.

Second, you’re going to need to install an SDK of one sort or another on your own machine. Partly this is because the pandemic has forced us to reduce the amount of on-campus time you will have to work with University-supported machines; partly it is because you need to be able to spend concentrated hours playing with the beasts, and there’s nowhere better than your own computer for experimenting.

To help you with these challenges we will:

If you don’t have a machine that you can use for this and need something cheap, you might try a Raspberry Pi 4 or a Pi 400 (preferably the 8GB models), both of which are very capable of running the basic IoT SDK that we work with.

If you’ve already got a machine running MacOS or Windoze (or OS/360) don’t panic:

And: look on the bright side, the more you do, the more you learn :)

1.1.2 Catch #3: ‘Click Here to Kill Everybody?’

Actually I lied: there are three catches:

As the perpetual beta style of service-oriented computing moves into physical spaces, we’re all at the mercy of suppliers’ willingness to use us as guinnea pigs – only this time our homes and our clothes and our handbags are the venue of choice.3 Will the future of the IoT be dominated by botnets and fraud? Or will we use the new machines to help face up to the massive survival challenges that a finite world poses to an expansionist system? I think you’re part of the answer.

1.1.3 IoT: From the General to the Specific

As humanity’s latest pandemic swirls across the world, wreaking havoc on the poor, the old and the unlucky (and giving the powerful their latest excuse for moulding us all into yet more profitable shapes), some small comfort may be found in the contemplation of the intricate. The history of machines is part of what distinguishes our species: cooperating to create the preconditions of our existence using ever more sophisticated engines.

Computational engines have brought a new level of generality to the picture: Turing machines exemplified by von Neumann architectures and running on some millions or billions of transistors have become a universal mechanism for information processing and automatic control, and collectively we are deploying more and more compute power, memory and storage at a truly astonishing rate. The extreme expression of this tendency is cloud computing, to the degree that it is now hard to imagine a compute problem which would exceed the combined power of the modern cloud.

At the opposite extreme from the general purpose computer are those engines that are tailored (ever more precisely) to a specific purpose. These machines, in the limit, form minimal solutions to complex problems. There are few fields that exhibit this minimalism as purely as what is currently often called the Internet of Things4. As computation has permeated almost all corners of technology a particular class of problems has become prominent, where we seek not generality but to consume the smallest amount of resource that can possibly succeed. The use cases for a general purpose machine are always expanding and we can always, potentially at least, justify the devotion of more resource to their construction. The uses case for the Internet of Things are inextricably tied to objects whose sizes, costs and operational environments present a constant resource challenge, a constant downwards pressure on compute cycles and power consumption.

The solutions that we build in the IoT are beautiful in their intricate simplicity, minimal expressions of the vast power of our universal machines, and their “…perfection is attained not when there is nothing more to add, but when there is nothing more to remove”5.

Of course we don’t always achieve minimalist elegance on our first tries. Heath Robinson captured the feeling I have when I look at some of my previous attempts at IoT devices:

6

This course is about transforming our computational selves from high priests of the general purpose to paragons of minimalism (probably via the joyfully messy intermediary of Heath Robinson).

Enjoy the ride!

1.1.4 Hope

What if we wish not simply for the comfort offered by contemplating (and, later, controlling) the miraculous intricacies of task-specific computational engines, but also for hope?

Take a minute. Switch off your phone. Look at the objects around you, remember the food on your plate at your last meal, feel the warmth of your clothing, the security of the building you are sitting in. The economic system that creates all of these things, all of these preconditions for your existence, is driven by a single imperative that equates profit with value. When that equation is combined with the vast and impersonal forces of transnational corporate competition the results are well known. We use our atmosphere as a sink for the carbon remains of an aeon of decayed flora and fauna. We use each other as expendable “human resources.” We crowd sick animals together in zoonotic melting pots and express surprise at the apocalyptic consequences.

In this somewhat gloomy context there are nevertheless several glimmers of hope connected to the IoT… but allow me, dear reader, a little suspense in my narrative: we’ll come back to how to save the world after we’ve done some of the spade work. Watch this space :)

1.2 How the Course Works

The course has two sides, practical and theoretical. The practical work is split in two halves:

  1. up to week 7 we program the basic functions of an IoT device, step by step
  2. from week 8 we choose a project from a range of options and work on it intensively until week 12

(The project, and hardware kit, is yours to keep at the end of the course.)

We provide information via three main channels:

(There is also a small amount of information on the Blackboard page for the course, and later in the term there may be hands-on sessions available in the Diamond Electronics Lab (DIA 2.02) on Monday afternoons, though due to the pandemic these are restricted in number.)

There is a notes section for each week of the course8. Each chapter ends with a “week N course notes” section describing our objectives for this part, and specifiying practical work that we want you to complete. By and large you can read the first parts of the chapters in lean-back mode: they are discursive, and intend to give a flavour of the field and its historical and technical contexts. The latter parts of the chapters are more directive, and you’ll need to sit at a keyboard and follow instructions; the intent is to lead you to practical outcomes and running code.

There is example code in the exercises tree of the course materials repository. This code covers many of the tasks that we ask you to complete week by week. You can, of course, copy the answers without trying to come up with your own solution, but you will learn little by doing so. The recommended method is to implement your own answer to the exercises, then go to the relevant example code and compare it with yours.9

This is a 10 credit (level 3) module; this means about 100 hours work. It is taught over one semester, so it averages out at about 7-8 hours per week. (Depending on your existing skills and previous experience you may need more or less time.)

1.2.1 Main Changes from Iteration 3

We needed to change quite a lot from last time the course ran! The 2021 version relies on 3 Ts: Toys@home, lots of Text, and Tech support :)

1.2.2 Assessment

Threshold and grading works by separating the process of passing the course from the process of attaining a good grade. The threshold assessments check that you have a basic understanding of all the material; the grading assessment evaluates the quality of that understanding in depth.

The course is assessed by two practicals (lab1, lab2) and a (multiple choice) exam.12 Lab1 happens at week 7 and lab2 at week 12 (and the exam during the standard exam period at the end of the semester). (A mock exam during term will allow you to practice the material.)

You must pass lab1 and the exam in order to pass the course. This will give you a score of 40%. In order to score more than 40% you must also complete lab2, the project.

Lab1 and the exam are assessed as pass/fail only (and no score is given for lab1). You will have a week to complete lab1 and 24 hours within which to sit the exam. A pass in both contributes 40 marks to your final result.

Lab2 is a project which students spend several weeks completing at the end of the course. The assessment is open-ended and qualitative in nature. It is marked out of 100 and contributes a maximum of 60 marks to your final score.

This is an advanced course and the project assessment in particular is intentionally open-ended. There is no single right answer! If you’re unsure about how much effort to expend the rule of thumb is to put in as many hours as is reasonable for a 10 credit module (see above; perhaps 10 hours per week).

We’ll give more detailed guidance in relation to each of the assessments as they occur; the things to remember to begin with are that:

1.3 COM3505 Week 00: Preliminaries

This section describes stuff to do before you start the course:

1. We will communicate with you using Git, both to deliver course materials and assignments to you, and for you to submit coursework to us. Therefore you are not fully enrolled for the course until you set up an appropriate git repository and give us the details. Do this ASAP!

2. Spend some time getting to know git and the other tools we’ll be using. (The Linux command line is powerful, elegant and, to newcomers, challenging. Using git from the command line adds another set of challenges. A bit of preparation will help!)

3. Refresh your memory of the Diamond Electronics lab safety protocols. You cannot participate in lab sessions without having completed the safety work.

Details follow.

1.3.1 Setting up your Git Repository

First register an account on GitLab.com if you don’t have one already. Use your Sheffield email address (ending in @sheffield.ac.uk) to register.

Second, create a private repository (project) there called com3505-student-2021:

(Make sure to select the Initialize repository with a README option.)

Third, go to the repository members settings…

…and add hamishcunningham

…as a project maintainer:

Lastly submit your GitLab user name to our server (see next section).

1.3.2 Tell us your Account User Name on GitLab

Your user name (or ID for our purposes) is the name you log in with, and appears in the URL of the site when you’re logged in. For example, Hamish’s user name / ID is hamishcunningham and his public GitLab projects appear at gitlab.com/hamishcunningham (and GitHub ones here , and SourceForge here…).

We need to know your username and match it up with your student record so that we can pull the coursework that you commit there. Tell us your username by making an HTTP GET request to the server that lives at https://HOSTNAME.shef.ac.uk:9194/ (where HOSTNAME is given on your MOLE/Blackboard page for the course). This server expects GET requests at the path com3505-2021 with parameters that include email and gitid. (Email addresses should end in @sheffield.ac.uk.)

Note that the server is only visible inside the University firewall so you’ll need to use the VPN if you’re working on a different network. Also, it uses a self-signed TLS certificate, so you’ll need to turn off certificate checking for this host if accessing from a browser. (The server is secure, but doesn’t embed a certificate from a known authority, so browsers give a warning.)

To send us your username, make a GET request something like the following (substituting MyGitUname for your gitlab account username and MyName for your email):

https://HOSTNAME.shef.ac.uk:9194/com3505-2021?gitid=MyGitUname&
  email=MyName@sheffield.ac.uk

You can do this via a web browser, or using wget or curl or any programming language or other tool that talks HTTP. If you feel like a challenge and you manage to figure out how do it with an ESP32 then:

  1. be careful not to flood the server with multiple requests!
  2. pat yourself on the back: you’re ahead of the game :)

For example, to use curl:

curl -k 'https://HOSTNAME.shef.ac.uk:9194/com3505-2021?gitid=MyGitUname&
  email=MyName@sheffield.ac.uk'

The response should look like this:

Received:  gitid=MyGitUname;  email=MyName@sheffield.ac.uk; 

Note: don’t send your numeric ID from GitLab, but the alphanumeric user name / path that appears in your project URLs.

(Later in the course we’ll use the same technique to gather data from the IoT devices you’ll be building, but that’s another story!)

1.3.3 Good Tools to Learn

We’ll be using most or all of the following tools in the course. It is a good idea to get to know them before starting (though you should be able to pick them up as we go along if needed).

In addition we will need a code editor or Integrated Development Environment (IDE) for writing IoT device firmware in C and C++, e.g.:

We’ll cover how to set up these tools next week.

1.3.4 STAYING SAFE in the Lab

The electronics lab in the Diamond (DIA 2.02) is a fantastic learning environment, and we give you as much time in there as we can during the course. It is also a potentially dangerous environment. Before coming to any lab session you must do the following:

Soldering is optional this year (but definitely a useful skill for IoT prototyping); if you want to do soldering in the lab, please also watch this tutorial video.

1.4 COM3505 Week 01 Course Notes

Have you done the preliminaries?? If not, now would be a good time :)

1.4.1 The Electronics Lab

Parts of the practical work for the course are best done in the electronics laboratory (DIA 2.02), but this laboratory is running with a reduced capacity to allow social distancing. That means each student will have at most two or three sessions in the lab during the 2021 course. You are also entitled to use the facilities in the iForge (the Diamond 1.01 project space) but again capacity is drastically reduced at present.

We have reduced the lab-based components of the work; please fit it in whenever you get lab access.

1.4.2 Learning Objectives

Our objectives this week are to:

1.4.3 Assignments

1.4.4 Working with your Git Repository

The assessment of your practical work on the course is delivered by checking into your git repository and pushing to the “origin remote” (on gitlab.com). If you don’t push anything, we won’t be able to give you feedback or to mark your work, so it is important to get used to this system early on.

Do this the first time you start work on a new machine:

check out your repository from gitlab into your own file space

Other useful commands:

Note: git (and GitLab / GitHub) have become industry standards in recent years, so it is important for you to get to know at least the basics. However, an independent survey of experts recently estimated that of 2,153 git command options fully 213% were either contradictory, confusing or error-inducing (or all three). [I may have made that bit up.] It is depressingly easy to get git into a mess! [I didn’t make that bit up.] What to do? Here’s one way to escape from git hell

Let’s say you have conflicts in your com3505-student-2021 repository and the process of resolving them is proving difficult. To re-create a clean version of the repository (in your home directory):

cd
mv com3505-student-2021 saved-repo
git clone https://gitlab.com/YourGitLabUname/com3505-student-2021.git

You’ve now got a fresh copy to work with; if you have changes in the saved version you can copy them over to the new, then commit and push from there.

Yes, it was called ‘git’ for a reason…15

1.4.5 First Lab Checklist

(There may be no of few lab sessions in 2021. Don’t panic! We will post out kits to you and help you work with them on-line.)

This section describes what to do when you get your first lab session:

If you finish early you might also want to learn about measuring simple values (using the multimeters and/or oscilloscope), but this is optional.

The hardware kits give you an ESP32 net-connected microcontroller, sensors and actuators, and the means to prototype experimental circuits: the foundations of IoT hardware.

For reference each kit should contain:

1.5 Further Reading

2 Definitions, and a Burning Question

When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind: it may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the stage of science. (Lord Kelvin)

Not everything that counts can be counted, and not everything that can be counted counts. (Albert Einstein)

Low cost networked computers are adding eyes and ears (or sensors) and arms, legs and voices (or actuators) to the Internet. These new devices are being connected to on-line brains (big data and deep learning in the cloud). This new field is the IoT, of course. Will the result be a ‘world robot?’ (Schneier 2017) This chapter will start to cover some of the context and history of the IoT. Chapters that follow will cover the hardware that makes it possible, the communications protocols and security systems it relies on, and the cloud-side analytics that make sense of the data it produces.

The practical work this week is to getting to know the SDKs and IDEs17 that allow us to develop software (or firmware) for the net-connected microcontrollers that are the foundations of IoT devices.

We’ll begin with a couple of definitions, then look at the genesis of the software ecosystem that we’ll be using to program our IoT devices, before moving on to the practical question of development tools.

2.1 Defining the IoT

There are many definitions of the Internet of Things (IoT); one of the most exciting is given by Bruce Schneier in his provocative piece Click Here to Kill Everyone:

…the Internet of Things has three parts. There are the sensors that collect data about us and our environment… Then there are the “smarts” that figure out what the data means and what to do about it… And finally, there are the actuators that affect our environment. … You can think of the sensors as the eyes and ears of the internet. You can think of the actuators as the hands and feet of the internet. And you can think of the stuff in the middle as the brain. We are building an internet that senses, thinks, and acts. This is the classic definition of a robot. We’re building a world-size robot, and we don’t even realize it. (Schneier 2017)

At the other extreme the IoT is about what becomes possible when networked microcontrollers become cheap enough to embed in very many everyday contexts, from central heating thermostats to garage doors. These devices face tight constraints of power usage and cost, and concomitant challenges to their security and functionality. They also quickly become extremely numerous, driving work on big data analytics and cloud computing.

The technology of networked devices goes back perhaps 50 years or more. The coining of the term itself is often credited to Kevin Ashton in 1999, while working at the Auto ID Center at MIT (Ashton 2011), of which more later.

2.2 Revolutionary Code: from MIT Printers to the Arduino

If we understand the past we stand a better chance of seeing into the future. Where did the IoT come from? Where did the systems we’re going to be coding come from? Later on we’ll learn about an embedded electronics ecosystem that brought the needs of Italian artists together with the origins of the operating system that drives more of the internet than any other and the compiler suite that has been ported to more architectures than any other and spawned the Arduino.

But first: do you own a phone?

You are very probably carrying a phone, but do you own it?! Call me an old stickler, but I think if I own something then:

Do those things apply to your phone? How about your laptop? Your tablet? In the 2020s we often buy electronics which the people who sell them claim we own, but if we take the trouble to read the licencing documents that accompany them we often find that we have few rights over them, and that repair, for example, is expensive or difficult or voids the manufacturer’s warranty. Much of the electronic and computational ecosystems we’ll spend most of our time with in this course arose from a similar realisation some 50 years ago.

2.2.1 Return with me to Boston in the 1970s…

Back when I was a wee small snotty thing, and many of you were only minute folds in the quantum potential of possible future universes, the PDP 10 was a cool computer:

18

It was the first machine to make time sharing (multitasking) common, and it had a huge maximum memory of… a whole megabyte!19

The operating system code (assembler) on the PDP 10 and its immediate cousins was routinely shared and improved by a community of programmers (or “hackers”20) at MIT, including one Richard Stallman.

2.2.2 Whaddya Mean, I can’t Fix It?!

As computing companies became bigger and more profitable in the 1980s practices began to change: there started to be no more access to source code, and users would have to sign an NDA21 even to get access to a binary. Stallman suffered the consequences of an aversion to NDAs when he was refused access to the source for a printer control program, even though his intention was to improve that program. He says: ’This meant that the first step in using a computer was to promise not to help your neighbor. A cooperating community was forbidden. The rule made by the owners of proprietary software was, “If you share with your neighbor, you are a pirate. If you want any changes, beg us to make them.”" [Ibid.]

2.2.3 What To Do?

’So I looked for a way that a programmer could do something for the good. I asked myself, was there a program or programs that I could write, so as to make a community possible once again?

The answer was clear: what was needed first was an operating system. That is the crucial software for starting to use a computer. With an operating system, you can do many things; without one, you cannot run the computer at all. With a free operating system, we could again have a community of cooperating hackers—and invite anyone to join. And anyone would be able to use a computer without starting out by conspiring to deprive his or her friends.’ (Ibid.)

And this is what lead to the kernel code that runs your Android phone, the GNU/Linux operating system that runs the majority of servers in the cloud, and the compiler code you’ll use to create firmware to run on the ESP32 in this course…

Why does this matter?

And, perhaps most important of all, we have been describing the genesis of the whole free and open source software movement, which has been responsible for huge portions of the code running our world, and which in combination probably constitute the most complex machine ever constructed. Where will it go? Where will humanity go, locked into a triple crisis of pandemic, environment and economy? The two questions are (perhaps uncomfortably) tightly linked.

Ok, enough context for now. We’ll come back to IoT history next week; now for something practical :)

2.3 Coding Support Tools: IDEs, SDKs, Libraries

When we write code to run on a microprocessor (e.g. on your laptop, phone, or that Cray you have in the garden shed) we use all the facilities of the operating system, a modern programming language and its library ecosystem to insulate ourselves from the underlying hardware and to provide a set of abstractions, APIs and components that can make us productive at high speed. In recent times we have started adding on-line APIs the mix, making our potential deployments of computational resource truly vast.

On a microcontroller like the ESP32 in your kits, we have a lot less support: programming is typically with low level languages (C, C++, occasionally bits of assembler), the ‘operating system’ is more like a task management library, and the development tools are often cranky and basic. Firmware usually has to be uploaded to the device over a serial line or JTAG connection, and the tooling to perform this task is different for every hardware family.

In some cases a new world of Javascript or Python programming and drag-and-drop upload (e.g. over USB OTG) is starting to become available for the IoT, but:

  1. this is not yet widespread
  2. it still uses the manufacturer’s C libraries under the hood

The second of these points in particular means that projects that need to use the hardware efficiently and to exploit all of the available facilities will most often write in C/C++.

This section begins by looking at the basic (toolchain, SDK and library) building blocks of development on the ESP32, then goes on to describe available development environments. Later on (section 2.4.3) will talk you through installing your own development environment and taking your first steps in programming an IoT device.

2.3.1 Toolchains: In the Beginning, There Was The C Compiler…

In any novel embedded system the foundational step is to port a C compiler to the new chip. The compiler, linker and the tools that take a binary image and burn it into the memory space of the device (in a form that can then be sucked up by a bootloader and turned into a running process) are called collectively the toolchain. The compiler is generally an unusual beast, as it has to run on one platform (your development machine, x86_64 perhaps) but produce binaries for a different architecture (Xtensa being the one on the ESP32 in your parts kit); i.e. we need a cross compiler. So when installing a development environment for the ESP32 we need to pick up the Xtensa port of GCC (the GNU Compiler Collection, aka GNU C Compiler).

There’s good news and bad news about this step:

There are snakes in the long grass, wear your wellingtons.

2.3.2 ESP-IDF, FreeRTOS and the ESP32 Arduino Core

A compiler, when ported to an IoT microcontroller, will open up the magical gates of productivity and allow you to do… not very much at all! The differentiators between the (many) different chips competing for IoT oxygen relate to their hardware facilities, and to access these we almost always need more than is provided by the standard C library. (On the smaller, more resource-constrained devices, the standard library may not even be present due to lack of space; in the case of the C++ library this is actually quite likely.) In order to sell chips, then, the microcontroller manufacturer needs to supply a set of libraries that expose the hardware functionality of its wares in as friendly and powerful way as possible.

2.3.2.1 ESP-IDF

Espressif, maker of the ESP32, provides an SDK and library set called ESP-IDF, or Espressif IoT Development Framework, whose development is hosted here on GitHub.

ESP-IDF has grown to be a large and successful open source project that incorporates ports of many other libraries to ESP hardware. For example, MbedTLS (source code) is a very popular small footprint cryptographics library (supporting certificate-based security and the TLS protocol that underlies security on the web via HTTPS). Espressif support a port of MbedTLS (documented here, source here) that exploits the ESP32’s hardware acceleration facilities for tasks like random number generation or hashing. This hides the peculiarities of the ESP’s underlying hardware from the programmer, who can instead use familiar and well-documented abstractions as surfaced by the MbedTLS library.

Other libraries that are supported for the ESP32 via ESP-IDF include:

In each case IDF adapts the library to particularities of the underlying hardware, making it easier for programmers to exploit the chip to its full potential without learning about the grungey details of which register does what under what conditions.

Whenever we write code for the ESP32 we will be using ESP-IDF APIs, either directly or indirectly. To begin with, the easiest way is to use them indirectly, letting the Arduino compatibility layer take the strain; see below.

As of early 2021 there are three main versions of IDF that we will come across:

From version 4 IDF changed its principal build system from GNU Make to CMake, and this has made supporting both versions quite complex. There are three scripts provided by the distribution that set up the environment needed for the build systems to work:

Each of these has changed and evolved over different versions of IDF, and they present a difficult moving target to IDEs and automation scripts like Eclipse, PlatformIO and VSCode. (It is also a rapidly changing system integrating many 3rd party libraries.) This means that it can be quite challenging to get a 4.x IDF setup working with the Arduino layer, so to begin with using 3.3.x is probably the best option. (In Feb 2021 it now looks much easier to do 4.2 with PlatformIO; see below.)

An additional complexity is that IDF provides a configuration tool, menuconfig (based on the KConfig language), originally developed for the Linux Kernel. This exposes many of the optional features of IDF and of the libraries it incorporates. To use it you have to be doing a native IDF build of one type or another (which means you can’t easily use it in the Arduino IDE).

2.3.2.2 FreeRTOS

Another important facility that we can access via the Espressif SDK is FreeRTOS, an open source real time ‘operating system’. FreeRTOS provides an abstraction for task initialisation, spawning and management, including timer interrupts and the ability to achieve a form of multiprocessing using priority-based scheduling. The ESP32 is a dual core chip (although the memory space is shared across both cores, i.e. it provides symmetric multiprocessing, or SMP). FreeRTOS allows us to exploit situations where several tasks can run simultaneously, or where several tasks can be interleaved on a single core to emulate multithreading.

This doesn’t necessarily make life simpler! Most microcontroller code, at least in the hobbyist and maker spaces, is written to target single core chips, and the potential for shooting yourself in the foot rises very quickly when trying to adapt to a world where multiple things may be happening (or at least seeming to happen) all at once. Who owns this piece of memory? What will happen when two tasks both try and talk over this bus? Why does my code behave differently if I pause a task for a few microseconds? To begin with at least, it may be easier to limit yourself to a single task that runs an infinite loop, and worry about FreeRTOS later on. (This is what happens, in fact, when we delegate our main program entry point to the Arduino “core” for ESP32: the setup and loop procedures which characterise Arduino programming are implemented for us using a FreeRTOS task, but we don’t have to worry about the details.)

Later on, interrupts, tasks, event queues, mutexes and semaphores will all become objects of interest, and if you’ve a mind to dive into the mysteries of FreeRTOS that’s where you’ll find them. Enjoy!

2.3.2.3 The Arduino Core for ESP32

Fairly early in the lifetime of the ESP32’s predecessor chip, the ESP826625, a Bulgarian developer going by the somewhat cryptic moniker of me-no-dev decided that what was needed was integration with the Arduino ecosystem. He set out to develop a compatibility layer between Arduino libraries and development tools and the ESP8266. The work rapidly captured the imagination of a large number of developers, and has become a mainstay of the ESP community: nowadays there are commits from more than 500 people in the github repos, and perhaps 10,000 commits overall. Espressif could see that this was a good thing: they hired me-no-dev and he now works full time on the “Arduino core” for the ESP32. The project can be found here.

What’s an Arduino core when it’s at home? I’m not sure where the term originated, but it basically means all the code that interfaces a device and its toolchain, compiler settings, and libraries to a) other Arduino library code and b) the Arduino IDE. This piece of kit is incredibly useful, because the Arduino ecosystem is massive. Pick up any electronic component (sensor, actuator, dog brush or kitchen sink) and the likelihood is that someone has published an Arduino-compatible library for it. This makes programming from the Arduino APIs hugely more productive than otherwise, because we’re almost always working from widely used running code. Win win win!

Motherhood, apple pie, happy ever after. Except: VERSION HELL!!!

2.3.2.4 Version Hell!!!

When we program the ESP32 we’re typically using C code from ESP-IDF, C code from the libraries that have been ported to the ESP chip family and included as part of ESP-IDF, C++ code that translates between ESP-IDF and the Arduino APIs, and C++ code that exposes sensor and actuator hardware within the Arduino ecosystem. (All of these are in active development, and all have their own release cycles.) We are then compiling this morass with a toolchain that has its own version trail, controlled by a build system that comes in a whole bunch of different and subtly incompatible flavours and, finally, uploading it to the IoT device using a Python script written in … erm, Python (possibly one of the least stable programming languages ever devised26).

Hmmm.

The types of things that tend to go wrong are:

So: if things go wrong, don’t worry. You’re not alone. Section 2.4.3 tries to map out a path of relatively easy and reliable options. But first, let’s round of this discussion with a look at the available build system CLIs and IDEs.

2.3.3 Developer Tools: CLIs and IDEs

The game, lest we forget amidst all this gratuitous verbiage, is to (cross-)compile our code against 1001 IoT and embedded systems libraries and burn the resultant firmware images to an ESP32. Along the way it might be nice to throw in a bit of:

We have two main families of options for doing (some subset of) these things:

Examples of IDEs include:

Examples of CLI build systems include:

I was talking to someone recently who works at one of the big semiconductor companies (the one that designed the chip in your phone) about the difference between microcontrollers and microprocessors and remembering all the weird challenges that the former tend to throw at you. “Welcome to embedded,” she smiled, where getting your toolchain installed can be a week’s work :)

The rest of this section will describe two of the most straightfoward options for getting started, and also provide a few notes on two options for using VSCode.

Note: you can try out the recipes here to get started on programming the device, or just read them and then follow the instructions in sec. 2.4 below.

2.3.3.1 Using the Arduino IDE (ArdIDE)

One of the simplest and most reliable ways of programming the ESP32 is using the Arduino IDE. This IDE is a little like me: quite robust and reliable, but a little antiquated. It is a good place to get started, but possibly not where you want to remain for the whole course. Your choice though: if you like it, go with it. 

To pick up the compiler toolchain and other specifics we need to install an IDE plugin known as a core. The easiest way to install the IDE and the ESP32 core is to follow the instructions on the core github site. You can choose the development release version; install core version 1.0.5 (rc4 or above), which should give you IDF release 3.3.4.33

For example:

Congratulations, you have successfully burnt your first ESP32 firmware!

Another useful example to try at this point is GetChipID:

If you got that lot to work, you have:

Finally, a note on versions and restrictions: as discussed in section 2.3.2.1 the rapid pace of change in IDF has meant that the Arduino core lags behind a little, and the instructions above will result in an installation based on IDF 3.3.4. Using the board manager installation method also means that the configuration utility for IDF (menuconfig) can’t be used (because the Arduino layer installs a pre-compiled version of IDF).

Probably time for a brew!

2.3.3.2 CLI on a Raspberry Pi

This section describes setting up an ESP-IDF CLI build. The instructions are for a Raspberry Pi running the Debian Buster port (which used to be known as Raspbian but now seems to have mutated into RaspiOS34). The instructions should work pretty much unchanged on Ubuntu 20.04. (If you are on other platforms you’ll need to adapt the instructions or use a VM or docker – see next section.)

If you need a cheap machine to develop for the IoT, Raspberry Pi models 4 or 400 are both capable of doing the job. They won’t be the fastest environment, but for around £/$/€100 you can get a capable computer that will plug into your TV or HDMI monitor and do everything you need to for the course. (A good choice would be this kit, for example, or this one.)

Below I describe how to get up and running with ESP32 development on the Pi. Note that the setup process took several hours (on a fast network), so don’t leave it until 10 minutes before your deadline :) (The complete setup uses around 2.5GB of the flash disk, depending on configuration.)

Here’s some more timing information, for install and build (using the firmware and scripts in the HelloWorld example, which will be introduced later):

Note that although setup and clean compiles are slow on the Pi, the compile/burn/test cycle is reasonably quick at under 30 seconds. This means that you can work productively on a Raspberry Pi, so long as you allow an hour for the initial setup. (This is from the command line, of course; running a big IDE like VS Code or Eclipse would no doubt be slower!)

To set up the operating system:

To get the ESP32 development environments running:

(Note: some things don’t work in the old versions, but they’re more stable. The version that the Arduino IDE install route detailed above pulls down is IDF 3.3.4 / Arduino core 1.0.5, and this currently is the most reliable combination.)

For example, using a pre-release IDF 4.3 in December 2020:

To try burning an example:

Well done! Please give yourself a pat on the back. If you’re feeling brave: VSCode for the Pi is available here :)

2.3.3.3 CLI Using Docker

If you don’t have an Ubuntu or RasbiOS environment available, you can try using docker (although you won’t be able to burn firmware directly unless you’re on another Linux platform).

If you’ve cloned the course repo (the-internet-of-things) and cd’d into it, and connected an ESP32 to USB, you should now be able to do the following:

(Command-line syntax for docker on non-Linux platforms may also be a little different; see the docker docs.)

If you’re on Linux, and you have the correct device specified (/dev/ttyUSB0, for example), then you should also be able to burn the firmware, e.g. by just doing ./magic.sh. This won’t work on MacOS or Windoze because no easy way to map the serial port driver model currenly exists In this case you’ll need to copy the the-internet-of-things/exercises/HelloWorld/build/HelloWorld.bin file out of your container and burn the firmware using a local install of ESP-IDF (which somewhat defeats the object!). If you want to try this, the command that is needed to burn the firmware can be copied from the output of the docker build, e.g.:

...
Generated /home/ubuntu/the-internet-of-things/exercises/HelloWorld/build/HelloWorld.bin
[1081/1082] cd /root/esp/esp-idf/components/esptool_py && /usr/bin/cmake -D IDF_PATH="/root/esp/esp-idf" -D ESPTOOLPY="/root/.espressif/python_en...D WORKING_DIRECTORY="/home/ubuntu/the-internet-of-things/exercises/HelloWorld/build" -P /root/esp/esp-idf/components/esptool_py/run_esptool.cmake
esptool.py --chip esp32 -p /dev/ttyUSB0 -b 921600 --before=default_reset --after=hard_reset write_flash --flash_mode dio --flash_freq 40m --flash_size 4MB 0x8000 partition_table/partition-table.bin 0x16000 ota_data_initial.bin 0x1000 bootloader/bootloader.bin 0x20000 HelloWorld.bin
esptool.py v3.0-dev
Serial port /dev/ttyUSB0
Connecting......

2.3.3.4 VSCode IDF Extension

A new kid on the block for ESP32 development support is the ESP-IDF VSCode plugin: see the installation guide for details. The plugin is still pretty new, and the process of getting it to work with a clone of both the IDF and the Arduino core is pretty involved. When it works, though, it looks like a nice environment to use :)

I got it to work like this (on Ubuntu 20.04):

See also: Quick User Guide for the ESP-IDF VS Code Extension.

2.3.3.5 Using VSCode with the Arduino Extension36

This section is about using VSCode in a more basic, but still useful mode.

Some more tweaks may need to be done for full IntelliSense compatibility, so you may need to hide squigglies until we figure out the additional paths you need to add. You can still use these instructions to do things like uploading your code to the ESP32, though.

First install VSCode itself; on Ubuntu you can do that like this on recent versions:

snap install code --classic

You can then run the beast from the command line with code, or from the launcher by vs....

Now install the C++ and Arduino extensions:

E.g.:

"arduino.path": "/.../arduino-PR-beta1.9-BUILD-115",
"C_Cpp.default.includePath": ["/.../arduino-PR-beta1.9-BUILD-115/libraries/"]

Make sure your JSON is still valid, and save it. You might also need to close and reopen Visual Studio Code for your include path to be recognised.

Doing all of the above will give you access to many of the Arduino IDE’s functions from within the Command Palette (usualy Ctrl+Shift+P) within VSCode under the same names.

2.3.4 FAQ

Ok, that’s the end of Chapter 2’s general material. The rest of the chapter gives specific tasks for Week 2.

2.4 COM3505 Week 02 Notes

2.4.1 Learning Objectives

Our objectives this week are to:

2.4.2 Assignments, Set Up, Exercises 1 & 2 (Ex01, Ex02)

Notes:

Exercises:

As noted in chapter 1’s “how the course works”, there are example solutions in the course materials tree. Have a go at doing your own version before looking at these!

2.4.3 Set up your Programming Environment

As discussed above there are lots of options, and lots of potential pitfalls, so give yourself plenty of time to do this task, and don’t be surprised if you need to come back to it multiple times to refine your toolset.

I recommend that you start with the Arduino IDE, then a command-line (CLI) build, e.g. pio run using PlatformIO. Once you have these working, if you wish you might then try one of the VSCode options. The ambitious amongst you might try Eclipse (if you’ve used it before and liked it), or Arduino Create, or PlatformIO/VSCode. The important thing is just to get one environment running (and if you’re happy with it feel free to stop there!).

Follow these steps:

2.4.3.1 Using PlatformIO and the Firmware Template

From Feb 10th 2021 the PlatformIO Espressif 32 development platform now supports ESP IDF v4.2 and a recent Arduino core (“2.0.0”). To run this with the firmware template (in HelloWorld or Thing) do this:

2.4.3.2 Using magic.sh and the Firmware Template

NOTE: these examples are designed for Ubuntu 20.04 and recent versions of RaspiOS. You’ll need a VM or emulation or etc. if you want to try them on other platforms.40

In our the-internet-of-things course materials git repository you can find a tree called exercises/HelloWorld. This contains example firmware code (functionally similar to the Blink example supplied with the Arduino IDE) and a script called magic.sh that (if it works) could simplify your install and build work.

For example, to install and setup IDF and try a CLI build you can do this:

# after cloning:
cd the-internet-of-things/exercises/HelloWorld

# get a list of options:
./magic.sh -h

# clone and check out IDF and the Arduino core: 
./magic.sh setup

# attempt an IDF build, flash and monitor:
./magic.sh

There’s also some support for the Arduino IDE:

# download and install:
./magic.sh setup-arduino-ide

# run:
./magic.sh arduino-ide

If you want to access other idf.py commands (see the IDF documentation for details), try

./magic.sh idf-py your-favourite-command

To take a copy of one of the examples, use the copy-me-to command from inside the example. E.g. if you have your gitlab repo checked out in your home directory, this would create a new tree there:

./magic.sh copy-me-to ~/com3505-student-2021/MyNewExample

For more details take a peak at the README or magic.sh.

2.4.3.3 Using the University Machines

You can (just about) do the programming for the course on both Windows and Linux on the University desktop machines. (The latter is rather out-of-date and slow, unfortunately, and although there is a development process running to improve the Linux installation it looks unlikely to bear fruit in time for the 2021 course.)

On both platforms you’ll probably need to use only the Arduino IDE; VSCode is also present but installing plugins for PlatformIO or ESP-IDF hasn’t worked so far.

To use University Linux, boot the PC (e.g. in DIA 2.02) into “Diskless Linux” (LTSP); more details and available locations here. ArdIDE should work, though slowly.

2.4.4 Hardware 2: Sensor/Actuator Board

We are using a breadboard (also known as proto-board) to assemble our circuits in the labs. Breadboards allow components and wires to be pushed into holes and connected without soldering. (In Hardware 1 you use one to experiment with voltages, resistors, and measuring a circuit’s performance with a multimeter. These are important skills for IoT device prototyping, and will be useful for your project work later in the term.)

To recap, a breadboard has some connections between holes already made – shown in this diagram:

Diagram of breadboard connections41

2.4.4.1 Using a Breadboard to Make a Sensor/Actuator Circuit

Take your ESP32 feather device and carefully lay it in place on the breadboard over the holes indicated:

Feather in breadboard

Now push gently on the edges of the feather to insert the pins into the holes – you are aiming to keep the device parallel to the breadboard as it goes into the holes, rather than push one side down and leave the other up. Keep moving around pushing in different places if it seems stuck – often new breadboards are a bit stiff at first. If the pins aren’t going in easily then check that they are all lined up above the holes correctly – if not you can bend them gently with a pair of fine pliers. You will need to use some force to push the device down onto the breadboard. To get an idea of how much force might be needed, take a jumper wire and try inserting that into a hole, then multiply that by 28. Try not to bend the header pins – if you do then use the pliers to return them to their correct orientation.

Once the feather device is inserted, add a LED, 120Ω (or thereabouts) resistor and push-button switch from your kit as shown:

Blinky components in breadboard

Pay attention to the orientation of the LED – as it is a (light emitting) diode it will only work one way round. The longer lead of the LED is the anode and it connects to the ESP32 output pin – the shorter lead is the cathode and it connects to the negative or ground connection. Add jumper leads to complete the circuit as shown – colour codes help to make the circuit more readable:

Blinky components with connections in breadboard

The circuit pictured above can be represented as a schematic – this is a more abstract representation of the components and connections:

Blinky schematic

The trick here is that the LED is connected to pin 32 and the switch to pin 14, so we might usefully define these in our C code as int pushButton = 14, externalLED = 32;.

The code for this sketch should make use of the internal pull-ups inside the ESP32. Simply pass the “input pullup” macro to the pinMode command: pinMode(pushButton, INPUT_PULLUP) These are optional resistors that connect to +v and the input pin. These make the inputs high, unless they are connected to ground. See this diagram:

Diagram of internal pull-up

Below are some pictures of an example board. Note that this one is using the ESP32 with “stacking headers” (additional sockets on top of the board) so we can use both the breadboard sockets or the stacking sockets to connect to.

You’re now ready to work with the LED and switch in firmware to answer Exercise 2 as above.

2.5 Further Reading

3 History; Blinking Things; WiFi

Where have we got to? By now you should have:

In this chapter we do two things:

There’s lots to do: crack on!

3.1 The Multiple Personalities of the Arduino Project

When we teach computing to beginners, we teach how to build something from the ground up. A new programming language: hello world. A machine learning method: the pseudo code for its algorithmic expression. This is necessary and valuable, but it hides a crucial fact about software development in the 2020s: we almost never build anything significant without starting from the work of tens of thousands of other people. These people form communities, and communities adopt and evolve tooling and workflows to create ecosystems. Choosing which ecosystems from the work of our predecessors we try to hitch a ride on is often one of the most influential decisions in any development project.

I think it fairly safe to say that few people would have predicted, around the turn of the millenium, that one of the most significant advances in embedded electronics and physical computing would be driven by “the development of contemporary art and design as it relates to the contexts of electronic media” (Barragán 2004). Starting in the early naughties, the Arduino project (Monk 2013; Banzi and Shiloh 2014b; Arduino 2017) had by the mid tennies come to provide the standard toolkit for experimentation in low power circuits. Today the ecosystem that resulted is first port of call for code to drive the sensors and actuators that give the IoT its interface to the physical world. And as we saw in the previous chapter, the Arduino IDE also provides us with one of the easiest ways to get started programming the ESP32.

We’ll start this chapter by looking in a little more detail at the various contributions that the project has made, and have a bit of a sniff around its development environment and the type of (C++) code it supports.

The Arduino Project

Arduino can refer to any or all of:

Arduino C++ refers to preprocessing, macros and libraries that are available via the IDE. (C++ is a medium-level language layered on the C (low level) systems programming language.) Over the last decade or so there has been a quite massive community of open source developers contributing to the ecosystem of code and documentation and forum posts and github repos and etc., which has made Arduino C++ a very productive environment for IoT development.

Remember that the ESP32 is not an Arduino (or even an AVR-based device), but there is a compatibility layer that interfaces it to the Arduino IDE. This makes lots of code for sensors and actuators and etc. magically available for the ESP.

An Arduino board

Having started out as a service project for arts and design students in Ivrea, Italy, in the early 2000s (who were using microcontrollers to create interactive exhibits) it brought cheaper hardware than the alternatives (based on Atmel’s AVR chips), and added an IDE derived from work on Processing and Wiring. Some millions (or possibly 10s of millions) of official and unofficial boards are now in existence, and it was for a long time the most popular platform for embedded electronics experimentation.

The Arduino IDE is a Java Swing desktop app that preprocesses your code (or “sketches”). Arduino’s programming language is C++ with lots of added libraries and a bit of code rewriting. It runs the GNU C++ toolchain (gcc) as a cross-compiler to convert the Arduino C++ dialect into executable binaries. The IDE includes a compilation handler that converts sketch firmware .ino files into a conventional .cpp file (poke around in /tmp or equivalent to see what the results look like).

Binaries are then uploaded (“burned”) to the board using various other tools (in our case a Python tool from Espressif). The IDE then allows monitoring of serial comms from the board, and provides access to libraries and example code. If you ask it nicely it will plot graphs for you, e.g. of the output of a microphone:

Plotting sound waves

As we noted in chapter 2 the IDE is pretty basic, and you may well want to move on to more sophisticated tools later on. It is well worth getting to know it to start with, however, as it is typically the fastest way to get started with new hardware, and the fastest way to find a working example from which to develop new functionality.

3.2 A Crossover Point

In sec. 2.1 we saw two ways very different to define the IoT: as a nascent world robot, or as the simple consequence of increases in computational power and network connectivity in microcontroller hardware. This section gives a bit more context to the origins of the field on the one hand, and the hardware space of IoT devices on the other.

When we look back at the antecedents of the IoT, it becomes clear that the field represents a crossover point between many earlier (and ongoing) areas of computer science and software engineering. Related and predecessor fields include:

If you understand roughly what each of these terms means then you have a good basis for understanding what the IoT means (and of being able to distinguish the marketing speak from the actuality of possible technology options).

3.2.1 The Early History of the IoT42

The term Internet of Things was coined by Kevin Ashton in 1999 for a presentation at Procter and Gamble. Evolving from his promotion of RFID tags to manage corporate supply chains, Ashton’s vision of the Internet of Things is refreshingly simple: “If we had computers that knew everything there was to know about things — using data they gathered without any help from us — we would be able to track and count everything, and greatly reduce waste, loss and cost. We would know when things needed replacing, repairing or recalling, and whether they were fresh or past their best.” (Ashton 2011)

Of course, devices had been ‘on the internet’ for several years before this, from at least 1982 in the case of a drink vending machine at Carnegie Mellon University in Pittsburg (Machine, n.d.). Using a serial port converter, the status lights of the front panel of the machine were connected to the Computer Science departmental computer, a DEC PDP-10 (for which today’s equivalent cost would be around $2 million!). The Unix finger utility was modified to allow it to report the level of coke bottles and whether they were properly chilled. Internet users anywhere could type “finger coke@cmua” and read the status of the machine. (It is notable that the world’s first IoT device was enabled by openly available Unix source code.)

A camera pointed at a coffee-pot in Cambridge’s computer science department was video-streamed on the internet from 1991, and when control from the web to the camera was established in 1993 the first webcam was born (Fraser 1995). Presaging very contemporary anxieties, a toaster had been connected to the internet in 1990 at the INTEROP exhibition (Romkey 2017), and nearly caused a strike as preparing food was an activity allocated to unionised labour. However it wasn’t until 2001 that a toaster became an IoT device in a modern sense, able to dynamically query a webservice for the current weather forecast, and then burn the appropriate pictogram onto a piece of toast (Ward 2001).

So we can see that from the earliest days of IoT (when it was often called pervasive or ubiquitous computing) our current concerns around open source, security (Denning and Denning 1977) and human obsolescence were already recognised.

3.2.2 The Current State of IoT Hardware43

“It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity…” (Dickens 1877)

We do indeed live in an epoch of belief and incredulity, with feverish hype of IoT widespread across the electronics and computing industries. Want a £400 wifi connected juicer that locks you into proprietary ingredient pouches anyone? Meanwhile ‘policy bloggers’ burble excitedly about techno-utopias such as smart cities that eliminate traffic — “Imagine a city with no traffic at all” (O’Muircheartaigh 2013). Security researchers are desperately trying to warn us of the dangers of letting our personal data leak out from our devices (Sarthak Grover and Feamster 2016). After FitBit users’ personal exercise data was exposed publicly (Loftus 2011) the company solemnly announced “We have also updated our default settings for new users for activity sharing to ‘private.’”

Current hardware is very diverse, with major companies such as Intel, Texas Instruments, Broadcom etc. competing to have their chips chosen by manufacturers. However the arrival of Espressif’s ESP8266 (and now the ESP32 (Community 2017; Kolban 2017) has shaken up the rest of industry by charging $2 per chip instead of $20. This has attracted a lot of attention and stimulated the creation of community-driven knowledge, documentation, libraries and tools; so much so that it is significantly quicker and easier to develop for this platform than most others.

IoT hardware can be classified according to its connectivity technology; currently the useful ones are ethernet, wifi, bluetooth, zigbee, z-wave and cellular. Whilst wired connections are still relevant for some applications, it seems that most recent developments have concentrated on wireless devices. If we are to buy hundreds of IoT devices each in the next few years, we certainly won’t be plugging them all into ethernet cables. Cellular technology remains stubbornly expensive both to buy and to run; fine for the now life-critical mobile phone, but not for those hundreds of devices.

Of the remaining mainstream wireless technologies, bluetooth’s USPs are it’s ultra low-power short-range attributes and that every phone has it. For devices with small batteries such as fitness trackers, this allows them to last a few days between charges. Wifi has major issues with power use and connection negotiation speed but it has emerged as a major IoT connectivity choice because of it’s ubiquity. Then there are the Z’s – Z-wave is proprietary but popular with blue chips like Honeywell and GE, Zigbee is an open standard also popular with big corporations – the Phillips Hue light bulbs use it as does the Nest thermostat.

Several ‘hubs’ have been launched by manufacturers such as Google, Amazon and Samsung that aim to bring all these devices together under one control, rather than having to manage dozens apps of physical remotes. For example, Samsung’s SmartThings hub has 4 wireless radios covering z-wave, zigbee, bluetooth and wifi plus an ethernet port.

3.3 COM3505 Week 03 Notes

3.3.1 Learning Objectives

Internet? What internet?! Up to now we’ve been programming the ESP as a standalone device. In the next period we’ll create and connect to networks, and figure out how to configure connections for devices without UIs.

Our objectives this week are to:

Practical work will include:

(There are a lot of exercises this week; you’ll benefit from completing all of them, but if you don’t have time don’t worry, just make sure to study the model solutions in exercises/Thing.)

3.3.2 Assignments

Exercises:

If you get the access point to work and then join it (e.g. from a phone or laptop), when you load http://192.168.4.1 in a browser, you should see something like this:

Now you’ve made a thing which is (almost) on the internet :)  Do a little jig, dance around the room, make a cup of tea.

3.3.3 Notes on the Model Code from Week 2

(You’ll find model versions of the exercises in exercises/Thing.)

3.3.3.1 Recap: Connecting to the ESP32

The ESP32 board we are using has a micro-usb socket to provide power and also allow communications between the microcontroller and your computer. Start by connecting the two together using the supplied cable. Start the Arduino IDE (e.g. by magic.sh arduino-ide). Ensure that the Tools>Board selected is the “Adafruit ESP32 Feather.” If you’re running the IDE locally (e.g. on your own machine or on the LTSP desktops) look in the Tools>Port menu to check that the serial connection has been established.

3.3.3.2 Various Arduino Functions

Note: in the Arduino IDE certain words such as OUTPUT, HIGH, TRUE etc. are pre-defined and shown in blue. Similarly functions such as pinMode, Serial.begin are coloured in orange – this can help you catch syntax errors.

Serial.begin(115200);           // initialise the serial line

Serial communication (sending or receiving text characters one by one) has to be initiated with a call to the begin function before it can be used. The serial communications between the ESP32 and computer can operate at various speeds or baud rates – we use 115200 baud. If you aren’t getting any response, or gibberish characters on the serial port monitor, then check you’ve got the correct speed set.

pinMode(BUILTIN_LED, OUTPUT);   // set up GPIO pin for built-in LED

pinMode is an Arduino command that tells the microcontroller to set up a single pin (first parameter) using a certain mode (second parameter). The two basic modes are INPUT and OUTPUT; INPUT_PULLUP connects an internal resistor between the pin and 3.3V (good for listening for pull down events; we’ll hear more about these later in the course).

delay(1000);                  // ...and pause

The delay function in Arduino takes milliseconds as it’s parameter – so a delay(1000); command pauses for 1 second. Note: this is a blocking method! Nothing else can happen on the core that is being paused for the duration of the call!

uint64_t mac = ESP.getEfuseMac(); // ...to string (high 2, low 4):

The ESP32 Arduino layer includes some helpful functions like this one to allow us to get (read) the status of the electronic fuses. After the silicon for the ESP32 is manufactured using a common mask, each one is programmed to give it a unique identity – including things like MAC addresses. These are one-time electronic ‘fuses’ that burn the MAC address into the chip so that it cannot be reprogrammed.

3.3.3.3 Reading from Switches

pinMode(14, INPUT_PULLUP); // pin 14: digital input, built-in pullup resistor

This pinMode call enables built-in pullup resistors connected between the pin and the positive supply voltage (3.3V). These prevent the input ‘floating’ when it isn’t connected to anything and instead make the input go high.

  if(digitalRead(14) == LOW) { // switch pressed

The digitalRead function returns the binary state of the input pin given as a parameter. Because we are using a pullup resistor and connecting the switch to 0V – the logic that digitalRead returns is reversed. Therefore, when the switch is pressed, the function returns LOW.

3.3.4 Exercise 03 Notes

3.3.5 Extension to Blinky (exercise 02)

Assemble the components shown in this schematic on your breadboard:

Blinky Traffic Lights schematic

You should then have a breadboard that looks like this:

Blinky Traffic Lights breadboard

Here’s a picture:

3.3.6 A Final Breadboard Prototype: 9 LEDs

Prototyping IoT devices has become much easier and cheaper in recent years. This makes development cycles much faster: from idea to prototype to product is now a cycle measured in weeks and months instead of years. We’ll finish our quick tour of prototyping skills (that began in Hardware 144 with soldering, breadboarding and using the multimeter and signal generator etc.) by building a more (physically) complicated circuit. This week we’ll build it on a breadboard, then when you’re in the lab you have the option to solder it onto matrixboard (which is a typical cycle in early prototyping; the next step would be to design a PCB and start manufacturing test boards).

In the second half of term you’ll have the opportunity to build projects that involve more complex hardware, e.g. to add GPS, or MP3 playing, or motor drivers, or ultrasonics, or etc. These may require soldering, and when they don’t work first time the multimeter and oscilloscope are our first ports of call to discover why.

(Hate soldering? No lab access? Don’t worry, there are also projects that only use the ESP32 itself.) The circuit is electrically very simple, being Week02’s Blinky with lots more LEDs, each protected by its own resistor:

9 LEDs breadboard 20

The cathode (short leg) of each LED is connected in series with a 180Ω resistor that connects in turn to ground. The anode is connected to one of the GPIO (general purpose input-output) pins of the ESP32.

3.3.6.1 Pinouts

Because we’re using lots of pins, it becomes a bit tricky to fit all the connections in, and we need to check the various references to see where is good to connect, e.g.:

Pinouts on the ESP32 Huzzah

An additional complexity is that (in common with other modules) the ESP has several names for many of its pins, e.g.:

ESP32 pinout schematic

There’s a good description of pinout detail here; and a good general discussion on which ESP32 pins to use here.

In this case we’ll use these pins:

// LEDs
uint8_t ledPins[] = {
  GPIO_NUM_26,
  GPIO_NUM_25,
  GPIO_NUM_21,
  GPIO_NUM_4,
  GPIO_NUM_27,
  GPIO_NUM_33,
  GPIO_NUM_15,
  GPIO_NUM_32,
  GPIO_NUM_14,
};

The course repo has example code called 9LEDs firmware.

3.4 Further Reading

4 Country of the Blind: Networking Devices Without UIs

This chapter covers two key tasks that the vast majority of all IoT devices must implement: provisioning and update. Provisioning is about how devices are supplied with network credentials, and update is about how devices are brought up-to-date with new versions of their firmware.

Over the next two weeks we’ll study these in some detail, and implement first provisioning and then update.

4.1 Provisioning and Update

When you first power up a Chromecast or a Firestick and plug it into your TV, for example, it is useless. These devices, sharing many typical characteristics of the IoT, are special purpose stream-and-decode machines that are fed a URL by their controller (your smartphone) and then act as an intermediary between the cloud and the television. Without a network connection, nothing can happen.

Whenever we supply IoT devices to third parties, we face the same problem; to solve it either:

  1. the device has to ship with network credentials, or
  2. support end-user configuration

Chromecast leverages Google’s Home phone app to achieve the second of these.

Secondly, all IoT devices must be considered vulnerable to attacks of one sort or another, just like any networked computer. This course is fairly new, but has already seen two major vulnerabilities in the ecosystems we rely on:

The first of these (WPA) was fixed fairly rapidly by patches to the relevant libraries. The second was impossible to fix in firmware, and as a result Espressif had to develop version 3 silicon (which is only becoming available now in 2021, e.g. as the ESP32-S3). The attack is quite difficult and restricted (being a. physical, b. complex and c. per-device), but no software patch is possible to fix the vulnerable existing hardware: all ESP32s in the field prior to this point will continue to be susceptible to this attack.

Quoting (Schneier 2017) (Click Here to Kill Everybody: Security and Survival in a Hyper-connected World):

“Everything is a computer. Ovens are computers that make things hot; refrigerators are computers that keep things cold. These computers—from home thermostats to chemical plants—are all online. The Internet, once a virtual abstraction, can now sense and touch the physical world. As we open our lives to this future, often called the Internet of Things, we are beginning to see its enormous potential in ideas like driverless cars, smart cities, and personal agents equipped with their own behavioral algorithms. But every knife cuts two ways. All computers can be hacked. And Internet-connected computers are the most vulnerable. Forget data theft: cutting-edge digital attackers can now crash your car, your pacemaker, and the nation’s power grid.”

Clearly, we need the ability to update our devices in the field; in the worst case we might even need to brick45 the hardware and send a replacement! How do we manage this process? Read on…

4.1.1 WiFi-based Provisioning

The most obvious way to provision (or provide network credentials to) a network-connected microcontroller like the ESP32 is to connect over WiFi and send the credentials over the radio. There have been several options developed to do this, including:

The SmartConfig control flow looks like this:

A more general solution, which is possible on all devices (like the ESP32) that support operating as a WiFi access point (AP), is to allow client connections over e.g. WPA + HTTP(S) (which is as secure as the rest of the web!) and then serve HTML to the client that allows triggering of a scan for available networks and entering of network credentials (which the device can then use to try and join itself, as a client). This is called WiFi-based provisioning, and is a robust and common approach. (The exercises this week are to implement this technique, using the webserver from last week. Flow of control is a little complex, involving the ESP32, your smartphone or laptop, and wireless router. Give yourself plenty of time to think it through, and try to develop your own solution before looking at how mine works.)

4.1.2 Over-the-Air Updates (OTA)

How can we ensure that IoT devices get patched promptly when security holes appear?

The answer is OTA, or over-the-air updates: when your microcontroller is connected via a fast network (e.g. over WiFi), it can download new firmware versions whenever they’re available. There are upsides, but also downsides:

Several partitions in flash are used for update staging, e.g.:

46

Here we see three partitions in use:

The sequence of an update then goes something like this:

Diagrammatically:

47

See below for an exercise to implement this style of OTA.

Note that the Arduino IDE also provides a form of OTA, accessible via the Tools>Port menu option when running an OTA capable sketch from a machine connected to the same network as the ESP. See File>Examples>ArduinoOTA>BasicOTA for an example sketch. (This only works via the IDE; more details on this video.)

We’ll move on to implementing OTA (on top of WiFi provisioning) next week.

4.1.3 WiFi Provisioning + OTA = ???

What happens when we combine both approaches under a web front end that provides management of the update binaries, device registration and the like? An early attempt at this type of system (for the ESP8266 and later ESP32) was built by Andreas Spiess and colleagues, called IoTAppStory:

This project provides a library for devices to poll a firmware repository and recognise when updates are available, then triggering OTA update. It follows an “app store” model as described in this video.

4.1.4 RainMaker: Official Provisioning & OTA

When we’re programming the ESP32, we can take advantage of RainMaker, which, from IDF 4.0 and above, is Espressif’s official solution to the provisioning and OTA tasks. (It also provides a simple device control and configuration mechanism via library calls and a phone app.)

We call the library from app_main, specifying any parameters we would like to expose on the device. We also upload firmware to a repository to support OTA. We can now manage the device from a phone app.

The advantages:

Disadvantages:

For example, if we have three devices integrated (or “claimed”) on our RainMaker account, the android app displays:

If we then open a device that exposes three controls, we see something like this:

All this is so neat, Espressif drew a pretty picture to go with it :)

Give it a try?

4.2 COM3505 Week 04 Notes

4.2.1 Learning Objectives

Our objectives this week are to:

4.2.2 Assignments

To turn things into internet things we need to connect them to a network. One way to connect a device without a user interface to a wifi network is to make it a wifi access point and serve a website that displays what network access points are available. We can then allow the user to choose an access point and enter the key.

4.2.2.1 Coding Hints

Ex07 is about templating: basic string manipulation to create HTML from component parts; the model solution uses an array of strings, but there are, as usual, many different valid solutions (inc. reusing 3rd party libraries).

Ex08 needs to use WiFiClientSecure, and talk to the server on the specified port from Blackboard, and get the URL and paths correct; under those circumstances it will return a line of text starting “Received….” Note that in some versions of the Arduino core you need a patch: see below.

Ex09 is about running a web server (and wifi access point to connect to it) which lists all the other access points the ESP is in range of, and allows you to connect to one of your choice.

We can use the solutions from Ex06 (create an access point and web server) along with Ex07 (utilities for creating web pages) to get us started here. (See the arrangements for running multiple setups and loops in main.cpp for an example of chaining the various exercises together.) Then for Ex08 and Ex09 we can use the WiFi and WiFiClientSecure classes, with calls to these methods (this isn’t complete!):

WiFi.begin(SSID, PSK);
WiFi.status();
WiFi.localIP();
webServer.handleClient();
myWiFiClient.connect(com3505Addr, com3505Port);
myWiFiClient.print(
  String("GET ") + url + " HTTP/1.1\r\n" +
  "Host: " + com3505Addr + "\r\n" + "Connection: close\r\n\r\n"
);
myWiFiClient.available();
myWiFiClient.stop();
myWiFiClient.readStringUntil('\r');

For Ex09 my code additionally uses:

WiFi.scanNetworks();
WiFi.SSID(i);
WiFi.RSSI(i);

Dive in!

4.2.2.2 Patch the core for Ex08

If you hit a problem getting your ESP to connect to the course server (using the WiFiClientSecure.connect() method) then it may be due to a mismatch between the way the server is secured and the version of the Arduino core library you’re using. (This library layers on top of a port of mbedTLS in ESP IDF to implement secure, socket-based web communication between clients and servers over HTTPS.) Make sure that your problem isn’t something simpler first (e.g. connection to the wrong wifi network!) but if you’re sure that your ESP can see the server but is refusing to connect, try the following.

The HTTPS server that we use to collect data from your ESP devices is hosted inside the University firewall, and has no external visibility. This means that we can’t use e.g. LetsEncrypt or similar to generate a signed X509 certificate, and have to use a self-signed certificate instead. What happens when we try to make a connection? A web browser will give the user a warning and allow them to make an exception if required, but in the IoT we typically have no user interface (and often no user present). So we need to tell the library code (in this case WiFiClientSecure) not to check the certificate authority (CA) chain.

In early versions of the Arduino core it is easy: just don’t present a certificate and it will connect without problems. In very recent versions it is also easy: supply a client certificate (self-signed) and call a setInsecure method and it will connect without checking the CA chain. Versions in the middle present a problem as neither option is available, and this includes the IDF 4.3 beta version that our default setup uses. To get over this problem, we must import (or “cherry pick”) a single commit that implements the setInsecure method. Assuming you’ve used the setup method magic.sh setup then you can do this as follows:

cd arduino-esp32
git pull
git describe
git cherry-pick ef99cd7fe7778719c92d6f8df0f10d3f0f7aa35e

Now edit libraries/WiFiClientSecure/src/ssl_client.cpp to resolve conflicts (search on “====” and delete that and the “>>>>” and “<<<<” parts). To tell git that the repository isn’t in conflict any more you can do:

git add `libraries/WiFiClientSecure/src/ssl_client.cpp` to register resolution

4.2.2.3 Which WiFi Network? What if it Doesn’t Connect?

It is possible to get an ESP to connect to Eduroam, but the Enterprise WPA security protocol can be tricky to navigate, so:

You may find it convenient to connect your ESP to a portable hotspot created via your phone or other device. On campus you can use the “other devices” network; see Blackboard for details.

If your ESP doesn’t connect, don’t panic. Try printing some diagnostics, phoning a friend or standing on your head and singing the Congolese national anthem backwards in Latin a couple of times. Then try some more diagnostics.

You may also hit problems getting other devices to connect to the ESP32’s internal access point. Different operating systems try to guess the characteristics of the networks they are asked to join in various ways, and this sometimes interacts badly with the model code. Android in particular may (or may not!) refuse to join the ESP’s access point, or join and then fail to deliver HTTP requests to the server we set up. Things to try if this happens:

When you’ve managed to get the ESP onto a wifi network, it will (on serial) report its IP address; you can then use that IP to connect to the web server instead of 192.168.4.1, which tends to be more reliable.

Finally, you probably don’t want to use the University’s WifiGuest network, as this requires sign-in.

4.2.2.4 Details of Our Cloud Server (for Ex08)

The internal cloud server for the course accepts simple GET requests with parameters that can be used to pass a little data to our persistent store.

The data format is key=value, e.g.: email=hamish@gate.ac.uk, and it requires a .ac.uk email address. A request to store the device’s MAC address would look like this: https://HOSTNAME.shef.ac.uk:9194/com3505-2021?mac=ThisESPsMAC&email=MyName@sheffield.ac.uk

(As before HOSTNAME is given on your Blackboard page for the course, and note that the server is inside the University firewall.)

4.2.3 Moving 9 LEDs to Matrixboard

Note: this requires soldering, which you may well not have done due to restricted lab access. No worries, just have a read through and (optionally) do it later on if you get in the lab.

Last week we built a 9 LED breadboard. This week the task is to transfer the circuit to a piece of matrixboard and solder it up into a more permanent form. (Breadboards are great for short-lived experiments, but some of the projects we’ll build later on are intended to be more robust, so we need to practice our soldering and circuitry skills a little more.)

You’ll need your breadboard, and some more parts from your kit:

Try folding the resistors in a group:

Place the resistors and LEDs first, ready for soldering:

Soldering this circuit is fiddly; remember to make sure both surfaces are hot before applying the solder.

Use bent wire from some of the components to form a ground rail:

When you’ve done the LEDs and resistors, add a ground jumper, and then move the other jumpers (connecting the GPIOs to the LEDs’ anodes) over:

Fold the wires together to make connections between jumpers, LEDs and resistors, and solder them:

Having an extra pair of hands helps!

Now burn the firmware in the (e.g. using the 9LEDs code).

4.3 Further Reading

Finally, (The Economist 2019)’s Chips with everything (from The Economist Sept 2019 Special Issue on the IoT) presented an interesting summary of the IoT gold-rush in general, and smart homes work in particular, which we excerpt below:

One way to think of it is as the second phase of the internet. This will carry with it the business models that have come to dominate the first phase—all-conquering “platform” monopolies, for instance, or the data-driven approach that critics call “surveillance capitalism.” Ever more companies will become tech companies; the internet will become all-pervasive. As a result, a series of unresolved arguments about ownership, data, surveillance, competition and security will spill over from the virtual world into the real one.

Start with ownership. As Mr Musk showed, the internet gives firms the ability to stay connected to their products even after they have been sold, transforming them into something closer to services than goods. That has already blurred traditional ideas of ownership. When Microsoft closed its ebook store in July, for instance, its customers lost the ability to read titles they had bought

Virtual business models will jar in the physical world. Tech firms are generally happy to move fast and break things. But you cannot release the beta version of a fridge.

In the virtual world, arguments about what should be tracked, and who owns the resulting data, can seem airy and theoretical. In the real one, they will feel more urgent.

The need for standards, and for iot devices to talk to each other, will add to the leaders’ advantages—as will consumer fears, some of them justified, over the vulnerability of internet-connected cars, medical implants and other devices to hacking.

The trick with the iot, as with anything, will be to maximise the benefits while minimising the harms. [p. 13]

Attracted by the lure of new business, and fearful of missing out, firms are piling in. Computing giants such as Microsoft, Dell, Intel and Huawei promise to help industries computerise by supplying the infrastructure to smarten up their factories, the sensors to gather data and the computing power to analyse what they collect. They are competing and co-operating with older industrial firms: Siemens, a German industrial giant, has been on an iot acquisition spree, buying up companies specialising in everything from sensors to office automation. Consumer brands are scrambling, too: Whirlpool, the world’s biggest maker of home appliances, already offers smart dishwashers that can be controlled remotely by a smartphone app that also scans food barcodes and conveys cooking instructions to an oven.

A world of ubiquitous sensors is a world of ubiquitous surveillance. Consumer gadgets stream usage data back to their corporate makers. Smart buildings—from airports to office blocks—can already track the people who move through them in real time. Thirty years of hacks and cyber-attacks have proved that computers are insecure machines. As they spread, so will that insecurity. Miscreants will be able to exploit it remotely and at a huge scale. [p. TQ 4]

Consumers can buy smart light bulbs, such as Hue from Philips, a Dutch electronics giant, which can be switched on or off by phone or voice and can generate thousands of tones and shades. Viewers of “12 Monkeys,” an American science-fiction tv series released in 2015, can download an app that will sync with their light bulbs, automatically changing their colour and brightness to match the mood of an episode moment by moment.

On the difficulties of hooking Smart Home systems together (Ben Wood):

“It’s a very Heath Robinson kind of patchwork, a jigsaw puzzle of connectivity.”

products from one manufacturer often fail to work well with those from another. Standards do exist: Zigbee and z-wave are wireless networking protocols designed for the type of low-power radios found in smart-home gadgetry. But many firms either use proprietary standards or implement existing standards in ways that prevent their products working with those from other companies.

Many companies are involved. Tim Hatt at gsma Intelligence says that telecoms firms are keen to find new, higher-margin businesses rather than simply acting as “bit pipes,” so they have built smart-home offerings as well. Vodafone, a telecoms company, advertises the v-Home hub as a central control point for smart-home devices. sk Telecom, a South Korean firm, has the Nugu. at&t, an American company, offers its Smart Home Manager. Others are startups, such as Wink, which launched with backing from General Electric. In Britain, even British Gas, a former state-owned energy monopoly, has got in on the act. It launched Hive, a smart-home ecosystem in 2013.

That fragmentation means risks

Until fairly recently, says Mr Wood, the assumption was that smart homes would be controlled from phones. But, he says, the reality is different. “Pulling out your phone, unlocking it, tapping an app, then using it to turn the lights on, is much more complicated and annoying than simply walking across the room and pushing a button.” Voice, he says, is by far the most convenient user-interface.

Amazon’s Alexa and Google Home, the two firms’ smart-speaker products, already have greater market penetration than rival smart-home hubs. [pp. TQ 5-6]

5 Sensing and Responding

We’ve looked at what the IoT is and where it came from. We’ve delved into the build and burn tools that we need to update the firmware on an IoT device. We’ve blinked (or actuated) LEDs and read (or sensed) from switches. This chapter will look in more detail at the various ways in which the microcontrollers in IoT devices (like the ESP32) talk to sensors and actuators, or, to put it another way, at the local protocols available on the device. We’ll also look in a bit more detail at what types of sensor and actuator we will typically encounter in the IoT.50

The practical side of the chapter will then continue from last week’s work on provisioning with an exercise on over-the-air (OTA) update.

5.1 Analog and Digital Sensors

A sensor is an electronic component designed to mesh the physical and digital worlds51 by converting a physical phenomenon into a signal which can be measured electrically. Sensors can be broadly classified as either analog or digital, and either active or passive. An analog sensor produces a continuous output signal whereas a digital sensor produces a discrete (usually binary coded) output signal. An active sensor requires an external power supply to generate an output reading whereas a passive sensor does not.

When we want to turn an analog signal into a digital one we need to think about both:

In other words, we have to consider how often we will sample the signal, and at what accuracy each of those samples will be taken. The consequences of these decisions dictate how much information loss the digitisation process will result in. Visually:

52

The X axis in these graphs represents time (“t”) and the Y axis is the strength of the analog signal (or amplitude, “A”). If the signal is varying very rapidly then we need to sample very often in order to capture the changes, and if the signal is varying between very large and very small amounts then we need a large number to store all its different possible levels. For sound waves, sampled from the output of a microphone, it is typical to take more than 40,000 samples per second and to use a 16 bit number to distinguish the loudness level at each of those points in time. (This is the standard which CD audio uses, for example.)

5.1.1 Two Ways to Sense Light Levels

As an example of analog vs. digital sensing, consider these two light sensors, both of which convert the amount of light incident upon them into electrical signals:

Many hundreds of different sensors are available, measuring everything you can think of and a few others besides. They range in price from a few pennies to many thousands.

In common with many Swiss people, Andreas Spiess lives in a building with an atomic shelter. Check out this video for his description of building a Gieger Müller Tube sensor (or Geiger Counter).

5.2 Reading from Analog Sensors

In general when designing sensing equipment in our IoT devices we prefer to find digital equipment that talks to the microcontroller on one of the standard buses that our SDK has library code support for (e.g. SPI, I2S or I2C; see below). In cases where we can’t find a digital version of the sensor, however, we can fairly easily perform direcy measurement of analog signals on the ESP32 as follows.

To begin with we need to check that the voltages produced by our sensor fall into an appropriate size range for the microcontroller to measure. If the values are too high we will need to reduce them; they are too small then we’ll need an amplifier. The former is easy, the latter not so :(

Let’s say that we’re using a range of between zero and one volt (or 0–1.1V as used in this example from Espressif). We can easily expand the range to say 0-5v using a potential divider, in which two resistors allow us to tap off a reduced signal like this:

The signal at “Vout” is found according to this formula:

Online tools can help with the maths, choosing the best fit of resistor values and so on.

We have expanded the range of signals we can deal with by reducing the signal that presents at the microcontroller (as “Vout”). In the other case (where we have signals too small to measure), reducing the input range (e.g. to 0-0.1v) requires active amplification of the signal, which we might typically address by adding a dedicated circuit board for the purpose. The amplification problem can be quite hard to solve reliably, as when we increase the signal we are likely to increase the amount of noise we’re picking up at the same time. Specialist sensing devices for very small analog signals are often quite expensive as a result!

After reading from an analog sensor, we perform the conversion to digital by means of an Analog to Digital Converter (ADC). An ADC samples the analog voltage and converts it into a digital number. The ADC has a voltage range and a number of bits; in our case on the ESP32 these are 0-1V and 12bit respectively. (Other ranges are available but aren’t very linear.) If we use 12 bits this means the digital number reported is between 0-4095 (2^12 = 4096). (Note that this limits the resolution of our measurements to ≅ 0.25mV.)

Here’s the code to take a reading from pin A0 (which connects to the ESP’s analog-to-digital converter two, ADC2):

const int sensorPin = A0; // select the input pin for the analog reading
short sensorValue = 0;    // 16 bit integer for value from the sensor

void setup() {
   Serial.begin(115200);  // initialise the serial monitor at 115200 baud
   pinMode(sensorPin, INPUT);  // set the sensor pin to be an input
}

void loop() {
   sensorValue = analogRead(sensorPin); // take analog reading and store
   Serial.println(sensorValue);    // print the value on the serial monitor
   delay(1000);    // wait for 1000ms (= 1 second)
}

Does it feel strange to see an integer (short sensorValue above) used for an analog voltage? As noted earlier, when we digitise an analog signal we’re shifting it from the continuous world (which might at first sight seem more naturally represented as floating point) to the discrete. As long as our integer is wide enough to store enough amplitude values for the purposes we are reading the sensor for, then it is perfectly appropriate.

In the exercises section below we’ll look at the ESP32’s (capacitative) touch sensing capabilities, which also read an analog signal and produce a digital output.

5.3 Digital Sensors

In some cases the signal produced by a sensor is binary to begin with – for example, switches may be either on or off, and we can read that binary state directly from the microcontroller’s general purpose input/output (GPIO) pins.

Example code:

const int switchPin = 12; // select the input pin for the switch
bool switchState = 1;     // for the value coming from the sensor

void setup() {
  Serial.begin(115200);  // serial monitor at 115200 baud
  pinMode(switchPin, INPUT_PULLUP);     // pullup, see below
}

void loop() {
  switchState = digitalRead(switchPin); // read digital input
  if (switchState == false) {
    // do stuff when switch pressed
    // (note reverse logic i.e. false means pressed)
  }
}

It is also now common for sensors to be pre-packaged with dedicated circuitry to do the analog-to-digital transformaion before the signal ever leaves the sensor housing. In this latter case, it is usual for the output from the sensor to use one of a handful of dedicated communications protocols intended; the next section describes several of these protocols.

5.3.1 Avoid Floating Voters

When using digital reads we need to be careful that our input pins aren’t left “floating” in order to minimise ghost signals that may otherwise be triggered by noise picked up on the circuitry we’re connected to. In other word, inputs don’t like to be left unconnected or “floating” – this results in fluctuating signals and can cause other, intermittent and difficult to diagnose problems.

It is common therefore to add a resistor that bridges the input to a stable value, either high or low. When connected to the positive supply voltage, it is a pull-up resistor. When the resistor is connected to ground the resistor is called a pull-down resistor. So long as the resistor is a moderately high value it will stop the input floating, but doesn’t interfere with normal operations (5-10Kohms is typical).

An example schematic of a pullup resistor:

pullup resistor

And a pulldown:

pulldown resistor

5.3.2 Vcc by any Other Name Would Smell as Sweet

While we’re talking about circuit schematics, let’s clear up some terminological issues. Confusingly there are several different terms for the positive and negative voltages that power electronics:

Clear? You’re a genius! Give yourself a huge slap on the back and treat yourself to a chocolate bar this instant.

5.4 Local Protocols: UART, SPI, I2C, 1-Wire…

IoT devices have more protocols associated with them than the average farm-yard dog has fleas. WiFi, LoRaWAN, Bluetooth, NB-IoT, Sigfox… although these vary in range from a few meters (Bluetooth) to several tens of kilometers (LoRaWAN etc.) they are all, relatively speaking, long-range protocols. In this section we look at short-range, or local protocols.

Local protocols define communications mechanisms for microcontrollers to talk to sensors and actuators using buses of various types. For example, the I2C (Inter-Integrated Circuit) protocol toggles the values of a Serial Data Line (SDA) and a Serial Clock Line (SCL) to transmit data up to a few megabits per second. On an oscilloscope an I2C transmission might look like this:

Different protocols use different transmission strategies, and their data rate, viable circuit length, reliability and ease of use all vary depending on how these strategies are organised. One thing that they all do is to provide an easier data transfer method than bit banging: toggling GPIO directly with firmware to send signals to or collect signals from peripheral sensors or actuators. Except for the very simplest of cases (like those exemplified above where we read a switch state from an input pin), local communications protocols are a more robust and simpler to implement option.

We’ll look at a small handful of common protocols below, after a note on terminology.

5.4.1 Terminology

In the old days, when I were a lad, we used to call some devices ‘master’ and others ‘slaves.’ Very, very slowly, engineers became aware that using these words was offensive to many, and now there is an ongoing project to replace these terms. You may still find these terms used, especially in older texts, my preferred alternative is ‘main’ and ‘secondary’ – as they preserve the initial letters (electronics loves its acronyms!) – but others use ‘controller’ and ‘peripheral.’ Please don’t use the obsolete terms, and if you find any reference to one that I’ve missed, please let me know!

5.4.2 UART

The Universal Asynchronous Reception and Transmission (UART) protocol (or serial port) is at the simple end of the food chain.

As in other cases, digital highs and lows are the basis for communications for the UART protocol. The most common setup is to have a single start bit (LOW), 8 data bits and then a stop bit (HIGH). Each group of 8 data bits is a byte that represents a character in ASCII.

53

Seeedstudio provide a useful comparison of UART, SPI and I2C:

UART I2C SPI
complexity: simple easy to chain many devices complex as devices increase
speed: slowest faster than UART fastest
# devices: up to 2 devices up to 127 but may get complex as devices increase many, but there are practical limits and may get complicated
# wires (plus ground): 1 2 4
duplex: full duplex half duplex full duplex
# mains / secondaries: single only multiple secondaries and mains only 1 main but can have multiple secondaries

There’s a code example for UART below.

5.4.3 SPI

The Serial Peripheral Interface (SPI) is a bit more complex, using synchronous signaling with a clock line in addition to the communications lines. This means that communications can be much faster than using a simple serial protocol like UART. In addition to these lines, each device has a chip select line that is used to select the device that you want to respond to the data. This means that several devices can use the same bus, only talking on the bus when their chip select line is pulled low.

54

Counting ground, this SPI needs four bus lines: MOSI (Main Out Secondary In); MISO (Main In Secondary Out); SCLK (Serial Clock). In addition each device needs its own chip select line.

5.4.4 I2C

The Inter-Integrated-Circuit (I2C, or I²C if we’re being posh) protocol is more complex still, using its two wire bus to potentially link up to 127 devices.

55

Secondary devices are addressed by the main device using signals on the bus itself instead of using separate select lines, and each device listens to the bus and only responds when it is addressed directly.

I2C is alone in the buses we’re covering here in providing a delivery guarantee, with confirmation that the listener device received the payload. Note that this sometimes results in bus lock-ups!

There’s a code example for I2C below.

5.4.5 1-WIRE

Yet more complex (and less common in the circuits we have worked with) is 1-Wire, a protocol that is somewhat similar to I2C but, as its name suggests, only uses a single data line (plus, as usual, ground).

Complex protocol, you should be aware of the details and where to find them if you need to, but happily, a well tested, documented library is available for use with minimum effort. Like I2C 1-Wire allows multiple devices. It supports a lower data rate than I2C but longer cable length (up to 50m+ vs. 1-2m).

Notably, 1-wire is used by Apple MagSafe power supplies (and by Dell and others) to exchange signals between the computer being powered and the circuitry controlling the PSU. No doubt there is an advantage to this arrangement somewhere, but as Gareth says, “Now my laptop won’t charge unless I have an approved charger – tor my own comfort and safety.” :(

5.4.6 Other Local Protocols

Other protocols we may come across include:

5.4.7 Talking the Talk: Local Protocol Examples

As with all but the most basic of communication protocols, the actual gorey details how data is signalled, how noise and errors are minimised, and how devices keep out of each other’s way when sharing buses are complex. Luckily, in the Arduino ecosystem other people have done the hard work of encapsulating all these details in libary code. To put it another way, open source allows us to see further by “standing on the shoulders of giants” :)57

So you (mostly) don’t need to worry about the details of the implementation, with start and stop bits, ack(nowledgement)s, bit order, etc. (At least, that is, until stuff starts going wrong!)

Here’s an example of using the Arduino versions of UART and I2C (the second of which is known confusingly as Wire). You have already used the built-in UART library to print to the serial monitor:

Serial.begin(115200);    // initialise the serial monitor at 115200 baud
Serial.print(“Hello world”);    // simple printing characters on the serial port

SPI and I2C are as easy to use (I2C example shown here):

#include <Wire.h>           // include the I2C library
Wire.begin();               // initialise the port;  pins 22 & 23 on ESP32
Wire.beginTransmission(44); // transmit to device #44 (0x2c)
Wire.write(byte(0x20));     // write the value 20 in hex (32 in decimal)
Wire.endTransmission();     // stop transmitting

This pattern (of using library code to do the heavy lifting is typical of our microcontroller/sensor/actuator ecosystem – or, to quote Gareth again, “the giants are stacked!”

Most of the time, you can use a pre-written library that deals with all of the communications details for you. These provide functions such as readSensor() – this typically requests a reading from the device, gets the response and formats it for you. For example the Adafruit library for the TSL2561 light sensor provides the functions setGain() and getLuminosity(); a getDustDensity() is provided by the dust sensor library we have used for air quality monitoring; and so on.

As usual, our good friend Andreas Spiess has a nice video that’s relevant, this one showing how to “connect many devices (sensors and displays) to one Arduino using the I2C bus. It starts with a simple hommade bus, shows how to find out the addresses of the different devices and ends with a demo of a system with three devices connected to an Arduino.” Recommended.

Ok, that completes our brief tour of local protocols. Now let’s finish off this discussion with a look at actuators, or the “arms and legs” of our “world robot,” to echo (Schneier 2017).

5.5 Actuators

Broadly speaking these devices change the physical world in response to electrical signals. These signals might be the amplitude of a sound pressure wave (to drive a loudspeaker), or the direction and speed of travel of a motor or solenoid.

Outputs can also be information, of course – such as a tweet, an SMS, a record in a database, or etc. – and the change that we want to create can just be a signal of some type – lighting an LED, perhaps, or running the vibration motor in a smartphone or smartwatch.

As with our protocol-based communications above, or our digital sensor readings before that, we’re generally in the game of picking up an existing library for whatever actuator hardware we’re trying to control. More giants :)

The ESP32 has several kinds of outputs that can drive actuators:

None of these outputs can supply much current – 12mA at maximum. (By Ohm’s law this 12mA limit at 3.3V implies we need a resistance of at least 275Ω in the circuit we’re driving.)

In practive we often need an amplifier or driver to supply the power the actuator needs, and we may also wish to isolate our low-voltage circuitry from the equipment we’re controlling – this is especially true of mains voltages, which can kill! (Never work with mains circuits if you’re not a qualified electrician!) Below we’ll talk about two ways to deal with higher power or higher voltage actuators.

5.5.1 High Power Actuators with Relays

If we want to switch a device which requires a current larger than a few miliamps we can use a mechanical device called a relay. (We might also commonly use transistors, especially MOSFETs, though these are a little more complex to wire up.)

In relays we have a control circuit, and a switching circuit. A small amount of power applied to the control circuit operates an electromagnet. The magnetic force causes the switch contacts to open and/or close, thus controlling the switch circuit. The switch is completely separate from the electromagnetic coil:

When the coil power is removed, the magnetic field collapses very quickly, inducing a massive voltage. We can protect against this voltage spike with a ‘snubber’ diode. Other devices that operate with a coil such as solenoids and motors also need snubbing.

5.5.2 High Voltage Actuators with Radio Control

How do you switch mains without a risk assessment?! We encourage our students to be adventurous in their learning… but please don’t touch any circuits that use mains electricity!

There’s an easier way, which is also very safe: use a remote control power socket that responds to radio control signals to switch mains electricity. We don’t need to touch the dangerous stuff. Instead, we send commands on the radio frequency (433MHz) that the sockets are tuned to. The transmitter is a low-voltage actuator component which is quite harmless:

The sockets are an off-the-shelf consumer item (which we never open or modify in any way!):

Again, a pre-written library does all the hard work – e.g. mySocket.send(4281651, 24);. More on this in Chapter 8 for our home automation projects.

5.5.3 Electric Blankets, Fish Farming and Liverpuddlians

What do cold nights, Liverpuddlian urban agriculture and the IoT have in common?!

This is a screenshot of a web page served from an ESP32 used to control

Now you know.

5.6 COM3505 Week 05 Notes

On the practical side of the course we’ve also now covered quite a lot of ground: from blinking LEDs and reading from switches to getting our devices to talk to WiFi and push data into the cloud, and last week we worked on provisioning. Let’s tie this together with a working demo of Over-the-Air update, and we will be fairly close to calling our first tour of IoT programming complete.

5.6.1 Learning Objectives

Our objectives this week are to:

5.6.2 Assignments

Exercises:

Lots of notes below!

5.6.2.1 ProUpdThing: Provisioning and Firmware Update

The course repository contains exercises/Thing/main/Ex10.cpp and also exercises/ProUpdThing/, which both implement firmware for WiFi-based provisioning and Over-the-Air (OTA) updates. The former uses a local HTTP server to make the .bin available for download by the ESP; the latter uses GitLab to host the .bin and does an HTTPS GET from there. For a third, more production-ready approach, have a look at Espressif’s RainMaker API, which hosts updates on AWS and provides an oven-ready C API for driving OTA on the ESP (via a webapp).

While working, think about possible future enhancements:

5.6.2.2 Configuring ProUpdThing

The ProUpdThing build assumes that a file called private.h exists in your home directory ($HOME). To allow the firmware to download from your (private) GitLab repo, you must create a Personal Access Token (PAT) on gitlab.com and add it to private.h:

Screenshots (with my token hidden!):

You also need to define your project ID (which is a unique identifier for your gitlab repository; it appears next to the project title on the top-level page). Copy it and define it in your private.h file:

#define _GITLAB_PROJ_ID "1234567"

Check that you have everything in place by building, flashing and performing updates as below.

The firmware does two jobs, the first being to allow wifi provisioning, the second to support OTA updates.

5.6.2.3 Provisioning

When the firmware starts up, it calls a library function (joinmeManageWiFi) to check if WiFi has previously been configured and if so try and join the network. If that fails, it starts up its own WiFi access point on the ESP32 and runs a webserver on port 80 (with DNS capture, which works most of the time; if not, try 192.168.4.1). If you join the access point from a phone or laptop, the webserver provides a list of other available access points, and allows you to enter the network key. It then shuts down its own access point and joins the network you specified.

A diagrammatic view:

5.6.2.4 OTA

When WiFi has been successfully configured and the ESP has an internet connection, another library function (joinmeOTAUpdate) checks to see if new firmware is available on gitlab. This is arranged via files pushed to a subdirectory called firmware/ as follows.

When a .bin has been exported from the IDE or built via IDF it is pushed to gitlab in a file called firmware/<<version>>.bin (e.g. firmware/6.bin for version 6). The highest version number available is stored in a file called firmware/version. In the code (see main.cpp) an int variable called firmwareVersion is set to the new version. We can then trigger an OTA update by pushing new versions of the .bin and version files, and restarting the ESP.

The project magic.sh script includes a command push-firmware that automatically takes the new version and pushes it to gitlab with an updated version file.

Experiment with the operation of the firmware, until you understand this process.

For example:

cd ProUpdThing
./magic.sh

Now we’re watching what is happening on the device on serial. In a different terminal edit main/main.cpp, make a trivial change (e.g. Serial.printf("now running v%d\n", firmwareVersion");), and increment the version number. Then:

In another terminal:

./magic.sh push-firmware

You should now see the device check gitlab, recognise the presence of new firmware, and update.

5.6.2.5 Hints

5.6.3 The ESP’s Sense of Touch

The ESP32 provides touch sensing capabilities via 10 of its GPIO pins. Each of these pins can measure capacitative variance incident on electrodes embedded in touch pads. (Or, if we’re doing things in a hurry, a bit of handy wire connected to the relevant pin!)

The ESP’s Arduino core defines shortcuts for the GPIO pins that are touch-capable (and accessible on your boards) as follows (in file .../arduino-esp32/variants/feather_esp32/pins_arduino.h):

static const uint8_t T0 = 4;
static const uint8_t T1 = 0;
static const uint8_t T2 = 2;
static const uint8_t T3 = 15;
static const uint8_t T4 = 13;
static const uint8_t T5 = 12;
static const uint8_t T6 = 14;
static const uint8_t T7 = 27;
static const uint8_t T8 = 33;
static const uint8_t T9 = 32;

As usual, these capabilities can be accessed in firmware via both an Arduino API (which is fairly basic) or an ESP IDF API (which is more powerful but more complex).

In Arduino-land, the following will read a value from GPIO 14 and store the current value in tval:

int tval = touchRead(T6);

(What is GPIO 14 and how do I find it?! See the discussion on pinouts in sec. 3.3.6.1.)

The values returned from touchRead (or the IDF equivalent touch_pad_read) are not binary (unlike digitalRead, for example), but represent an analog measure of capacitance. To control an OTA process, we probably only need a binary output (“is the user touching the magic pad or not?”), so your code will need a method for translating the analog signal to a binary yes/no decision. There is also noise in the signal, so you may need to discard outlying values or perform averaging of values.

Below some example output from the Arduino IDE serial plotter, with a jumper cable attached to GPIO pin 14, and Serial.printf("%d\n", touchRead(T6)); in loop(). The high values (around 75) are where there is no touch happening; the first low range (around 60) is where I’m touching the outside of the cable (which is covered in plastic); the second low range (around 15) is where I’m touching the metal end of the cable. In all cases there are random spikes of noise!

Serial monitor touch

(The serial monitor is accessible via the IDE’s Tools menu.)

The IDF API for touch sensing is detailed here. Using this API in filtered mode (with a filter period of 100mS) results in a graph like this (for similar test conditions as above):

Serial monitor touch, IDF API

There’s considerably less noise in this version (though note the relatively slow descent to the low state for a pin touch).

6 Machine Learning and Analytics in the Cloud

It is time to step back a little from the device and its local protocols for talking to sensors and actuators. Below we’ll discuss processing your data to produce efficient prediction models that can run at the edge, and how machine learning plays out in the IoT. But first, a little digression about AI.

6.1 Is AI about Intelligence?

I was talking to a friend of mine recently who consults for UK universities on e-learning tech and is writing a book about critical reasoning in the age of algorithmic intelligence. In the 1980s we studied cognitive science and artificial intelligence (AI) together, and have watched with interest the recent resurgence in the latter. When we started out AI was first and foremost a project to understand human intelligence by modelling it on computer. By contrast, in the twenty twenties AI is about using applied statistics58 to estimate the probability of a translation fragment, or steer a self-driving car, or recommend a product to someone who just surfed to the page of a similar item. It is also much better funded and much more noise is made about how transformative it may be or become.59 How has this happenned? Were there specific breakthroughs? Changes in social and economic context?

My friend and I ended up swapping emails on the following question:

To what extent – if at all – does the claim survive that AI replicates, reproduces or tells us something useful about human intelligence? Is there an acknowledged end to that narrative (and if so, when and why?)

I think that what happened is that it became vastly profitable to use applied statistics (aka Machine Learning) to tailor ecommerce websites to recommend products to customers that other customers have previously bought when shopping for similar items. This is the foundation of the business models of Google and Facebook, for example. In the case of the former, their initial rise to prominence (and ability to sell advertising) was, of course, based on web search, and this led by various paths to them becoming key defenders of an open web (in the sense invented by Berners-Lee). If Google can’t index it, you can’t search it, and someone else (e.g. Facebook in their partly walled garden, or Apple in theirs) gets to sell the advertising instead. This polarisation of digital corporate conflict encouraged Google in providing all sorts of Really Useful Infrastructure, from satnav and maps to calendars and gmail, and to making these offerings all work as well as possible, mostly for free – or in exchange for your data, of which they amassed a volume for which the term humungous rapidly became quite inadequate….

And it turns out that if you have a really huge number of examples of human beings behaving in particular ways (especially if those ways involve a finite set of decisions: which book to buy, or who to include in an email cc. list, or which route to drive around a traffic jam), then you can get computers to do Really Useful Stuff in supporting human intelligence. (Not least: get speech recognition to work well enough that lots of people now want to use it in their homes, on Alexa or Home or Siri.)

Is this AI, in any sense that the original researchers of that field would have recognised? Probably not, but by now few people remember or care. The amounts of money being made are vast, and they drive the research agenda and reward the acolytes who, of course, invent new rafts of terminology to repel the casual boarder and valourise their knowledge. AI became almost synonymous with Machine Learning (counting lots of occurrences to inform probabilistic choice); Deep Learning (using perceptrons and other network representation models, and stacking them) became one of the chief mysteries; “Big Data”(and data analytics) came to be touted as the saviour of high tech UK60; and so on.

The project of understanding human intelligence, then, is now rather eclipsed (at least in my neck of the woods), by the “let’s get some data and count features and do prediction” school. Will this trend connect up again with the original project? Will progress be more substantial now that we have increased our data set sizes by orders of magnitude? Or is it, as one of the first web browser engineers61 once said, the fate of our generation to spend our best years working on advertising?!

You have more chance of finding out than I do. Enjoy the chase!

6.2 IoT, Big Data Analytics, and Deep Learning

6.2.1 Machine Learning at the Edge62

Machine Learning has gone through some tremendous growth during the last decade, mostly driven by progress in training deep neural networks. Their benefits are impacting more and more applications, used in image classification, object detection, speech recognition, translations and many more. Wherever data is being generated, it is likely that machine learning can be used to automate some tasks. That is why it is important to study machine learning in the context of data generated by IoT devices.

Currently, machine learning operates under a common computing paradigm. Data is moved to the cloud (or a centralised server) for data processing. There, data is labeled and sanitised for quality control, and can be used to train machine learning models. How these models are used can differ: either continue to upload data to the cloud for run-time inference (e.g., voice recognition with Alexa, or Siri), or move the model to the edge to perform inferences close to the data source. Due to privacy concerns the latter is likely to grow in adoption over the coming year. Here, we are going to explore the process of training a neural network model and how this can be made available and used at the edge (on your IoT device).

6.2.1.1 Motivation

Edge computing devices are usually constrained by resources (battery, memory, computing power, etc.). The amount of computation they do is thus limited, and so should the machine learning models running at the edge.

If you have ever used the Google personal assistant on your phone, you would know that this can be triggered by using a keyword “Okay Google.” This relies on a voice recognition model running continuously on the phone to detect when you are calling it. Because the CPU is often turned off in idle mode to save battery, on most devices the keyword spotting task runs on a DSP (Digital Signal Processing) unit. The DSP consumes just milliWatts (mW) of power, but it is limited in its computing resources, having just kilobytes (KB) of memory available. So the keyword spotting model needs to be really small. For instance, the Google assistant voice wake-up model is just 14 KB. The same resource constraints we find on microcontrollers, having just KB or memory. We will see in this section how we can build such small machine learning models and how these can be deployed on small devices.

6.2.1.2 Introduction to Machine Learning

Although Machine Learning may see daunting at first, involving a lot of maths and statistics, some basic concepts can be applied out of the box, without going too much into details of how these work and how you can develop the optimal training process for your task. The latter is still an active research area, and even machine learning specialists cannot agree on the optimal methods. But getting satisfactory results from your machine learning models can be achieved with default options and some parameter tuning as we will learn in this section.

We use Machine Learning when the patterns in data are not immediately obvious to us for how to programme about them. For instance, we know for sure that water boils at 100°C, so if we have a reliable thermometer we can programme with a hard threshold on that value. That is how the common thermostats in boilers and refrigerators are built. But if we work with noisy data, or the data exhibits patterns that are too complex for us to spot at a quick glance, it is better to use machine learning for those tasks. Voice recognition is one example, where direct thresholds are hard to impose. That is because each time we utter a word or phrase there is some slight deviation in the tone, pronunciation, opening of the mouth, angle to the microphone, background noise and many other altering conditions. But machine learning can extract information from a lot of data examples to automatically focus on the more meaningful attributes during training. Once such a model is trained, we can use it in our programmes to make inferences in the real-world without understanding the complexity of the data ourselves.

There are many different approaches to machine learning. One of the most popular is deep learning, which is based on a simplified idea of how the human brain might work. In deep learning, an artificial neural network is trained to model the relationships between various inputs and outputs. We call it deep because modern network architectures have many stacked layers of parameters or weights (resembling the connections between brain neurons). Each architecture is designed for specific tasks and a lot of effort has been going in research over the last few years to determine the best architectures for each task. For instance, image classification works well with Convolutional Neural Networks, whereas text translations benefit from Recurrent Neural Networks.

In the following sections we are going to have a crash course on the most essential part of deep learning to get you started with training and deploying a machine learning model. It is by no means complete of what you can do in machine learning, but we hope this will get you curious and excited about this topic so you will study it in greater details in other courses.

6.2.1.3 Training Deep Neural Networks

Training is the process by which a model learns to produce the correct output for a given set of inputs. It involves feeding training data through a model and making small adjustments to it until it makes the most accurate predictions possible.

Training a neural network will require a sizable dataset of plausible examples. It is important that your model is trained on a distribution of data which is as close as possible to the real data. Assuming you have a dataset of labeled instances (annotated with the ground-truth class, for example the uttered word in keyword spotting), you will assign a portion of that data for training, a smaller fraction for validation (to evaluate the choices of parameters you make during training) and finally a test set that you never touch until the very end when you are happy with your model’s performance on the validation set and you want to determine its final performance (inference accuracy) on the test set. A common split is 60%:20%:20%, with the largest set always for training.

During the training process, internal parameters of the network (weights and biases) are adjusted so that prediction errors on your training set are slowly diminished. In the forward-pass through the network, the input is transferred at each layer in the network, all the way to the final layer which produces a data format that is easily interpretable. The most common output format of a classification neural network is a vector of activations where each position is associated with a different class. The vector position with the highest value indicates the winning class. For instance, if we want to detect a short spoken command (Yes/No), we estimate between the following classes “Yes,” “No,” “Silent” and “Unknown” (in that order in the output vector). If our network produces the following output vector: [0.9, 0.02, 0.0, 0.08], we will associate the estimation to the class “Yes.” The network is trained by adjusting the weights of the network such that the output will get as close as possible to [1, 0, 0, 0] if “Yes” is indeed the ground-truth for the spoken word. This is done by training on many more examples at once, each contributing to the adjustment of weight values. This is repeated several times over the same training set (epochs). The training stops when the model’s performance stops improving.

During training we look at two metrics to assess the quality of the training: accuracy and loss, in particular on samples selected from a validation set (different from the training set). The loss metric gives us a numerical indication of how far the model is from the expected answer, and the accuracy indicates the percentage of times the model chooses the correct class. The aim is to have high accuracy (close to 100%) and low loss value (close to 0). The training and validation sets should have a fair distribution of samples from all the classes. Of course, these are task dependent and for other more complex tasks even getting to 70% accuracy is considered a very good achievement.

The figure above shows a common training behaviour pattern. Over a number of epochs (iterations over the training set), the accuracy increases and stabilizes at a certain value – at which point we say that the model has covered; and the loss decreases towards its convergence point.

Training neural networks is usually based on try and error in choosing the right training parameters (or hyperparameters because they are empirically determined). After each adjustment, the network training is repeated and metrics visualised until we get to an acceptable performance. The common hyperparameters adjusted during training are: learning rate, which indicates how much the values of weights should be updated at each epoch. A common value to start with is 0.001, network structure – number of layers and the number of neurons per layer, learning optimisation parameters (such as momentum, etc.), activation functions (sigmoid, tanh, ReLU), number of training epochs.

Once we are happy with the model produced during training, we test it on the test set to assess its final accuracy. We are then ready for deployment to the device where inferences will be produced. But there are some more optimisations that can be performed to make the model run more efficiently on the device.

6.2.1.4 Neural Network Quantization

When deep neural networks first emerged, the hard challenge was getting them to work at all. Naturally, researchers paid more attention to training methods and model recognition accuracy, and little on inference performance on hardware. It was much easier for researchers to work with floating-point numbers, and little attention was given to other numerical formats.

But as these models become more relevant for many applications, more focus is now going into making the inference more efficient on the target device. While the cost of training a model is dominated by the cost of the data engineer time, the operational cost of running inefficient models on millions of devices dramatically outbalance the development cost. This is why companies are now so interested in designing and training the best machine learning models that can run efficiently on edge computing devices.

Quantization is one popular solution to make neural networks smaller. Instead of storing the weights of neural networks in the traditional 32-bit floating-point numerical representation, quantization allows us to reduce the number of bits required to store weights and their activations. Here we will look at quantization, reducing weights to 8-bit fixed-point representation.

Definition: Quantization is an optimisation that works by reducing the precision of numbers used to represent the model’s parameters, which by default are 32-bit floating-point numbers. This results in smaller model size, better portability and faster computations.

Floating-point representations are essential during the training process for very small nudges of the weight values based on gradient descent. While that data representation makes a lot of sense for infinitely small adjustment of the weights in several passes over the training set, during the inference time an approximated result for the estimation will likely still determine the same class (although with less intensity on the last layer of the network). This is possible because deep neural networks are generally robust to noise in the data, and quantized values could be interpreted as such.

The effect of applying quantization is a little bit of loss in accuracy (the approximated result will not always match that produced by the trained model in 32-bit floating-point representation). But the gains in terms of performance on latency (inference time) and memory bandwidth (on device and in network communication) are justifiable.

6.2.1.5 The Quantization Method

Neural networks can take a lot of space. The original AlexNet, one of the first deep neural networks for computer vision, needed about 200 MB in storage space for its weights.

To take an example, let’s say we want to quantize in 8-bit fixed-point (using integer values between 0 and 255), the weights of a neural network layer that has values between -2.5 and 6.5 in 32-bit floating-point representation. We associate the smallest weight value with the start of the quantized interval (-2.5 will be represented as 0) and the largest with the maximum value (6.5 will become 255). Anything between the min and max will be associated with the closest integer value in a linear mapping of [-2.5, 6.5] to [0, 255]. So with this mapping, 2.4912 will be 127 because it falls approximately at the middle of the interval.

scale = (max(weights) - min(weights)) / 256

x_code = quant(x) = (x - min(weights)) / scale

x_reconstruct = dequant(x_code) = min(weights) + x_code * scale

The first benefit is that the storage of the model is reduced by 75% (from 32 bits down to 8 bits representation of each weight). This can give substantial benefits if only just for communication cost and flash storage and loading. In the simplest form of device implementation, the numbers can be converted back to 32-bit representation and using the same computation approach in 32-bit precision to perform the inference. But with newer computation frameworks (e.g., TensorFlow Light) the weights can be used directly in 8-bit representation for integer operations (inputs are also quantized to 8-bit precision). This will also take advantage of the hardware support for accelerating to perform the operations in parallel in a single instruction with SIMD (single instruction multiple data) configurations. With this, 4 times more 8-bit values are loaded from flash in the same instruction and more values are stored in a single register and in the cache. If implemented efficiently, theoretically this can speed up the execution by 4x (performing 4 different 8-bit operations in the same clock time needed for one 32-bit operation). Not to mention that integer operations are generally more efficient in hardware, so they consume less energy compared to floating-point computations.

The advantages of quantization:

In summary, quantizations are great, because if you reduce the model representation from 32-bit floating-point representations to 8-bit fixed-point representations you save storage space, memory bandwidth on loads, and even speed up computations and energy consumption where hardware permits it. And for some devices (such as 8-bit microcontroller) using quantization may be the only option to run inference models on them.

6.2.1.6 Keyword spotting exercise

Has a lot of applications to enable the triggering of speech recognition. For instance okay google, or Alexa are keywords that trigger the device to start recording your commands. Your voiced commands are then sent through the internet to the cloud for speech recognition, to interpret your commands into computer instructions. However, knowing when to start recording your commands is important. It would be too costly to stream continuous recording to the cloud, not to mention privacy invasive. But running the keyword spotting algorithm locally, on your device in the home is safe and cost effective. The audio is streamed to the microcontroller which analyses with small and efficient algorithms to spot when the enabling word is used.

Step 1 Data collection

For Keyword Spotting we need a dataset aligned to individual words that includes thousands of examples which are representative of real world audio (e.g., including background noise).

Step 2: Data Preprocessing

For efficient inference we need to extract features from the audio signal and classify them using a NN. To do this we convert analog audio signals collected from microphones into digital signals that we then convert into spectrograms which you can think of as images of sounds.

Step 3: Model Design

In order to deploy a model onto our microcontroller we need it to be very small. We explore the tradeoffs of such models and just how small they need to be (hint: it’s tiny)!

Step 4: Training

We will train our model using standard training techniques explored in Course 1 and will add new power ways of analyzing your training results, confusion matrices. You will get to train your own keyword spotting model to recognize your choice of words from our dataset. You will get to explore just how accurate (or not accurate) your final model can be!

Step 5: Evaluation

We will then explore what it means to have an accurate model and running efficient on your device.

6.3 COM3505 Week 06 Notes

6.3.1 Learning Objectives

Our objectives this week are to:

6.3.2 Assignments

(Note: normally at this point we ask you to choose project hardware. Unfortunately due to the pandemic we’re starting off with only one project, so that we can post out kits to everyone who won’t be in Sheffield. We will try to allow you to switch, or augment, the hardware for that project after Easter if circumstances allow. The good news is that the hardware we’re giving out straight away is this really cool ESP32-based smartwatch.)

6.3.2.1 Coding Hints

6.3.2.2 Setting up an IFTTT Applet

IFTTT stands for “if this, then that.” We can think of the “this” as an incoming notification (or source), and the “that” as a triggered action (or sink). IFTTT “applets” accept notifications from diverse sources and trigger actions on diverse sinks. For example, we can set up an HTTP-based trigger (using their “webhooks” service) that causes Twitter to tweet a message.

First create an account on ifttt.com and/or download the app for your phone. (You’ll need a Twitter account too if you want to tweet.) In IFTTT navigate your way to new applet creation. This varies depends on what app or URL you’re accessing through, but e.g.:

When you’ve found the “create applet” dialogue, it will bring up If +__This__ Then That. Run through these steps:

This brings up If (hook logo) Then +__That__. Now:

We’ve now set up most of what we need on IFTTT, but we need to copy a token to allow our ESP to authenticate against the server. To do this either

This will give you a page saying something like:

Your key is: my-long-key-string
Back to service
To trigger an Event
Make a POST or GET web request to:

https://maker.ifttt.com/trigger/{event}/with/key/my-long-key-string
...
You can also try it with curl from a command line.

curl -X POST https://maker.ifttt.com/trigger/{event}/with/key/my-long-key-string

Replace {event} with your event name, e.g. my-first-ifttt-trigger and click Test it, or copy the curl statement and try it from the command line.

NOTE: IFTTT services sometimes have quite a high latency, and this seems particularly true for Twitter. You might have to wait 10 minutes or more for the tweet to appear in public! (You should be able to test the service without waiting for that to happen though, as the POST request will return a Congratulations! You've fired the {event} event message if it succeeds. Also check the Webhook>Settings>Activity page to see if Twitter accepted the trigger or if some other problem may have occured. YMMV!

Make a note of your key, and the URL that the service lives at. (Following the style of previous weeks’ exercises, you could put the key in your private.h file, e.g.: #define _IFTTT_KEY "j7sdkfsdfkjsdflk77sss".

6.3.2.3 Accessing the IFTTT Applet from Firmware

Now that we have a working service, we just need to get the ESP32 to call it over HTTP(S). This example from the ESP32 Arduino core can be adapted to talk to IFTTT with a little work. If we define a doPOST method (based on the example) that takes a URL and a JSON fragment in IFTTT format, we can do the service trigger job like this:

void doPOST(String url, String body) {
// ...
}

void iftttTweet(String messageDetails) {
  String url(
    "https://maker.ifttt.com/trigger/{event}/with/key/" _IFTTT_KEY
  );
  doPOST(url, messageDetails);
}

(You need to define the _IFTTT_KEY in your private.h file in the normal way.)

Good luck!

The course GitLab repository contains exercises/IFTTTThing, a model solution to the exercise. (Note: this code can be controlled by an ultrasonic sensor attached to pins A0 and A2 like this. By default this is turned off; define USE_ULTRASONICS to turn it on.)

6.4 Further Reading

Pete Warden and Daniel Situnayake, TinyML: Machine Learning with TensorFlow Lite on Arduino and Ultra-Low-Power Microcontrollers, O’Reilly Media, 2019 (Warden and Situnayake 2019).

7 Scheduling Tasks, Gestating New Devices

This chapter begins by introducing techniques for time sharing on microcontrollers, and for minimising power consumption. We then look at what the process of IoT device development entails, using the example of Sheffield’s unPhone IoT platform. The practical work this week is to begin the first lab assessment – for details see Blackboard.

7.1 Timers, Interrupts, Tasks, Events

Multiprocessing (Tanenbaum and Bos 2015) has become such a ubiquitous feature of modern computing that we take it for granted. A quick sneak peak at my desktop’s promiscuous innards (“ps ax |wc - l” for the curious) reveals no less than three hundred separate processes all “running” at the same time: what profligacy! My desktop doesn’t contain 300 CPUs, of course, or even that number of cores, but the miracles of multi-GHz multicore processors, super fast RAM, paging, context switching and scheduling make it seem as if it does, and that makes the job of anyone trying to program the beast enormously easier than it would otherwise be. The ESP32, by way of contrast, has two diddy little cores running at a couple of hundred MHz, and most of one core is needed to cope with protocols like WiFi or Bluetooth. This chapter looks at four strategies that we can adopt to cope with this somewhat constrained environment:

  1. time slicing
  2. interrupts
  3. timers
  4. FreeRTOS tasks 

Let’s take these in turn.

7.1.1 Time Slicing

We’ll take the simplest approach first. This works, but doesn’t extend well as complexity increases.

When our microcontroller has only a few simple tasks to perform, which happen in a linear sequence and don’t have any strict timing requirements, we can simply program these as a series of imperative statements, e.g.

void loop() {
  readSensors(&sensorData);
  postReadings(sensorData);
  Serial.println(ESP.getFreeHeap());
}

Job done. Hmmm, but maybe the memory print-outs are happening so quickly they’re difficult to read, or maybe the expensive operation of talking to WiFi and cloud HTTP server is draining the battery too quickly, and we only need readings every few seconds. What to do? In code we’ve seen in previous examples, we might add a delay, e.g.:

...
  postReadings(sensorData);
  delay(2000); // wait a couple of seconds
  Serial.println(ESP.getFreeHeap());
...

That works. Hmmm, but maybe we also need to show a warning signal if something in the sensor readings looks problematic.63 Shall we split that delay and do it in the middle? What if we only want to do one of the actions every fourth time through? Or what if we need to do some housekeeping if we get a control signal of some sort, but otherwise speed on through?

Flow of control can quickly become complex and error-prone as we add more and more cases, and the interaction with delay (during which the core we’re running on does nothing) is often tricky to manage (not to mention an inefficient use of hardware resource). A simple way to improve things is to add a loop counter, and to trigger events in slices based on this counter. For example:

int loopCounter = 0;
const int LOOP_ROLLOVER = 100000;   // how many loops per action slice
const int TICK_MONITOR = 0;         // when to take sensor readings
const int TICK_POST_READINGS = 200; // when to POST data to the cloud
const int TICK_HEAP_DEBUG = 100000; // do a memory diagnostic

void loop() {
  if(loopCounter == TICK_MONITOR) { // monitor levels
    readSensors(&sensorData);
  } else if(loopCounter == TICK_POST_READINGS) {
    ...
  } else if(loopCounter++ == LOOP_ROLLOVER) {
    loopCounter = 0;
  }
}

Various arrangements using additional counters can be used to organise different frequencies for different tasks, and the size of the action slices can be adjusted to change the overall duration of each sequence.

This approach works reasonably well, until we start to need to respond urgently to particular events, or perform certain tasks at particular times. The next section looks at solutions to that type of requirement.

7.1.2 Interrupts and Timers

So far we have relied on our firmware code to deal with all aspects of scheduling, monitoring and responding. The hardware we’re running on, however, provides two facilities that can lead us to much more powerful solutions to these types of problem:

The programming model for interrupts is to define very short and fast procedures called Interrupt Service Routines (ISRs), and to register these routines as callbacks from the timer or external state change that we wish to respond to. ISRs impose some additional constraints on your code, being normally written to avoid being interrupted themselves and to avoid running for more than some tiny number of cycles. Therefore they use mutual exclusion resource locking (or mutex locking), volatile variables (which the compiler will not attempt to optimise) and forced residence in fast RAM (using the IRAM_ATTR flag). Due to these constraints an ISR will rarely do any complex processing itself, but just push an event description into a queue which is then drained from some other process (or task – see also next section).

The exercises tree contains an example called TimingThing that responds to external interrupts from a push-button switch connected between GPIO 14 and ground. It contains two methods of note for this discussion:

uint8_t SWITCH_PIN = 14, LED_PIN = 32; // which GPIO we're using
volatile bool flashTheLED = false;     // control flag for LED flasher
...
// gpio interrupt events
static void IRAM_ATTR gpio_isr_handler(void *arg) { // switch press handler
  uint32_t gpio_num = (uint32_t) arg; // data from ISR service; not used
  flashTheLED = ! flashTheLED;                  // toggle the state
}
static void eventsSetup() {                     // call this from setup()
  // configure the switch pin (INPUT, falling edge interrupts)
  gpio_config_t io_conf;                        // params for switches
  io_conf.mode = GPIO_MODE_INPUT;               // set as input mode
  io_conf.pin_bit_mask = 1ULL << SWITCH_PIN;    // bit mask of pin(s) to set
  io_conf.pull_up_en = GPIO_PULLUP_DISABLE;     // disable pull-up mode
  io_conf.pull_down_en = GPIO_PULLDOWN_DISABLE; // disable pull-down mode
  io_conf.intr_type = GPIO_INTR_NEGEDGE;        // interrupt on falling edge
  (void) gpio_config(&io_conf);                 // do the configuration

  // install gpio isr service & hook up isr handlers
  gpio_install_isr_service(0); // prints an error if already there; ignore!
  gpio_isr_handler_add(        // attach the handler
    (gpio_num_t) SWITCH_PIN, gpio_isr_handler, (void *) SWITCH_PIN
  );
}
...
  if(flashTheLED) ...write HIGH on the LED pin, wait a little, write LOW...

The board that the example runs with looks like this:

Here:

(Note the 10k pull-up from the sensing side of the switch to V+; this is to prevent phantom reads when using the interrupt-driven code.)

Previous examples we’ve looked at have polled the switch (or other sensor) regularly during execution, looking for a state change. Often it is possible to miss a change if it happens during some other part of the execution path. The approach described here, although a little more complex at first sight, has the significant advantage of using the hardware to monitor and trigger a response to changes, and is the preferred method for any non-trivial system.

7.1.3 FreeRTOS Tasks

In section 2.3.2.2 I wrote that

Another important facility that we can access via the Espressif SDK is FreeRTOS, an open source real time ‘operating system’. FreeRTOS provides an abstraction for task initialisation, spawning and management, including timer interrupts and the ability to achieve a form of multiprocessing using priority-based scheduling. The ESP32 is a dual core chip (although the memory space is shared across both cores, i.e. it provides symmetric multiprocessing, or SMP). FreeRTOS allows us to exploit situations where several tasks can run simultaneously, or where several tasks can be interleaved on a single core to emulate multithreading.

In order to do this FreeRTOS tasks are allocated their own processing context (including execution stack) and a scheduler is responsible for swapping them in and out according to priority. (Note that the usual case for FreeRTOS ports is single core MCU, and that therefore the ESP32 port has made a number of changes to support dual core operation.)

For example, main.cpp in the MicML example64 in the exercises tree forwards data from the I2S bus to a server (over WiFi) whenever it is available from a microphone board. The code uses a FreeRTOS task containing an infinite loop that continually waits for a notification from a parallel I2S reader task (in I2SSampler::addSample in I2SSampler.cpp, via xTaskNotify):

// write samples to our server
void i2sWriterTask(void *param) {
  I2SSampler *sampler = (I2SSampler *) param;

  while (true) { // wait for some samples to save
    uint32_t ulNotificationValue =
      ulTaskNotifyTake(pdTRUE, pdMS_TO_TICKS(100));
    if (ulNotificationValue > 0) {
      // read the i2s buffer, post to server ...
    }
  }
}
...
void setup() {
  ...
  // set up i2s to read from our microphone
  i2s_sampler = new I2SSampler();

  // set up the i2s sample writer task
  TaskHandle_t writer_task_handle;
  xTaskCreate(
    i2sWriterTask, "I2S Writer Task", 4096,
    i2s_sampler, 1, &writer_task_handle
  );

  // start sampling from i2s device
  i2s_sampler->start(
    I2S_NUM_1, i2s_pins, i2s_config,
    32768, writer_task_handle
  );
  ...
}

Having set up these two tasks, there is nothing else to do, as FreeRTOS schedules all the processing without further intervention!

void loop() {
  // nothing to do here!
  // all driven by the i2s peripheral reading samples
}

This has been a whistle-stop tour of a number of complex and powerful tools in the MCU programmers’ armoury. It is well worth reading up on background sources and experimenting with the IDF, Arduino and FreeRTOS examples to get a better sense of how this all hangs together. Best of luck!

7.2 IoT Device Gestation: Creating the unPhone

Over the last half a dozen years or so, myself and colleagues have worked on control and monitoring systems for a sustainable urban food growing technology called aquaponics (Hamish Cunningham and Kotzen 2015; Rakcozy 2011). This is pretty close to the lowest environmental impact footprint that intensive agriculture can manage, pairing a closed-loop recirculating aquaculture system with hydroponic vegetables fed via naturally occuring nitrifying bacteria. No pesticides, no growth hormones, motherhood, apple pie, yada yada yada.

The catch? Complexity. We have three quite different types of organism (fish, plants, bacteria) all sharing an ecosystem; to maintain conditions favourable to all three whilst simultaneously harvesting food is quite the balancing act.

What to do?

In contrast to hydroponics, the growing of vegetables in dissolved fertilisers, aquaponics lacks mature control, monitoring, data sharing and analytics systems to support it. Our research programme has been to try and contribute IoT-based systems in this space, to chip away at the complexity problem by partial automation, knowledge sharing, the wisdom of crowds and the frequent wearing of silly hats in bright colours. To test out the approach, we first built a little aquaponics system in an unused alcove on campus:

(System design followed that from the FAO given in (Somerville et al. 2014).)

Soon afterward Richard Nicolle of Garden Up cornered me in the pub and beat me repeatedly with a packet of cheese and onion crisps until I just couldn’t bear the pain any longer and agreed to pitch an aquaponic green wall idea to Gripple’s special projects manager Gordon Macrae (who’s a bit like an SAS Colonel only based in a secret headquarters underneath Sheffield’s River Don). Gareth Coleman couldn’t resist an opportunity to muck about with insanely complicated bleeding-edge technology, and we managed to convince Dave Hartley of ShowKoi that we needed at least one real engineer on the team to stop people laughing (too loudly) at us. The die was cast, and a year later we had transformed this:

into this:

(We had a lot of help from Gripple’s Ninja Plumber contractors too. They’re a bit like Robert de Niro’s character in the film Brazil, only better with pipes and fittings.)

Eureka!

Except: when we did the figures, it turned out that only the Queen of England would actually be able to afford one. Drawing board, back to, etc. :(

More recently Gareth installed our concourse system at Heeley City Farm, Duncan Cameron and Tony Ryan helped us develop a next iteration of the system in the AWEC controlled environment center, and in 2021 as part of the Institute of Sustainable Food we’re building a new minifarm with Regather.

Anyhow, I’m getting ahead of myself. Scroll back half a decade, and as is normal for the first prototypes of a new device, we initially relied on breadboards and jumper cables to hook together the various different sensors, actuators and microcontrollers that made up the control and monitoring systems. As time went on and we added more and more functions to our rigs, we started to develop an attractive line in spaghetti wiring looms!

Reliability? Hmmm. Breathing on it was usually ok, but anything more violent would run the risk of dislodging a connection, and even when everything was plugged together just right, the presence of several high frequency bus signals on unshielded wires waving in the breeze was just asking for trouble. That meant lots of work for Gareth and Gee, slaving over a hot oscilloscope at all hours:

At the time, we were building a lot of different devices for various projects, and we could see the advantages in combining as many functions as possible under one roof. Martin Mayfield was running a project called Urban Flows which, amongst other things, was interested in measuring the life cycle impacts of the Sheffield food system, and agreed to fund version one of a new IoT device.

Soon after, with lots of help from Pimoroni, the unphone went from a thought experiment on my office whiteboard…

…to a prototype in a neat grey case:

It soon became clear that there was a wider need for something along the lines of the unphone as an IoT development platform. By the time the Pandemonium arrived, we were at hardware iteration 6-and-a-bit (or, as Pimoroni’s Jon Williamson insists on calling it, “7”), and all was set fair for world domination, ushering in the socialist paradise and finally putting to bed all those nasty rumours about computer scientists being a bunch of useless no-marks, nerds and geeks who should on no account ever be invited to parties.

Quoting from unphone.net:

Developing for the Internet of Things is a bit like trying to change a light bulb in the dark while standing on a three-legged chair with your shoelaces tied together. By the time you’ve configured your development environment, assembled the libraries your hardware demands, fixed the compatibility issues and figured out how to burn new firmware without bricking your circuit board you’re all ready to discover that the shiney new networking function that you bought the board for in the first place doesn’t actually work with your local wifi access point and that the device has less memory space than Babbage and Lovelace’s difference engine.

Hey ho.

The unPhone is an IoT development platform from the University of Sheffield, Pimoroni and BitFIXit that builds on one of the easiest and most popular networked microcontrollers available (the ESP32), and adds:

Untie your shoelaces and let’s get cracking :-)

The externally accessible components are arranged like this:

And other components are attached via an expander cable and daughter board:

Later versions also have three pushbutton switches across the lower front of the case. Here’s one running an infra-red control test from the IR LEDs to a sensor mounted on another unit:

The daughter board accepts three feather sockets and has a small matrixboard-style prototyping area:

The expander can be housed in a 3D-printed case extender of varying thickness (designed by Peter Hemmins; thanks Pete!):

Robots, TV remotes, battleship games, air quality monitors, dawn simulator alarms: the beast has proven quite versatile!

7.2.1 Steps in Device Creation

The rest of this section returns, belatedly, to the point, and looks at the various stages that the prototype unphone device made its way through on the way to (almost!) maturity, and attempts to draw out a few lessons for IoT device gestation in general.

A typical IoT device creation process goes through these steps:

From circuit to board:

Logical level circuit diagrams

PCB (printed circuit board) routing diagrams

Then send away for PCBs and a stencil, and… bung it in the pick’n’place machine:

Load in a bunch of rolls of components:

Applying solder paste to the bare board:

Getting it right: 6¼ hardware iterations:

From left to right:

(Versions 5 and 6 removed the mic and added front-of-case buttons; we don’t talk about why we needed version 6¼, at least not in public.)

It’s a thing! (31st Jan to 8th Nov 2018: 9 months from proposal to delivery!)

7.2.2 Some Lessons

Bringing up a new board is, in a word, messy! When things don’t work as expected there’s no knowing whether it is the hardware or the code that’s at fault. This means lots of painful debugging with an oscilloscope! Allow plenty of time for this step!

It is easy to hit unintended consequences. At some point we were getting low on IO pins to connect peripherals to, and one of us said “I know, let’s add an IO expander!” This little chip adds lots of connectivity by shunting the chip select lines of the various modules onto an expander chip (TCA9555, on I2C). Unfortunately, it also means that none of the libraries work any more, as then need an extra step when writing to the device. The libraries use use digitalWrite(PIN, HIGH) internally, but we need to activate the expander to do chip select toggling first. So now we have an IOExpander class that we inject into libs:

#include "IOExpander.h"
#define digitalWrite IOExpander::digitalWrite
#define pinMode IOExpander::pinMode
#define digitalRead IOExpander::digitalRead

The overridden methods use bit 2 of the PIN int to represent an unPhone expander pin (why not the top bit? casts from uint to int make it -ve!):

void IOExpander::digitalWrite(uint8_t pin, uint8_t value) {
  if(pin & 0x40) {
    pin &= 0b10111111;  // mask out the high bit
    IOExpander::writeRegisterWord(0x02, new_output_states); // trigger 9555
...

Summarising the good bits and the not so good bits:

Overall, the process of creating a new IoT device is still a pretty big job, but it is undeniably shrinking as time goes on. What do you want to create?!

7.3 COM3505 Week 07 Notes

This week we publish “Lab” Asssessment 1, which you will have until week 8 (after Easter) to complete. For details see Blackboard.

7.4 Futher Reading

8 Applications

There are two ways of constructing a software design: one way is to make it so simple that there are obviously no deficiencies; the other way is to make it so complicated that there are no obvious deficiencies. (C.A.R. Hoare)

If you try to make something beautiful, it is often ugly. If you try to make something useful, it is often beautiful. (Oscar Wilde)

Welcome to the end of the beginning: from now on we’re in project territory. The most important part of the rest of the course is to design, build, test and document an IoT device. Most often this involves both hardware and firmware (and sometimes cloud-based software), but it is also possible to do a good project that is only in firmware.

The first half of this chapter does three things:

The rest of the chapter then details project options.65

(NOTE: in 2021 the pandemonium makes it difficult to manage supply of lots of different options, so by default we’re giving everyone an ESP32-based smartwatch. There are also options that only use the ESP32 featherwing which has already been supplied. Other possibilities are included below for reference and for those able to pick up hardware in Sheffield.)

8.1 Beep my Earing Whenever I Start Sounding Like a Donkey

Applications of IoT technology are many and varied. Existing products are as diverse as RFID warehousing and logistics trackers, exercise products such as Fitbit or the Polar heart rate monitor, the Good Night Lamp or the NEST thermostat. DIY projects are similarly numerous and diverse. For example:

It sometimes feels easier to define what isn’t IoT, than what is – particulary as the marketing departments of most major companies decided somewhere in the middle tennies that everything lying around in their product portfolio up to and including that old COBOL point-of-sale system that last sold in the late 1850s is actually a blazingly relevant IoT innovation that will transform your life, or at least help trim off that irritating positive bank balance that you’ve been worrying about. If a device…

…then it probably qualifies for inclusion in the IoT. (If it is permanently powered, talks to the outside world via teletype or Morse code, and deploys enough compute power to run the entire world’s automation capability in the 1980s, it may not be.)

8.2 Projects: Design, Build, Document

The course project is both a great opportunity to learn about the entirety of an IoT device and the main way that we assess the depth of your learning. (See chapter 1 for more details of the threshold and grading assessment method in use from 2021.) The latter point means that you should pay a good deal of attention to documentation and presentation of your work: functionality is important, but the best projects will be those that combine great functionality with great documentation, both of the device itself and of the process of its creation.

You now need to start:

As always you should keep checking in and pushing to your repository as you iterate through design, development, testing and documentation phases.

8.2.1 Possible Projects

There are many possible projects, some using additional hardware, some just using the ESP32 on its own. We supply a variety of add-on hardware for you to use in project work, but if you prefer it is also possible to do a project using just the ESP32 itself, for example:

Projects using additional hardware:

Below we detail hardware build issues, relevant libraries and example code etc., after a look at the LiPo batteries used by some project options.

8.3 LiPo Safety

8.3.1 What are Lithium Polymer Batteries?

Lithium Polymer (or LiPo) cells are one of the most effective commercially available rechargeable batteries, having high energy and power density. These batteries are used in all manner of mobile applications and are particularly popular with remote control (RC) hobbyists.

If needed you will be given a LiPo battery as part of the project hardware for COM3505. The battery is potentially dangerous if damaged, or the electronics connected to the battery are damaged, or the battery is connected to inappropriate hardware. If in doubt, stop work and ask for help!

8.3.2 What are the Dangers?

Although these cells are very useful, they can be dangerous when mistreated or misused. There are three main causes of cell failure: puncture, over-heating and over-charging. If the battery fails as a result of any of these, hazardous electrolyte leakage, fire or explosions may occur.

The rate of transfer of energy within the battery is strongly linked to its internal heat; when over-heated the cell’s long term performance will degrade, more energy is released and the battery gets hotter, creating a dangerous feedback loop. If this continues the electrolyte will vaporise and cause the battery to expand, which may lead to fire or even an explosion.

This same effect can be caused by over-charging the battery, or in some cases even just leaving it charged in the wrong circumstances. Henry (one of our previous teaching assistants on the course) used to fly an RC helicopter that ran off a multi-cell LiPo pack. Having forgotten to discharge it, it was left in a container in his shed. Many months later the cell exploded in a ball of flame nearly burning down the shed!

The sensitive chemistry of the batteries can also lead to fire if the battery gets punctured and vents into the air.

8.3.3 Avoiding Problems

ALWAYS take the following precautions when using LiPos in COM3505:

8.4 Build and Development Notes

8.4.1 Smart Watches

The Lilygo T-Watch-2020 is a smart watch development kit based on the ESP32.

The watch includes an accelerometer, touch display, infra-red sensor, loudspeaker and vibration motor. In other words it fits out definition of IoT devices: a networked microcontroller with sensors (accelerometer; touch input) and actuators (screen; vibration motor; the ESP32’s radio). It is powered by a LiPo battery managed by an AXP202 PMU66 and has its own RTC67.

There are many possible projects you can build with the watch, for example:

If you want to emphasise the hardware component of the project, you could also connect the watch to your feather (over WiFi or BT or BLE) and use the feather / LED / etc. combination as an actuator (e.g.: turn the TV on or off over IR when I enter the room), perhaps using a gesture language. In the other direction you might experiment with control of your watch via a mic on your feather, for example.

How to get started? The most comprehensive firmware setup currently available is probably Bill Dudley’s github project, and there is some example shell code illustrating how to set this up in the exercises tree.

8.4.2 DIY Alexa

Chris Greening’s ESP32-based voice-controlled home automation system, or “DIY Alexa”, uses a custom microphone board for the ESP32 that he has developed based on a low-noise MEMS mic:

(We have one each of the mics, and speakers and speaker driver boards.)

Other links:

The sound input stage is also documented below.

8.4.3 A Simple Robot Car

The arrival of robot overlords has been much anticipated. Get ahead of your fellow humans by building a simple robot car! The one we’ll be making is from Adafruit:

robot

When following the build instructions, read through them first before jumping off to the linked details and starting assembly – there are some steps in the earlier instructions that you are later told to modify.

Note: some of the images below use the unphone platform to drive the robot; in 2021 this isn’t available so just use the breadboard version instead.

Follow Adafruit’s build instructions to construct the robot chassis and then attach the breadboard on top. (Ignore the “Temporarily unable to load embedded content” message.)

(See also the other Adabox002 videos. but again, remember that we’ll be using the ESP32 feather as our processing board rather than the bluetooth board supplied in AdaBox002.)

You will need a motor driver board, and use a small pozidrive screwdriver and a small pair of pliers (available in the electronics lab or project space – DIA 2.01 or 1.01) to put the headers into the stepper motor connections as shown:

motor driver with headers

Now we can connect the motors to the header pins:

robocar front

Then screw the battery box wires into the connector at the side of the motor driver – make sure to put the red lead in the + connection and the black one into the – connection:

robocar side

You can follow the next section here: basic code for robot to test out the robot.

We are using the aREST library to provide a RESTful interface – one that responds to HTTP calls to URI’s like //esp32/forward.

The code is on github here and the code is explained on this Adafruit page.

Modify the sketch to use your local wifi access point (phones on tether work well as they report the IP of connected devices). Also change the first two lines to adapt this code to run on the ESP32; they should read:

    #include <WiFi.h>
    #include <aREST.h>

Once you have programmed your ESP32 you can discover it’s IP address either on the serial port or via your hotspot. Using a laptop, join the same network as the ESP32. Edit the script.js file inside the interface directory to replace the IP address with that for your ESP32. Now you can run the demo.html file in a browser for wifi control of the robot!

8.4.3.1 Robot Car: Kit List

8.4.4 Binary Diff for Incremental OTA

(This project doesn’t require any additional hardware, using just a bare ESP32, so is a good choice if you’re not interested in the electronics prototyping side of things.)

As we discussed earlier in the course, the resource-constrained nature of IoT devices makes keeping them up-to-date with security patches and new features a significant challenge, and in situations where network connectivity may be interrupted it is common to allocate two or three times the requisite amount of flash memory to store firmware: one for the current version and one to download the next (updated) version into (and sometimes another one to hold a “factory reset” version). This adds a significant cost to the device.

This project is to port an approach to incremental updates that is used to minimise the size of updates in several contexts including Ubuntu’s snap package manager and Google Chrome’s courgette updates process. Incremental updates work by analysing the difference (or delta) between a new version of (in our case) firmware and the previous version, then sending only the list of changes. A client-side routine is then required to check that the change bundle has come down the pipe successfully and if so apply the change list to the firmware image.

This isn’t a new approach; binary diff for software update is quite common. This blog post has a good summary; systems often use bsdiff or xdelta. The disadvantage is that when differencing object files small changes in source can result in huge changes in the binary: “compiled code is full of internal references where some instruction or datum contains the address (or offset) of another instruction or datum.” The Chrome courgette system is particularly interesting however, because it performs a limited decompile of the object code and then analyses the changes between versions with the address changes separated out from instruction-level changes. This can result in a much smaller diff. (Recent versions of Firefox have taken up the same system, I believe.) Chromium (the open source version of Chrome), for example, sometimes achieves a huge reduction in update size using courgette:

(Posted here.)

To do this type of update on an ESP32 we would need to:

This project is not an easy option, but if successful would be a real contribution to the community :)

8.4.4.1 Advanced Topic: Drag&Drop Update

As an advanced topic, it would be interesting to consider the possibility of closing the gap between the old world of C++ firmware burn tooling and the new one of drag-and-drop Python or JavaScript script update that has started to be a common development mode for microcontrollers such as the BBC’s Micro:bit or boards supporting CirciutPython or the Espruino JavaScript port. Can we compress binary firmware updates sufficiently to rival the new approaches?

Another issue to consider here is board support for USB OTG: if we can treat the board as a mass storage device then drag-and-drop becomes trivial in most operating systems. This is one of the key differences between the ESP32 and the ESP32S2 / ESP32S3, for example.

8.4.5 TV Remote, TV-B-Gone: IR-Remote Projects

Using an IR LED, start by reviewing the TV-B-Gone codes and code inspiration, then read on for how to get started. Make sure you have collected an infra-red reciever: IR reciever that demodulates the IR transmissions and produces digital signals straight into the ESP32. (You can use these for testing and also to demo the operation of your project in the documentation.)

If we attach infra-red (IR) LED’s to our ESP, we can use it to send remote control commands – for example to turn off TV’s, or perhaps issue a series of commands such as turn on the TV, turn on a satellite box, change the input source, etc.

The mechanics of using the timers is complex and also different on the arduino uno and the ESP32. Luckily the well established IRremote Arduino Library by z3t0 handles this for us. The latest version lists support for ESP32 receiving IR only!

Good old Andreas Spiess has implemented the missing send functionality from the IR remote library.

The libary should be modified to reflect the IR LED pin used on your board – so if you’re installing this yourself then modify the IRremote.h file – line 262 – change: byte timerPwmPin=3; to read: byte timerPwmPin=12; (assuming you’re connecting on pin 12).

In order to know the codes that are used in a certain device (there are hundreds of different propriatary formats!) you can search on the internet – noting that there are at least three ways to write the binary formats and several other ways to express the codes even in a particular protocol.

This page gives good if dense info – combined with this page listing Sony TV device codes.

Ok – so, from the code page, we see that the first table lists basic codes, such as code 21 for power. The Sony:1 at the head of the table tells us that of the various device codes used by Sony TVs, these codes are part of device 1.

So the worked example gives us a template for how to proceed:

Our command is code 21 – convert to the 7 bit binary value 0010101 and reverse it to get 1010100. My device code 1 gets expressed as a 5 bit binary value 00001 and reverse it to get 10000. Put these together to get 101010010000, which is A90 hex. Whew! The 12-bit nature of the codes explains the second parameter in the call.

And looking at the example sketches included with the library, IRsendDemo does indeed use code 0xA90 – this provides confidence that maybe I’ve got my sums right – and lo and behold – it turns my TV off!

It is often easier to connect an IR receiver device: IR receiver device and read the codes that are produced by an existing remote control. In order to do this we use a TSOP4838 photoreceiver device that has a sensor, amplifier, demodulator and signal conditioning circuitry all built in. It connects directly to the ESP32 as illustrated:

sensor in socket

Pin 1 is Data Out, pin 2 is Ground and pin 3 is Vcc – 3.3V in our case. This means you can just put the sensor into the expander as shown above, and the data comes into pin A0 (on this image). (However it isn’t very robust!)

NOTE: on the latest (2019) unPhone, pin A0 won’t work for this purpose; use A1 (next along) instead:

sensor in socket

For a more reliable connection, solder the sensor directly into the board (though not on A0!):

sensor in holes

When you have a device that sends IR codes and a receiver that decodes them, you can put them together for testing like this:

two devices

8.4.6 Light Sensor

We are using the TSL2591 light sensor in a handy breakout from Adafruit.

You will need to solder some jumper wires to four points on the sensor - these are the same whether you have the square or round board:

Light sensor

Then it’s a simple matter of plugging the wires into the expander like so:

Light sensor expander

8.4.7 Remote Control Power Sockets for Home Automation

Control the world!!! Or at least, things that plug into a mains socket… This project adds remote mains power switching to your ESP32 using a 433 MHz radio transmitter. This type of transmitter is a common mechanism for remote controls, e.g. for your central heating. The actuator we are using is a radio transmitter circuit that uses the 433MHz frequency:

433MHz transmitter

This is part of the ISM band of frequencies that consumer electronic devices can use without license – in our case we are using electric sockets that switch on and off in response to codes issued on that frequency:

433MHz mains socket

By using a radio-controlled mains socket we cunningly avoid having to interface directly with the mains. Being able to switch a mains powered device such as a light, fan, heater etc. opens up the possibilities of control greatly – hopefully it’s already giving you some ideas…

Power Socket circuit schematic

As you can see from the schematic we can just connect the transmitter without needing any additional components. This transmitter is designed to operate at 5V – but it does work at 3.3V with a lower power consumption but reduced range. If you want to use this transmitter on batteries, you might like to experiment with running it on 5V vs 3.3V and see whether the trade-off is desirable for your project needs.

Soldering the transmitter into the expander should look something like this:

Expander and 433MHz transmitter

Here I’ve soldered wires into the holes just in front of the ones for the transmitter. On the expander the prototype holes aren’t connected to each other, so on the back I made sure the wires connected to the pins of the transmitter:

Expander bottom showing 433MHz connections

The sockets come in packs of three and with a single push button remote control. Eight different sets of codes are used to signal to the sockets so that you can use more than three sockets in one location. The sockets are programmed to match the remote (which can’t be changed), but can also be re-programmed to change the code they respond to. For approx. 3 seconds when power is applied, the socket is in learning mode; if a control code is broadcast during this period the socket will learn this code and begin responding. If no code is received during the first 3 seconds, the socket goes into normal mode and responds to the last code learned. Each socket code has an ‘on’ command and an ‘off’ command – there seems to be no other commands and no feedback – so no ability to read the status of the socket for example.

You can initialise the handy RC Switch library like thus:

#include <RCSwitch.h>
RCSwitch mySwitch = RCSwitch();

Then a couple of setup configurations:

// Transmitter is connected to esp32 Pin #12
mySwitch.enableTransmit();

// We need to set pulse length as it's different from default
mySwitch.setPulseLength(175);

Now we can send a command to the switch:

mySwitch.send(4281651, 24);

I’ve mapped out the codes for all 8 sets of sockets here:

Without an antenna the range of the transmitter is only a meter or so; but a simple aerial can be made with a piece of wire. Using this aerial range can often reach 20-30m and usually can work through walls and ceilings. Instructions can be found here (credit: Ben Schueler).

Aerial

Once you’ve made your aerial you should solder it so it connects with the pin on the right of the transmitter – marked ‘ANT’ so it will look something like this:

Expander with aerial

8.4.7.1 Home Automation: Kit List

8.4.8 Sound Input

We used to use Adafruit’s I2S microphone breakout board. Figuring out how to wire and drive it is a little tricky but not impossible :)

More recently we’ve been experimenting with Chris Greening’s ICS-43434 mic board. There’s documentation and code from his github here.

8.4.8.1 CMG ICS-43434

Chris Greening’s ESP32-based Alexa-alike uses a custom microphone board:

I wired this up to the Huzzah ESP32 like this:

There’s code to drive it and collect the audio here. Note that the Feather Huzzah pins to connect the mic to are:

(Config field .data_out_num is for outward communication on I2S and should be set to I2S_PIN_NO_CHANGE as we’re only listening to the mic in this case.)

To (re-)create the lib/tfmicro tree in MicML follow his instructions here.

8.4.8.2 Adafruit SPH0645LM4H

Try this:

Then you can try this to test:

cd ...the-internet-of-things/exercises/MicML
./magic.sh arduino-ide

And open Tools>SerialPlotter.

You should see something like this:

The hardware setup will look something like this (with a white wire for ground, instead of black):

If you hit problems, there’s useful info in this post, and Espressif’s I2S documentation here.

8.4.9 MP3 Player

The ESP32 feather can be attached to a featherwing with a VS1053 MP3 decoder chip which can also do a nice job of synthesising MIDI input. Store your .mp3 files on an SD card (after formatting it to FAT), and see the example code for how to play via the headphone jack.

Note that streaming audio is a blocking operation for at least one of the ESP32’s cores, so your UI will have to find clever ways to squeeze into available processing power!

8.4.10 NeoPixels; Dawn Simulator Alarm

“Show me someone who is bored of LED’s and I will show you someone who is bored of life.” Ahem. This project uses beautiful and useful NeoPixel LEDs to simulate dawn:

Adafruit’s NeoPixels are bright multicolour LEDs controlled via a single signal wire (plus two for power). Read up on them here - we’re using the NeoPixel stick and the NeoPixel Fether.

There used to be a bug with the ESP32 having difficulty with the timing see here, but hopegfully it was fixed here.

To get them working, first solder on three connections for power and data:

Neopixel wired.

Then connect these to the USB 5V, GND and pin A0 as shown:

Neopixel on expander

8.4.10.1 Dawn Simulator: Kit List

NeoPixel array featherwing

8.4.11 Peer-to-Peer Voting Systems

The challenge with this project is to provide mechanisms to prevent vote stuffing, interception and ensure privacy of votes.

The first thing to do is to use the code for Ex09 so that you have a way to configure your ESP on new networks:

void setup09() { // wifi UI
  ...
}

void loop09() {
  dnsServer.processNextRequest(); // handle DNS  requests
  webServer.handleClient();       // handle HTTP requests
  ...
}

Secondly, find the location of your board using Google’s geolocation API. In order to make the things we’re voting on localised, we need a way to find our location and report this.

This could be provided by a GPS chip, however these consume a moderate amount of power and take a fair amount of time to initialise and get a ‘first fix.’ In addition they don’t work well in urban canyons (between buildings) or indoors.

Alternatively we can use a cunning technique that relies on the relative strength of wireless signals. By comparing the signal strength seen by a device with a map of access points the location of the device can be inferred through triangulation.

Google has already done the tedious task of collecting the location of millions of wireless access points and makes an API available (to developers) that takes a list of wireless signals in the location, and returns a latitude and longitude co-ordinate.

Github user @gmag11 has contributed code for the ESP32 that uses the google API to return location data - Google location code

You will need to get a google API key from Google Geolocation API

Note: in order to Get a list of available votes, you could use a Twitter account to store and access votes, using Twitter’s location function, or implement an IFTTT notification mechanism.

8.4.12 Panic Button

This project uses Google’s geolocation API and/or the GPS featherwing to figure out where you are when you press the button.

The idea is to provide a way to signal a “panic” (or need for urgent assistance) to others, e.g. security staff. In order to be a useful panic button, we need a way to find our location and report this.

This could be provided by a GPS chip, however these consume a moderate amount of power and take a fair amount of time to initialise and get a ‘first fix.’ In addition they don’t work well in urban canyons (between buildings) or indoors.

Alternatively we can use a cunning technique that relies on the relative strength of wireless signals. By comparing the signal strength seen by a device with a map of access points the location of the device can be inferred through triangulation.

Google has already done the tedious task of collecting the location of millions of wireless access points and makes an API available (to developers) that takes a list of wireless signals in the location, and returns a latitude and longitude co-ordinate.

Github user @gmag11 has contributed code for the ESP32 that uses the google API to return location data - Google location code.

You will need to get a google API key from Google Geolocation API.

8.4.13 Ultrasonic sensors

The HC-SR04 ultrasonic sensor is a cheap and cheerful way of sensing distance to solid objects like a wall. The device is well described here: tutorial on ultrasonic sensor.

We can connect it to an ESP32 without needing any additional components:

ultrasonic sensor on breadboard.

Following the schematic here:

ultrasonic sensor on breadboard.

You can use this sensor without a library, as its operation is so simple. The sensor is triggered by a 10ms pulse to the TRIG pin, from GPIO pin 25. The sensor sends an ultrasonic pulse and waits for the response, triggering the ECHO pin to GPIO pin 34. The time between the trigger and the echo is directly proportional to the distance between the sensor and the object. The sensor doesn’t work properly with small objects and complex fields of view - it’s really designed for detecting obstacles like walls. In code, once the trigger pulse has been set, the arduino function pulseIn is useful to time the response:

int duration = pulseIn(echoPin, HIGH);
int distance = (duration/2) / 29.1;
Serial.print(distance);
Serial.println(" cm");

However, this sensor is not very well behaved! It occasionally gives wrong values, sometimes the pulse never returns to the sensor correctly, and sometimes a spurious signal will be reported. Simple functions to average readings or exclude results that are more than 50% larger or smaller than the average of the last few results can help clean up the data.

8.4.14 Predictive Text UI

There’s a simple predictive text library (see predictor.cpp) in your exercises tree (which was originally written by Mark Hepple – thanks Mark! – and then ported to the ESP32). Currently the UI, using a touchscreen featherwing, is very basic – can you improve it? How many words can you support on the ESP without running out of memory?

8.4.15 Musical Instrument

Using the VS1053’s MIDI synthesis capability and e.g. the ESP’s touch sensors as control mechanisms many musical instruments become possible. Pick up a music player featherwing if you want to build one of these.

8.4.16 Bedtime Tracker

Eyes drooping? C programming not seeming quite as exciting as usual? Perhaps you should be getting more sleep :) Can you track your screen-off shut-eye time using a light sensor, and present the data back in an easy to interpret form?

8.4.17 Battleships Game

(This requires a pair of ESPs, both with touchscreens, so check that these are available to you before chosing!)

This is a popular game with simple UI requirements, perfect for a connected microcontroller like the ESP32.

8.4.18 A Note on UIs

These libaries might be worth looking at if you want to do sophisticated UI stuff:

8.4.19 Air Quality Monitor

NOTE: we’re retiring this project, partly because it is quite challenging, especially without access to the lab, and partly because there are newer and better solutions that we haven’t had time to integrate into the course as yet. If you’re super confident and really want to give this a go, please get in touch, but if not please choose another project.

The project used three sensors to measure various aspects of Air Quality:

Air Quality assembled

This project is complex to wire up!

The old instructions, for reference:

We’ll be using two low-cost sensors to give an indication of air quality, a dust sensor (Sharp GYP2Y1010AU0F) and a Volatile Organic Compounds sensor (MQ-135). In addition we’re using a digital temperature and humidity sensor, the AM2302, a clone of the popular DHT22.

Both air quality sensors are analog sensors that are powered by 5V. They return a voltage that is between 0V-5V; the level of the voltage is proportional to the level of dust or VOC’s sensed.

Because our ADC (Analog to Digital Converter) in the ESP32 only has a limited input voltage range (0-1.5V with other ranges available) we need to scale the sensor output voltage with a potential divider.

A potential divider is made of two resistors that divide the voltage according to the formula:

    Vout = Vin ( R2 / R1+R2 )

We need to divide the Vout by 5 to scale it correctly, giving Vout approx 1.3 and Vin=5; working out the equation to fit into the available resistor values is a nice excercise for the reader.

If the reader does not feel like exercising then online calculators, e.g. this one will select resistors that minimise the error.

For further reading about the voltage divider see Sparkfun’s tutorial here.

The output voltages from these dividers go to the ESP32 – it has multiple ADC channels – we are using A2 and A3.

In our circuit, resistors R4 & R5 form a potential divider for the VOC sensor, reducing it’s range from the 0-5V the sensor outputs to 0-1.25V. The ADC has a switchable attenuator and so we can set it for 1.5V range with analogSetPinAttenuation(A0, ADC_2_5db). We’re using this range as it is more linear than the 11db range, but still gives a convenient voltage divider, and we avoid going to the end of the range where the conversion looses accuracy.

For the dust sensor, we can also use the same voltage divider to reduce the 5V output down to a measurable 1.25V, resistors R2 & R3 form this potential divider.

The AM2302 sensor uses a proprietary digital protocol but luckily a library hides that from us. We add a 10K resistor R6 to act as a pull-up on the data signals from this sensor to the microcontroller.

The other components in our circuit, R1 and C1, are needed to make the dust sensor operate correctly. For more information on the dust sensor, read the application note. Note – the capacitor C1 is an electrolytic type that has polarity – make sure you wire it in the correct way round!

a schematic

For construction, I recommend using breadboard at first, and then once the circuit is working you can convert it to soldered connections. Stick the breadboard and dust sensors down to the assembled expander board using sticky pads:

Expander with bb dust

The idea is to place the dust sensor so that the hole in the centre of the sensor is below the board, so that air (and dust) can be measured:

Dust sensor

Using the schematic as a guide, connect your component together on the breadboard. You may find the wires from the dust sensor difficult to insert into the breadboard – try twisting the strands of wire together. You may need to use a soldering iron to melt some solder onto the twisted wire – this is called ‘tinning’ the wire:

Tinned wires68

Then I suggest starting by connecting the power wires – there are three voltages in this circuit, two potential dividers and three signals going into the ESP32! With all those connections it really helps to use colour codes to help keep them all in the right places. I’ve used:

After all the sensors, wires and passive components have been placed on the breadboard it might look something like this:

Breadboard with sensors

Seriously – once you have got the circuit working – you are strongly advised to solder the wires neatly in place otherwise you will be spending a lot of time debugging loose wires!

8.4.19.1 Air Quality: Kit List

8.5 COM3505 Week 08 Notes

8.5.1 WARNINGS!!!

If you’re now using LiPo batteries – please review the safety instructions above and be sure to follow them. If in doubt ask a member of staff.

8.5.2 Learning Objectives

Our objective this week is to:

8.5.3 Assignments and Assessment

8.6 Further Reading

Following on from the work we did in the last couple of weeks (on provisioning and over-the-air updates), have a look at these:

9 Learning in the Fog – AI on the Edge

If there is a subject with a greater reliance on impenentrable jargon than computing, I have yet to find it. Fog? Edge? (Mist, even?) Since “cloud” has become the accepted term for distributed computing, computation that happens locally has started to acquire its own terminology building on the same metaphor. A smart phone or an IoT device is an edge device (a droplet perhaps, or edge node). The set of edge devices on a network make a fog or mist.69

We start this chapter by putting machine learning and AI in the context of edge devices in the fog. After motivation, the first issue to address is the relatively low power of computational resource available outside of the datacenter, which leads to a discussion of federated learning.

9.1 Why Learn at the Edge?70

The vast majority of current applications are designed for data processing to be performed predominantly in the cloud. There are several advantages for this model, such as:

But more recently, privacy is becoming a growing concern in the population. Data has long been seen as an important asset by large companies, but only now individuals are starting to recognise this fact about their own data. Streaming data to the cloud may not be the model of future IoT applications.

So what can we do instead to protect data privacy? One answer is to train Machine Learning models at the edge. The method with the most traction in current research is Federated Learning.

9.2 Federated Learning

Definition:

Federated learning is a machine learning setting where multiple entities (clients) collaborate in solving a machine learning problem, under the coordination of a central server or service provider. Each client’s raw data is stored locally and not exchanged or transferred; instead, focused updates intended for immediate aggregation are used to achieve the learning objective. Focused updates are updates narrowly scoped to contain the minimum information necessary for the specific learning task at hand; aggregation is performed as earlier [sic.] as possible in the service of data minimization … (Kairouz et al. 2019)

In essence, instead of sending your data to the cloud to train an ML model, with Federated Learning the ML model is sent from the server to the edge (on the client). Using the local data of the client, the model is updated to reflect a better performance on the local data. This local update of the model is then sent to the server for aggregation with updates from many other clients. The local data is discarded after the model update and never leaves the device in its raw form.

Federated Learning enhances the following principles:

Federated Learning has been receiving substantial interest recently, both from research and industry, so it’s worth exploring it in the context of IoT data. The underlying conditions for this method to work in the IoT space are that sensing devices (or nearby computing hosted on the local network, e.g. gateways or desktops) have sufficient computing resources to store a fraction of the data and perform the model updates.

In 2016, McMahan et al. introduced FL: “We term our approach Federated Learning, since the learning task is solved by a loose federation of participating devices (which we refer to as clients) which are coordinated by a central server.” (Brendan McMahan et al. 2016) An unbalanced and non-IID (identically and independently distributed) data partitioning across a massive number of unreliable devices with limited communication bandwidth are the best defining set of challenges.

9.2.1 How does FL work?

Developing a FL framework:

  1. Problem identification: The model engineer identifies a problem to be solved with FL.
  2. Client instrumentation: If needed, the clients (e.g. an app running on mobile phones) are instrumented to store locally (with limits on time and quantity) the necessary training data. In many cases, the app already will have stored this data (e.g. a text messaging app must store text messages, a photo management app already stores photos). However, in some cases additional data or metadata might need to be maintained, e.g. user interaction data to provide labels for a supervised learning task.
  3. Simulation prototyping (optional): The model engineer may prototype model architectures and test learning hyperparameters in an FL simulation using a proxy dataset.
  4. Federated model training: Multiple federated training tasks are started to train different variations of the model, or use different optimization hyperparameters.
  5. (Federated) model evaluation: After the tasks have trained sufficiently (typically a few days), the models are analyzed and good candidates selected. Analysis may include metrics computed on standard datasets in the datacenter, or federated evaluation wherein the models are pushed to held-out clients for evaluation on local client data.
  6. Deployment: Finally, once a good model is selected, it goes through a standard model launch process, including manual quality assurance, live A/B testing (usually by using the new model on some devices and the previous generation model on other devices to compare their in-vivo performance), and a staged rollout (so that poor behavior can be discovered and rolled back before affecting too many users). The specific launch process for a model is set by the owner of the application and is usually independent of how the model is trained. In other words, this step would apply equally to a model trained with federated learning or with a traditional datacenter approach.

The steps in training processes:

  1. Client selection: The server samples from a set of clients meeting eligibility requirements. For example, mobile phones might only check in to the server if they are plugged in, on an unmetered wi-fi connection, and idle, in order to avoid impacting the user of the device.
  2. Broadcast: The selected clients download the current model weights and a training program (e.g. a TensorFlow graph) from the server.
  3. Client computation: Each selected device locally computes an update to the model by executing the training program, which might for example run SGD on the local data (as in Federated Averaging).
  4. Aggregation: The server collects an aggregate of the device updates. For efficiency, stragglers might be dropped at this point once a sufficient number of devices have reported results. This stage is also the integration point for many other techniques which will be discussed later, possibly including: secure aggregation for added privacy, lossy compression of aggregates for communication efficiency, and noise addition and update clipping for differential privacy.
  5. Model update: The server locally updates the shared model based on the aggregated update computed from the clients that participated in the current round.

Typical numbers found in the deployments used at Google’s scale are presented in the table below.

Scale
Total population size 106 – 1010 devices
Devices selected for one round of training 50 – 5000
Total devices that participate in training one model 105 – 107
Number of rounds for model convergence 500 – 10000
Wall-clock training time 1 – 10 days

9.2.2 Applications of FL

Google makes extensive use of FL in the Gboard mobile keyboard and in Android Messages. Each time you type on the Gboard and it predicts the word you intend to type next based on the text context, that is used as signal to strengthen the model being used for word prediction. This model is then pulled from your phone and from other users’ phones to update a large global model that is capable of predicting the next word more robustly. Note here, not all the words you type are considered for sharing in the global model, such as usernames or passwords. But anything that falls in the general sense of using the language makes it into the global model.

Apple is using cross-device FL in iOS 13 for applications like the QuickType keyboard and the vocal classifier for “Hey Siri.” The keyboard strategy is very similar to that of Google’s. For the vocal keyword it stores samples that have a lower confidence and did not activate the device. If these are followed by a stronger example that wakes up the device, the immediately previous samples are considered for training and updating the local model.

A growing application of FL is now taking place in medicine and banking, where data is highly confidential, but actors can allow computing to happen in their system closer to the data.

9.2.3 Research challenges

Non-Identical and Independently Distributed (non-IID) phemomena in FL is usually explored in two directions – in hardware and in data:

Two bottlenecks appear in practice:

These bottlenecks can be addressed by compression using quantisation or model reduction:

9.2.4 Summary

Federated Learning is a distributed learning setting. This is built on many clients contributing updates to a shared model. A centralised server chooses a set of clients to update the model. These clients update the global model with their own local data and push that back to the server. The server aggregates the updates from the contributing clients and pushes the updated model for many more clients to use (even if they don’t participate in the update).

9.2.5 Hands on experience

This course has no lab associated to the theoretical concepts. However, there are many resources online if you want to start playing with FL. The more popular repository is FedML. This has a set of examples and benchmarks you can explore independently. A dedicated implementation for IoT devices is proposed here https://github.com/FedML-AI/FedML-IoT.

9.3 COM3505 Week 09 Notes

9.3.1 Learning Objectives

Our objective this week is to get cracking on the projects, continuing the course theme of learning by doing. Four more weeks and you’ll have had a real taste of creating a new IoT device, from breadboard to prototype, and be all set to become a practising IoT engineer :)

You should have chosen your project now. If not, do so ASAP!

10 WaterElves, Gripples and Fish Poo: IoT Case Studies

10.1 Aquaponic Control Systems

In section 7.2 I referred to aquaponics, a sustainable intensive agriculture technology. Myself and colleagues work on monitoring and control systems that attempt to make the technology simpler to use, and we build those systems using IoT technology. This section will spend a little bandwidth looking at an IoT system for intensive sustainable agriculture with aquaponics: the WaterElf. First a little context.

10.1.1 Urban Agriculture

Why bother growing things in towns and cities? Most of our agricultural land is already being farmed, and most of our food production is part of a system which prioritises profit over costs like pollution, soil degradation or injustice.

How can we grow more food in more places (and grow it quickly)? My colleague Jill Edmondson ran a project called My Harvest (for which my team in Computer Science built the database backend). Jill’s team used the data from that project (collected from allotment growers across the UK) and a geospatial analysis of Sheffield city green (and grey71) spaces to estimate the amount of food that could be produced within the city boundaries. The results (published in Nature food (Edmondson et al. 2020)) are surprising: urban spaces have the potential to host large-scale growing that can make a significant contribution to our food supplies.

To do this we need intensive agricultural methods that don’t need soil. Whilst the go-to option there is hydroponics (growing vegetables in disolved fertiliser), the difficulty is that fertilisers are either a scarce natural resource (e.g. potash) or produced using the Haber-Bosch nitrogen fixation process (which is currently responsible for something on the order 1 or 2% of the world’s total energy consumption). The combination of fish and vegetables in a closed loop system — or aquaponics — has both low environmental impact and high productivity.

72

At Regather’s Club Garden the Institute for Sustainable Food are building an urban farming demonstrator to showcase the potential of the technology, and hopefully to become part of an established urban food hub, on the edge of Sheffield’s https://www.facebook.com/LansdowneEstate/.

Aquaponics replaces the fertilisers used in hydroponic growing with fish food, and uses only as much water as the plants need. It works all year round, and can fit into the nooks and crannies of urban spaces. Vegetarians use ornamental fish; others use edible species.

73

The technique creates a symbiotic ecosystem of fish and plants that is necessarily clean — if we tried to add fertilisers or pesticides to the growbeds we would soon damage the fish, as would growth factors or prophylactic antibiotics fed to the fish. The widespread use of agrochemicals in our current food production industries drives down resilience — most are dependent on oil, and many inject toxicity into our environment and compromise the long-term sustainability of agriculture. The integrated ecosystem present in aquaponics (combining fish, vegetables, bacteria and worms) results in less disease and faster growth, removing the need for agrochemicals and pharmaceuticals.

74

10.1.2 Research Questions

The catch? Balancing the ecosystem is tricky, and we don’t know enough about the optimum setups for cooler climates (or about the systems sizes that suit typical UK greenhouses or sheds). Research questions we’re addressing to meet these challenges include:

Connectivity. New-generation microcontrollers like ESP8266 and ESP32 bring a mature wifi stack and modern micro compute resource to the existing Arduino ecosystem, and WiFi is winning the Internet of Things (IoT) connectivity battle in cases where power is easily available. In other cases LoRaWAN, Sigfox, LTE-M and NB-IoT are all candidates vying for uptake. We’re combining WiFi and LoRaWAN, and developing multi-mode device provisioning that will work in both well-resourced, WiFi and power-rich environments and in low resource economies where long range low power radio is more appropriate.

Monitoring. Aquaponics relies on three interdependent sets of organisms (plants, fish and nitrifying bacteria), all of which have to operate within certain parameters. Monitoring water quality (pH, temperature), light levels (lux), and air quality (temperature and humidity), and providing cloud-based data recording for produce outputs, can make managing this balancing act easier and bring the technology in reach of many more users.

75

Control. The bacterial populations that we rely on for converting ammonia to nitrates thrive in some types of hydroponic growth media that are highly oxygenated, and the resultant combination of nitrate concentrations and oxygen also promote high plant growth rates. A good way to create that type of environment is ebb and flow (or flood and drain) irrigation, which requires solenoid control of growth bed water flows. Industrial solenoids are expensive; we’re experimenting with several cheaper valves appropriate to low resource environments and driving them via MOSFET circuits. We also control lighting, air and water pumps using a 433 MHz transmitter and COTS (commodity) mains switching devices which are widely available (and therefore remove the need to deal with mains voltages in our circuitry).

Analytics. Our connected devices interface with the new generation of IoT cloudside logging, visualisation, event generation, platform integration and analytics (for example on Adafruit.io or IFTTT), along with lower-level data logging on AWS-based cloud VMs that offer high-reliability (EBS) multiregion data storage. An example dashboard:

76

At the heart of the approach is an IoT device called the WaterElf, which pairs an ESP32 microcontroller with environmental sensing (for lux, air and water temperature, humidity and water levels) and water flow control. The device is open hardware supported by open source firmware on the unphone gitlab repository. An example schematic:

WaterElf is an ESP32-based control, monitoring and communications device for aquaponics. It combines several features characteristic of different greenhouse control systems and water quality monitors used in aquaculture and hydroponics. The latest version of the WaterElf features wireless sensors to measure pH, light intensity (LUX), water temperature, air temperature and humidity as well as three ultrasonic sensors that enable the monitoring of water levels in three growbeds from a single device. A programme controlling the opening and closing of valves, thereby regulating water levels, runs locally in the Elf. The timing of this is controlled by the water level (high water levels sending a signal to stop the flow) – if for any reason this signal does not come, filling of tanks will automatically stop after 15 minutes to prevent overflow.

Saving the world with fish poo plus electronics. You heard it here first :)

10.2 COM3505 Week 10 Notes

10.2.1 Learning Objectives

Our objectives this week are to:

10.3 Further Reading

As noted in an earlier chapter, Adafruit’s “All the IoT” series has lots of useful background on IoT connectivity options. Hera are the Adafruit-hosted copies: - episode 1, transports - episode 2, protocols - episode 3, services - episode 4, Adafruit.io - episode 5, security

More details on application-layer communication protocols:

Cautionary tales:

11 Gateway to the Future

There are two dishes on the dessert menu.

First, as is frequently noted in the literature, you can’t say “IoT” without mentioning the network, and we’ll fill in one of the blanks in our earlier discussions of connectivity by summarising the more common examples of off-board communications protocols.

Second, we promised right back in Chapter 1 to return to the subject of hope, and so we will do.

11.1 Non-Local Communications Protocols

We build our things using local protocols (or buses) to integrate their sensors and actuators (e.g. UART, SPI, I2C) — we saw examples of these in chapter 5. We get our things connected using network transports, and get them talking to each other (or to gateways or to the cloud) using network protocols.

Adafruit’s IoT video series (see section 10.3) is excellent on transports (and their key criteria of “power, distance and bits”), including:

Transports deliver bits between devices, and between devices and gateways or devices and the cloud. Transports are like the telegraph, or smoke signals: they pass along data, but without imposing an interpretation on that data. Protocols sit on top of transports; we can think of them as the language that devices use to speak to each other (a bit like morse code, or “three puffs means the Vikings are coming!”). The rest of this section looks at off-board (or non-local) networking options for the IoT, and finishes with a look at LPWAN options.

So far we’ve used: HTTP(S) over WiFi. Other important options exist at the application layer (and LPWAN is important):

77

HTTP, and REST (a set of conventions plus JSON or XML on top of HTTP), is probably the most popular protocol (see below). The big three non-HTTP protocols are:

More recently web sockets have become an options, lower overhead than HTTP but also lower level).

Important differences include the security models (often not baked in to the earlier protocols, later layered on with SSL/TLS), QoS, levels of support:

78

Their relative popularity was summarised by an Eclipse Foundation survey:

11.1.1 Lower Power WANs and TTN

The rest of this section concentrates on Lower Power Wide Area Networks (LPWAN) IoT connectivity options, and especialy LoRaWAN and The Things Network.

WiFi is popular, but it is designed for high bandwidth devices, which may be quite wasteful in this context (our IoT gizmos are probably not going to be streaming HD video!). Because of this a variety of LPWAN options are coming that more closely mirror IoT specificities (very low data rates, low power radio options). The leading examples:

One solution to all this choice is the ‘throw in the kitchen sink’ approach — e.g. Pycom supporting 5 networks.

The LoRa (Long Range) radio protocol is a (physical layer), spread spectrum (~125 kHz) protocol employing a frequency-modulated (FM) chirp. LoRaWAN is a media access control (MAC) protocol (network layer) layered on top of LoRa. Upload / download are symmetrical. More details in Andreas Spiess’ video:

LoRaWAN has come to particular prominence recently because of the success of the Things Network. This crowdfunded LoRaWAN initiative enables distributed LoRaWAN has been very successful in spreading across the world, based on its democratising manifesto:

Everything that carries power will be connected to Internet eventually. Controlling the network that makes this possible means controlling the world. We believe that this power should not be restricted to a few people, companies or nations. Instead this should be distributed over as many people as possible without the possibility to be taken away by anyone.

79

There are two main families of alternative to LoRaWAN: UNB (e.g. Sigfox) and the new cellular offerings (LTE-M and NB-IOT). Ultra Narrow Band (UNB) uses less spectrum than LoRa, and experiences lower noise (interference) levels as a result. Upload / download are asymmetrical (download, or network to device, is lower bandwidth than upload, or device to network). Both Sigfox and NB IoT use UNB.

Sigfox is a global LPWAN network operator that: tries to build adoption at the device (hardware) level by minimising connections costs; makes money by selling bandwidth; competes with current mobile telecoms providers. NB IoT and LTE-M are from the big telecoms companies.

The mobile operators (working together as 3GPP) have made several responses to LPWAN competition:

The key criteria are bandwidth vs. range and vs. power:

(The other protocols: mentionned are: ZigBee, but this is more about PANs (personal area networks) than WANs; BlueTooth and BlueTooth LE, ditto. These make sense as device-to-gateway protocols, and if we assume that the gateway has mains power, then it is likely to use WiFi or wired ethernet.)

This has been a whistle-stop tour of IoT network transports and protocols.

11.2 Hope, Revisited

On day one of the fourth iteration of the IoT course from which these notes originate, I had the unexpected and slightly unsettling pleasure of inviting one hundred (mostly) strangers into my bedroom (mediated, of course, by our Zoom-alike video conferencing tool) for my first lecture of 2021. The forward view from my Sheffield home was of more lockdown, more illness, more poverty, and less of the pleasures of human company that sustain and nourish us as members of our miraculously social and cooperative species. There had to be, I thought, some silver linings hiding in this vast and ugly grey cloud. There were three that I came up with then.

First, chaos is the new normal. Don’t worry if you’re blown off course. Don’t stress about that code that doesn’t compile. We’re only human, and the world is a difficult place to live. Try your best to keep smiling, expect others to sympathise and they probably will, and make sure to have a silly hat ready to hand at all times.

Second, like banging your head against a wall, it will feel really good when it stops!80

We’ll get to the third in a minute, but I’ve since thought of a fourth silver lining, which is how well the need to isolate to protect each other has proved our respect and love for our fellow strugglers. Most especially the wonderful extent to which the young and strong and healthy have carefully isolated and distanced in order to protect others from infection. Everyone is a hero, and you are all beautiful.

The one that I’d like to delve into here, though, was silver lining #3, viz.: it is more obvious than ever that our social systems are profoundly broken, and that the old ways don’t work. Those ways are what got us into this nightmare in the first place! So we have to build back differently. I’ll finish this section by summarising a little of why that has to be true (which is all a bit of a downer; feel free to skip ahead!) and then share a few glints of optimism, a few shards of hope, poking their bright beams through the overcast. And hope there is; so heads up!

11.2.1 The Depressing Bit

(Skip to the next section if you’re feeling low!)

The pandemic, climate change, a decade of austerity ended by a massive spree for corrupt procurement: we are not, my friends, floating gently along the River of Contentment. As noted by Peter Wadhams in his Farewell to Ice (Wadhams 2017),

…the existing level of carbon dioxide in the atmosphere is sufficient to cause unacceptable amounts of warming in the future. We no longer have a ‘carbon budget’ that we can burn through before feeling worried that we have caused massive climate change. We have burned through the budget and are causing the change now. … By now it is too late. The CO₂ levels in the atmosphere are already so high that when their warming potential is realized in a few decades, the resulting temperature rise will be catastrophic. (p. 192)

What to do? The first thing is to recognise that the problems are neither the accidental consequence of imperfect electoral systems nor the outcome of unpleasant character or greedy individuals. Not to say that these things don’t exist and aren’t significant at any particular point in time, but they are not the causes of the underlying tendency towards disaster that we are, unfortunately, firmly routed on.

No, the problems are systemic, and structural.

Lord Stern, who was commissioned by the UK government in the mid-naughties to study The Economics of Climate Change (Stern and UK, Treasury 2007) advocated aiming for a likely 3 degree temperature rise as being “economically viable,” and rejected all other options as non-viable. The report then included all sorts of evidence that shows the 3 degree rise to be a huge gamble, with odds of 50:50 in some cases of much worse consequences.

In other words, as set out by Stern at the behest of the UK Treasury, our economic system cannot support odds better than the toss of a coin for avoiding catastrophic change. Can we conclude anything other than that the economic system itself is at fault? And the situation has only worsenned since then. If it is not economically viable to save the planet, then the economic system is wrong.

It is common to think that markets find efficient solutions. They don’t: they find profitable solutions. And our markets are dominated by a quite small number of truly humungous corporations, vying to become even larger. In a 2011 article on the Network of Global Corporate Control, complex systems specialists from a Swiss university (Vitali, Glattfelder, and Battiston 2011) analysed “the relationships between 43,000 transnational corporations” and “identified a relatively small group of companies, mainly banks, with disproportionate power over the global economy” (New Scientist, October 2011). The study “combines the mathematics long used to model natural systems with comprehensive corporate data to map ownership among the world’s transnational corporations (TNCs)” and uses data on “37 million companies and investors” with details of “all 43,060 TNCs.” The research “revealed a core of 1318 companies” that “represented 20 per cent of global operating revenues” and “the majority of the world’s large blue chip and manufacturing firms — the ‘real’ economy — representing a further 60 per cent of global revenues.” Further, “it found much of it tracked back to a ‘super-entity’ of 147 even more tightly knit companies … that controlled 40 per cent of the total wealth … Most were financial institutions. The top 20 included Barclays Bank, JPMorgan Chase & Co, and The Goldman Sachs Group.” In other words, 150 or so organisations (mostly banks) control the lion’s share of world production. Another 1000 or so control much of the rest.

One thing that the markets dominated by these huge corporations do often drive is competition. (Not all the time: when the banks cease to be competitive the state bails them out; when a powerful group of companies stitches up one sector then they’ll inflate prices. But as a general rule, markets are competitive arenas.) Competition in our economy means that each company has to grow (or else your competitor will get big enough to buy you or undercut you or otherwise get their hands on your share of the pie). Infinite growth is built in to the fundamental model of our economy. Of course there are, in reality, limits to growth: common sense can tell you all you need to know here, but if that doesn’t cut it then get a few of your friends or colleagues to rendezvous in the bathroom or the stationary cupboard and then just keep on packing them in. Economists may tell you that ‘externalities’ mean that growth is unlimited, but your friends will tell you that things are getting pretty stuffy already and to please stop being such a dozy wazzock. As globalisation has spread corporate competition across the world, so growth gets less and less viable within existing markets.

Competition also means that if one corporation or country manages to drive down the wages and social services of their workforce then they automatically put pressure on their competitors to do the same. Otherwise the higher profits of the cheapskates will let them encroach on the markets of the higher paying. Over time this creates a race to the bottom.

And all this destructive chaos is the basis of our food system, and the zoonotic melting pots that it has created; as Wallace wrote presciently in 2016, “Highly capitalized agriculture may be farming pathogens as much as chickens or corn” (Wallace 2016).

So much, so depressing. Bleugh.

11.2.2 The Third Certainty: Change

It is sometimes said that the only certainties in life are death and taxes. I think we can safely add a third certainty: change. The systemic tendencies (towards infinite growth or minimal wages) which we can see working themselves out in our food system (progenitor of the covid pandemic81) or our environment (and the melting ice caps) or our health systems (where PPE stocks became an afterthought) create a structure which is both massively dynamic and permanently unstable. I think that if we raise our heads and look far enough ahead, even as we judder and shake atop this swirling cauldron there is every chance that we can steer a course to a more rational world, and save ourselves from the whirlpool at the center.

Why? Three reasons.82

11.2.2.1 Democratising… Stuff?

The net revolutionised the virtual world and made publishing free to all. But what about the chair your sit on or the fork you eat with? What if we get the ability to share, modify and build anything, in the same way we can publish anything? What if we can democratise the creation and recreation of the physical world? What if we can devolve manufacturing to individuals and communities? Perhaps we would make different choices? Perhaps we wouldn’t put profit before people?

If the last 20 years were about the web, then the next 20 will be about making. Why? Ubiquitous connectivity and decentralised production in the virtual world have made revolutions in creating, sharing and consuming on-line. Now the same changes are starting in the world of manufacturing, and the consequences are likely to be massive.

Remember how hard it used to be to publish? Photocopiers spawned a whole generation of fliers and fanzines, but the big-time of global distribution used to be a very closed world. When we publish we share, and the web has let us share as never before – but, until recently, we’ve mostly used the web to share information (in the form of bit streams of one sort or another). The next revolutionary wave of technology brings the ability to share into the physical world – it brings the information revolution from bits to atoms. And as Chris Anderson writes in his Makers83 the physical world dwarfs the virtual. (There’s perhaps an 80-20 ratio between economic activity devoted to atoms in comparison to bits.)

Capitalism drives innovation, which brings with it continual waves of technological revolution.84

Anderson’s book quotes Cory Doctorow saying that increasingly “the money on the table is like krill” – many many tiny chunks of nutrition that suits a new type of sieve, smaller and more distributed (a “long tail”). Both authors imagine the changes that will take place when the means of production become minituarised, localised, and – in a sense – democratised. At least under some circumstances the small and the open and the fast moving can sneak beneath the corporate radar long enough to become viable alternatives – like my friends at Pimoroni in Sheffield, for example, who sold tens of thousands of locally-made boxes for the Raspberry Pi.

The new methods of manufacturing (CAD-CAM designs driving CNC routers, 3D printing and laser cutting), and the new culture of open source and dynamic virtual organisations start to challenge corporate dominance, at least around the edges. China’s explosive growth and its willingness to ignore the west’s definition of “intellectual property” helps too (though bringing with it the labour relations of the sweatshop).

Anderson talks of a “future where the Maker Movement is more about self-sufficiency… than it is about building businesses….” This, he says, is “closer to the original ideas of the Homebrew Computing Club or the Whole Earth Catalogue. The idea, then, was not to create big companies, but rather to free ourselves from big companies” (pp. 225-226)

We can also make a link into the argument for localist economics made by organisations like the Transition Network (e.g. in Rob Hopkins’ books) – peak oil, social instability and environmental crisis all point to the local and the small scale as a key source of sustainability and resilience. The more stuff we can manufacture within short distances of where we live, the safer we are (not to mention the saved carbon in long-distance transport).

Welcome to the future – perhaps it will be of our own making :-)

11.2.2.2 IoT: from their Cloud to our Fog?

Industry was pumping private data into its clouds like the hydrocarbon barons had pumped CO₂ into the atmosphere. Like those fossil fuel billionaires, the barons of the surveillance economy had a vested interest in sowing confusion about whether and how all this was going to bite us in the ass. By the time climate change can no longer be denied, it’ll be too late: we’ll have pumped too much CO₂ into the sky to stop the seas from swallowing the world; by the time the datapocalypse is obvious even to people whose paychecks depended on denying it, it would be too late. Any data you collect will probably leak, any data you retain will definitely leak, and we’re putting data-collection capability into fucking lightbulbs now. It’s way too late to decarbonize the surveillance economy. Cory Doctorow, Attack Surface (Doctorow 2020)

Cloud computing has driven miraculous reductions in the difficulty of large scale data collection, analysis and distribution. This has allowed us to conquer many previouly insoluble problems, and, to a degree, democratised access to massively scaleable computation. It has also had two more negative consequences. In the context of the market-driven imperative to sell the next plastic widget to ever more passive and isolated consumers, advertising has become the rationale for a surveillance system more pervasive and all-encompassing than the worst dystopian nightmares or our forebears. (As part of the same process, the carbon footprint of the global datacenter has reached a significant fraction of our total energy uses, including a troubling quantity of energy devoted to blockchains85.)

And then there’s the spies. Half a dozen years ago I wrote:

If you’ve been paying more than a gnat’s hair’s width of attention to All Things Internet in the past year or so, you’ll know that the US and the UK have been spending the odd spare billion of taxpayers’ hard-earned on a programme of indiscriminate surveillance of everything you, I and the dog do on-line.

(This is fine of course. I’ve nothing to hide. You’re welcome to pop round and put a microphone in my toilet and a webcam in my bedroom — though I may demand the right to fit the same gear in your house first… That ok? And I reserve the right to point out that a couple of hundred thousand people have the same access to all your data that Edward Snowden had shortly before he walked out of a US government building in Hawaii with several gigabytes of leak. If he can do it, how many others? And do you trust them all? You do? Great! Now, please email your credit card numbers and a selection of explicit selfies to me. It’s for your own good, honest.)

The spies like to cultivate back doors in the cryptography that protects on-line transactions. Even if that was ok, there’s no way to have a backdoor that only a spy can use. Four years ago, in the wake of a massive attack on UK medical computing, I wrote that:

The ransomware cyber attack on the NHS is horrifying — and as a computer scientist I feel ashamed that the world my field helped create is now at the mercy of such destructive scammers. It didn’t have to be this way!

This note looks at the context of the attack — why did the NSA help the attackers?! — and explains the “kill switch” and how it slowed the spread of the WannaCry worm. It concludes with ways we can avoid this type of nightmare in the future.

First, the spooks: the NSA (and GCHQ) believe that they need (and have the right) to see every piece of digital communication made by any citizen at any time under any circumstances. More than that — they also believe that they’re entitled to turn on your computer or phone or TV and listen on its microphone or watch on its camera. (That’s why Facebook’s Mark Zuckerberg tapes over his laptop’s webcam!)

There’s a problem: just as we don’t leave home without locking our doors, we don’t leave our computer systems unguarded. How are the spies to cope? They do two things:

  1. “Persuade” software companies to leave deliberate holes in their security. (This is a little like convincing all lock installers to post a copy of every key to the local constabulary, only worse: digital keys are much easier to copy or steal. When Amber Rudd says WhatsApp needs a handy backdoor for law enforcement purposes, this is what she means!)

  2. Break into computer systems, subvert their security mechanisms and suck up the data from your email, chats, documents, etc. etc.

This second activity is what has helped bring the NHS’s computer systems to their knees. One of the NSA’s programs for breaking into Microsoft software (codenamed Eternalblue) was stolen and publicly released in April. The black hat hackers behind WannaCry adapted it to their own nefarious purposes, and we’re now suffering the results.

Government not only supports the spies in these efforts, they allow them to do their worst in total secrecy, even in the courts. In the UK we’re now banned by the Investigatory Powers Act from hearing in court about what evidence was collected in this way and how — giving a whole range of government agencies and employees carte blanche to compromise our online security with impunity.

Since then the Solar Winds hack has demonstrated even more powerfully how vulnerable this process (and the closed-source software that makes it possible) makes us all. Open source is part of the answer, but the situation is urgent; what to do?

Tim Berners-Lee’s answer, from his company Inrupt, is the Solid project to decentralise personal data storage. With a longer pedigree, Freedombox has been building personal servers to move the basic functions of cloud-based SaaS into our homes. Hook these up with the IoT, with programmable personal devices and open systems, and we’re starting to see ways to claw back our data: to build our own fog from our domestic gateways and personal IoT devices.

There’s another potential upside here, to do with prospects for deconvergence, disaggregation, and device respecialisation. Sherry Turkle’s fantastic but terrifying book Alone Together explains how “Technology has become the architect of our intimacies. Online, we fall prey to the illusion of companionship, gathering thousands of Twitter and Facebook friends, and confusing tweets and wall posts with authentic communication. But this relentless connection leads to a deep solitude.” The mental health penalties of this contradiction are increasingly commonplace, especially amongst the young, our digital natives.

The project I would like to do is about making it easier not to look at my phone so often (and about sharing less of my personal data with the internet behemoths). We have spent a decade squeezing all the information processing functionality in our lives into a single device (and a marvellous device it has become: smartphones aren’t going away any time soon). My problem is that whenever I decide to step away from the beast for a little while (to get a break from all the messaging, for example, or rest my eyes on a paper book for a change) one of the other functions of the phone calls me back almost immediately (I want to put some music on, or check a recipe, or etc. etc. etc.). What I would like is to separate out some of those functions into special-purpose devices: informational appliances that are more restricted, less all-encompassing. This is partly what smarthome gadgets like Echo or HomePod or Nest attempt to do, but at the cost of sending yet more data to the cloud (and from there to identity thieves, or, whenever it feels the need, to the state – see above).

In this context I’m excited by the possibilities of connected microcontrollers like the ESP32, which are low energy and low cost enough to be used in volume, but powerful enough to do useful things in a smartwatch package like this one, for example. IoT devices exist at the edge of the cloud. If they become more peer-to-peer, more foggy, (and if we include the substantial computate power now available on the gateway ARM processors86) then we can reduce reliance on the cloud.

So: informational device disagreggation! New privacy in the fog! Unlimited free biscuits for all nerds! You heard it here first.

11.2.2.3 Transition: from Sustainability to Resilience?

When not seen through the distorted lens of their supposed efficiency, markets are destructive. Instead we need to think of efficiency as what best meets human needs. This means that:

…“what’s good is what’s good for the biosphere.” In light of that principle, many efficiencies are quickly seen to be profoundly destructive, and many inefficiencies can now be understood as unintentionally salvational. Robustness and resilience are in general inefficient; but they are robust, they are resilient. And we need that by design. Kim Stanley Robinson, The Ministry for the Future (Robinson 2020)

Resilience is the ability to bounce back from stresses, strains and shocks. A society is resilient if it can supply its needs (e.g. for food, shelter or health) without relying on systems outside of its control. A society is not resilient if it relies on shipping large parts of its needs half way around the world. We are vulnerable to disruption brought by war, or the chaotic weather systems brought by climate change, or the volatile price of oil.

The Transition Network make a strong case for the benefits of working towards community resilience, not least because the act of making positive change, even when that change is necessarily partial and incomplete, is a great way to enjoy life! Connecting to our locality, to the people around us, is a great way to feel genuine togetherness and to reduce our reliance on the virtual variety analysed by Turkle. And the spin-offs are to make us all stronger: for example, increasing local food production can help us adapt to change and make our communities more resilient.

There’s lots to do! Dive in!

11.2.3 The Main Reason

…the strongest, in the existence of any social species, are those who are the most social. In human terms, most ethical… There is no strength to be gained from hurting one another. Only weakness. (Ursula le Guin, The Dispossessed)

Ok, I’m not very good at counting (that’s perhaps why I’ve spent most of my working life with computers; they do the numbers!). There are more than three reasons for hope; almost uncountable reasons, and the main one is you!

Thanks for coming out to play, and best of luck with your journey! Let’s leave the last word to our old mate Albert, who knew a thing or two…

A human being is a part of this whole, called by us “Universe,” a part limited in time and space. He experiences himself, his thoughts and feelings as something separated from the rest — a kind of optical delusion of consciousness. This delusion is a kind of prison for us, restricting us to our personal desires and to apportion for a few persons nearest to us. Our task must be to free ourselves from this prison by widening our circle of compassion to embrace all living creatures and the whole of nature in its beauty. (Albert Einstein.)

11.3 COM3505 Weeks 11 & 12 Notes

11.3.1 Learning Objectives

Our objective this week and next is to finish, commit and push the project work. Good luck, and bon voyage!

11.4 Colophon

These notes were:

All opinions expressed are those of the author(s). YMMV.

Bibliography

Adelantado, F, X Vilajosana, P Tuset-Peiro, B Martinez, J Melia-Segui, and T Watteyne. 2017. Understanding the Limits of LoRaWAN.” IEEE Commun. Mag. 55 (9): 34–40.
Arduino. 2017. Arduino IDE Guide.” https://www.arduino.cc/en/Guide/Environment. https://www.arduino.cc/en/Guide/Environment.
Ashton, Kevin. 2011. That ‘internet of things’ thing.” RFiD Journal 22 (7).
Banzi, Massimo, and Michael Shiloh. 2014b. Getting Started with Arduino: The Open Source Electronics Prototyping Platform. Maker Media, Inc.
———. 2014a. Getting Started with Arduino: The Open Source Electronics Prototyping Platform. Maker Media, Inc. https://market.android.com/details?id=book-Xd3SBQAAQBAJ.
Barragán, H. 2004. Wiring: Prototyping physical interaction design.” Interaction Design Institute, Ivrea, Italy. https://scholar.google.ca/scholar?cluster=4073615779562206947&hl=en&as_sdt=0,5&sciodt=0,5.
———. 2016. The untold history of Arduino.” Luettavissa: Https://Arduinohistory. Github. Io/Luettu. https://scholar.google.ca/scholar?cluster=15872095708386818356&hl=en&as_sdt=0,5&sciodt=0,5.
Bassi, Alessandro, Martin Bauer, Martin Fiedler, Thorsten Kramp, Rob Van Kranenburg, Sebastian Lange, and Stefan Meissner. 2013. Enabling things to talk.” Designing IoT Solutions with the IoT Architectural Reference Model, 163–211.
Blenn, Norbert, and Fernando Kuipers. 2017. LoRaWAN in the Wild: Measurements from The Things Network.” arXiv [Cs.NI], June. http://arxiv.org/abs/1706.03086.
Brendan McMahan, H, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. 2016. Communication-Efficient Learning of Deep Networks from Decentralized Data,” February. http://arxiv.org/abs/1602.05629.
Community, Esp32. 2017. Esp32 Forum.” https://esp32.com/. https://esp32.com/.
Cunningham, H. 2012. Agile Research.” ArXiv e-Prints http://arxiv.org/abs/1202.0652v1 (February). http://adsabs.harvard.edu/abs/2012arXiv1202.0652C.
Cunningham, Hamish, and Benzion Kotzen. 2015. Meet the sustainable vegetables that thrive on a diet of fish poo.” The Conversation, November.
Denning, Dorothy E, and Peter J Denning. 1977. Certification of programs for secure information flow.” Commun. ACM 20 (7): 504–13. https://doi.org/10.1145/359636.359712.
Dhanjani, Nitesh. 2015. Abusing the Internet of Things: Blackouts, Freakouts, and Stakeouts. “O’Reilly Media, Inc.”
Dickens, Charles. 1877. A Tale of Two Cities: And, Great Expectations. Lee; Shepard. https://play.google.com/store/books/details?id=opBBAQAAMAAJ.
Dizdarević, Jasenka, Francisco Carpio, Admela Jukan, and Xavi Masip-Bruin. 2019. A Survey of Communication Protocols for Internet of Things and Related Challenges of Fog and Cloud Computing Integration.” ACM Comput. Surv. 51 (6): 1–29. https://doi.org/10.1145/3292674.
Doctorow, Cory. 2012. Pirate cinema. Macmillan. https://freekidsbooks.org/wp-content/uploads/2020/01/FKB-Stories-Cory_Doctorow_-_Pirate_Cinema.pdf.
———. 2019. Unauthorized Bread.” https://craphound.com/category/unauthorizedbread/. https://craphound.com/category/unauthorizedbread/.
———. 2020. Attack Surface. Head of Zeus Ltd. https://play.google.com/store/books/details?id=G2XWDwAAQBAJ.
Doukas, Charalampos. 2012. Building Internet of Things with the Arduino. USA: CreateSpace Independent Publishing Platform.
Edmondson, Jill L, Hamish Cunningham, Daniele O Densley Tingley, Miriam C Dobson, Darren R Grafius, Jonathan R Leake, Nicola McHugh, et al. 2020. The hidden potential of urban horticulture.” Nature Food 1 (3): 155–59.
Fraser, Quentin Stafford-. 1995. Trojan Room Coffee Pot Biography.” https://www.cl.cam.ac.uk/coffee/qsf/coffee.html. https://www.cl.cam.ac.uk/coffee/qsf/coffee.html.
Greenfield, Adam. 2017. Rise of the machines: who is the ‘Internet of things’ good for? Guardian, June. https://goo.gl/uIUCrD.
Grover, Sarthak, and Nick Feamster. 2016. The internet of unpatched things.” Proc. FTC PrivacyCon. https://www.ftc.gov/system/files/documents/public_comments/2015/10/00071-98118.pdf.
Grover, Siddharth. 2017. The Internet of Things (2016).” International Journal of Computer Science and Engineering. https://doi.org/10.14445/23488387/ijcse-v4i8p101.
Hunn, Nick. 2018. British Smart Meters cost £28 million EACH.” https://www.nickhunn.com/british-smart-meters-cost-28-million-each/. https://www.nickhunn.com/british-smart-meters-cost-28-million-each/.
Kairouz, Peter, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, et al. 2019. Advances and Open Problems in Federated Learning,” December. http://arxiv.org/abs/1912.04977.
Kolban, Neil. 2017. Kolban’s Book on ESP32.
Kranenburg, Rob van, and Alex Bassi. 2012. IoT Challenges.” Proc. Int. Wirel. Commun. Mob. Comput. Conf. 1 (1): 9. https://doi.org/10.1186/2192-1121-1-9.
Kurniawan, Agus. 2016. Smart Internet of Things Projects. Packt.
Li, Y, and H Cunningham. 2008. Geometric and Quantum Methods for Information Retrieval.” SIGIR Forum 42 (2): 22–32. http://www.sigir.org/forum/2008D-TOC.html.
Loftus, Jack. 2011. Dear Fitbit Users, Kudos On the 30 Minutes of ‘Vigorous Sexual Activity’ Last Night.” Gizmodo.
MacDermott, A, T Baker, and Q Shi. 2018. Iot Forensics: Challenges for the Ioa Era.” In 2018 9th IFIP International Conference on New Technologies, Mobility and Security (NTMS), 1–5.
Machine, Coke. n.d. The "Only" Coke Machine on the Internet.” https://www.cs.cmu.edu/~coke/history_long.txt. https://www.cs.cmu.edu/~coke/history_long.txt.
Margolis, Michael. 2011. Arduino Cookbook: Recipes to Begin, Expand, and Enhance Your Projects. “O’Reilly Media, Inc.”
McEwen, Adrian, and Hakim Cassimally. 2013. Designing the Internet of Things. John Wiley & Sons.
Monk, Simon. 2013. Programming Arduino Next Steps: Going Further with Sketches. McGraw Hill Professional.
Naik, N. 2017. Choice of effective messaging protocols for IoT systems: MQTT, CoAP, AMQP and HTTP.” In 2017 IEEE International Systems Engineering Symposium (ISSE), 1–7. https://doi.org/10.1109/SysEng.2017.8088251.
NERC. 2016. NERC Guidance on the Safe Use of Lithium Batteries.” http://www.nerc.ac.uk/about/policy/safety/procedures/guidance-lithium-batteries/. http://www.nerc.ac.uk/about/policy/safety/procedures/guidance-lithium-batteries/.
Nold, Christian, and Rob van Kranenburg. 2011. The Internet of People for a Post-oil World. Lulu.com.
Nordrum, Amy. 2016. The internet of fewer things [News].” IEEE Spectrum. https://doi.org/10.1109/mspec.2016.7572524.
O’Muircheartaigh, Fionan. 2013. The Smart City: Using Technology to Reduce Congestion in London.” https://www.sustainablebusinesstoolkit.com/the-smart-city-using-technology-to-reduce-congestion-in-london/. https://www.sustainablebusinesstoolkit.com/the-smart-city-using-technology-to-reduce-congestion-in-london/.
O’Reilly, Tim, and Cory Doctorow. 2015. Opportunities and Challenges in the IoT. O’Reilly Media, Inc. https://www.oreilly.com/library/view/opportunities-and-challenges/9781492048220/.
Perzanowski, Aaron, and Jason Schultz. 2016. The End of Ownership.” https://mitpress.mit.edu/books/end-ownership. https://mitpress.mit.edu/books/end-ownership.
Pfister, Cuno. 2011. Getting Started with the Internet of Things: Connecting Sensors and Microcontrollers to the Cloud. “O’Reilly Media, Inc.”
Rakcozy, B. 2011. Aquaponics Q and A: The Answers to Your Questions about Aquaponics. Nelson; Pade. http://books.google.co.uk/books/about/Aquaponics_Q_and_A.html?hl=&id=scVKMwEACAAJ.
Robinson, Kim Stanley. 2020. The Ministry for the Future: A Novel. Orbit. https://play.google.com/store/books/details?id=dvHNDwAAQBAJ.
Romkey, J. 2017. Toast of the IoT: The 1990 Interop Internet Toaster.” IEEE Consumer Electronics Magazine 6 (1): 116–19. https://doi.org/10.1109/MCE.2016.2614740.
Rubell, Brent. 2018. Welcome to Adafruit IO.” https://learn.adafruit.com/welcome-to-adafruit-io; Adafruit. https://learn.adafruit.com/welcome-to-adafruit-io.
Schneier, Bruce. 2017. Click Here to Kill Everyone.” NY Mag, January. http://nymag.com/selectall/2017/01/the-internet-of-things-dangerous-future-bruce-schneier.html.
Schwartz, Marco. 2016a. Home Automation with the ESP8266: Build Home Automation Systems Using the Powerful and Cheap ESP8266 Wifi Chip. CreateSpace Independent Publishing Platform.
———. 2016b. Internet of Things with ESP8266. Packt Publishing Ltd.
Sivaraman, V, H H Gharakheili, A Vishwanath, R Boreli, and O Mehani. 2015. Network-level security and privacy control for smart-home IoT devices.” In 2015 IEEE 11th International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob), 163–67.
Slama, Dirk, Frank Puhlmann, Jim Morrish, and Rishi M Bhatnagar. 2015. Enterprise IoT: Strategies and Best Practices for Connected Products and Services. “O’Reilly Media, Inc.”
Somerville, Christopher, Moti Cohen, Edoardo Pantanella, Austin Stankus, and Alessandro Lovatelli. 2014. Small-scale Aquaponic Food Production: Integrated Fish and Plant Farming. FAO Fisheries and Aquaculture Technical Paper No. 589. FAO. http://books.google.co.uk/books/about/Small_scale_Aquaponic_Food_Production.html?hl=&id=DpRirgEACAAJ.
Stallman, Richard. 2002. Free Software, Free Society: Selected Essays of Richard M. Stallman. Lulu.com.
Sterling, Bruce. 2014. The Epic Struggle of the Internet of Things. Strelka Press. https://www.amazon.co.uk/Epic-Struggle-Internet-Things-ebook/dp/B00N7EKIJ4.
Stern, Nicholas, and UK, Treasury. 2007. The Economics of Climate Change: The Stern Review. Cambridge University Press. https://play.google.com/store/books/details?id=U-VmIrGGZgAC.
Tablan, V, K Bontcheva, I Roberts, and H Cunningham. 2014. Mı́mir: an Open-Source Semantic Search Framework for Interactive Information Seeking and Discovery.” Journal of Web Semantics. https://doi.org/10.1016/j.websem.2014.10.002.
Tablan, V, I Roberts, H Cunningham, and K Bontcheva. 2013. GATECloud.net: a Platform for Large-Scale, Open-Source Text Processing on the Cloud.” Philos. Trans. R. Soc. Lond. A 371 (1983).
Tanenbaum, A S, and H Bos. 2015. Modern operating systems.” http://lib.bvu.edu.vn/bitstream/TVDHBRVT/19439/1/Modern-Operatin-systems.pdf.
Thakur, Manoj. 2016. Zero to Hero ESP8266. Circuits4you.com.
The Economist. 2019. How the world will change as computers spread into everyday objects.” https://www.economist.com/leaders/2019/09/12/how-the-world-will-change-as-computers-spread-into-everyday-objects; The Economist. https://www.economist.com/leaders/2019/09/12/how-the-world-will-change-as-computers-spread-into-everyday-objects.
The Things Network. 2018. The Things Network Manifesto.” https://github.com/TheThingsNetwork/Manifest. https://github.com/TheThingsNetwork/Manifest.
Upton, Eben, and Gareth Halfacree. 2014. Raspberry Pi user guide. John Wiley & Sons.
Vitali, Stefania, and Stefano Battiston. 2014. The Community Structure of the Global Corporate Network.” PLoS ONE. https://doi.org/10.1371/journal.pone.0104655.
Vitali, Stefania, James B Glattfelder, and Stefano Battiston. 2011. The network of global corporate control.” PLoS One 6 (10): e25995. https://doi.org/10.1371/journal.pone.0025995.
Wadhams, Peter. 2017. A farewell to ice: a report from the Arctic. Oxford University Press.
Wallace, Rob. 2016. Big Farms Make Big Flu: Dispatches on Influenza, Agribusiness, and the Nature of Science. NYU Press. https://play.google.com/store/books/details?id=lQE9DAAAQBAJ.
Ward, Mark. 2001. BBC News.” BBC. http://news.bbc.co.uk/1/hi/sci/tech/1264205.stm.
Warden, Pete, and Daniel Situnayake. 2019. TinyML: Machine Learning with TensorFlow Lite on Arduino and Ultra-Low-Power Microcontrollers. “O’Reilly Media, Inc.” https://play.google.com/store/books/details?id=tH3EDwAAQBAJ.
Wiebers, David O, and Valery L Feigin. 2020. What the COVID-19 Crisis Is Telling Humanity.” Neuroepidemiology 54 (4): 283–86. https://doi.org/10.1159/000508654.

  1. Wilgengebroed, via Wikimedia.↩︎

  2. Reading this on GitLab.com? There’s also a PDF version or a GitLab Pages version.↩︎

  3. In February 2021, for example, The Register reported that Amazon-owned “smart” doorbell maker Ring “is suffering a major outage with many of its video doorbells effectively dead, turning smart homes into very dumb ones.”↩︎

  4. There are many other terms we can correctly apply to this field, and we’ll look at these later on as part of putting our work into its historical, social and economic contexts.↩︎

  5. Airman’s Odyssey, Antione de St. Exupery.↩︎

  6. From Socks Studio.↩︎

  7. See an error, omission or something that could be improved? Please raise an issue or make a pull request.↩︎

  8. Guess which chapter this is. Did you say “one?” Pat yourself on the back, you’re getting the hang of this IoT stuff already!↩︎

  9. Your answer better than ours? Well done! Send me a pull request if you have time.↩︎

  10. For this iteration pretty much everything is now open and freely accessible in its up-to-the minute form (apart from the exam questions!).↩︎

  11. All statements in this document are true to the best of our knowledge, except those that aren’t.↩︎

  12. In 2021 all of these are on-line and asynchronous.↩︎

  13. Or an equivalent emulation environment like Cygwin, WSL or a VM or a container.↩︎

  14. Emacs? What’s that?↩︎

  15. Linus Torvalds quipped “I’m an egotistical bastard, and I name all my projects after myself. First ‘Linux,’ now ‘git.’”↩︎

  16. If you’re reading this as a GitLab ‘.mkd’ file (e.g. on gitlab.com or a checked-out repository), citations don’t link anywhere. Try the PDF or the GitLab Pages version to find the references.↩︎

  17. SDK: Software Development Kit; IDE: Integrated Development Environment.↩︎

  18. Wikimedia.↩︎

  19. Recently the sucessor computer to the PDP10, the PDP11, has been completely emulated on an ESP32!↩︎

  20. ‘The use of “hacker” to mean “security breaker” is a confusion on the part of the mass media. We hackers refuse to recognize that meaning, and continue using the word to mean someone who loves to program, someone who enjoys playful cleverness, or the combination of the two.’ (Stallman 2002)↩︎

  21. NDA: Non-Disclosure Agreement. If you’re considering signing one of these read it VERY carefully!↩︎

  22. Python’s release policy: if it isn’t incompatible with all previous and future versions, add some library dependencies to abandoned projects that fail to compile on all target architectures, a new virtualisation environment to deal with different Python versions and a new language feature that gets activated at random depending on what day of the week it is and what flavour of tea you’re drinking.↩︎

  23. This will be called something different (e.g. export.bat) on non-unix platforms.↩︎

  24. When this happens, deleting the relevant tree from ~/.espressif/python_env and re-running esp-idf/install.sh can help.↩︎

  25. During our tale Espressif’s love of easily understandable product names and narrative coherence will often be in evidence; a recent example is an apparent intention to name an ESP32 successor that uses a completely different instruction set and architecture – RISC V – the ESP32-C3. Good to see them sticking to their guns and being just as thoroughly mystifying for new stuff as they were for old.↩︎

  26. Python undoubtedly has many fine qualities. In fact I wish I knew it better, and used it more. It also, unfortunately, has its own particular VERSION HELL, which does tend to complicate life a little here and there.↩︎

  27. No! I’m not coming! You’ll never take me a….↩︎

  28. Subsequent lectures will be provided by our backup lecturer.↩︎

  29. Don’t worry, they’re quite sane.↩︎

  30. So far.↩︎

  31. If you try Arduino Create (or Arduino CLI) and have spare time, why not fork this repository on GitLab, add a section about how it went and make a pull request?↩︎

  32. This was true in late 2020 at least. As the release cycles of the Arduino core and ESP IDF progress PlatformIO support will likely stabilise again, so it is probably worth a try, especially if you like VSCode, with which it integrates well. (Stop press: Feb 2021 4.2 release looks good!)↩︎

  33. This is quite an old release but is the best supported by the Arduino core (as of December 2020).↩︎

  34. Actually the names seem to relate to some change in responsibilities; delve here if you care.↩︎

  35. Why the wierd checkout numbers? These are git commit hashes, which identify unique commits in the version history. We need them because of version hell! (Less flippantly, there are only a subset of versions where IDF and Arduino core match up, especially if we want to use recent developments.)↩︎

  36. This section contributed by Simon Fish.↩︎

  37. Guy says you’re still going to need to wear the silly hat though. At least I think that’s what he said. I always listen very carefully.↩︎

  38. The magic.sh script supports an argument copy-me-to which copies and renames a firmware tree. E.g. cd HelloWorld; ./magic.sh copy-me-to ~/NewExample.↩︎

  39. Hardware 1 is the electronics lab intro sheet – see last week’s notes. Because lab access is restricted you’re likely to be doing this one (“hardware 2”) before hardware 1. Don’t worry, but do have a read of Hardware 1 first if you get time.↩︎

  40. Or feel free to port them! And send me a pull request!↩︎

  41. From Kornakproto.↩︎

  42. This section contributed by Gareth Coleman.↩︎

  43. This section contributed by Gareth Coleman.↩︎

  44. You may not have done hardware 1 yet of course; don’t worry, we’ll get to it later, with luck!↩︎

  45. If you damage your hardware beyond repair, it turns from an interesting electronic gadget into an inanimate lump: it becomes, in other words, pretty much like a brick. Hence the colloquial verb “to brick,” which bears an intimate relationship with the term “FUBAR” in the engineer’s vocabulary. It is recommended to avoid exploring these in practice as far as you can manage.↩︎

  46. From Luca Dentalla.↩︎

  47. From Andreas Schweizer.↩︎

  48. If you’re using the magic.sh script, this can set up environment variables for you and call idf.py to erase flash like this: magic.sh idf-py erase_flash.↩︎

  49. Hitting it won’t help of course, but might make you feel better. Temporarily: when Andy asks you to explain why you need a new one the happy feeling will likely evapourate. You could try blaming it on the silly hat that Guy made you wear?↩︎

  50. This material is derived from a lecture by Gareth Coleman. Thanks G!↩︎

  51. In fact we’re simplifying history a bit if we say that sensors were originally designed to mesh the physical and digital, as the original electrical sensors pre-date the use of numerical information processing in digital computing machines. Early examples included thermistors and microphones, in the 1800s. In modern usage, however, we’re almost always interested in turning whatever electrical signal is generated by our sensor into a digital representation of that signal, in order to manipulate it in the digital realm that underlies almost all contemporary computational processes.↩︎

  52. From rpi.edu/dept/phys.↩︎

  53. From electronics.stackexchange.com.↩︎

  54. From Sparkfun.↩︎

  55. From: Sparkfun.↩︎

  56. Thinking about the vunerability of car control systems reminds me of a particularly hair-raising landing in Athens a couple of decades ago, in an Airbus A320, one of the first planes to use a “fly by wire” digital control system. The old Athens airport was beside the sea and used to suffer from swirling cross-winds when a storm blew up, and on this occasion we went round three times, aborting each of the first two runs due to massive turbulence. The passenger beside me had to make use of the vomit bag, and none of us felt very confident that we were going to end up safe and sound on the ground any time soon. To reassure us one of the crew came on the intercom and said “Don’t worry, we’re quite safe, the plane is controlled by computer!” At which point the anxiety levels of the computer scientists on the flight went up around a thousand percent :) When we had eventually landed and were waiting for baggage, I caught sight of the pilot. (It was an old airport, and the crew picked up their bags from the same place as the passengers.) The pilot’s hair was plastered to his forhead by sweat, and he was dragging heavily on a cigarette. Luckily the control systems did their job that evening, but it was still a pretty hairy time.↩︎

  57. Wikipedia: in English the most familiar expression of this concept is by Isaac Newton in 1675: “If I have seen further it is by standing on the shoulders of Giants.”↩︎

  58. Aren’t deep learning algorithms (and the neural network models that underlie them) more than just “applied statistics?” Maybe. It is true that neural net architectures are inspired by human brain biology, but also true that in their typical current form they have little biological plausibility: the neurons in our heads are more complex, more analogue, more connected. Our brains are also at the center of hugely complex organisms with vast input ranges and rich feedback loops – and wants, desires, needs… The neural nets that are used for deep learning are good at generalising over a large amount of data, and perhaps they are better at doing that than other more obviously statistical methods. I suspect, however, that the key to the success of their modern applications are the truly enormous datasets that have been created (by human beings) as part of our rush to move all types of information persistence and sharing onto the web. They are learning count-based abstractions over human authored data, rather than modelling the processes by which humans originate that data in the first place.↩︎

  59. Until around half a dozen years ago my day job was computational infrastructure for the extraction of information from human language text, which, depending on which funding agency or research lab was paying the bills, was either down-to-earth software engineering or flight-of-fancy pioneering AI. There is an argument that the crossover space between literary and technical competence is what allows the EU to gain competitive advantage and still avoid falling foul of World Trade Organisation anti-subsidy rules: people like me write the science fiction wrapper that allows states to fund “research” which might otherwise just be “development.” One unfortunate downside of this arrangement is that those who have only the literary (or marketing) competence and no real technical skill or commitment can also sometimes slip in to join the party, with the result that the core pre-requisites of science and engineering (evidence, repeatability, transparency) sometimes become lip-service window dressing, and the corporatisation of university research takes another step towards the inconsequential mirroring of advertising fantasy. But I digress!↩︎

  60. A decade ago I was luck enough to spend a short year working as ANR Chaire d’Excellence at the Internet Memory Foundation in Paris with Julien Masanès and colleagues. They used to run the UK national archives web provision, so for several years if you went to a UK government webpage that no longer existed you would be forwarded to an archival copy served out of a datacenter in Amsterdam managed by French sysadmins, partly paid for by EU high tech research budgets. They worked on early big data infrastructure and analytics using map reduce over HDFS in Hadoop, sucked down from the web by their custom crawler. Ahead of their time!↩︎

  61. Jamie Zawinski’s website is so beautiful! I want one!↩︎

  62. This section contributed by Valentin Radu.↩︎

  63. One of the systems we built was an aquaponic green wall (the Aquamosaic) at Gripple’s Riverside Works factory in Carbrook. The factory floor was covered in expensive machines for making various types of complicated metal objects, and our wall was covered in around a hundred water flow control solenoids (repurposed from washing machine spare parts). To drive water flow to the top of the wall, some 7 meters above floor level, a large high pressure pump was fitted to the pipework. When it worked, it was poetry; but if any of the hundreds of pipes and fittings sprang a leak, the factory got wet, and it quickly became obvious that our hosts, though gracious in the extreme, were smiling a little more fixedly than was their norm. We needed to trap leak events very rapidly, and turn off the pump, so we added a pressure sensor to the plumbing, and a current monitor and 433MHz relay to the mains supply, and wrote some remedial code. Factory dry :)↩︎

  64. Code from Chris Greening and others: thanks folks!↩︎

  65. Many of the project circuits and their descriptions were developed by Gareth Coleman.↩︎

  66. PMU: Power Management Unit.↩︎

  67. RTC: Real Time Clock.↩︎

  68. Image from thesolderblog.blogspot.co.uk.↩︎

  69. It works for me. I live in a country where rain is a way of life.↩︎

  70. These sections on machine learning contributed by Valentin Radu.↩︎

  71. “Grey spaces” are transient, informal and other less easily categorised areas of urban development.↩︎

  72. The virtuous circle of aquaponics.↩︎

  73. Small scale aquaponics: (Somerville et al. 2014).↩︎

  74. Graphic courtesy of I. Karonent, adapted for aquaponics by S. Friend.↩︎

  75. The DripDash cloudside control and analytics suite.↩︎

  76. From Todmorden’s Incredible Edible aquagarden (now sadly closed).↩︎

  77. Image source.↩︎

  78. Choice of Effective Messaging Protocols for IoT Systems: MQTT, CoAP, AMQP and HTTP, N. Naik, IEEE Xplore, 2017.↩︎

  79. Example TTN architecture.↩︎

  80. Writing this in May 2021 the view from my (fortunate, privileged) desk is more optimistic, with the NHS vaccination programme and a long period of lockdown having brought down local covid numbers very positively. Space to live again. If this isn’t true where you are, my sympathies.↩︎

  81. As Wiebers and Feigin state in (Wiebers and Feigin 2020), “…in the midst of all of the pandemonium and destruction, and as we begin to find our way through this crisis, it is imperative for us as a society and species to focus and reflect deeply upon what this and other related human health crises are telling us about our role in these increasingly frequent events and about what we can do to avoid them in the future. Failure to do so may result in the unwitting extermination of all or a good part of our species from this planet. Although it is tempting for us to lay the blame for pandemics such as COVID-19 on bats, pangolins, or other wild species, it is human behavior that is responsible for the vast majority of zoonotic diseases that jump the species barrier from animals to humans.”↩︎

  82. Some of the following previously appeared as articles on my blog.↩︎

  83. The title and the theme echo Cory Doctorow’s Makers; read them both!↩︎

  84. The unfortunate thing, of course, is that it doesn’t do this in service of human need, but as part of the competition for corporate profit – hence our inability to stop the degradation of our environment, or the banker-oriented response to the economic crisis, or the continual wars over oil in the Middle east. This isn’t about bad people, or even bad ideas – it is the central logic of the system that revolves around competition between vast corporations, and everything else is secondary. See Joel Bakan’s The Coporation for a good description of how this works (or doesn’t!).↩︎

  85. As Michael Roberts writes, “A particular negative of the NFT craze is that encoding artwork or an idea onto a blockchain involves complex computations that are highly energy intensive. In six months, a single NFT by one crypto artist consumed electricity equivalent to an EU citizen’s average energy consumption over 77 years. This naturally results in a significant carbon footprint. And this is an issue that applies to blockchain technology more generally.”↩︎

  86. The Pi 4 makes a great P2P secure hosting and home gateway when installed with FreedomBox (and is now capable of being a decent desktop replacement, fact).↩︎