What it does

You talk to Cleo and it answers, then acts. A USB microphone picks up the instruction, Whisper turns it into text on the Pi itself, and a camera frame goes through an object detector at the same time. A language model takes both and returns the next action as structured output, which the gait code turns into movement across eight servos.

Everything in that loop runs on the robot. The speech recognition is offline and needs no API key, and the object detection runs locally. Unplug the network and Cleo still hears you and still walks.

Seventy-two hours

We started with nothing printed and no code written. What follows is roughly the order it happened in.

← → 01 / 10

Seeing and hearing at the same time

The interesting engineering is in how those two inputs meet. The camera does not tell the model what is in the room in pixels. It runs MobileNet SSD locally, reduces the frame to a list of labels and rough positions, and hands over something like "person, centre; bottle, right". The model gets that alongside whatever you just said, and returns one action.

Keeping the vision output small is what made the whole thing fit on a Pi. A description of the scene is a few dozen characters. An image is a few hundred kilobytes and a round trip we did not have time for.

The face

A small OLED on the front shows a face that changes with what the robot is doing: idle, thinking, searching, happy, surprised. Each state has several variants that cycle at random, so it never looks the same twice in a row.

It sounds like decoration and it is the most useful thing we built. Speech recognition takes a second or two on a Pi, and in that gap a silent robot looks broken. A robot with a thinking face looks like it is thinking. People stopped repeating themselves once the face went in.

How it is put together

Operating modes

Cleo scans for attached sensor modules at boot and starts in whichever mode matches what it finds. You can switch at any time by saying so, because the language model reads the intent rather than matching a fixed phrase. Saying "go into security mode" and "keep an eye on the room" both land in the same place.

• General. No sensor module attached. Conversational.
• Security. Starts when the PIR sensor is present. Patrols and watches for movement.
• Environment. Starts when the DHT11 is attached. Reports temperature and humidity and flags anything unusual.
• Search and rescue. Voice command only. Sweeps an area looking for people.

There is also a manual mode for demos, driven from a keyboard with a live annotated camera window: W to step forward, A and D to turn, space to ask what it currently sees. That mode exists because a hackathon judging table is a bad place to rely on voice recognition.

What I would fix

Whisper takes a second or two on a Pi after you stop talking, which is long enough that people repeat themselves even with the thinking face. MobileNet SSD only knows 20 object classes, so anything outside that list is invisible to the robot. The chat history lives in memory and resets when the mode changes, which means Cleo forgets what you told it the moment you switch. Text to speech needs the internet, so the one part of the loop that is not offline is the part that talks back.

None of these were worth solving inside 72 hours. All four are the first things I would pick up if I went back to it.

Why it is open source

We wrote it as boilerplate rather than a one-off. The servo and display layers can run in a mock mode that prints instead of driving hardware, so someone with a Pi and a webcam and none of the rest of it can still get the voice and vision loop working. The point was that the next person starts from a robot that already stands up.

What it does

Six buttons give you the digits 1 to 6. Enter four of them and the ESP32 compares what you typed against the stored PIN. If it matches, a 3 kHz tone plays and the servo swings the latch from 0° to 90°. If it does not, you get a 2 kHz tone, one of the three status LEDs goes out, and you have two tries left.

Three failures in a row puts the system into a 60 second lockout. The LCD switches to a countdown and ignores every button until the timer runs out. Once you are in, one button relocks the door and another starts a PIN change: confirm the old code, type a new one, done, with no reflashing.

The state machine

This is the part of the project I care about. I drew the diagram first, six states and every transition between them, and wrote the firmware against that drawing rather than the other way round.

It is a Mealy machine, so the outputs depend on the current state and on the button event that caused the transition, not on the state alone. Read any arrow on that diagram and you get both halves: "wrong password ×3" going in, "Timed Lockout" coming out. What the LCD shows, which tone plays, where the servo sits and how many LEDs are lit all fall out of those pairs.

Writing it this way is why the lockout countdown updates smoothly without ever blocking. millis() gets sampled against the lockout start each time through the loop, so nothing sits in a delay() waiting for sixty seconds while the rest of the system goes deaf.

Walking the diagram

Ten frames from the demo, in the order they happen. Every state on that diagram gets visited except the two second splash at power on.

← → 01 / 10

The companion app

The same state machine drives a phone app, so the lock can be operated without reaching for the keypad. It mirrors the hardware exactly rather than working around it: the same four digit entry, the same three attempts, the same sixty second lockout with the same countdown. A second interface that quietly allowed more tries would make the lock on the door a decoration.

Hardware

The poster

The course deliverable included a one page summary, which is a useful compression test: everything that matters about the build, at a size you can read across a room.

Where it falls short

These are the trade-offs I made for a course prototype, and I would rather list them than pretend they are not there.

• The PIN lives in RAM, so it goes back to 1234 on every power cycle. Writing it to NVS would fix that.
• The LCD prints the default PIN at startup. Convenient for a demo, wrong for a lock.
• The comparison runs digit by digit and stops early, so it is not constant time.
• Button release is handled with blocking while loops, which stall the main loop briefly.
• Only six digits are available, so the keyspace is 6⁴, or 1,296 combinations. A patient attacker with no lockout would be through in under an hour, which is exactly what the lockout is for.

The brief

Our client has Usher syndrome, which costs her both vision and hearing. She cooks constantly and can't read a digital kitchen scale. We had six weeks, limited prototyping resources, and a requirement that the whole thing be cheap enough to build from parts anyone can buy.

The idea we dropped first

Our first concept was a walking stick with obstacle detection: a camera, a speaker, high contrast colours, even glow in the dark surfaces. It was the more impressive design and I argued for it. It also needed a wireless link to a server, and our client had told us her hearing aids already use Bluetooth and that extra connections could interfere with them.

That one constraint killed it. Stepping back and looking at her actual day, cooking was the thing she loved and the thing she was losing to her vision, so we aimed at that instead. Choosing the smaller, less impressive idea turned out to be the best decision we made.

How the five of us split it

We divided the work by what each of us was actually good at rather than by slicing the project into equal parts. Some people took 3D modelling and speaker integration, others took physical prototyping, sensor calibration and circuit organisation. I took the button interface end to end.

That only works if people talk, because a mechanical change and an electrical change land on each other constantly. The enclosure had to have room for my breadboard. My button caps had to clear the weighing plate. We checked in often enough that neither of those turned into a rebuild.

Designing it

We used a morph chart to compare options in each part of the design. How would ingredients be held: a bowl, a funnel, a detachable tray? How would weight be measured: a load cell, a pressure sensor, a strain gauge? How would she control it: voice commands, physical buttons, LED indicators? Laying it out that way let us trade complexity against usability instead of guessing.

The design had to weigh without any visual display, speak the result as it changed, give her buttons she could find and press by feel, and survive spills. That last one sounds minor until you remember the thing lives on a kitchen counter.

My part: the button interface

I owned how she actually touches the scale, which meant the circuit, the firmware and the physical buttons.

• The circuit. I wired the buttons to an Arduino on a breadboard and kept the layout compact enough to fit inside the final enclosure without a redesign.
• The firmware. I wrote the first version of the button code covering four functions: increase, decrease, tare and repeat the last reading. By the final prototype the first two had been repurposed into a power button and a unit conversion button, and tare and repeat stayed as they were.
• The caps. The stock tactile switches were tiny and hard to press, which is a problem when the user can't see them. I designed and printed larger yellow caps with different surface textures on each, so she could tell them apart by feel alone.
• Assembly. I tested the full loop from button to Arduino to serial output before we integrated the DFPlayer Mini for audio.

I also presented our reasoning for picking the scale over the walking stick, walked through the testing results, and wrote half of the executive summary and half of the conceptual design section of the report.

Why buttons and not a screen

A few choices shaped everything else. Physical tactile buttons instead of a touchscreen or a companion app, because a flat glass surface gives you nothing to find with your hands. Distinct shapes and textures on each cap so they can be told apart by touch rather than by position. Audio feedback given priority over any visual indicator. And a load cell paired with an Arduino, because it measures reliably and integrates simply, which matters more than sophistication when the thing has to work every day.

Testing it

We tested accuracy by weighing known masses and comparing them against what the scale announced, and durability by running a displacement simulation of the enclosure under load. Percent error stayed low across most of the range, and it fell as the applied weight went up, which is the pattern you want on a kitchen scale where the interesting measurements aren't the tiny ones. The simulation showed minimal deformation under normal use.

The qualitative half was repeated hands-on use. The scale stayed stable, the buttons stayed responsive, and the audio came out consistently across trials.

Six weeks, in order

We ran the project off a Gantt chart with weekly milestones, which mattered more than it sounds like it should. By week seven we had sourced the load cell and amplifier and had a first prototype doing the basic job of turning a weight into a number. The two weeks after that were integration: my button system went in, the audio module went in, and the second prototype became something you could hand to someone.

The last stretch was validation. Each function got tested on its own, then the whole thing assembled, then the pitch and the report. Leaving integration until the end would have been the obvious mistake and we managed to avoid it.

What went wrong, and what fixed it

The audio came out unclear in testing. The files we'd loaded onto the DFPlayer Mini were taking up too much storage and playing badly, and re-encoding them fixed it outright. It was the single change that improved the final product most, and we only caught it because we tested the audio early instead of at assembly.

Where it ended up

The final prototype has four tactile buttons with textured markers, a weighing plate mounted over the load cell, and a case that wipes clean. During the live presentation it measured accurately and spoke every reading without a hitch.

The lesson I took from it: I used to assume the best idea was the most engineered one. Now I think the best idea is the one that fits the person using it, and dropping an idea you like is often the step that gets you there.

What I'd improve

The enclosure should be a more durable, water resistant material. It lives on a kitchen counter and ours would not survive a year of that. The internal wiring could be far more compact, which would help both reliability and anyone trying to repair it. The audio needs work in a noisy kitchen, and the range of spoken units is narrower than it should be.

The brief

Baggage handling at airports is expensive and disorganised, and bags end up in the wrong place. We had to design a system that sorts luggage and moves it to the right destination quickly, with both a physical mechanism and the software driving it.

The constraints were geometric. Nothing could extend past a restriction line between the two plates when the ramp was retracted, every part had to stay inside a defined physical envelope, and the luggage had to arrive undamaged.

What we built

The mechanism pairs a conveyor belt turned by a rotary actuator with a scissor lift ramp extended by a linear actuator. The Python software drives a Q-arm for the precise handling and puts passenger and baggage data behind an interface you can actually read.

The software side

The team wrote three core functions between us. Mine tracked how many passengers on each plane had both "late" status and a layover, reading from a supplied data file. That meant file input and two dimensional lists, neither of which I had used before, and a fair amount of printing things out to find where my indices were wrong.

Debugging that function is what made the Q-arm work. The arm needed a lot of trial and error to position correctly, and by then I had a habit of checking my assumptions one line at a time instead of staring at the whole thing hoping to spot it.

My part

• The ramp. Each of us designed a concept in Inventor and the team combined the strongest pieces. Mine was a linear actuator driving a scissor lift ramp to push luggage horizontally, and it became a core part of the final assembly.
• The software. I wrote the time delay function and the code that moves the Q-arm. Tracking passengers with late and layover status meant reading from a supplied file and building two dimensional lists, both new to me at the time.
• Printing. I took on 3D printing every part the mechanism needed, including reprinting a modified bracket after we found screws catching on the supports.
• Coordinating. As coordinator I called and recorded every meeting held outside our design studio sessions, and kept the logbook that documented them.

Testing it

We tested in two directions. Load simulations confirmed the structure held up under the weights we expected without deforming enough to matter. Then we ran the mechanism repeatedly to check that horizontal and vertical transport both stayed smooth, and that nothing crossed the restriction boundary when the ramp was retracted. Watching it run is how we found the screws catching on the supports, which no simulation would have told us.

The choices that shaped it

Separating horizontal from vertical motion is the decision the whole design rests on. Once the conveyor owned lifting and the ramp owned reaching, both could be tuned for one job and the restriction line stopped being a problem. We picked actuator-driven mechanisms over anything more improvised because they repeat the same motion precisely, and precision was worth more to us than saving weight.

Keeping the team on the same page

As coordinator I called and recorded every meeting that happened outside our design studio sessions, and kept a logbook of all of it: the date, where we met, and what actually got decided. Some of those entries are the project's real history. Meeting at Thode because three of our biggest parts were going to take longer to print than our studio time allowed. Finishing one function in the evening after the other two were done in studio. Booking extra test time before the interview so we could find problems while there was still room to fix them.

I had never kept a record like that before and assumed it was paperwork. It made me pay closer attention, and the log got more useful the more involved I became.

The problem that took longest

My ramp moved luggage horizontally across one plane. It couldn't lift. We needed movement in two dimensions and I had solved one of them, which took the team a while to work around. The answer was to add the conveyor belt for vertical travel and leave the ramp doing what it was good at, pushing bags past the restriction line.

What I took from it

Keeping the logbook was new to me and I expected it to be busywork. It made me pay closer attention to what the team was actually doing, and the record got better because I was more involved. The harder lesson was time management. We ran out of design studio hours for 3D printing and had to find the Thode makerspace instead, and the team code and report both came together in a last minute push. We got there, but the stress was self-inflicted and avoidable.

What I'd improve

The mechanical tolerances are loose enough to cause friction we didn't need, and tightening them is the first thing I'd do. Actuator response time is the next, since a lot of the cycle is spent waiting. On the software side the sorting logic works but isn't quick, and it would take real rewriting rather than tuning. All of that assumes more physical testing than eight weeks allowed.

The brief

Design a filtration system that pulls ciprofloxacin, a pharmaceutical pollutant, out of hospital wastewater. The site was a rural hospital in Byron Bay, Australia, where discharge regulations are strict. The system had to survive being wet all the time, draw very little energy, comply with Australian environmental law, and stay affordable across its whole life rather than just at purchase.

What we were bound by

The constraints did most of the narrowing for us. The system sits in water permanently, so anything that degrades wet was out. It had to draw very little energy, because a rural hospital is not going to run a power hungry filter around the clock. It had to comply with Australian environmental law. And it had to be affordable across its whole life rather than just at purchase, which rules out cheap things that need replacing.

Against that, what we wanted: remove the ciprofloxacin effectively, use sustainable and recyclable materials, and keep the carbon footprint down.

Narrowing the options

We started with an objective tree to separate what we wanted from what we were bound by, then a morph chart to generate options for the filtration mechanism, flow control, degradation and monitoring.

The objective tree is the one I would keep. Writing "filter antibiotic disposal into clean water" at the top and then forcing every branch underneath to be a real sub-goal made it obvious which of our ideas were goals and which were just features we liked.

My research covered three filtration approaches: deep-bed, membrane-based and electro filtration. Comparing them is what pointed the team toward membrane-based systems as the most effective and economical way to catch ciprofloxacin specifically, and it cut several branches off the morph chart.

The material analysis

This was my main technical contribution. We screened materials in Granta EduPack against four criteria: cost, carbon footprint, stiffness and strength. That got us to a shortlist of concrete, cement and coir fibre.

• Deriving the indices. I derived and applied the Material Performance Index equations for our case, working from strength to density ratios and adjusting for cost and carbon footprint so we were ranking on sustainability as well as performance. I ran the equivalent derivation for stiffness.
• Porosity. A filter is full of holes, so the bulk material properties are the wrong numbers to use. I estimated pores per unit volume, calculated pore and fibre volume, and from those worked out the effective Young's modulus and yield strength. Coir fibre held up mechanically at around 10% porosity.
• Checking the answer. I rebuilt the Ashby chart using the porosity-adjusted values to confirm coir fibre still landed in the acceptable region for stiffness against density. It did, which is what let us commit to it.

We ran a weighted decision matrix over the shortlist, scoring each material on cost, environmental friendliness, filtration effectiveness and sustainability. Coir fibre took 19 points against 10 for cement and 9 for concrete, which is not a close result. It's naturally porous, has a low footprint, and is cheap because it's a byproduct of agricultural waste. We checked the design against the Environment Protection and Biodiversity Conservation Act and the Sustainable Procurement Guide to confirm it met national standards.

The eco-audit surprise

Almost all of the energy and carbon in the coir fibre option came from transporting it. Transport accounted for 94.6% of the energy and 93.9% of the CO2, against essentially nothing for use and disposal. That wasn't what we expected and it forced us to re-run the comparison properly rather than assume the natural material won by default. It still came out ahead once we weighed the full picture, and the life cycle analysis confirmed it's recyclable and biodegradable at end of life. If the design were built for real, sourcing the fibre locally is the single change that would matter most.

What I took from it

I went in assuming concrete or cement would win, because coir fibre looks like a handful of string. Reading into recyclability and environmental impact changed my mind, and that changed what I think a good solution means. It was also the first project where government regulation was part of the design rather than a box to tick afterwards. If I did it again I'd read the Australian standards in week one instead of week four.