How It Works

Desk Buddy is a full ecosystem — hardware, firmware, cloud, and AI — all connected over MQTT.

the ecosystem

The Desk Buddy Ecosystem

Every piece of the system communicates through a single MQTT broker. When you send a command from Desk Buddy Web, it travels as a JSON message to the ESP32 on your desk. Requests that need intelligence — object detection, motion planning, language — are routed to the Desk Buddy AI server.

01 / CLOUD ENCRYPTED MESH
WEB.CLIENT

Desk Buddy Web

Browser control panel

FREE + PRO
MQTT / JSON COMMANDS ⇄ STATUS
ROUTER.00

MQTT Broker

Dedicated account / user

ROUTING
02 / HARDWARE + INTELLIGENCE EDGE NODES
EDGE.ESP32-S3

Physical Desk Buddy

Servos · camera · firmware

ARMED
REAL-TIME 21ms LATENCY
CLOUD.AI-01

Desk Buddy AI

Language · vision · planning

STANDBY
// mqtt api

Controlled by JSON over MQTT

Every action is a JSON message published to your device's command topic. The firmware replies with in_progress then completed (or failed). Every message includes a unique action_id so you can correlate requests and responses.

Topics

CMD esp32_5/test Send commands to the arm
HB esp32_5/HEARTBEAT Live servo angles & telemetry

Available Actions

  • baseRotate — spin the 360° continuous base
  • servo — move elbow, wrist, or twist
  • gripper — grab, drop, soft hold, or set angle
  • controlik — inverse kinematics by distance
  • perch — move to stored rest position
  • detect_object — vision: find objects by phrase
  • detect_color — vision: identify dominant colors
  • photo — capture image from onboard camera
  • calibrate — save hover & perch calibrations
  • ota_update — remote firmware update over Wi-Fi

Example — Move the Elbow

// command
{
  "action":    "servo",
  "action_id": "20",
  "servoName": "ELBOW",
  "position":  135,
  "speed":     10,
  "sender":    "ai_server"
}

Example — Base Rotate

// command
{
  "action":      "baseRotate",
  "action_id":   "69",
  "controlType": "MAGNET",
  "direction":   "RIGHT",
  "speed":       "slow",
  "value":       3,
  "sender":      "ai_server"
}
Full MQTT API Reference →
// hardware

Hardware Details

Desk Buddy is built entirely from off-the-shelf components. No custom PCBs, no proprietary parts — everything can be sourced independently.

ESP32-S3 Cam

The brains of Desk Buddy. Handles Wi-Fi, MQTT, servo control, and onboard camera capture — all in one compact module.

3 Servo Joints

Standard hobby servos for elbow, wrist, and twist. Each moves 0–180° with configurable speed.

360° Continuous Base

A continuous rotation servo drives the base. Controlled by magnet count, encoder ticks, or by homing to one of 12 origin magnets.

Gripper

Servo-driven gripper with GRAB, DROP, and SOFTHOLD presets, plus raw angle positioning.

Onboard Camera

The S3 Cam module captures images on demand for object detection, color detection, and AI vision tasks.

Common Power & Frame

Frame, brackets, and power supply are all generic and available from standard hobby and electronics suppliers worldwide.

4
Degrees of Freedom

Base rotation + elbow + wrist + twist — plus a gripper end effector.

Full Hardware List →
// mqtt architecture

How MQTT Works

Instead of HTTP's request/response model, MQTT uses a broker as a central hub. Clients publish messages to a topic and subscribers receive them instantly — no polling, no overhead. For Desk Buddy, the web app publishes a JSON command, the broker delivers it to your ESP32, and the ESP32 publishes its reply back — all in under a second.

Your Mosquitto Broker

Hosted for you

No broker to install. Just point your ESP32 at the host and use your credentials.

TLS encrypted

All traffic is encrypted in transit. Your commands and telemetry stay private.

Command flow

1
Publish JSON command

From the web app, workflow, or your own MQTT client.

2
Broker routes it

Mosquitto delivers the message to your ESP32 subscriber.

3
ESP32 executes & replies

status: "in_progress" then status: "completed", matched via action_id.

Connect your own client

// python — paho-mqtt
import paho.mqtt.client as mqtt
import json

client = mqtt.Client()
client.username_pw_set("your_username", "your_password")
client.connect("your_broker_host", 1883)

client.publish("esp32_5/test", json.dumps({
    "action":    "servo",
    "action_id": "my-001",
    "servoName": "ELBOW",
    "position":  90,
    "speed":     10,
    "sender":    "my_script"
}))

Any MQTT client works — same host, same credentials:

  • MQTT Explorer — GUI desktop client, great for testing
  • Node-RED — visual flow editor, connect to other services
  • Home Assistant — add Desk Buddy as a smart home device
  • paho-mqtt — Python library for scripts and automation
  • MQTT.js — JavaScript/Node client for web integrations
// desk_buddy_ai

The AI Server

Desk Buddy AI is our own server that brings intelligence to the arm. It combines models we built ourselves with best-in-class open-source AI.

Linear Actuation Models

Homegrown models trained specifically for Desk Buddy's geometry. Given a target position, they output the servo angles needed to get there — no generic IK solver required.

Visual Detection

Open-source vision models identify objects and colors in images captured by the onboard camera. Results feed back to the firmware as pick coordinates.

LLM Integration

An open-source LLM lets you describe tasks in plain language. The AI translates your instructions into a sequence of MQTT actions sent to the arm.

All via MQTT

The AI server communicates with the arm the same way everything else does — JSON over MQTT. No special integration required on the firmware side.