Optical Pairing: Streaming K-Means for Zero-Shot IoT Communication

IoT devices need to communicate with smartphones for initial setup, but traditional methods have significant limitations. Bluetooth pairing requires users to navigate confusing menus and enter codes. QR codes fail in bright sunlight or when printed on reflective surfaces. NFC requires close proximity and specific hardware. What if devices could simply flash their screen to securely transmit data to any smartphone camera, working reliably outdoors and at distances up to 25 feet?

View Patent (US 10,116,394 B2)

The Solution

I developed a system that encodes data as sequences of colored light pulses. The transmitting device (an IoT device with an LED or screen) shows sequences of colors, with the delta between colors in the color space representing different bits of information. The receiving device (a smartphone) uses its camera to detect these color sequences from within the complete scene and decode them back into data.

The challenge wasn't just sending colored light - it was making it work reliably in the real world. Varying lighting conditions from outdoor sunlight to dark interiors with other point sources dramatically change how colors appear to a digital camera. Different phone cameras see colors differently. The devices might not be perfectly aligned. Traditional color-based vocabulary and detection is unreliable in these conditions.

The Innovation: Streaming K-Means in Color Space

The key innovation was using a streaming k-means clustering algorithm to decode symbols encoded as modulation deltas in color space. Instead of looking for absolute "red" or "blue" frequencies, the system clusters the captured color data and identifies symbols based on the transitions between clusters.

Each IoT device has a different set of LEDs with unique spectral characteristics. Each smartphone camera captures colors differently, also varying with lighting conditions. The streaming k-means algorithm adapts to these variations in real-time - as it receives color data from the camera, it continuously updates its clusters to match the specific combination of this transmitter's LEDs and this receiver's camera response.

The symbols themselves are encoded in the modulation deltas - the changes in color space as the transmitter switches between different LED combinations. The receiver doesn't need to know the absolute colors; it just needs to detect the pattern of color shifts from one cluster to another.

The clock signal recovery is embedded in the modulation itself. The receiver extracts timing information from the transitions between symbols, eliminating the need for a separate clock synchronization channel. This self-clocking property makes the system robust to varying frame rates and processing delays in the camera pipeline.

How It Works

The transmitting device encodes data symbols as specific rotations within the color space, via sequences of varying hues. As the LEDs modulate through different states, they trace a path through color space that the receiver captures via camera frames.

The receiver runs a streaming k-means algorithm on the incoming color data. As camera frames arrive, the algorithm:

1. Extracts color values from the region of interest (the transmitting device)
2. Updates cluster centroids based on the new observations
3. Identifies transitions between clusters as symbol boundaries
4. Recovers timing from the pattern of transitions
5. Decodes the symbol sequence into the transmitted data

The beauty of this approach is its adaptability. The varying hues enable detection of the light source. The receiving device's captured colors are a linear combination of the transmitter's LED spectra and intensities, filtered by the transmission medium (air, glass, etc.), plus ambient light. But the streaming k-means doesn't care about absolute values - it clusters whatever hues it sees, and detects the modulation patterns within that color space.

This approach handles real-world variations that would break simpler systems:

• Bright sunlight: The clusters adapt to include the ambient light offset and white balance
• Different cameras: Each device sees different absolute colors, but the same relative transitions
• Varying LED characteristics: The algorithm clusters whatever color combinations the specific LEDs produce
• Timing variations: The self-clocking modulation recovers timing from the data stream itself
• Orientation: Tracking enables independence from the angle of the device and the position within the frame
• Distance: Validated working up to approximately 25 feet from a smart plug with an LED button

Applications

The system enables several use cases for IoT devices:

• Device Discovery: Identify nearby devices without pre-pairing
• Secure Key Exchange: Transmit encryption keys visible only to devices in line-of-sight
• Authentication: Verify device identity through optical signatures
• Status Reporting: Display diagnostic information readable by a phone app

Implementation

I implemented this system in C for the embedded IoT device side (running on ThreadX RTOS with limited processing power) and Swift + OpenGL for the iPhone camera processing. The machine learning components needed to run efficiently on mobile GPUs to process camera frames in real-time.

An interesting design choice: we set the data rate much slower than the system could support. The algorithm could have transmitted data much faster, but we deliberately slowed it down because the sequence of changing colors looked beautiful. Users watching their phone pair with the device saw a smooth, pleasant light show rather than rapid flickering. Function met aesthetics.

We demonstrated light communication for device pairing at CES 2017 as part of SolPad's solar energy platform, which won recognition as part of AVING News' Best of CES 2017 Top 10. The patent (US 10,116,394 B2) I authored was granted in 2018.

Technical Summary

Rather than trying to calibrate these responses to known absolute values, the streaming k-means approach works directly in the observed color space. The algorithm treats the problem as unsupervised clustering - it doesn't need to know what "red" should look like, it just needs to distinguish between the different LED states based on their positions in the captured color space.

The symbols are encoded in the modulation deltas - the vectors between cluster centroids as the LEDs transition from one state to another. The receiver identifies these transitions and extracts both the data (which transitions occurred) and the timing (when they occurred) from the same signal. This self-clocking, adaptive approach was the key to making optical communication practical for real-world IoT applications across diverse hardware and environmental conditions.