pctablet505
Work / Computer vision

Recognising many faces at once on edge devices

A multi-face recognition engine at 99.2% accuracy and 30+ FPS on edge devices, with face search cut from 1,200 ms to 87 ms.

  • Computer vision
  • Samsung R&D Institute India
  • Mar 2023 – Aug 2024
  • Shipped
  • 1 min read
99.2%
recognition accuracy on our own test set
○ reported, from my LinkedIn
30+ FPS
frames per second on edge devices
○ reported, from my LinkedIn
1,200 ms → 87 ms
time to search the face index once
○ reported, from my LinkedIn
10,000+
registered users
○ reported, from my LinkedIn
old detector: missed facesnew detector: found themembeddingclose to the same person10,000+ registered usersone by one1,200 ms per lookuptree index87 ms per lookupknownknownknownunknownthe nearest stored face names the person; close to none means unknownphotos are the easy casephoto✓front camera✓back camera✓video✓checked on live camera feeds and recordings, not only on photosAndroidWindows, Linux✓exam proctoring99.2% on our own test set30+ FPS on edge devices

1 of 6Find every face

The old detector missed faces. A different detector finds the faces it missed.

6 steps · 42 s
Illustration drawn by code, not a recording: faces drawn from simple curves are found, turned into numbers and matched through a tree index.

Recognises several faces at once in live camera frames, video and photos, on Android, Windows and Linux, for access control. I replaced a face detector that missed faces, trained a classifier on a custom set of face embeddings, and moved face search from a linear scan to an O(log n) index: 1,200 ms to 87 ms per lookup across 10,000+ registered users, at 99.2% accuracy and 30+ FPS.

Problem

Access control needs every face in the frame recognised while people walk past, on modest edge hardware. The detector in place missed faces, and searching a growing set of registered people one by one got slower with every person added.

Approach

  • Find every face. A different face detector, so the faces the old one missed were found.
  • Know the person. A custom set of face embeddings, and a classifier trained on it.
  • Search in steps, not one by one. Face search moved from a linear scan to an O(log n) index, so a lookup grows with the logarithm of the number of people, not with the number itself.
  • Run anywhere. Android, Windows and Linux, with Python, OpenCV and ONNX.
  • Test on live feeds. Photos are the easy case. Front and back phone cameras and video each behaved differently, so the engine was checked on live camera feeds and recordings, not only on photos.
  • One engine, many apps. A desktop app finds every registered face in photos and video, and an Android app does it on the live camera. The exam-proctoring platform is one app built on it. I was the engine's only researcher and developer.

Results

99.2% accuracy at 30+ FPS on edge devices, and 1,200 ms to 87 ms per lookup across 10,000+ registered users.

More in computer vision

  • Reading damaged barcodes on phonesJul 2022 – Aug 2024

    Read 90%+ of damaged barcodes correctly on its damaged test set, where a commercial scanning SDK read 70%, with models trained on 10M+ synthetic images. · Samsung R&D Institute India · Built

  • Generative inpainting on a phone's NPU and DSPSep 2024 – Jan 2025

    Redesigned a generative inpainting feature, object removal and image completion, so its models fit NPU and DSP latency budgets for the Snapdragon Summit demo. · Qualcomm · Demo

  • Telling attention from distraction, with emotion recognitionSep 2023 – Nov 2023

    Detects whether a person at a screen is attentive or distracted, at 87% accuracy; checks across frames cut false alarms by 40%. · Samsung R&D Institute India · Shipped

  • An AI exam-proctoring platform (Satyapan)Jul 2023 – Oct 2023

    An anti-cheat exam platform used for 2,000+ employees across 100+ tests, with 60% less administrative overhead. · Samsung R&D Institute India · Shipped