教程:VENTUNO Q/Real-Time Eye Gaze
来自WindwizPi wiki
在 VENTUNO Q 上实时跟踪视线
概述 本教程在 Arduino® VENTUNO™ Q 上跑一个实时视线跟踪演示，模型是 Qualcomm® AI Hub 上的 eyegaze。演示从 USB 相机画面里找到人脸，勾出双眼的眼睑和虹膜，并画一条箭头，表示视线方向。用到的三个模型都在开发板 Qualcomm® Dragonwing™ QCS8275 的 Qualcomm® Hexagon™ NPU 上运行。
EyeGaze 模型只接受一只眼睛的小块裁剪，不能直接对着整幅相机画面。演示因此把三个模型串起来：
| 阶段 | 模型 | 运行时 | 作用 |
|---|---|---|---|
| 1 | mediapipe_face 人脸检测
|
LiteRT (TFLite) | 在画面里找到脸 |
| 2 | mediapipe_face 人脸关键点
|
LiteRT (TFLite) | 定位双眼的眼角 |
| 3 | eyegaze
|
ONNX Runtime | 每只眼睛的眼部关键点和视线方向 |
本指南包括：
- 给 VENTUNO Q 供电并进入 shell。
- 配置带两套 NPU 运行时的 Python 虚拟环境。
- 直接在开发板上下载模型文件。
- 创建两个脚本并运行演示。
- 为什么 EyeGaze 要先改写，才能在 NPU 上跑得快。
说明：
本教程从空目录开始，不必先做其他 AI Hub 指南。如果已经做过 VENTUNO Q 上的 ONNX Runtime 或 实时人脸网格，环境大部分已经就绪，后面有些步骤会眼熟。
硬件与软件要求
硬件
- Arduino® VENTUNO™ Q
- Arduino® USB-C Power Supply (65W)，或接在电源插座上的 7–24 V 直流电源
- 接到 USB-A 口的 USB 相机
- 接到开发板的显示器、键盘和鼠标（脚本会在屏幕上打开实时窗口）
说明：
把 VENTUNO Q 当 SBC 用时，鼠标不是必须的（有则会方便一些）。
软件
- （可选）主机上可用的
adb（Android® Platform Tools）或ssh - Python 3.12（VENTUNO Q 已预装）
- 开发板上数 GB 空闲存储，用来放 Python 软件包
本教程可以在 SBC 模式下完成。adb 和 ssh 只是用你自己的电脑远程连上开发板的可选工具。
警告：
先用电源给 VENTUNO Q 供电，然后再用 USB-C® 线连到主机，否则开发板可能死机。建议电源至少 65 W，电压在 7–24 V。
访问开发板 Shell
做完本教程需要进入开发板的 shell。可以用：
adb（USB 接到开发板的 USB-C® 口）ssh（同一局域网）- 把开发板当作 SBC，接上显示器、键盘和鼠标
本教程里的示例要求开发板以 SBC 方式运行，并接上显示器和键盘。adb 和 ssh 只方便你在自己的电脑上完成安装、改脚本；示例本身必须在 SBC 模式下跑。
Shell 访问（SBC 模式）
在 SBC 模式下进入 shell：给开发板上电，接上显示器、键盘，鼠标可选。登录后，在 Ubuntu 桌面底栏点 >_ 图标打开终端。
把开发板设成 SBC 的步骤见 SBC模式。
Shell 访问（ADB 或 SSH）
用 adb 时，用 USB-C® 线连接 VENTUNO Q 和电脑，然后运行：
# Using adb (Android Debug Bridge)
adb shell
用 ssh 时，先让 VENTUNO Q 和电脑在同一网络，然后运行：
# Using ssh (Secure Shell)
ssh arduino@<ip-address>
如果不知道开发板 IP，接上键盘和显示器，在开发板上运行 hostname -I；也可以先用 sudo nmtui 配好 Wi-Fi®。
说明：
更多远程访问方式见 Remote Access 教程。
配置 Python 环境
这一节的命令都在 VENTUNO Q 上执行。
1. 建立项目目录和虚拟环境
先建一个空的工作目录，再在里面创建并激活 Python 虚拟环境：
mkdir -p /home/arduino/eye-gaze
cd /home/arduino/eye-gaze
python3 -m venv .venv
source .venv/bin/activate
提示符前面出现 (.venv) 就表示环境已激活。每次新开一个会话，都要再执行一次 source。
2. 安装 AI Hub Models 软件包
qai-hub-models 提供下载模型用的命令行工具，也会带上演示需要的 OpenCV、NumPy 和 ONNX：
pip install qai-hub-models
说明：
这一步会装一大批库，需要等一段时间。
3. 安装带 QNN 支持的 ONNX Runtime
qai-hub-models 会装上只有 CPU 的 onnxruntime，它和带 NPU 的构建冲突（两者都提供名为 onnxruntime 的模块）。先卸掉它，再把 onnx 固定到 QNN wheel 要求的版本：
pip uninstall -y onnxruntime
pip install onnx==1.18.0
然后下载并安装 ONNX Runtime QNN wheel。这个构建带有 QNNExecutionProvider，推理会走到 NPU：
wget https://cdn.edgeimpulse.com/qc-ai-docs/wheels/onnxruntime_qnn-1.23.0-cp312-cp312-linux_aarch64.whl
pip install onnxruntime_qnn-*-linux_aarch64.whl
4. 把 LiteRT 固定到指定版本
两个人脸模型走 LiteRT（ai-edge-litert）。qai-hub-models 装的是 2.x，不能配合 NPU delegate，所以要装回 1.3.0：
pip install ai-edge-litert==1.3.0 opencv-python numpy
说明：
pip 会打出一行以 ERROR: 开头的提示，说 qai-hub-models 需要更新的 ai-edge-litert。降级仍然会成功，之后 qai-hub-models fetch 也能用，这条提示可以忽略。固定这个版本的原因见 实时人脸网格 教程。
确认版本正确：
pip list | grep -i -E "^onnxruntime|^ai-edge-litert|^onnx "
输出里应有 ai-edge-litert 1.3.0、onnx 1.18.0、onnxruntime-qnn 1.23.0，且没有单独的 onnxruntime。
5. 安装 Qualcomm AI Runtime 库
NPU 执行需要 Qualcomm® AI Runtime (QAIRT) 库，任何 pip 包都不会把它带进来。从开发板的 apt 仓库安装：
sudo apt update
sudo apt install qairt-libs qairt-dsp-binaries
继续之前，确认演示用到的两个后端都在：
ls /usr/lib/libQnnHtp.so /usr/lib/libQnnTFLiteDelegate.so
下载模型文件
演示需要三份下载。全部放进工作目录里的 models/，可以直接在开发板上获取：
| 文件 | 说明 | 来源 |
|---|---|---|
eyegaze-onnx-w8a16/
|
量化后的 EyeGaze 模型（.onnx 图和 .data 权重）
|
Qualcomm AI Hub |
mediapipe_face-tflite-float/
|
浮点人脸检测和人脸关键点模型 | Qualcomm AI Hub |
anchors_face_back.npy
|
解码检测器输出用的锚点表 | MediaPipePyTorch 参考实现 |
1. 从 AI Hub 拉取模型
虚拟环境处于激活状态时运行：
cd /home/arduino/eye-gaze
# EyeGaze -> extracts models/eyegaze-onnx-w8a16/
qai-hub-models fetch eyegaze --runtime onnx --precision w8a16 --output-dir models/
# Face detector + face landmarks -> extracts models/mediapipe_face-tflite-float/
qai-hub-models fetch mediapipe_face --runtime tflite --precision float --output-dir models/
说明：
人脸模型故意用 float 精度拉取。NPU 不能直接跑 FP32，但 QNN 后端加载时会把浮点图转成 FP16，所以两个模型仍在 NPU 上跑，大约分别是 1 ms 和 0.4 ms。这样可以避开 实时人脸网格 里说的校准有问题的 w8a8 人脸检测器。不同精度的取舍见 NPU 指南。
2. 下载锚点表
anchors_face_back.npy 没有放在 AI Hub 上。它是 Apache-2.0 的 zmurez/MediaPipePyTorch 参考仓库里的静态 BlazeFace 锚点表，qai-hub-models 自己的 mediapipe_face 后处理也用这个仓库：
wget -P models/ https://raw.githubusercontent.com/zmurez/MediaPipePyTorch/master/anchors_face_back.npy
3. 核对结果
列出已下载的文件：
find models -type f | sort
应正好看到这 7 个文件：
models/anchors_face_back.npy
models/eyegaze-onnx-w8a16/eyegaze.data
models/eyegaze-onnx-w8a16/eyegaze.onnx
models/eyegaze-onnx-w8a16/metadata.json
models/mediapipe_face-tflite-float/face_detector.tflite
models/mediapipe_face-tflite-float/face_landmark_detector.tflite
models/mediapipe_face-tflite-float/metadata.json
说明：
每个 metadata.json 列出该模型输入、输出的形状、dtype，以及量化的 scale 和 zero point。eyegaze.py 开头的常数取自 models/eyegaze-onnx-w8a16/metadata.json。
创建脚本
演示是放在 models/ 旁边的两个 Python 文件：
| 文件 | 作用 |
|---|---|
prepare_model.py
|
改写下好的 EyeGaze 模型，使它整段在 NPU 上跑。第一次启动演示时会自动执行 |
eyegaze.py
|
演示本身 |
在 VENTUNO Q 上进入工作目录，创建第一个文件：
cd /home/arduino/eye-gaze
nano prepare_model.py
把下面 代码示例 里的 prepare_model.py 贴进编辑器。在 nano 里用 Ctrl+X，然后 Y，再 Enter 保存退出。第二个文件同样处理：
nano eyegaze.py
工作目录里这时应有 .venv/、models/、prepare_model.py 和 eyegaze.py（还有下载的 .whl，已经不需要了）。
运行演示
虚拟环境激活后运行：
cd /home/arduino/eye-gaze
source .venv/bin/activate # if not already active
python3 eyegaze.py
第一次运行会多打一行，表示正在转换 EyeGaze 模型，并把结果写到 models/eyegaze_npu.onnx。以后的运行会复用这个文件：
Camera: /dev/video2
/home/arduino/eye-gaze/models/eyegaze-onnx-w8a16/eyegaze.onnx -> /home/arduino/eye-gaze/models/eyegaze_npu.onnx: 48 BatchNormalization -> Conv, 3 Reshape fixed
Models on the NPU. Press q to quit.
会打开一个标题为 "EyeGaze" 的窗口，画面像镜子一样左右镜像。找到脸之后，每只眼睛上会画：
- 浅蓝色的眼睑轮廓，
- 虹膜外一圈 绿色环，
- 从虹膜中心指出视线方向的 红色箭头。
左上角显示帧率、人脸阶段和眼睛阶段的耗时，以及以度为单位的视线角度。右上角两张小的灰度图，就是 EyeGaze 模型实际看到的眼部裁剪，用来最快确认眼睛有没有找对。
允许远程主机在这台显示器上运行程序：
键盘操作
窗口获得焦点时：
| 键 | 作用 |
|---|---|
q 或 Esc
|
退出 |
c
|
校准：先直视相机，再按 c，把这个方向当作零点
|
x
|
清除校准 |
m
|
切换镜像 |
p
|
切换眼部裁剪预览 |
s
|
切换平滑，用来和模型原始输出对比 |
命令行选项
| 选项 | 说明 |
|---|---|
--cpu
|
三个模型都在 CPU 上跑，不用 NPU |
--source N
|
用 /dev/videoN，不用自动选中的第一台 USB 相机
|
--source FILE
|
处理图像或视频文件，不用相机 |
--save PATH
|
把标注后的结果写成图像或视频文件 |
--headless
|
不打开窗口（和 --save 一起用，适合 ssh）
|
--width、--height
|
相机采集分辨率（默认 1280 × 720） |
--no-mirror
|
不镜像 |
--no-smoothing
|
一开始就关掉平滑 |
注意：
VENTUNO Q 板载相机管道占用 /dev/video0 和 /dev/video1。USB 摄像头会占两个节点，只有第一个出画面。脚本因此不假定编号，而是选 uvcvideo 驱动下的第一个节点。如果报告 No USB webcam found，用 lsusb 看相机是否出现，并用下面的命令列出采集设备：
for d in /sys/class/video4linux/video*; do echo "$d -> $(cat $d/name)"; done
用 --source N 传入你的相机节点编号，即可覆盖自动选择。
怎样得到较好的结果
视线估计好不好，主要看眼睛拍得清不清楚。下面三件事比其他因素都重要：
- 坐近一点，并尽量在画面正中。 每只眼睛在画面里至少要有 60 像素宽。1280 × 720 下，脸大约占画面高度的三分之一。模型是靠虹膜在眼睛里的位置读视线的，脸又小又远时，一个像素的误差就已经是好几度。
- 从正面打光。 窗户或灯在侧面时，半边脸过曝、半边脸在阴影里。光线太暗时，多数摄像头会加长曝光，帧率会掉下去（本教程所用相机上大约 7–15 fps），眼睛也会糊。
- 做校准。 直视镜头，按
c。这样去掉由脸型、以及相机相对屏幕位置带来的固定偏差。
性能
在运行 Ubuntu 24.04 的 VENTUNO Q 上，对 1280 × 720 的一帧测得：
| NPU (burst) | CPU（--cpu）
| |
|---|---|---|
| 人脸阶段（检测器加两次关键点） | 6.2 ms | 121 ms |
| 眼睛阶段（双眼各跑一次 EyeGaze） | 8.2 ms | 179 ms |
| 每帧合计 | ~15 ms | ~316 ms |
| 单次 EyeGaze 推理 | 1.5 ms | 85 ms |
在 NPU 上整条管道大约每帧 15 ms，所以你看到的帧率由相机决定，不是由开发板决定。同一条管道在 CPU 上大约每秒三帧。
代码示例
两个脚本全文如下。保存到工作目录，和 models/ 放在一起。
prepare_model.py
脚本也可以从这里下载。
#!/usr/bin/env python3
"""Rewrite AI Hub's EyeGaze w8a16 ONNX so that it runs entirely on the NPU.
onnxruntime-qnn 1.23 rejects two patterns in the published graph. Each rejected
node runs on the CPU instead, which cuts the model into 36 NPU pieces with a CPU
round trip between them: ~20 ms per inference instead of ~1.5 ms.
* BatchNormalization with per-channel quantized parameters
("QNN BatchNorm doesn't support dynamic scale"), 48 of them. Each becomes the
equivalent 1x1 Conv, quantized like the graph's other Convs (per-channel int8
weight, int32 bias). The weight is a diagonal matrix: a depthwise Conv would
be the obvious choice, but the HTP backend runs those ~100x slower here.
* Reshape with allowzero=1 ("QNN Reshape doesn't support dynamic shape"). No
target shape contains a zero, so the attribute is simply cleared.
The result matches the original to within 0.06 heatmap pixels / 0.04 degrees.
eyegaze.py runs this automatically when models/eyegaze_npu.onnx is missing.
Usage: python prepare_model.py [SOURCE.onnx] [OUTPUT.onnx]
"""
import sys
from pathlib import Path
import numpy as np
import onnx
from onnx import helper, numpy_helper
MODELS = Path(__file__).resolve().parent / "models"
SOURCE = MODELS / "eyegaze-onnx-w8a16" / "eyegaze.onnx"
OUTPUT = MODELS / "eyegaze_npu.onnx"
def convert(source=SOURCE, output=OUTPUT):
model = onnx.load(str(source))
graph = model.graph
initializers = {t.name: t for t in graph.initializer}
producer = {name: node for node in graph.node for name in node.output}
def constant(name):
return numpy_helper.to_array(initializers[name])
def dequantized(name):
"""Float value of a BN parameter: what its (Quantize ->) Dequantize chain yields."""
dq = producer[name]
assert dq.op_type == "DequantizeLinear", dq.op_type
if dq.input[0] in initializers:
q = constant(dq.input[0]).astype(np.float64)
else:
quantize = producer[dq.input[0]]
assert quantize.op_type == "QuantizeLinear", quantize.op_type
zero_point = constant(quantize.input[2])
limits = np.iinfo(zero_point.dtype)
q = np.rint(constant(quantize.input[0]).astype(np.float64) / constant(quantize.input[1]))
q = np.clip(q + zero_point, limits.min, limits.max)
return (q - constant(dq.input[2]).astype(np.float64)) * constant(dq.input[1]).astype(np.float64)
nodes, new_initializers = [], []
batch_norms = reshapes = 0
for node in graph.node:
if node.op_type == "Reshape":
for attribute in node.attribute:
if attribute.name == "allowzero" and attribute.i:
assert 0 not in constant(node.input[1])
attribute.i = 0
reshapes += 1
if node.op_type != "BatchNormalization":
nodes.append(node)
continue
x, *parameters = node.input
gamma, beta, mean, variance = (dequantized(p) for p in parameters)
epsilon = next(a.f for a in node.attribute if a.name == "epsilon")
# gamma * (x - mean) / sqrt(variance + epsilon) + beta == w * x + b
w = gamma / np.sqrt(variance + epsilon)
b = beta - mean * w
# Each output channel has a single non-zero weight, so per-channel int8
# represents it exactly (as +-127).
x_scale = float(constant(producer[x].input[1]))
w_scale = np.maximum(np.abs(w), 1e-12) / 127.0
b_scale = x_scale * w_scale
b_q = np.rint(b / b_scale)
assert np.abs(b_q).max() < 2**31 - 1, f"{node.name}: bias does not fit int32"
name, channels = node.name, len(w)
new_initializers += [
numpy_helper.from_array(np.diag(w).astype(np.float32).reshape(channels, channels, 1, 1),
f"{name}_w"),
numpy_helper.from_array(w_scale.astype(np.float32), f"{name}_w_scale"),
numpy_helper.from_array(np.zeros(channels, np.int8), f"{name}_w_zero"),
numpy_helper.from_array(b_q.astype(np.int32), f"{name}_b_q"),
numpy_helper.from_array(b_scale.astype(np.float32), f"{name}_b_scale"),
numpy_helper.from_array(np.zeros(channels, np.int32), f"{name}_b_zero"),
]
nodes += [
helper.make_node("QuantizeLinear", [f"{name}_w", f"{name}_w_scale", f"{name}_w_zero"],
[f"{name}_w_q"], name=f"{name}_w_quantize", axis=0),
helper.make_node("DequantizeLinear", [f"{name}_w_q", f"{name}_w_scale", f"{name}_w_zero"],
[f"{name}_w_dq"], name=f"{name}_w_dequantize", axis=0),
helper.make_node("DequantizeLinear", [f"{name}_b_q", f"{name}_b_scale", f"{name}_b_zero"],
[f"{name}_b_dq"], name=f"{name}_b_dequantize", axis=0),
helper.make_node("Conv", [x, f"{name}_w_dq", f"{name}_b_dq"], list(node.output),
name=f"{name}_as_conv", kernel_shape=[1, 1]),
]
batch_norms += 1
# Drop the BN parameters' now-unused Quantize/Dequantize chains.
outputs = {o.name for o in graph.output}
while True:
used = {name for node in nodes for name in node.input} | outputs
kept = [node for node in nodes if any(name in used for name in node.output)]
if len(kept) == len(nodes):
break
nodes = kept
kept_initializers = [t for t in graph.initializer if t.name in used] + new_initializers
del graph.node[:], graph.initializer[:], graph.value_info[:]
graph.node.extend(nodes)
graph.initializer.extend(kept_initializers)
onnx.checker.check_model(model)
onnx.save(model, str(output)) # one file, weights embedded
print(f"{source} -> {output}: {batch_norms} BatchNormalization -> Conv, "
f"{reshapes} Reshape fixed")
if __name__ == "__main__":
convert(*sys.argv[1:3])
eyegaze.py
脚本也可以从这里下载。
#!/usr/bin/env python3
"""Real-time eye gaze estimation on the Arduino VENTUNO Q (Qualcomm AI Hub EyeGaze).
Per frame:
camera frame
-> MediaPipe face detector
-> MediaPipe face landmarks -> the four eye corners
-> one level 160x96 crop per eye -> EyeGaze (EyeNet), once per eye
-> 34 eye landmarks per eye + one gaze direction from both, smoothed and drawn
Usage:
python eyegaze.py # first USB webcam, models on the NPU
python eyegaze.py --cpu # same, on the CPU (slow)
python eyegaze.py --source 2 # /dev/video2
python eyegaze.py --source clip.mp4 --save out.mp4 --headless
Keys: q/Esc quit, c calibrate (look into the camera first), x clear calibration,
m mirror, p eye previews, s smoothing.
"""
import argparse
import math
import os
import threading
import time
from pathlib import Path
import cv2
import numpy as np
import onnxruntime as ort
from ai_edge_litert.interpreter import Interpreter, load_delegate
HERE = Path(__file__).resolve().parent
MODELS = HERE / "models"
GAZE_SOURCE_MODEL = MODELS / "eyegaze-onnx-w8a16" / "eyegaze.onnx" # as downloaded from AI Hub
GAZE_MODEL = MODELS / "eyegaze_npu.onnx" # made by prepare_model.py
FACE_DETECTOR = MODELS / "mediapipe_face-tflite-float" / "face_detector.tflite"
FACE_LANDMARKS = MODELS / "mediapipe_face-tflite-float" / "face_landmark_detector.tflite"
FACE_ANCHORS = MODELS / "anchors_face_back.npy"
# --- EyeGaze model I/O (models/eyegaze-onnx-w8a16/metadata.json) ---
EYE_W, EYE_H = 160, 96
INPUT_MAX = 65535.0 # uint16 input, scale 1/65535: 0..65535 <-> 0..1
HEATMAP_SCALE, HEATMAP_ZERO = 0.000017086889783968218, 11910
GAZE_SCALE, GAZE_ZERO = 0.000015449146303581074, 38104
HEATMAP_TO_CROP = 2.0 # heatmaps are 80x48, half the crop resolution
SOFTARGMAX_BETA = 100.0 # same sharpness the model uses internally
# The 34 landmarks: eyelid outline, iris outline, iris centre, eyeball centre.
EYELID, IRIS, IRIS_CENTRE = slice(0, 16), slice(16, 32), 32
# --- Eye crops ---
# MediaPipe face-mesh indices. "right"/"left" are the subject's own right and left;
# corner pairs are ordered left-to-right in the (unmirrored) image.
RIGHT_EYE_CORNERS = (33, 133)
LEFT_EYE_CORNERS = (362, 263)
RIGHT_EYE_CONTOUR = (33, 7, 163, 144, 145, 153, 154, 155, 133, 173, 157, 158, 159, 160, 161, 246)
LEFT_EYE_CONTOUR = (362, 382, 381, 380, 374, 373, 390, 249, 263, 466, 388, 387, 386, 385, 384, 398)
EYE_CROP_WIDTH = 1.5 # crop width, in corner-to-corner distances
# Mean iris heatmap peak: ~0.5 for an open eye, <0.1 when closed or not an eye.
# An eye appears above the first value and stays until it drops below the second.
EYE_CONFIDENCE_ON, EYE_CONFIDENCE_OFF = 0.2, 0.12
# --- Face tracking ---
DETECT_THRESHOLD = 0.6
DETECT_ROI_SCALE = 1.5 # detector box -> landmark crop
TRACK_ROI_SCALE = 1.5 # previous landmarks' bounding box -> landmark crop
LANDMARK_THRESHOLD = 0.5
MESH_PASSES = 2
# --- Smoothing (One Euro filter: cutoff = min_cutoff + beta * speed) ---
CORNER_FILTER = dict(min_cutoff=0.3, beta=10.0) # eye corners, in face sizes
LANDMARK_FILTER = dict(min_cutoff=0.5, beta=0.1) # eye landmarks, in pixels
GAZE_FILTER = dict(min_cutoff=0.3, beta=3.0) # pitch/yaw, in radians
IMBALANCE_RATE = 0.05 # per frame, for the left/right difference estimate
# --- Drawing (BGR) ---
EYELID_COLOR = (255, 220, 120)
IRIS_COLOR = (120, 255, 120)
GAZE_COLOR = (60, 60, 255)
ARROW_LENGTH = 3.0 # in eye widths, for a gaze at 90 degrees
SHIFT = 4 # draw with 1/16 px precision so overlays glide
class OneEuroFilter:
"""Low-pass filter whose cutoff rises with speed: steady at rest, quick to follow motion."""
def __init__(self, min_cutoff, beta, d_cutoff=1.0):
self.min_cutoff, self.beta, self.d_cutoff = min_cutoff, beta, d_cutoff
self.x = self.dx = None
@staticmethod
def _alpha(cutoff, dt):
return 1.0 / (1.0 + 1.0 / (2.0 * math.pi * cutoff * dt))
def __call__(self, x, dt):
x = np.asarray(x, dtype=np.float64)
if self.x is None or dt <= 0:
self.x, self.dx = x, np.zeros_like(x)
return x
self.dx += self._alpha(self.d_cutoff, dt) * ((x - self.x) / dt - self.dx)
speed = np.linalg.norm(self.dx, axis=-1, keepdims=True)
self.x = self.x + self._alpha(self.min_cutoff + self.beta * speed, dt) * (x - self.x)
return self.x
def crop_transform(centre, angle, scale, out_w, out_h):
"""Affine (2x3) taking frame pixels to an upright out_w x out_h crop centred on
`centre`, plus its inverse. `angle` is the tilt, in the frame, of the crop's x axis."""
c, s = math.cos(angle) * scale, math.sin(angle) * scale
forward = np.array([[c, s, 0.0], [-s, c, 0.0]])
forward[:, 2] = np.array([out_w / 2.0, out_h / 2.0]) - forward[:, :2] @ centre
return forward, cv2.invertAffineTransform(forward)
def transform_points(affine, points):
return points @ affine[:, :2].T + affine[:, 2]
def litert_model(path, use_npu):
if use_npu:
# htp_performance_mode 2 = "burst", the fastest Hexagon clock profile.
delegates = [load_delegate("libQnnTFLiteDelegate.so",
options={"backend_type": "htp", "htp_performance_mode": "2"})]
interpreter = Interpreter(model_path=str(path), experimental_delegates=delegates)
else:
interpreter = Interpreter(model_path=str(path), num_threads=4)
interpreter.allocate_tensors()
return interpreter
class FaceTracker:
"""Finds one face and returns its 468 MediaPipe mesh landmarks in frame pixels."""
def __init__(self, use_npu):
self.detector = litert_model(FACE_DETECTOR, use_npu)
self.mesh = litert_model(FACE_LANDMARKS, use_npu)
self.det_in = self.detector.get_input_details()[0]
self.mesh_in = self.mesh.get_input_details()[0]
self.det_size = int(self.det_in["shape"][1])
self.mesh_size = int(self.mesh_in["shape"][1])
# Detector outputs: box coordinates (.., 16) and scores (.., 1) for two anchor
# grids; the larger grid comes first in the anchor table.
outs = sorted(self.detector.get_output_details(), key=lambda d: -int(d["shape"][1]))
self.det_coords = [d["index"] for d in outs if d["shape"][-1] == 16]
self.det_scores = [d["index"] for d in outs if d["shape"][-1] == 1]
outs = self.mesh.get_output_details()
self.mesh_score = next(d["index"] for d in outs if np.prod(d["shape"]) == 1)
self.mesh_points = next(d["index"] for d in outs if np.prod(d["shape"]) > 1)
# (896, 2, 2): [[x_centre, y_centre], [w, h]] per anchor, normalised to 0..1.
self.anchors = np.load(FACE_ANCHORS).astype(np.float32).reshape(-1, 2, 2)
self.roi = None # (centre xy, size, angle) of the landmark crop
def _detect(self, rgb):
h, w = rgb.shape[:2]
scale = self.det_size / max(h, w)
new_w, new_h = round(w * scale), round(h * scale)
pad_x, pad_y = (self.det_size - new_w) // 2, (self.det_size - new_h) // 2
canvas = np.zeros((self.det_size, self.det_size, 3), np.float32)
canvas[pad_y:pad_y + new_h, pad_x:pad_x + new_w] = cv2.resize(rgb, (new_w, new_h)) / 255.0
self.detector.set_tensor(self.det_in["index"], canvas[None])
self.detector.invoke()
scores = np.concatenate([self.detector.get_tensor(i).reshape(-1) for i in self.det_scores])
best = int(np.argmax(scores))
if scores[best] < math.log(DETECT_THRESHOLD / (1.0 - DETECT_THRESHOLD)): # logit
return None
coords = np.concatenate([self.detector.get_tensor(i).reshape(-1, 16) for i in self.det_coords])
# Anchor-relative [box centre, box size, right eye, left eye, nose, mouth, ears].
offset, size = self.anchors[best]
points = coords[best].reshape(8, 2) * size
points[[0, 2, 3]] += offset * self.det_size
points[[0, 2, 3]] = (points[[0, 2, 3]] - (pad_x, pad_y)) / scale
box_size = points[1].max() / scale
eye_a, eye_b = points[2], points[3]
angle = math.atan2(eye_b[1] - eye_a[1], eye_b[0] - eye_a[0])
return points[0].astype(np.float64), box_size * DETECT_ROI_SCALE, angle
def _landmarks(self, rgb):
"""Runs the landmark model on the current ROI and moves the ROI onto the result."""
centre, size, angle = self.roi
forward, inverse = crop_transform(centre, angle, self.mesh_size / size,
self.mesh_size, self.mesh_size)
crop = cv2.warpAffine(rgb, forward, (self.mesh_size, self.mesh_size),
borderMode=cv2.BORDER_REPLICATE)
self.mesh.set_tensor(self.mesh_in["index"], (crop.astype(np.float32) / 255.0)[None])
self.mesh.invoke()
if float(self.mesh.get_tensor(self.mesh_score).reshape(-1)[0]) < LANDMARK_THRESHOLD:
self.roi = None
return None
points = self.mesh.get_tensor(self.mesh_points).reshape(-1, 3)[:, :2] * self.mesh_size
points = transform_points(inverse, points.astype(np.float64))
lo, hi = points.min(axis=0), points.max(axis=0)
eye_a, eye_b = points[RIGHT_EYE_CORNERS[0]], points[LEFT_EYE_CORNERS[1]]
self.roi = ((lo + hi) / 2.0, float((hi - lo).max()) * TRACK_ROI_SCALE,
math.atan2(eye_b[1] - eye_a[1], eye_b[0] - eye_a[0]))
return points
def __call__(self, rgb):
# Detect on every frame (about 1 ms on the NPU). Following the previous frame's
# landmarks alone can slide onto a hand or a beard and stay there; that is only
# the fallback for frames where the detector misses, e.g. a turned head.
self.roi = self._detect(rgb) or self.roi
if self.roi is None:
return None
# Twice: the first pass centres the ROI on the face, the second measures on
# that well-centred crop. The landmarks shift slightly with the face's
# position in the crop otherwise.
for _ in range(MESH_PASSES):
points = self._landmarks(rgb)
if points is None:
return None
return points
class EyeGaze:
"""Qualcomm AI Hub EyeGaze (EyeNet): one eye crop in, eye landmarks and gaze out."""
def __init__(self, use_npu):
if not GAZE_MODEL.exists():
from prepare_model import convert
convert(GAZE_SOURCE_MODEL, GAZE_MODEL)
if use_npu:
providers = [("QNNExecutionProvider",
{"backend_type": "htp", "htp_performance_mode": "burst"})]
else:
providers = ["CPUExecutionProvider"]
self.session = ort.InferenceSession(str(GAZE_MODEL), providers=providers)
self.on_npu = self.session.get_providers()[0] == "QNNExecutionProvider"
self._xs = np.arange(EYE_W // 2, dtype=np.float32)
self._ys = np.arange(EYE_H // 2, dtype=np.float32)
def __call__(self, gray, centre, width, roll, flip):
"""gray: full frame. centre/width/roll: where the eye is, how wide (corner to
corner, pixels) and how tilted. flip: True for the subject's right eye. EyeNet
only knows left eyes, so right eyes are mirrored on the way in and back on
the way out.
Returns (landmarks (34, 2) in frame pixels, [pitch, yaw] in radians relative
to the level crop, confidence, the crop).
"""
forward, inverse = crop_transform(centre, roll, EYE_W / (EYE_CROP_WIDTH * width), EYE_W, EYE_H)
# Replicated border: black beyond the frame edge would wreck the equalisation.
crop = cv2.equalizeHist(cv2.warpAffine(gray, forward, (EYE_W, EYE_H),
borderMode=cv2.BORDER_REPLICATE))
image = crop[:, ::-1] if flip else crop
tensor = (image.astype(np.float32) * (INPUT_MAX / 255.0)).astype(np.uint16)[None]
heatmaps, gaze = self.session.run(["heatmaps", "gaze_pitchyaw"], {"image": tensor})
# Landmarks: soft-argmax over the last hourglass stack's heatmaps. Doing it here
# in float is ~3x closer to the reference than the NPU's own 16-bit softmax.
maps = (heatmaps[0, -1].astype(np.float32) - HEATMAP_ZERO) * HEATMAP_SCALE
peaks = maps.max(axis=(1, 2))
weights = np.exp(SOFTARGMAX_BETA * (maps - peaks[:, None, None]))
weights /= weights.sum(axis=(1, 2), keepdims=True)
points = np.stack([weights.sum(axis=1) @ self._xs, weights.sum(axis=2) @ self._ys],
axis=1).astype(np.float64) * HEATMAP_TO_CROP
pitch_yaw = (gaze[0].astype(np.float64) - GAZE_ZERO) * GAZE_SCALE
if flip:
points[:, 0] = EYE_W - points[:, 0]
pitch_yaw[1] = -pitch_yaw[1]
return transform_points(inverse, points), pitch_yaw, float(peaks[IRIS].mean()), crop
def gaze_vector(pitch, yaw, roll):
"""Unit vector in camera space (x right, y down, z away from the camera) for a
gaze measured in an eye crop that is tilted by `roll` in the frame."""
x, y = -math.cos(pitch) * math.sin(yaw), math.sin(pitch)
c, s = math.cos(roll), math.sin(roll)
return np.array([c * x - s * y, s * x + c * y, -math.cos(pitch) * math.cos(yaw)])
class Eye:
"""One eye's filters and latest result."""
def __init__(self, corners, flip):
self.corners, self.flip = list(corners), flip
self.reset()
def reset(self):
self.landmark_filter = OneEuroFilter(**LANDMARK_FILTER)
self.gaze_filter = OneEuroFilter(**GAZE_FILTER)
self.landmarks = self.pitch_yaw = self.crop = None
self.width = self.roll = 0.0
self.visible = False
class GazePipeline:
def __init__(self, use_npu=True, smoothing=True):
self.tracker = FaceTracker(use_npu)
self.eyegaze = EyeGaze(use_npu)
self.smoothing = smoothing
self.eyes = [Eye(RIGHT_EYE_CORNERS, flip=True), Eye(LEFT_EYE_CORNERS, flip=False)]
self.corner_filter = OneEuroFilter(**CORNER_FILTER)
self.imbalance = np.zeros(2) # running (left - right) / 2 of [pitch, yaw]
self.offset = np.zeros(2) # calibration: [pitch, yaw] read when looking at the camera
self.pitch_yaw = self.gaze = None
self.face_ms = self.gaze_ms = 0.0
def _eye_corners(self, mesh, dt):
"""The four eye corners, steadied. The model reads gaze from where the iris
sits in the crop (about 2.5 degrees per pixel at this scale), so the corners
that place the crop must not jitter, nor lag behind a moving head. Hence they
are filtered relative to the face (its centroid, size and tilt, which average
over many landmarks) rather than in frame coordinates."""
corners = mesh[[c for eye in self.eyes for c in eye.corners]]
if not self.smoothing:
return corners
origin = mesh.mean(axis=0)
size = math.sqrt(((mesh - origin) ** 2).sum(axis=1).mean())
across = mesh[list(LEFT_EYE_CONTOUR)].mean(axis=0) - mesh[list(RIGHT_EYE_CONTOUR)].mean(axis=0)
c, s = across / np.linalg.norm(across)
to_face = np.array([[c, s], [-s, c]])
in_face = self.corner_filter((corners - origin) @ to_face.T / size, dt)
return in_face * size @ to_face + origin
def process(self, frame, dt):
"""Updates self.eyes and self.gaze from a BGR frame taken `dt` seconds after
the previous one."""
t0 = time.perf_counter()
mesh = self.tracker(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))
t1 = time.perf_counter()
self.face_ms = (t1 - t0) * 1000.0
if mesh is None:
for eye in self.eyes:
eye.reset()
self.corner_filter = OneEuroFilter(**CORNER_FILTER)
self.pitch_yaw = self.gaze = None
self.gaze_ms = 0.0
return
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
corners = self._eye_corners(mesh, dt).reshape(2, 2, 2)
for eye, (a, b) in zip(self.eyes, corners):
eye.width = float(np.linalg.norm(b - a))
eye.roll = math.atan2(b[1] - a[1], b[0] - a[0])
landmarks, pitch_yaw, confidence, eye.crop = self.eyegaze(gray, (a + b) / 2.0, eye.width,
eye.roll, eye.flip)
was_visible = eye.visible
eye.visible = confidence >= (EYE_CONFIDENCE_OFF if was_visible else EYE_CONFIDENCE_ON)
if not eye.visible:
continue
if self.smoothing:
if not was_visible: # don't glide in from wherever the eye was before a blink
eye.landmark_filter = OneEuroFilter(**LANDMARK_FILTER)
eye.gaze_filter = OneEuroFilter(**GAZE_FILTER)
landmarks = eye.landmark_filter(landmarks, dt)
pitch_yaw = eye.gaze_filter(pitch_yaw, dt)
eye.landmarks, eye.pitch_yaw = landmarks, pitch_yaw
self._fuse()
self.gaze_ms = (time.perf_counter() - t1) * 1000.0
def _fuse(self):
"""One gaze direction from both eyes.
The right eye goes through the model mirrored, so any constant bias in the
model's yaw (or in where the face mesh puts the eye corners) comes out with
opposite signs in the two eyes: on its own each eye is several degrees off
and the two arrows splay apart. Their mean cancels that. The running half
difference is kept so that one eye alone can be corrected the same way while
the other is hidden."""
right, left = self.eyes
if right.visible and left.visible:
self.imbalance += IMBALANCE_RATE * ((left.pitch_yaw - right.pitch_yaw) / 2.0 - self.imbalance)
self.pitch_yaw, roll = (left.pitch_yaw + right.pitch_yaw) / 2.0, (left.roll + right.roll) / 2.0
elif left.visible:
self.pitch_yaw, roll = left.pitch_yaw - self.imbalance, left.roll
elif right.visible:
self.pitch_yaw, roll = right.pitch_yaw + self.imbalance, right.roll
else:
self.pitch_yaw = self.gaze = None
return
self.gaze = gaze_vector(*(self.pitch_yaw - self.offset), roll)
def calibrate(self):
"""Call while the user looks into the camera: the current reading becomes zero."""
if self.pitch_yaw is not None:
self.offset = self.pitch_yaw.copy()
def clear_calibration(self):
self.offset = np.zeros(2)
def gaze_angles(self):
"""(pitch, yaw) in degrees, or None. Seen from the camera: positive pitch
looks down, positive yaw looks to the image's left."""
if self.gaze is None:
return None
x, y, z = self.gaze
return math.degrees(math.atan2(y, math.hypot(x, z))), math.degrees(math.atan2(-x, -z))
def fixed(points):
return np.round(np.asarray(points) * (1 << SHIFT)).astype(np.int32)
def draw_eye(frame, eye, gaze):
if not eye.visible:
return
thickness = max(1, round(frame.shape[0] / 400))
cv2.polylines(frame, [fixed(eye.landmarks[EYELID])], True, EYELID_COLOR, thickness, cv2.LINE_AA, SHIFT)
cv2.polylines(frame, [fixed(eye.landmarks[IRIS])], True, IRIS_COLOR, thickness, cv2.LINE_AA, SHIFT)
start = eye.landmarks[IRIS_CENTRE]
end = start + gaze[:2] * ARROW_LENGTH * eye.width
cv2.arrowedLine(frame, tuple(fixed(start).tolist()), tuple(fixed(end).tolist()), GAZE_COLOR,
thickness + 1, cv2.LINE_AA, SHIFT, tipLength=0.25)
cv2.circle(frame, tuple(fixed(start).tolist()), (thickness + 1) << SHIFT, GAZE_COLOR, -1,
cv2.LINE_AA, SHIFT)
def draw_text(frame, text, origin, scale=0.6):
cv2.putText(frame, text, origin, cv2.FONT_HERSHEY_SIMPLEX, scale, (0, 0, 0), 4, cv2.LINE_AA)
cv2.putText(frame, text, origin, cv2.FONT_HERSHEY_SIMPLEX, scale, (255, 255, 255), 1, cv2.LINE_AA)
def render(frame, pipeline, fps, mirror, previews):
for eye in pipeline.eyes:
draw_eye(frame, eye, pipeline.gaze)
if mirror:
frame = cv2.flip(frame, 1)
device = "NPU" if pipeline.eyegaze.on_npu else "CPU"
draw_text(frame, f"{fps:4.1f} fps face {pipeline.face_ms:4.1f} ms gaze {pipeline.gaze_ms:4.1f} ms {device}",
(12, 26))
angles = pipeline.gaze_angles()
if angles is not None:
draw_text(frame, f"pitch {angles[0]:+5.1f} yaw {angles[1]:+5.1f} deg", (12, 52))
elif all(eye.crop is None for eye in pipeline.eyes):
draw_text(frame, "no face", (12, 52))
draw_text(frame, "q quit c calibrate (look into the camera) x clear m mirror p previews s smoothing",
(12, frame.shape[0] - 12), 0.45)
if previews:
# What the model sees. Ordered as on screen: the eye drawn on the left goes left.
eyes = pipeline.eyes[::-1] if mirror else pipeline.eyes
for i, eye in enumerate(eyes):
if eye.crop is None:
continue
x = frame.shape[1] - (len(eyes) - i) * (EYE_W + 8)
crop = eye.crop[:, ::-1] if mirror else eye.crop
frame[8:8 + EYE_H, x:x + EYE_W] = cv2.cvtColor(crop, cv2.COLOR_GRAY2BGR)
cv2.rectangle(frame, (x, 8), (x + EYE_W - 1, 8 + EYE_H - 1), (255, 255, 255), 1)
return frame
def find_usb_camera():
"""Index of the first USB (UVC) capture node. On the VENTUNO Q /dev/video0 and
/dev/video1 belong to the onboard camera pipeline and never deliver frames."""
nodes = sorted(Path("/sys/class/video4linux").glob("video*"), key=lambda p: int(p.name[5:]))
for node in nodes:
driver = os.path.basename(os.path.realpath(node / "device" / "driver"))
# A UVC camera exposes two nodes; index 0 is video, index 1 is metadata.
if driver == "uvcvideo" and (node / "index").read_text().strip() == "0":
return int(node.name[5:])
return None
class Camera:
"""Webcam read on a background thread, so the main loop always gets the newest
frame instead of one that has been waiting in the driver's queue."""
def __init__(self, index, width, height):
self.cap = cv2.VideoCapture(index, cv2.CAP_V4L2)
# MJPG: uncompressed YUYV is limited to a few fps above 640x480 over USB 2.
self.cap.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter_fourcc(*"MJPG"))
self.cap.set(cv2.CAP_PROP_FRAME_WIDTH, width)
self.cap.set(cv2.CAP_PROP_FRAME_HEIGHT, height)
self.cap.set(cv2.CAP_PROP_FPS, 30)
self.cap.set(cv2.CAP_PROP_BUFFERSIZE, 1)
if not self.cap.isOpened():
raise SystemExit(f"Could not open /dev/video{index}.")
self.fps = 30.0
self._frame = self._time = None
self._count = self._taken = 0
self._running = True
self._new = threading.Condition()
self._thread = threading.Thread(target=self._reader, daemon=True)
self._thread.start()
def _reader(self):
while self._running:
ok, frame = self.cap.read()
with self._new:
self._frame, self._time = (frame if ok else None), time.monotonic()
self._count += 1
self._new.notify()
if not ok:
return
def read(self):
"""Blocks until there is a frame newer than the last one returned.
Returns (frame, capture time in seconds), or (None, None) if the camera stops."""
with self._new:
if not self._new.wait_for(lambda: self._count != self._taken, timeout=5.0):
return None, None
self._taken = self._count
return self._frame, self._time
def release(self):
self._running = False
self._thread.join(timeout=2.0)
self.cap.release()
class VideoFile:
"""Image or video file, played frame by frame (never dropping any)."""
def __init__(self, path):
self.cap = cv2.VideoCapture(path)
if not self.cap.isOpened():
raise SystemExit(f"Could not open {path}.")
self.fps = self.cap.get(cv2.CAP_PROP_FPS) or 30.0
self.is_image = self.cap.get(cv2.CAP_PROP_FRAME_COUNT) <= 1
self._count = 0
def read(self):
ok, frame = self.cap.read()
self._count += 1
return (frame, self._count / self.fps) if ok else (None, None)
def release(self):
self.cap.release()
def open_source(args):
if args.source is not None and not args.source.isdigit():
return VideoFile(args.source)
index = int(args.source) if args.source is not None else find_usb_camera()
if index is None:
raise SystemExit("No USB webcam found. Plug one in, or pass --source N (see `ls /dev/video*`) "
"or --source FILE.")
print(f"Camera: /dev/video{index}")
return Camera(index, args.width, args.height)
def parse_args():
parser = argparse.ArgumentParser(description="Real-time eye gaze estimation (Qualcomm AI Hub EyeGaze).")
parser.add_argument("--source", help="camera index, or an image/video file (default: first USB webcam)")
parser.add_argument("--cpu", action="store_true", help="run the models on the CPU instead of the NPU")
parser.add_argument("--use-npu", action="store_true", help=argparse.SUPPRESS) # now the default
parser.add_argument("--width", type=int, default=1280, help="camera capture width (default 1280)")
parser.add_argument("--height", type=int, default=720, help="camera capture height (default 720)")
parser.add_argument("--no-mirror", action="store_true", help="don't mirror the view")
parser.add_argument("--no-smoothing", action="store_true", help="show raw model output")
parser.add_argument("--save", metavar="PATH", help="write the annotated output to an image/video file")
parser.add_argument("--headless", action="store_true", help="don't open a window")
return parser.parse_args()
def main():
args = parse_args()
ort.set_default_logger_severity(3)
source = open_source(args)
is_image = getattr(source, "is_image", False)
pipeline = GazePipeline(use_npu=not args.cpu, smoothing=not args.no_smoothing)
print(f"Models on the {'NPU' if pipeline.eyegaze.on_npu else 'CPU'}. Press q to quit.")
# Mirrored like a mirror for a live camera; files are shown as they are.
mirror = isinstance(source, Camera) and not args.no_mirror
previews = True
writer = None
fps = 0.0
last_time = last_loop = None
try:
while True:
frame, now = source.read()
if frame is None:
break
pipeline.process(frame, 0.0 if last_time is None else now - last_time)
last_time = now
loop = time.perf_counter()
if last_loop is not None:
rate = 1.0 / max(loop - last_loop, 1e-6)
fps = rate if fps == 0.0 else 0.9 * fps + 0.1 * rate
last_loop = loop
output = render(frame, pipeline, fps, mirror, previews)
if args.save:
if is_image:
cv2.imwrite(args.save, output)
else:
if writer is None:
writer = cv2.VideoWriter(args.save, cv2.VideoWriter_fourcc(*"mp4v"), source.fps,
(output.shape[1], output.shape[0]))
writer.write(output)
if not args.headless:
cv2.imshow("EyeGaze", output)
key = cv2.waitKey(0 if is_image else 1) & 0xFF
if key in (ord("q"), 27):
break
if key == ord("c"):
pipeline.calibrate()
elif key == ord("x"):
pipeline.clear_calibration()
elif key == ord("m"):
mirror = not mirror
elif key == ord("p"):
previews = not previews
elif key == ord("s"):
pipeline.smoothing = not pipeline.smoothing
except KeyboardInterrupt:
pass
finally:
source.release()
if writer is not None:
writer.release()
cv2.destroyAllWindows()
if __name__ == "__main__":
main()
原理
https://docs.arduino.cc/tutorials/ventuno-q/eye-gaze/
每一帧相机画面按下面的步骤处理：
- 人脸检测（NPU）。 画面加边缩放到 256 × 256，送进人脸检测器。得分最高的锚点对照锚点表，解出人脸框和两个眼睛关键点，由此得到绕脸旋转的正方形感兴趣区域。
- 人脸关键点（NPU）。 裁出这块区域，送进关键点模型，跑两遍。第一遍把区域移到脸上，第二遍在居中的裁剪上测量。468 个关键点里，演示只留四个眼角。
- 眼部裁剪。 每只眼睛切出一块摆正的 160 × 96 灰度图，宽度是两眼角距离的 1.5 倍，并做直方图均衡。模型只在左眼上训练过，所以右眼先镜像再推理，结果再镜像回来。
- EyeGaze（NPU）。 每只眼睛跑一次，返回三路输出：
heatmaps（每个关键点一张 80 × 48 热图）、landmarks和gaze_pitchyaw。演示自己用浮点从热图算出 34 个关键点位置，比 NPU 上 16 位算出来的landmarks更接近 CPU 参考，大约近三倍。 - 融合。 两只眼睛平均成一个视线方向。右眼是镜像后进模型的，模型 yaw 上的固定偏差在两只眼睛上符号相反，所以单看一只眼会偏好几度，两支箭头也会叉开。平均可以把这个偏差消掉。
- 平滑。 眼角、关键点和视线角度各自经过一个 One Euro filter。这是一种低通滤波：数值不动时平滑很重，变化快时平滑很轻。眼角是相对脸来滤波的，不是在画面坐标里滤，所以转头时裁剪不会拖在后面。
34 个关键点的顺序是：眼睑一圈 16 点、虹膜一圈 16 点、虹膜中心、眼球中心。视线角度沿用模型训练数据的约定，从相机看：pitch 为正是往下看，yaw 为正是朝向（未镜像）画面的左侧。
说明：
如果要按视线做动作，而不只是画出来，在主循环里
pipeline.process(...)之后读pipeline.gaze_angles()。它返回融合后的(pitch, yaw)，单位是度；没有眼睛可见时返回None。
为什么要改写模型
把下载的 eyegaze.onnx 直接交给 QNNExecutionProvider 可以跑，但每次推理大约 20 ms，比 AI Hub 在 VENTUNO Q 上给出的大约 1 ms 慢十倍以上。
原因是这份模型是为比开发板上 1.23 QNN wheel 更新的 ONNX Runtime 构建的，这个 wheel 会拒绝图里的两种写法：
- 48 个
BatchNormalization节点，因为参数是按通道量化的。 - 3 个
Reshape节点，因为带了allowzero=1属性。
每个被拒绝的节点都退回 CPU。图被切成 36 段独立的 NPU 分区，段与段之间数据还要回到 CPU。
prepare_model.py 去掉这两种写法，不改变模型的计算结果：
- 每个
BatchNormalization换成数学上等价的 1 × 1Conv，量化方式和图里其他卷积相同。 - 清掉
allowzero属性。目标形状里都没有 0，这样做是安全的。
改写后的模型作为 单独一段 NPU 分区加载，大约 1.5 ms 跑完，视线和原模型相差在 0.04° 以内。
注意：
检查是否退回 CPU： 哪怕只有少数形状运算退回 CPU，sess.get_providers() 也会列出 CPUExecutionProvider，所以它分不出健康模型和这里这种模型。要看模型实际被切成几段，创建会话时打开详细日志。把下面存成 check.py：
import onnxruntime as ort
so = ort.SessionOptions()
so.log_severity_level = 0
ort.InferenceSession("models/eyegaze-onnx-w8a16/eyegaze.onnx", so,
providers=[("QNNExecutionProvider", {"backend_type": "htp"})])
然后运行，并过滤日志里的分区数量和节点被拒绝的原因：
python3 check.py 2>&1 | grep -E "Number of partitions|REASON" | sort | uniq -c
在 NPU 上跑得好的模型会报告一个分区，或很少的几个分区。
已知限制
- 精度受模型限制。 Qualcomm 报告 EyeGaze 在 MPIIGaze 基准上的平均角度误差大约 15°。箭头能看出视线大致方向，但还不够用来判断你在读屏幕上的哪一个词。
- 结果取决于眼部裁剪落在哪里。 模型根据虹膜在裁剪里的位置推断视线。眼睛宽 67 像素时，大约每个像素 2.5°。眼角偏几个像素，结果就会偏十度以上。所以脸小或光线差时，箭头会抖。
- 只跟踪一张脸。 演示跟随画面里得分最高的那张脸。
- 眼镜和很强的侧光会同时降低眼角和眼部裁剪的质量。
总结
本教程在 VENTUNO Q 上配好两套 NPU 运行时，从 Qualcomm AI Hub 下载了三个模型，并运行脚本实时跟踪双眼、估计视线方向，每个模型都在 Qualcomm® Hexagon™ NPU 上加速。你也看到怎样识别一个悄悄退回 CPU 的模型，以及改写两种不支持的图结构后，EyeGaze 从每次推理 20 ms 降到 1.5 ms。接下来可以在视线输出上继续做，例如判断有没有人在看屏幕，或驱动随注意力变化的界面。
