高通手機跑AI系列之——穿衣試裝算法

環境準備

手機

測試手機型號：Redmi K60 Pro

處理器：第二代驍龍8移動--8gen2

運行內存：8.0GB ，LPDDR5X-8400，67.0?GB/s

攝像頭：前置16MP+后置50MP+8MP+2MP

AI算力：NPU 48Tops INT8 && GPU?1536ALU x 2 x 680MHz = 2.089 TFLOPS

提示：任意手機均可以，性能越好的手機速度越快

軟件

APP：AidLux 2.0

系統環境：Ubuntu 20.04.3 LTS

提示：AidLux登錄后代碼運行更流暢，在代碼運行時保持AidLux APP在前臺運行，避免代碼運行過程中被系統回收進程，另外屏幕保持常亮，一般息屏后一段時間，手機系統會進入休眠狀態，如需長駐后臺需要給APP權限。

算法Demo

代碼功能詳解

這段代碼實現了一個基于計算機視覺的實時人臉變化的試衣應用。通過攝像頭捕獲實時視頻流，檢測人臉并提取關鍵點，然后將用戶選擇的目標人臉圖像與實時檢測到的人臉進行變換，最終實現人臉變換效果。整個系統包含人臉檢測、關鍵點定位、人臉變形與融合、用戶界面交互等多個核心模塊。

核心模塊詳細解析

1. 人臉檢測模塊

# 人臉檢測預處理函數
def preprocess_img_pad(img, image_size=128):# 圖像填充與預處理，適應模型輸入要求shape = np.r_[img.shape]pad_all = (shape.max() - shape[:2]).astype('uint32')pad = pad_all // 2img_pad_ori = np.pad(img,((pad[0], pad_all[0] - pad[0]), (pad[1], pad_all[1] - pad[1]), (0, 0)),mode='constant')# 顏色空間轉換與尺寸調整img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)img_pad = np.pad(img,((pad[0], pad_all[0] - pad[0]), (pad[1], pad_all[1] - pad[1]), (0, 0)),mode='constant')img_small = cv2.resize(img_pad, (image_size, image_size))img_small = np.expand_dims(img_small, axis=0)img_small = (2.0 / 255.0) * img_small - 1.0img_small = img_small.astype('float32')return img_pad_ori, img_small, pad

?人臉檢測模塊使用 BlazeFace 算法，通過face_detection_front.tflite模型實現：

對輸入圖像進行填充和尺寸調整，適應模型輸入要求 (128x128)
使用錨點 (anchors) 進行邊界框預測和解碼
通過非極大值抑制 (NMS) 過濾重疊檢測框
輸出包含人臉位置和關鍵點的檢測結果

2. 人臉關鍵點檢測模塊

# 人臉關鍵點檢測預處理
def preprocess_image_for_tflite32(image, model_image_size=192):image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)image = cv2.resize(image, (model_image_size, model_image_size))image = np.expand_dims(image, axis=0)image = (2.0 / 255.0) * image - 1.0image = image.astype('float32')return image

?關鍵點檢測模塊使用face_landmark.tflite模型：

對人臉區域圖像進行預處理，調整為 192x192 尺寸
模型輸出 468 個人臉關鍵點坐標
這些關鍵點覆蓋了眉毛、眼睛、鼻子、嘴巴和臉部輪廓等區域
關鍵點用于后續的人臉對齊和變形操作

3. 人臉變換算法模塊

# 人臉變換核心函數
def faceswap(points1, points2, img1, img2):img1Warped = np.copy(img2)# 計算人臉凸包hull1 = []hull2 = []hullIndex = cv2.convexHull(np.array(points2), returnPoints=False)for i in range(0, len(hullIndex)):hull1.append(points1[int(hullIndex[i])])hull2.append(points2[int(hullIndex[i])])# 計算Delaunay三角形if img2 is None:return NonesizeImg2 = img2.shaperect = (0, 0, sizeImg2[1], sizeImg2[0])dt = calculateDelaunayTriangles(rect, hull2)if len(dt) == 0:quit()# 對每個三角形應用仿射變換for i in range(0, len(dt)):t1 = []t2 = []for j in range(0, 3):t1.append(hull1[dt[i][j]])t2.append(hull2[dt[i][j]])try:warpTriangle(img1, img1Warped, t1, t2)except:return None# 計算掩碼并進行無縫克隆hull8U = []for i in range(0, len(hull2)):hull8U.append((hull2[i][0], hull2[i][1]))mask = np.zeros(img2.shape, dtype=img2.dtype)cv2.fillConvexPoly(mask, np.int32(hull8U), (255, 255, 255))r = cv2.boundingRect(np.float32([hull2]))center = ((r[0] + int(r[2] / 2), r[1] + int(r[3] / 2)))output = cv2.seamlessClone(np.uint8(img1Warped), img2, mask, center, cv2.NORMAL_CLONE)return output

?人臉變換模塊實現了完整的人臉變換流程：

使用 Delaunay 三角剖分算法將人臉區域劃分為多個三角形
對每個三角形應用仿射變換，實現人臉形狀對齊
通過凸包計算確定人臉輪廓區域
使用 OpenCV 的seamlessClone函數進行無縫融合，避免明顯邊界
整個過程保證了人臉紋理和形狀的自然過渡

4. 界面交互模塊

# 自定義應用類
class MyApp(App):def __init__(self, *args):super(MyApp, self).__init__(*args)def main(self):# 創建垂直布局容器main_container = VBox(width=360, height=680, style={'margin': '0px auto'})self.aidcam0 = OpencvVideoWidget(self, width=340, height=400)# 添加圖像選擇按鈕和保存按鈕self.lbl = Label('點擊圖片選擇你喜歡的明星臉：')bottom_container = HBox(width=360, height=230, style={'margin': '0px auto'})self.img1 = Image('/res:' + os.getcwd() + '/' + back_img_path[0], height=80, margin='10px')self.img1.onclick.do(self.on_img1_clicked)# 保存按鈕功能self.bt1 = Button('保存合成圖片', width=300, height=30, margin='10px')self.bt1.onclick.do(self.on_button_pressed1)return main_container

?界面交互模塊提供了用戶操作接口：

實時顯示攝像頭畫面和人臉變換結果
提供目標人臉圖像選擇功能 (支持 4 張預設圖像)
實現合成圖像保存功能
使用了自定義的 UI 組件 (如 VBox、HBox、Image、Button 等)
支持點擊事件處理和用戶交互反饋

模型作用解析

1. face_detection_front.tflite

這是一個基于 BlazeFace 算法的人臉檢測模型，主要作用：

檢測圖像中的人臉位置，輸出邊界框坐標
同時預測 6 個人臉關鍵點 (眼睛、鼻子、嘴巴等位置)
模型輸入尺寸為 128x128，適合在邊緣設備上運行
使用錨點機制提高檢測精度和速度
輸出包含邊界框坐標和分類分數

2. face_landmark.tflite

這是人臉關鍵點檢測模型，主要功能：

對檢測到的人臉區域，預測 468 個精確關鍵點
關鍵點覆蓋了面部所有重要特征點
模型輸入尺寸為 192x192，輸出維度為 1404x4
這些關鍵點是人臉變換算法的基礎，用于形狀對齊和變形
模型在 GPU 上運行以提高實時性

aidlite 推理引擎介紹

基本功能

aidlite 是一個專為高通邊緣設備設計的輕量級推理引擎，具有以下核心功能：

支持多種深度學習框架模型轉換和部署 (TensorFlow Lite, ONNX 等)
提供統一的 API 接口，簡化模型推理流程
支持 CPU 和 GPU 加速，根據設備資源自動選擇最佳計算方式
優化內存使用，適合在資源受限的設備上運行
提供模型編譯和優化工具，提高推理效率

架構特點

# aidlite模型加載與配置示例
model = aidlite.Model.create_instance(model_path)
config = aidlite.Config.create_instance()
config.implement_type = aidlite.ImplementType.TYPE_FAST
config.framework_type = aidlite.FrameworkType.TYPE_TFLITE
config.accelerate_type = aidlite.AccelerateType.TYPE_CPU
config.number_of_threads = 4
interpreter = aidlite.InterpreterBuilder.build_interpretper_from_model_and_config(model, config)

?aidlite 的架構設計具有以下特點：

模塊化設計，支持插件式擴展
硬件抽象層，屏蔽不同設備的差異
高效的內存管理機制，減少內存拷貝
支持多線程并行計算，充分利用多核 CPU
針對移動設備和嵌入式設備進行了專門優化

性能優勢

aidlite 在邊緣設備上的性能優勢：

低延遲：針對實時應用場景優化，響應時間短
高效率：計算資源利用率高，能耗低
輕量級：庫文件體積小，占用系統資源少
跨平臺：支持多種嵌入式系統和硬件平臺
靈活配置：可根據設備性能動態調整計算參數

代碼應用場景分析

主要應用場景

娛樂與社交媒體
- 實時人臉變換濾鏡，用于短視頻和直播
- 社交媒體特效，增加用戶互動性
教育與演示
- 計算機視覺原理教學演示
- 機器學習模型應用實例展示
- 人臉處理技術科普

技術特點與限制

技術優勢
- 實時性：支持攝像頭實時視頻流處理
- 易用性：提供圖形界面，操作簡單
- 靈活性：支持自定義目標人臉圖像
- 輕量級：可在嵌入式設備上運行
應用限制
- 對人臉姿態和表情變化的魯棒性有限
- 復雜光照條件下效果可能下降
- 多人臉場景下需要進一步優化
- 高精度需求場景下可能需要更復雜的模型

總結

這段代碼實現了一個完整的實時人臉變化試衣Demo，結合了人臉檢測、關鍵點定位和圖像變形等多種計算機視覺技術。系統使用 aidlite 推理引擎在邊緣設備上高效運行深度學習模型，實現了實時的人臉分析和圖像合成。該應用具有廣泛的娛樂和實用價值，同時也展示了輕量級深度學習在邊緣設備上的應用潛力。通過進一步優化模型和算法，可以將該系統應用于更多場景。

示例代碼

import cv2
import math
import sys
import numpy as np
import os
import subprocess
import time
from cvs import *
import aidlite# 背景圖片路徑列表,圖片可以替換
back_img_path = ('models/Biden.jpeg', 'models/zh.jpeg', 'models/zh1.jpeg', 'models/kt1.jpg')# 讀取第一張背景圖片
faceimg = cv2.imread(back_img_path[0])
mod = -1
bfirstframe = True
saveimg = faceimg# 從文件中讀取關鍵點
def readPoints(path):# 創建一個點的數組points = []# 讀取文件中的點with open(path) as file:for line in file:x, y = line.split()points.append((int(x), int(y)))return points# 應用仿射變換
def applyAffineTransform(src, srcTri, dstTri, size):# 計算仿射變換矩陣warpMat = cv2.getAffineTransform(np.float32(srcTri), np.float32(dstTri))# 應用仿射變換到源圖像dst = cv2.warpAffine(src, warpMat, (size[0], size[1]), None, flags=cv2.INTER_LINEAR, borderMode=cv2.BORDER_REFLECT_101)return dst# 檢查點是否在矩形內
def rectContains(rect, point):if point[0] < rect[0]:return Falseelif point[1] < rect[1]:return Falseelif point[0] > rect[0] + rect[2]:return Falseelif point[1] > rect[1] + rect[3]:return Falsereturn True# 計算Delaunay三角形
def calculateDelaunayTriangles(rect, points):# 創建Subdiv2D對象subdiv = cv2.Subdiv2D(rect)# 將點插入到Subdiv2D對象中for p in points:subdiv.insert(p)# 獲取三角形列表triangleList = subdiv.getTriangleList()delaunayTri = []pt = []for t in triangleList:pt.append((t[0], t[1]))pt.append((t[2], t[3]))pt.append((t[4], t[5]))pt1 = (t[0], t[1])pt2 = (t[2], t[3])pt3 = (t[4], t[5])# 檢查三角形的三個點是否都在矩形內if rectContains(rect, pt1) and rectContains(rect, pt2) and rectContains(rect, pt3):ind = []# 通過坐標獲取人臉關鍵點的索引for j in range(0, 3):for k in range(0, len(points)):if abs(pt[j][0] - points[k][0]) < 1.0 and abs(pt[j][1] - points[k][1]) < 1.0:ind.append(k)# 如果索引列表長度為3，則將其添加到Delaunay三角形列表中if len(ind) == 3:delaunayTri.append((ind[0], ind[1], ind[2]))pt = []return delaunayTri# 對三角形區域進行變形和融合
def warpTriangle(img1, img2, t1, t2):# 找到每個三角形的邊界矩形r1 = cv2.boundingRect(np.float32([t1]))r2 = cv2.boundingRect(np.float32([t2]))# 偏移點以矩形左上角為原點t1Rect = []t2Rect = []t2RectInt = []for i in range(0, 3):t1Rect.append(((t1[i][0] - r1[0]), (t1[i][1] - r1[1])))t2Rect.append(((t2[i][0] - r2[0]), (t2[i][1] - r2[1])))t2RectInt.append(((t2[i][0] - r2[0]), (t2[i][1] - r2[1])))# 創建掩碼mask = np.zeros((r2[3], r2[2], 3), dtype=np.float32)cv2.fillConvexPoly(mask, np.int32(t2RectInt), (1.0, 1.0, 1.0), 16, 0)# 提取源圖像的矩形區域img1Rect = img1[r1[1]:r1[1] + r1[3], r1[0]:r1[0] + r1[2]]size = (r2[2], r2[3])# 對源圖像的矩形區域進行仿射變換img2Rect = applyAffineTransform(img1Rect, t1Rect, t2Rect, size)img2Rect = img2Rect * mask# 將變換后的三角形區域復制到目標圖像中img2[r2[1]:r2[1] + r2[3], r2[0]:r2[0] + r2[2]] = img2[r2[1]:r2[1] + r2[3], r2[0]:r2[0] + r2[2]] * ((1.0, 1.0, 1.0) - mask)img2[r2[1]:r2[1] + r2[3], r2[0]:r2[0] + r2[2]] = img2[r2[1]:r2[1] + r2[3], r2[0]:r2[0] + r2[2]] + img2Rect# 人臉變換函數
def faceswap(points1, points2, img1, img2):img1Warped = np.copy(img2)# 找到凸包hull1 = []hull2 = []hullIndex = cv2.convexHull(np.array(points2), returnPoints=False)for i in range(0, len(hullIndex)):hull1.append(points1[int(hullIndex[i])])hull2.append(points2[int(hullIndex[i])])# 計算凸包點的Delaunay三角形if img2 is None:return NonesizeImg2 = img2.shaperect = (0, 0, sizeImg2[1], sizeImg2[0])dt = calculateDelaunayTriangles(rect, hull2)if len(dt) == 0:quit()# 對Delaunay三角形應用仿射變換for i in range(0, len(dt)):t1 = []t2 = []for j in range(0, 3):t1.append(hull1[dt[i][j]])t2.append(hull2[dt[i][j]])try:warpTriangle(img1, img1Warped, t1, t2)except:return None# 計算掩碼hull8U = []for i in range(0, len(hull2)):hull8U.append((hull2[i][0], hull2[i][1]))mask = np.zeros(img2.shape, dtype=img2.dtype)cv2.fillConvexPoly(mask, np.int32(hull8U), (255, 255, 255))r = cv2.boundingRect(np.float32([hull2]))center = ((r[0] + int(r[2] / 2), r[1] + int(r[3] / 2)))# 無縫克隆output = cv2.seamlessClone(np.uint8(img1Warped), img2, mask, center, cv2.NORMAL_CLONE)return output# 對圖像進行預處理，用于TFLite模型
def preprocess_image_for_tflite32(image, model_image_size=192):image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)image = cv2.resize(image, (model_image_size, model_image_size))image = np.expand_dims(image, axis=0)image = (2.0 / 255.0) * image - 1.0image = image.astype('float32')return image# 對圖像進行填充和預處理
def preprocess_img_pad(img, image_size=128):shape = np.r_[img.shape]pad_all = (shape.max() - shape[:2]).astype('uint32')pad = pad_all // 2img_pad_ori = np.pad(img,((pad[0], pad_all[0] - pad[0]), (pad[1], pad_all[1] - pad[1]), (0, 0)),mode='constant')img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)img_pad = np.pad(img,((pad[0], pad_all[0] - pad[0]), (pad[1], pad_all[1] - pad[1]), (0, 0)),mode='constant')img_small = cv2.resize(img_pad, (image_size, image_size))img_small = np.expand_dims(img_small, axis=0)img_small = (2.0 / 255.0) * img_small - 1.0img_small = img_small.astype('float32')return img_pad_ori, img_small, pad# 繪制檢測到的人臉框
def plot_detections(img, detections, with_keypoints=True):output_img = imgprint(img.shape)x_min = 0x_max = 0y_min = 0y_max = 0print("找到 %d 個人臉" % len(detections))for i in range(len(detections)):ymin = detections[i][0] * img.shape[0]xmin = detections[i][1] * img.shape[1]ymax = detections[i][2] * img.shape[0]xmax = detections[i][3] * img.shape[1]w = int(xmax - xmin)h = int(ymax - ymin)h = max(w, h)h = h * 1.5x = (xmin + xmax) / 2.y = (ymin + ymax) / 2.xmin = x - h / 2.xmax = x + h / 2.ymin = y - h / 2. - 0.08 * hymax = y + h / 2. - 0.08 * hx_min = int(xmin)y_min = int(ymin)x_max = int(xmax)y_max = int(ymax)p1 = (int(xmin), int(ymin))p2 = (int(xmax), int(ymax))cv2.rectangle(output_img, p1, p2, (0, 255, 255), 2, 1)return x_min, y_min, x_max, y_max# 繪制人臉網格
def draw_mesh(image, mesh, mark_size=2, line_width=1):image_size = image.shape[0]mesh = mesh * image_sizefor point in mesh:cv2.circle(image, (point[0], point[1]),mark_size, (0, 255, 128), -1)# 繪制眼睛輪廓left_eye_contour = np.array([mesh[33][0:2],mesh[7][0:2],mesh[163][0:2],mesh[144][0:2],mesh[145][0:2],mesh[153][0:2],mesh[154][0:2],mesh[155][0:2],mesh[133][0:2],mesh[173][0:2],mesh[157][0:2],mesh[158][0:2],mesh[159][0:2],mesh[160][0:2],mesh[161][0:2],mesh[246][0:2], ]).astype(np.int32)right_eye_contour = np.array([mesh[263][0:2],mesh[249][0:2],mesh[390][0:2],mesh[373][0:2],mesh[374][0:2],mesh[380][0:2],mesh[381][0:2],mesh[382][0:2],mesh[362][0:2],mesh[398][0:2],mesh[384][0:2],mesh[385][0:2],mesh[386][0:2],mesh[387][0:2],mesh[388][0:2],mesh[466][0:2]]).astype(np.int32)cv2.polylines(image, [left_eye_contour, right_eye_contour], False,(255, 255, 255), line_width, cv2.LINE_AA)# 獲取人臉關鍵點
def getkeypoint(image, mesh, landmark_point):image_size = image.shape[0]mesh = mesh * image_sizefor point in mesh:landmark_point.append((point[0], point[1]))return image# 繪制人臉關鍵點和輪廓
def draw_landmarks(image, mesh, landmark_point):image_size = image.shape[0]mesh = mesh * image_sizefor point in mesh:landmark_point.append((point[0], point[1]))cv2.circle(image, (point[0], point[1]), 2, (255, 255, 0), -1)if len(landmark_point) > 0:# 繪制左眉毛cv2.line(image, landmark_point[55], landmark_point[65], (0, 0, 255), 2, -3)cv2.line(image, landmark_point[65], landmark_point[52], (0, 0, 255), 2, -3)cv2.line(image, landmark_point[52], landmark_point[53], (0, 0, 255), 2, -3)cv2.line(image, landmark_point[53], landmark_point[46], (0, 0, 255), 2, -3)# 繪制右眉毛cv2.line(image, landmark_point[285], landmark_point[295], (0, 0, 255), 2)cv2.line(image, landmark_point[295], landmark_point[282], (0, 0, 255), 2)cv2.line(image, landmark_point[282], landmark_point[283], (0, 0, 255), 2)cv2.line(image, landmark_point[283], landmark_point[276], (0, 0, 255), 2)# 繪制左眼睛cv2.line(image, landmark_point[133], landmark_point[173], (0, 0, 255), 2)cv2.line(image, landmark_point[173], landmark_point[157], (0, 0, 255), 2)cv2.line(image, landmark_point[157], landmark_point[158], (0, 0, 255), 2)cv2.line(image, landmark_point[158], landmark_point[159], (0, 0, 255), 2)cv2.line(image, landmark_point[159], landmark_point[160], (0, 0, 255), 2)cv2.line(image, landmark_point[160], landmark_point[161], (0, 0, 255), 2)cv2.line(image, landmark_point[161], landmark_point[246], (0, 0, 255), 2)cv2.line(image, landmark_point[246], landmark_point[163], (0, 0, 255), 2)cv2.line(image, landmark_point[163], landmark_point[144], (0, 0, 255), 2)cv2.line(image, landmark_point[144], landmark_point[145], (0, 0, 255), 2)cv2.line(image, landmark_point[145], landmark_point[153], (0, 0, 255), 2)cv2.line(image, landmark_point[153], landmark_point[154], (0, 0, 255), 2)cv2.line(image, landmark_point[154], landmark_point[155], (0, 0, 255), 2)cv2.line(image, landmark_point[155], landmark_point[133], (0, 0, 255), 2)# 繪制右眼睛cv2.line(image, landmark_point[362], landmark_point[398], (0, 0, 255), 2)cv2.line(image, landmark_point[398], landmark_point[384], (0, 0, 255), 2)cv2.line(image, landmark_point[384], landmark_point[385], (0, 0, 255), 2)cv2.line(image, landmark_point[385], landmark_point[386], (0, 0, 255), 2)cv2.line(image, landmark_point[386], landmark_point[387], (0, 0, 255), 2)cv2.line(image, landmark_point[387], landmark_point[388], (0, 0, 255), 2)cv2.line(image, landmark_point[388], landmark_point[466], (0, 0, 255), 2)cv2.line(image, landmark_point[466], landmark_point[390], (0, 0, 255), 2)cv2.line(image, landmark_point[390], landmark_point[373], (0, 0, 255), 2)cv2.line(image, landmark_point[373], landmark_point[374], (0, 0, 255), 2)cv2.line(image, landmark_point[374], landmark_point[380], (0, 0, 255), 2)cv2.line(image, landmark_point[380], landmark_point[381], (0, 0, 255), 2)cv2.line(image, landmark_point[381], landmark_point[382], (0, 0, 255), 2)cv2.line(image, landmark_point[382], landmark_point[362], (0, 0, 255), 2)# 繪制嘴巴cv2.line(image, landmark_point[308], landmark_point[415], (0, 0, 255), 2)cv2.line(image, landmark_point[415], landmark_point[310], (0, 0, 255), 2)cv2.line(image, landmark_point[310], landmark_point[311], (0, 0, 255), 2)cv2.line(image, landmark_point[311], landmark_point[312], (0, 0, 255), 2)cv2.line(image, landmark_point[312], landmark_point[13], (0, 0, 255), 2)cv2.line(image, landmark_point[13], landmark_point[82], (0, 0, 255), 2)cv2.line(image, landmark_point[82], landmark_point[81], (0, 0, 255), 2)cv2.line(image, landmark_point[81], landmark_point[80], (0, 0, 255), 2)cv2.line(image, landmark_point[80], landmark_point[191], (0, 0, 255), 2)cv2.line(image, landmark_point[191], landmark_point[78], (0, 0, 255), 2)cv2.line(image, landmark_point[78], landmark_point[95], (0, 0, 255), 2)cv2.line(image, landmark_point[95], landmark_point[88], (0, 0, 255), 2)cv2.line(image, landmark_point[88], landmark_point[178], (0, 0, 255), 2)cv2.line(image, landmark_point[178], landmark_point[87], (0, 0, 255), 2)cv2.line(image, landmark_point[87], landmark_point[14], (0, 0, 255), 2)cv2.line(image, landmark_point[14], landmark_point[317], (0, 0, 255), 2)cv2.line(image, landmark_point[317], landmark_point[402], (0, 0, 255), 2)cv2.line(image, landmark_point[402], landmark_point[318], (0, 0, 255), 2)cv2.line(image, landmark_point[318], landmark_point[324], (0, 0, 255), 2)cv2.line(image, landmark_point[324], landmark_point[308], (0, 0, 255), 2)return image# 自定義應用類
class MyApp(App):def __init__(self, *args):super(MyApp, self).__init__(*args)def idle(self):self.aidcam0.update()def main(self):# 創建垂直布局容器main_container = VBox(width=360, height=680, style={'margin': '0px auto'})self.aidcam0 = OpencvVideoWidget(self, width=340, height=400)self.aidcam0.style['margin'] = '10px'i = 0exec("self.aidcam%(i)s = OpencvVideoWidget(self)" % {'i': i})exec("self.aidcam%(i)s.identifier = 'aidcam%(i)s'" % {'i': i})eval("main_container.append(self.aidcam%(i)s)" % {'i': i})main_container.append(self.aidcam0)self.lbl = Label('點擊圖片選擇你喜歡的明星臉：')main_container.append(self.lbl)# 創建水平布局容器bottom_container = HBox(width=360, height=230, style={'margin': '0px auto'})self.img1 = Image('/res:' + os.getcwd() + '/' + back_img_path[0], height=80, margin='10px')self.img1.onclick.do(self.on_img1_clicked)bottom_container.append(self.img1)self.img2 = Image('/res:' + os.getcwd() + '/' + back_img_path[1], height=80, margin='10px')self.img2.onclick.do(self.on_img2_clicked)bottom_container.append(self.img2)self.img4 = Image('/res:' + os.getcwd() + '/' + back_img_path[3], height=80, margin='10px')self.img4.onclick.do(self.on_img4_clicked)bottom_container.append(self.img4)self.img3 = Image('/res:' + os.getcwd() + '/' + back_img_path[2], height=80, margin='10px')self.img3.onclick.do(self.on_img3_clicked)bottom_container.append(self.img3)self.bt1 = Button('保存合成圖片', width=300, height=30, margin='10px')self.bt1.onclick.do(self.on_button_pressed1)main_container.append(bottom_container)main_container.append(self.bt1)return main_containerdef on_img1_clicked(self, widget):global faceimgbgnd = cv2.imread(back_img_path[0])faceimg = bgndglobal modmod = 0print('mod', mod)def on_img2_clicked(self, widget):global faceimgbgnd = cv2.imread(back_img_path[1])faceimg = bgndglobal modmod = 1print('mod', mod)def on_img3_clicked(self, widget):global faceimgbgnd = cv2.imread(back_img_path[2])faceimg = bgndglobal modmod = 2print('mod', mod)def on_img4_clicked(self, widget):global faceimgbgnd = cv2.imread(back_img_path[3])faceimg = bgndglobal modmod = 3print('mod', mod)def on_button_pressed1(self, widget):cv2.imwrite('result.jpg', saveimg)aidlite.makeToast("保存合成圖片result.jpg成功!")# 獲取攝像頭ID
def get_cap_id():try:# 構造命令，使用awk處理輸出cmd = "ls -l /sys/class/video4linux | awk -F ' -> ' '/usb/{sub(/.*video/, \"\", $2); print $2}'"result = subprocess.run(cmd, shell=True, capture_output=True, text=True)output = result.stdout.strip().split()# 轉換所有捕獲的編號為整數，找出最小值video_numbers = list(map(int, output))if video_numbers:return min(video_numbers)else:return Noneexcept Exception as e:print(f"發生錯誤: {e}")return None# 主處理函數
def process():cvs.setCustomUI()# 加載人臉檢測模型inShape = [[1, 128, 128, 3]]outShape = [[1, 896, 16], [1, 896, 1]]model_path = "models/face_detection_front.tflite"model = aidlite.Model.create_instance(model_path)if model is None:print("創建face_detection_front模型失敗!")model.set_model_properties(inShape, aidlite.DataType.TYPE_FLOAT32, outShape,aidlite.DataType.TYPE_FLOAT32)config = aidlite.Config.create_instance()config.implement_type = aidlite.ImplementType.TYPE_FASTconfig.framework_type = aidlite.FrameworkType.TYPE_TFLITEconfig.accelerate_type = aidlite.AccelerateType.TYPE_CPUconfig.number_of_threads = 4fast_interpreter = aidlite.InterpreterBuilder.build_interpretper_from_model_and_config(model, config)if fast_interpreter is None:print("face_detection_front模型build_interpretper_from_model_and_config失敗!")result = fast_interpreter.init()if result != 0:print("face_detection_front模型解釋器初始化失敗!")result = fast_interpreter.load_model()if result != 0:print("face_detection_front模型解釋器加載模型失敗!")print("face_detection_front模型加載成功!")# 加載人臉關鍵點檢測模型model_path1 = "models/face_landmark.tflite"inShape1 = [[1 * 192 * 192 * 3]]outShape1 = [[1 * 1404 * 4], [1 * 4]]model1 = aidlite.Model.create_instance(model_path1)if model1 is None:print("創建face_landmark模型失敗!")model1.set_model_properties(inShape1, aidlite.DataType.TYPE_FLOAT32, outShape1,aidlite.DataType.TYPE_FLOAT32)config1 = aidlite.Config.create_instance()config1.implement_type = aidlite.ImplementType.TYPE_FASTconfig1.framework_type = aidlite.FrameworkType.TYPE_TFLITEconfig1.accelerate_type = aidlite.AccelerateType.TYPE_GPUconfig1.number_of_threads = 4fast_interpreter1 = aidlite.InterpreterBuilder.build_interpretper_from_model_and_config(model1, config1)if fast_interpreter1 is None:print("face_landmark模型build_interpretper_from_model_and_config失敗!")result = fast_interpreter1.init()if result != 0:print("face_landmark模型解釋器初始化失敗!")result = fast_interpreter1.load_model()if result != 0:print("face_landmark模型解釋器加載模型失敗!")print("face_landmark模型加載成功!")# 加載錨點數據anchors = np.load('models/anchors.npy').astype(np.float32)# 0-后置，1-前置camid = 1capId = get_cap_id()if capId is None:print("使用MIPI攝像頭")camid = camidelse:print("使用USB攝像頭")camid = -1cap = cvs.VideoCapture(camid)bFace = Falsex_min, y_min, x_max, y_max = (0, 0, 0, 0)fface = 0.0bfirstframe = Truefacepath = "models/Biden.jpeg"global faceimgfaceimg = cv2.imread(facepath)faceimg = cv2.resize(faceimg, (480, 640))roi_orifirst = faceimgpadfaceimg = faceimgf_x_min, f_y_min, f_x_max, f_y_max = (0, 0, 0, 0)fpoints = []spoints = []global modmod = -1temp = faceimgwhile True:frame = cvs.read()if frame is None:continueif camid == 1:frame = cv2.flip(frame, 1)if bfirstframe or mod > -1:frame = cv2.resize(faceimg, (480, 640))bfirstframe = Trueroi_orifirst = faceimgpadfaceimg = faceimgf_x_min, f_y_min, f_x_max, f_y_max = (0, 0, 0, 0)fpoints = []spoints = []bFace = Falsefface = 0.0x_min, y_min, x_max, y_max = (0, 0, 0, 0)start_time = time.time()img_pad, img, pad = preprocess_img_pad(frame, 128)if bFace == False:result = fast_interpreter.set_input_tensor(0, img.data)if result != 0:print("face_detection_front模型解釋器set_input_tensor()失敗")result = fast_interpreter.invoke()if result != 0:print("face_detection_front模型解釋器invoke()失敗")raw_boxes = fast_interpreter.get_output_tensor(0)if raw_boxes is None:print("示例: face_detection_front模型解釋器->get_output_tensor(0)失敗!")classificators = fast_interpreter.get_output_tensor(1)if classificators is None:print("示例: face_detection_front模型解釋器->get_output_tensor(1)失敗!")detections = blazeface(raw_boxes, classificators, anchors)[0]if len(detections) > 0:bFace = Trueif bFace:for i in range(len(detections)):ymin = detections[i][0] * img_pad.shape[0]xmin = detections[i][1] * img_pad.shape[1]ymax = detections[i][2] * img_pad.shape[0]xmax = detections[i][3] * img_pad.shape[1]w = int(xmax - xmin)h = int(ymax - ymin)h = max(w, h)h = h * 1.5x = (xmin + xmax) / 2.y = (ymin + ymax) / 2.xmin = x - h / 2.xmax = x + h / 2.ymin = y - h / 2. - 0.08 * hymax = y + h / 2. - 0.08 * hx_min = int(xmin)y_min = int(ymin)x_max = int(xmax)y_max = int(ymax)x_min = max(0, x_min)y_min = max(0, y_min)x_max = min(img_pad.shape[1], x_max)y_max = min(img_pad.shape[0], y_max)roi_ori = img_pad[y_min:y_max, x_min:x_max]roi = preprocess_image_for_tflite32(roi_ori, 192)result = fast_interpreter1.set_input_tensor(0, roi.data)if result != 0:print("face_landmark模型解釋器set_input_tensor()失敗")result = fast_interpreter1.invoke()if result != 0:print("face_landmark模型解釋器invoke()失敗")mesh = fast_interpreter1.get_output_tensor(0)if mesh is None:print("示例: face_landmark模型解釋器->get_output_tensor(0)失敗!")stride8 = fast_interpreter1.get_output_tensor(1)if stride8 is None:print("示例: face_landmark模型解釋器->get_output_tensor(1)失敗!")print(f"stride8.shape: {stride8.shape}")ffacetmp = stride8[0]bFace = Falsespoints = []mesh = mesh.reshape(468, 3) / 192if bfirstframe:getkeypoint(roi_ori, mesh, fpoints)roi_orifirst = roi_ori.copy()temp = roi_orifirst.copy()bfirstframe = Falsepadfaceimg = img_pad.copy()f_x_min, f_y_min, f_x_max, f_y_max = (x_min, y_min, x_max, y_max)mod = -1else:getkeypoint(roi_ori, mesh, spoints)roi_orifirst = faceswap(spoints, fpoints, roi_ori, temp)if roi_orifirst is None:continuef_img_pad = padfaceimg.copy()x_min, y_min, x_max, y_max = (f_x_min, f_y_min, f_x_max, f_y_max)f_img_pad[y_min:y_max, x_min:x_max] = roi_orifirstx, y = f_img_pad.shape[0] / 2, f_img_pad.shape[1] / 2img_pad = f_img_padshape = frame.shapeglobal saveimgsaveimg = img_pad[max(0, int(y - shape[0] / 2)):int(y + shape[0] / 2),max(0, int(x - shape[1] / 2)):int(x + shape[1] / 2)]t = (time.time() - start_time)lbs = 'Fps: ' + str(int(100 / t) / 100.) + " ~~ Time:" + str(t * 1000) + "ms"cvs.setLbs(lbs)cvs.imshow(saveimg)time.sleep(1)# BlazeFace人臉檢測類
class BlazeFace():def __init__(self):# 類別數量self.num_classes = 1# 錨點數量self.num_anchors = 896# 坐標數量self.num_coords = 16# 分數裁剪閾值self.score_clipping_thresh = 100.0# 縮放因子self.x_scale = 128.0self.y_scale = 128.0self.h_scale = 128.0self.w_scale = 128.0# 最小分數閾值self.min_score_thresh = 0.75# 最小抑制閾值self.min_suppression_threshold = 0.3# Sigmoid函數def sigmoid(self, inX):if inX >= 0:return 1.0 / (1 + np.exp(-inX))else:return np.exp(inX) / (1 + np.exp(inX))# 將原始張量轉換為檢測結果def tensors_to_detections(self, raw_box_tensor, raw_score_tensor, anchors):assert len(raw_box_tensor.shape) == 3assert raw_box_tensor.shape[1] == self.num_anchorsassert raw_box_tensor.shape[2] == self.num_coordsassert len(raw_box_tensor.shape) == 3assert raw_score_tensor.shape[1] == self.num_anchorsassert raw_score_tensor.shape[2] == self.num_classesassert raw_box_tensor.shape[0] == raw_score_tensor.shape[0]# 解碼邊界框detection_boxes = self._decode_boxes(raw_box_tensor, anchors)thresh = self.score_clipping_threshraw_score_tensor = raw_score_tensor.clip(-thresh, thresh)# 計算檢測分數detection_scores = 1 / (1 + np.exp(- raw_score_tensor)).squeeze(axis=-1)# 過濾掉分數低于閾值的檢測結果mask = detection_scores >= self.min_score_threshoutput_detections = []for i in range(raw_box_tensor.shape[0]):boxes = detection_boxes[i, mask[i]]scores = np.expand_dims(detection_scores[i, mask[i]], axis=-1)output_detections.append(np.concatenate((boxes, scores), axis=-1))return output_detections# 解碼邊界框def _decode_boxes(self, raw_boxes, anchors):boxes = np.zeros(raw_boxes.shape)x_center = raw_boxes[..., 0] / self.x_scale * anchors[:, 2] + anchors[:, 0]y_center = raw_boxes[..., 1] / self.y_scale * anchors[:, 3] + anchors[:, 1]w = raw_boxes[..., 2] / self.w_scale * anchors[:, 2]h = raw_boxes[..., 3] / self.h_scale * anchors[:, 3]boxes[..., 0] = y_center - h / 2.  # yminboxes[..., 1] = x_center - w / 2.  # xminboxes[..., 2] = y_center + h / 2.  # ymaxboxes[..., 3] = x_center + w / 2.  # xmaxfor k in range(6):offset = 4 + k * 2keypoint_x = raw_boxes[..., offset] / self.x_scale * anchors[:, 2] + anchors[:, 0]keypoint_y = raw_boxes[..., offset + 1] / self.y_scale * anchors[:, 3] + anchors[:, 1]boxes[..., offset] = keypoint_xboxes[..., offset + 1] = keypoint_yreturn boxes# 加權非極大值抑制def weighted_non_max_suppression(self, detections):if len(detections) == 0: return []output_detections = []# 按分數從高到低排序remaining = np.argsort(-detections[:, 16])while len(remaining) > 0:detection = detections[remaining[0]]first_box = detection[:4]other_boxes = detections[remaining, :4]# 計算IoUious = overlap_similarity(first_box, other_boxes)mask = ious > self.min_suppression_thresholdoverlapping = remaining[mask]remaining = remaining[~mask]weighted_detection = detection.copy()if len(overlapping) > 1:coordinates = detections[overlapping, :16]scores = detections[overlapping, 16:17]total_score = scores.sum()weighted = (coordinates * scores).sum(axis=0) / total_scoreweighted_detection[:16] = weightedweighted_detection[16] = total_score / len(overlapping)output_detections.append(weighted_detection)return output_detections# BlazeFace人臉檢測函數
def blazeface(raw_output_a, raw_output_b, anchors):if raw_output_a.size == 896:raw_score_tensor = raw_output_araw_box_tensor = raw_output_belse:raw_score_tensor = raw_output_braw_box_tensor = raw_output_aassert (raw_score_tensor.size == 896)assert (raw_box_tensor.size == 896 * 16)raw_score_tensor = raw_score_tensor.reshape(1, 896, 1)raw_box_tensor = raw_box_tensor.reshape(1, 896, 16)net = BlazeFace()# 后處理原始預測結果detections = net.tensors_to_detections(raw_box_tensor, raw_score_tensor, anchors)# 非極大值抑制filtered_detections = []for i in range(len(detections)):faces = net.weighted_non_max_suppression(detections[i])if len(faces) > 0:faces = np.stack(faces)filtered_detections.append(faces)return filtered_detections# 將檢測結果轉換為原始圖像坐標
def convert_to_orig_points(results, orig_dim, letter_dim):inter_scale = min(letter_dim / orig_dim[0], letter_dim / orig_dim[1])inter_h, inter_w = int(inter_scale * orig_dim[0]), int(inter_scale * orig_dim[1])offset_x, offset_y = (letter_dim - inter_w) / 2.0 / letter_dim, (letter_dim - inter_h) / 2.0 / letter_dimscale_x, scale_y = letter_dim / inter_w, letter_dim / inter_hresults[:, 0:2] = (results[:, 0:2] - [offset_x, offset_y]) * [scale_x, scale_y]results[:, 2:4] = results[:, 2:4] * [scale_x, scale_y]results[:, 4:16:2] = (results[:, 4:16:2] - offset_x) * scale_xresults[:, 5:17:2] = (results[:, 5:17:2] - offset_y) * scale_yresults[:, 0:16:2] *= orig_dim[1]results[:, 1:17:2] *= orig_dim[0]return results.astype(np.int32)# 計算IoU
def overlap_similarity(box, other_boxes):def union(A, B):x1, y1, x2, y2 = Aa = (x2 - x1) * (y2 - y1)x1, y1, x2, y2 = Bb = (x2 - x1) * (y2 - y1)ret = a + b - intersect(A, B)return retdef intersect(A, B):x1 = max(A[0], B[0])y1 = max(A[1], B[1])x2 = min(A[2], B[2])y2 = min(A[3], B[3])return (x2 - x1) * (y2 - y1)ret = np.array([max(0, intersect(box, b) / union(box, b)) for b in other_boxes])return retif __name__ == '__main__':initcv(startcv, MyApp)process()

效果樣圖：