Hey!!
At Jagberry Vision, we are dedicated to pushing the boundaries of technology through innovative projects in the fields of Embedded Systems and Artificial Intelligence. Our mission is to provide cutting-edge insights, practical tutorials, and in-depth explorations into the world of embedded solutions and AI integration. Whether you're an industry professional, a student, or an enthusiast, we aim to empower our community with the knowledge and tools needed to bring smart, efficient solutions to life.
In this section, we’ll learn how to build a facial verification login system for websites.
Here I kept the web design simple enough for two reasons:
1. I haven’t explored web development a lot till now.
2. Secondly, To keep the deployment stuff relatively easier for the audience to understand the procedure and mainly utilise their time in building the project instead of resolving version-related issues for different environment the audience are working on during their practical implementation.
Building a Facial Verification Model: *Stage-01: Face Detection - *A very first task for the model is to detect a face in the given frame/image. For this, we have multiple approaches: - MTCNN - Haar Cascade - CNN - DeepFace *For this project, we’ll be using MTCNN because of its better performance in face detection.
Create a Python file and name it Extract_face_Mtcnn_stage01.py
""" Goal: """ # Extract Face using MTCNN from Images
""" Libraries """ from PIL import Image import numpy as np from mtcnn.mtcnn import MTCNN
x1, y1, x2, y2 =0, 0, 0, 0
# detected face from image def extract_face_n_labels(mImage, imageIsArray=True):
# using MTCNN for face detector in an image detector = MTCNN()
cropped_face = []
global x1, y1, x2, y2
if imageIsArray == True:
pixels = mImage
# MTCNN for face detection face_detected = detector.detect_faces(pixels) print('face_detected, proccessed values: ', face_detected)
if face_detected: # if face_detected list not empty
# getting the bounding box of detected face x1, y1, w, h = face_detected[0]['box']
Don’t go towards the length of the code here we are performing 3 things: 1. imageIsArray - Checking if the passed image to the function is an array. (Well this if statement is optional if you are sure you are going to call this function by passing Image converted into an array then no need to use this parameter and this if statement either)
2. Face detecting - Using MTCNN
3. Cropping Image - With the help of the coordinates of the detected face we are going to crop the image, So now the new image consists of only the detected face so that further facial feature processing could occur only over the face, not over the unwanted background.
*Stage-02: Create Face Embeddings -Now that we have cropped the image, it’s time to dive deeper Into the facial verification model. Now we want our model to tell the difference between the faces. Whether the two faces are similar or not. One way to do this is by passing an image to the model and allowing it to learn the features based on its pixel values and bringing the output accordingly. But, this approach takes time and is inefficient for this situation. So, instead, we’ll be calculating the distances of some major facial features instead of the whole face.
This approach also helps in avoiding errors while processing images in different lighting conditions, because now the model is not learning over the pixel values instead it has calculated the distances of the facial features which is not affected by any lighting conditions.
So these distances are nothing but embedded features(1D vector). In order to calculate embedded features we again have a lot of approaches available here: - FaceNet - DeepFace - OpenFace - VGG-Face **Accuracy of these models**
We’ll be using VGG-Face for our model building. VGG-model produces random facial measurements called embedded features. Well, you don’t have to worry about which random facial features are they.
Create a Python file and name it Embedded_Features_VggNet_stage02.py
""" Goal: """ # Getting Embedded Features using VggNet from Images
""" Libraries """ from keras_vggface.vggface import VGGFace #model.py change to tensorflow.keras.utils from keras.layers import MaxPooling2D, ZeroPadding2D, Convolution2D, Dropout, Flatten, Activation from keras.models import Model, Sequential
def vgg_model_building():
"""Method-01: Build Vgg model (BASED ON RESNET50)""" vgg_model = VGGFace(model='resnet50', weights='vggface', include_top=False, input_shape=(224, 224, 3), pooling='avg')
Please Note: As you will run the code it might raise an error at the line ‘from keras_vggface.vggface import VGGFace’ as No module named ‘keras.engine.topology’ . for this just open models.py you will get the link of the models.py file at error terminal itself, open it and replace ‘from keras.engine.topology import get_source_inputs’ with ‘from tensorflow.keras.utils import get_source_inputs’
In the vgg_model_building() function I have shown two methods for building the vgg model. One is simply importing vgg-face and specifying state-of-the-art architecture(RestNet50) and the second one is building the layers of the architecture(VGG16). Both architectures are different so it's up to you whichever you find compatible with your system follow it accordingly and comment out the one you don’t follow.
Also please notice in the implementation of VGG16 architecture at the end we have flattened the CNN layers which produce 1D vectors consisting of 2622 elements, well these vectors are nothing but the embedded vectors only consisting of the facial features. Whereas, if we use the prebuilt model of ResNet50 as vgg_model then we get 2048 facial features.
*Stage-03: Face Verification - If you followed till now congrats we are now at the last stage of the model building, at this stage we’ll be calculating the distances of embedded features of the stored database users(db_user_embedding) and the current login user(current_user_embedding) For the time being, if you don’t understand what’s the meaning of database user here. So, hold down your horses, I’ll be explaining it in the deployment stage you will get to know the whole picture behind using these terms. For now, just think of it as if we are going to compare both faces by calculating the distance of embedded features(facial measurements). If the distance is large that means both images are different and if the distance is close to 0 then the face is similar which means the user is similar to the one stored in a database. To calculate the similarity of the face we can use: 1. Euclidean distance 2. Cosine Similarity We’ll be seeing both of them in the code, in order to understand how to implement them, but for the final stage Euclidean distance we’ll going to be the major deciding factor as it performs really well in this situation compared to cosine similarity.
Create a Python file and name it Face_Verification_stage03.py
""" Goal: """ # Finally Face Verifiction step using Euclidean distance
Finally Congrats!!! we are done with the model-building part.
2. Deployment:
Building up the model is never enough until the model is deployed so that model can be utilized by the community too.
Deployment Thoughts:
We’ll create a signup page that consists of email and password as input fields as well as the live streaming window. It captures the image and sends it to the model which returns the embedded feature of the input image.
Signup Page
Store the email ID, password and embedded feature of the new user in the Firebase database.
Now we’ll create a login page that takes the email ID and password from the user and in the backend, we’ll search for the current user in the database using its email ID as a unique identifier in this project. If the user didn’t remember his password can able to switch to Forget Password Page.
Login Page
The idea behind this project is generally if the user forgets their password they get an OTP via email for the verification process. But we’ll optimize this process by replacing OTP verification with the Facial Verification process.
So finally Forget Password page consists of a Live streaming window that takes the current user image and sends it to the model, the model returns the embedded feature calling it current_user_embedding. Now before switching to Forget Password page user needs to enter the email ID so as to find the current user email-ID existence in the database and then retrieve the respective embedded feature from the database calling it db_user_embedding.
Forget Password Page for Facial Verification
Now the final goal is to do the verification of the user by calculating the Euclidean distance/Cosine similarity of both the embeddings (db_user_embedding & current_user_embedding). If the verification is successful send the user to the home page.
Hustling in this step will make you explore more about firebase, if your are new to firebase database.
We performed various operations in all the 3 stages, hence in order to keep the code easy to understand we’ll going to merge all of them in an optimized manner in one Python file. So, if we want to perform any model operation we can simply import this Python file.
Create a Python file and name it ForgetPassword_page.py
""" Goal: Gathering all the stages in one file for the clean implementation """
"""Process: 1. Prepare MTCNN function 2. Prepare Embeddings function 3. Merge them all and reshape the embedded feature """
currentUser_embedded_face = embedded_face[0, :] # [0,:] -> to bring the shape from (1,2048) to (2048,)
return currentUser_embedded_face
Starting with the deployment implementation part:
Well, if you are a beginner in Python this section will going to be quite lengthy and a bit difficult to understand, hence I have commented on each line of the code for a better understanding.
Create a new Python file and name it Signup_page_addNewuser.py
""" Goal: Run Flask to:
Add new User into our database (Signup Page)
Build Login Page (verify current user email_id + if password not remember switch to forget passwprd page)
Build Forget Password Page (Use Facial verification in forget password)
## Need to save 3 things into the database:
Email_ID
Password
Facial Embedded feature of the new user."""
"""Process:
1. Database connectivity
2. Signup Page: # Takes input - Email ID - Password - Image (Live stream using webcam) ** Finally Saves all the 3 thing into the database.
3. Login Page: # Takes input - Email - Password (If not remember switch to forget pass page) - Before switching verify whether email_id is existing in Db, if not then current user goes back to the Signup Page. - If current user email id verified then fetch its appropriate Face embedding from database using its email id as unique identifiers for searching.
4. Forget Password page: # Takes input: - Image(From live streaming Face Detection [used Haar Cascade]) - Calculate Current User Face Embedding. - Match the Db Embedding and Current User Embedding If, matches then go to Home page. else, back to login page
"""
""" Libraries """ import cv2 import numpy as np import ForgetPassword_page_FaceVerification import firebase_admin from firebase_admin import credentials,db import yaml import json from json import JSONEncoder from flask import Flask, render_template, request, Response, redirect, url_for, flash from PIL import Image import Face_Verifiction_stage03 import threading import time
"""Getting Config file""" with open('config.yml', 'r') as file: config_data = yaml.safe_load(file)
app = Flask(__name__) app.secret_key = "super secret key" # for displaying msg for custom duration after successful signup app.config['MESSAGE_FLASHING_OPTIONS'] = {'duration': 100}
# Initialize the app with a service account, granting admin privileges firebase_admin.initialize_app(cred, { 'databaseURL': config_data['databaseURL']}) # getting url stored in config file
# check whether the entered value is empty if request.form['email'] == '' or request.form['password'] == '': error = 'Please enter values in the input field'
else: capture = 1 #if signup button pressed then set capture=1
# sending email,password for database operation(saving these values) push_data2Database(email, password)
# Throw msg if user data successfully saved successfulMsg = "Data Saved" flash(successfulMsg, 'success') # flash a msg after task is successfully done
return redirect(url_for('login'))
# otherwise if login button pressed then switch to login page elif request.form.get('login_action') == 'Login':
# Generate frames for Live streaming on Signup page def SignupVideoGen():
""" Opencv cam """ # cam = cv2.VideoCapture(2)
global capture, ImageArray
# Take user image: [OpenCV] while True:
ret, frame = cam.read()
# Convert the image into array and store into global np array variable ImageArray = np.array(frame)
if ret:
if (capture): # if capture button clicked then capture has set 1 and click img
capture = 0 cv2.imwrite('img.jpg',frame)
# process the image and produce facial embeddings convert2FacialEmbeddings()
# encode the frame so that flask can return it as a response ret, buffer = cv2.imencode('.jpg',frame)
# convert each frame to a byte object frame = buffer.tobytes()
# concat frame one by one and show result yield (b'--frame\r\n' b'Content-Type: image/jpeg\r\n\r\n' + frame + b'\r\n')
# return Frames to the flask signup template @app.route('/video_feed') def video_feed(): return Response(SignupVideoGen(), mimetype='multipart/x-mixed-replace; boundary=frame')
################# """Login Page"""
#################
# Step 3 # Task: # 1. check if current user email exist in db. # 2. If yes then fetch the appropriate embedded feature for facial verification @app.route('/login', methods=['POST', 'GET']) def login():
global currentUser_embedded_face, dbUser_embedding, error, database
# refrencing the database ref = database.reference('/Users')
if request.method == 'POST':
# pressed forget password key if request.form.get('forget_action') == 'Forget Password':
if request.form['email'] == '': error = 'Please enter your Email Address'
# 1.check if email exist in db
else:
# get the email id from current user email = request.form['email'] print(email)
# creating list of child node of db child_id = [] for i in range(1, 5): # lets us assume we have only 5 users in db child_id.append(str(i))
# search for child node for i in child_id: user = ref.child(i).get()
if user != None:
if user['email'] == email: # searching current user in db via email print('found email at child node:', i)
# 2. Fetched appropriate Face embidding using emailID as unique identifier encodded_embedding = user['embeddings'] print(type(encodded_embedding))
# Step 4 # If current User is the existing User in the DB...Then for auth forget password page """ Forget Password Page: 1. Consist of Live streaming Face detection. 2. Button to click the photo for verification """ @app.route('/forget_password', methods=['GET', 'POST']) def forget_password():
global mcapture global euclid_distance, cosine_similarity, threshold_cosine, threshold_euclid
if request.method == 'POST':
# if Verify button pressed Start verfication process(MTCNN, Emebedd, Euclid) if request.form.get('verify_action') == 'Verify Image':
print("Verify button clicked") mcapture = 1
# use try except block to release lock because if release function run done twice then error arises. try: forgetPage_lock.release()
mainThread_lock.acquire() #blocking main thread to avoid ForgetPassVideo run at first call
# Generate frames for streaming on ForgetPassword page """ It performs: 1. Live streaming with Face detection. 2. Facial Verification """ def ForgetPassVideoGen():
global ImageArray, mcapture, x, y, w, h, currentUser_embedding, imgArray global euclid_distance, cosine_similarity, threshold_cosine, threshold_euclid
# after generator run once lock the thread forgetPage_lock.acquire()
# Using Haar cascade for realtime face detection # [Motive] -> To display user its detected face which guide them # to click image at correct timing.
embedded_face = embedded_face[0, :] # [0,:] -> to bring the shape from (1,2048) to (2048,)
return embedded_face
##################################### # converting np.array in json format
#####################################
# used for converting numpy array emmbedded feature json for pushing in db class NumpyArrayEncoder(JSONEncoder): def default(self, obj): if isinstance(obj, np.ndarray): return obj.tolist() return JSONEncoder.default(self, obj)
# Push user data into database def push_data2Database(user_email, user_password):
""" Connecting to database """ global database
""" get user data for storing """ embedded_face = convert2FacialEmbeddings() email, password = user_email, user_password
""" Store values into database """
""" We Have three value to store: 1. Email as string 2. Password as String 3. Embedded face as array, so convert into json and then save to firebase. """
### Before pushing np.array embedded_face to db convert it to json ##
# Serialization embedded_face numpyData_embedding = {"embeddings": embedded_face} encoded_EmbeddedData = json.dumps(numpyData_embedding, cls=NumpyArrayEncoder) # use dump() to write array into file
# prepare data to insert data = {'email': email, 'password': password, 'embeddings':encoded_EmbeddedData}
# created reference of database ref = database.reference('/Users')
# Save data into database ref.child('1').set(data)
"""Small Note""" # activate thread2 when we are at ForgetPassword_page # else, keep it lock # lock.acquire() -> lock thread # lock.release() -> unlock thread
The threading concept was used in this project in order to make the two tasks run separately that is forget_password() function and ForgetPassVideoGen().
Earlier without threading, forget_password() runs only once because of which global variable euclid_distance, cosine_similarity, threshold_cosine, and threshold_euclid gets updated in ForgetPassVideoGen(). This does not make the forget_password() function aware of these variable changes.
Now the question arises why do we want forget_password() to get aware of these changes? because after all, the forget_password() runs over the server and can perform a page switching to the home page after successful verification. whereas ForgetPassVideoGen() is a generator function whose task is to generate multiple frames for live streaming.
So to avoid this problem we have to use the threading concept to run both of these tasks parallelly over 2 independent threads.
At the same time, we used the lock and release concept of multi-threading to have control over their activities during the program.
With the threading concept if you would have noticed in forget_password() we made the function sleep for 4.5 sec, by that time the ForgetPassVideoGen() updates the global variables euclid_distance, cosine_similarity, threshold_cosine, and threshold_euclid on a different thread and then these updated variables can be utilized back in forget_password() function for the verification process after sleep duration is over.
