Using IoT to Smart Security Door System with Face Tracking Camera ()
1. General Introduction
1.1. Introduction
Humans have always wanted to have a sense of security and prevent others from entering their areas and spaces throughout centuries. They try to hide their precious belongings and important items by using locks and any other sort of security. It was believed that the first lock found was made in ancient Egyptian civilization 4000 DC ago [1], and from that time till now, it has evolved to be more secure and provide more security to humans.
The modern pin tumbler lock, or the Cylinder lock, is one of the most important inventions and discoveries in decades. That was invented in 1861 by Linus Yale Sr. This one has a key with a specific shape to make the lock open only with keys that have the same shape.
Now in digital age and modern technology times, smart doors are introduced to society. They depend on internet, and humans control their locks from distances and monitor them. Smart doors can send an alert to user in different ways, which brings additional security than normal doors. Smart locks do not require holding any physical keys. They are considered as a part of a home Automation. Another advantage of smart door is that it can control the access of people to enter through the door and monitor their actions. [2]
1.2. Introduction to IoT
For many years, people have always been interested in collecting information and data. Humans depend on internet every day in their daily lives nowadays and it has become a major means of communication with others. Now what happens if we want to connect every device that has on/off switch, every object that is able to access internet, human beings, everything to all, with data sharing from one to all, through huge and giant network.
2. System Design and Its Components
2.1. Introduction
Several applications of IoT require basic components and internet connection. In this chapter, the components of the system and the block diagram structure are presented, including the process of controlling and monitoring the system within IoT.
2.2. The Block Diagram and Its Components
The system’s block diagram is shown in Figure 1. The lock is controlled by the microcontroller that is connected to fingerprint, keypad and camera. This type of controller has the ability to connect to internet and send data to a certain server. The user can have control of the microcontroller through an application.
The diagram block of the system was designed based on putting the controller in the middle, and then the controller was connected to the server by a WIFI router, and then the user was connected to the controller through Biometric Door Lock such as fingerprint machine or password..etc. Accordingly, we can get closed loop.
2.2.1. ESP-32 Microcontroller
In this study, ESP32 WROOM Module is chosen (specifications shown in Table 1), as shown in Figure 2. This chip is called “The chip of the future”, and it is widely used in IoT applications and projects. It has several advantages such as built-in Wi-Fi and Bluetooth, many pins for input and output, and 10 touch sensors that can be used in different applications. It is programmed by Arduino software IDE.
Why do we use ESP 32 Series of microcontrollers?
The ESP32 Series of microcontrollers is the most popular among its siblings. They boast an extensive range of features: Dual-Core Processor, Wireless Connectivity, Low-Power Modes, Rich Peripherals, Analog-to-Digital Conversion
Figure 1. System’s block diagram.
Table 1. ESP-32 WROOM module specifications.
Main Processor |
Tensilica Xtensa 32-bit LX6 |
Clock Frequency |
240 MHz |
Operating Voltage |
3.3 V |
DC current per I/O pin |
40 mA |
GPIOs |
28 Pins |
Analog to digital converters |
6 Pins |
Digital I/O pins |
14 Pins |
SRAM |
520 KB |
ROM |
448 KB |
Flash Memory |
16 MB |
Operating temperature |
−40˚C to +125˚C |
Price |
11 Jordanian Dinar |
Figure 2. ESP32.
(ADC), Secure Boot and Flash Encryption, Support for Various Protocols, Extensive Memory, RTOS Support, OTA (Over-The-Air) Updates, Bluetooth Audio and Classic Bluetooth, Rich Development Ecosystem and Cost-Effective that is making it accessible to hobbyists and professionals alike “this feature is applicable also on ESP32 Camera”.
In addition, THE ESP32 Development tools and frameworks are ESP-IDF (Espressif IoT Development Framework) is the official software development framework for Espressif’s series of microcontrollers. It provides a comprehensive set of libraries, tools, and documentation for developing applications on these chips.
ESP-IDF is designed to make it easier for developers to harness the full potential of the ESP32 microcontroller and build a wide range of IoT and embedded applications. [3]
Note: The 5 V pin can be supplied with a Regulated 5 V, this voltage will again be regulated to 3.3 V through the on-board voltage regulator. Remember ESP32 operated with 3.3 V only.
Reasons for Choosing This Type of Microcontroller:
Wi-Fi & Bluetooth Dual Mode: The combination of Wi-Fi, Bluetooth, and Bluetooth LE allows the microcontroller to target a wide variety of applications, making it truly powerful [4].
High Integration: With minimal PCB specifications, the chip adds priceless functionality and versatility to your applications because it is embedded with antenna switches, RF balun, power amplifier, low noise receiver amplifier, filters, and power management modules [4].
Configurability and Customization: Different antenna configurations (e.g. PCB antenna, antenna connector) and flash sizes are available for ESP32 modules to meet the needs of various applications [4].
Application-Ready: All modules in the ESP32 Series have a broad operating temperature range of −40˚C to 105˚C and are suitable for commercial use [4].
2.2.2. Camera
With ESP32-CAM (specifications shown in Table 2), this chip is from ESP32 module family, it has fewer GPIOs than the ESP-32, because many of the GPIOs are used internally for the camera and the microSD card port. Figure 3 shows how it looks. Its main focus is on IoT projects and systems, this board allows you to create a localhost server for video steaming with a lot of resolutions, with external storage micro-SD card slot up to 4 GB with advanced functions like image tracking, and the camera is OV2640 module 2 MP.
In this project, we will use it as a second microcontroller, and it allows us to have visual feedback in case of incorrect entry of the system.
Figure 3. ESP32-CAM.
Table 2. ESP32-CAM module specifications.
Main Processor |
Tensilica Xtensa 32-bit LX6 |
Clock Frequency |
160 MHz |
Operating Voltage |
5 V |
GPIOs |
9 Pins |
Photosensitive Array |
1622 × 1200 |
Resolution |
2 Megapixel |
SRAM |
520 KB |
ROM |
448 KB |
Flash Memory |
16 MB |
Operating temperature |
−20˚C - 85˚C; |
Price |
Jordanian Dinar |
2.2.3. Viewing Pictures
In this system, there are different platforms in order to view the data the captured Images:
Telegram (Figure 4):
Using the telegram bot, we can also view the captured images from the ESP32-CAM microcontroller. All you need is to have a telegram account which can be created very easily and for free on your smart phone.
Figure 4. Telegram application icon.
2.2.4. Mobile Application (Figure 5)
In this system, the Blynk applications will be chosen. This platform main focus is about IoT application. It allows the user to control and interact with different microcontrollers like Arduino, NodeMCU, ESP32, etc. in different ways, like, Bluetooth, USB, etc. The important one is via Wi-Fi [5]. With Blynk, we have the ability to control the system through your mobile.
Figure 5. Blynk application icon.
2.2.5. Fingerprint Scanner
Every person has unique friction ridges on their fingers, known as fingerprints. A fingerprint scanner is a hardware sensor that reads these unique fingerprints to verify a person’s identity. It is used to unlock content or services and is now common in smartphones and laptops. Figure 6 illustrates how a human fingerprint is converted into digital code.
Figure 6. Biometric fingerprint to digital code.
Figure 7. Optical fingerprint scanner.
The optical scanner, capacitive scanner, and ultrasonic scanner are the three common types of fingerprint scanners. Each type of scanner’s basic purpose is to image a picture of a person’s fingerprint and look for a match in its database. The accuracy of a fingerprint image is measured in dots per inch (DPI). Figure 7 shows the optical fingerprint scanner that will be used and Table 3 shows the Optical Fingerprint Scanner Specifications. [6]
Table 3. Optical fingerprint scanner specifications.
Supply voltage |
3.6 - 6.0 V |
Number of pins needed |
6 |
Operating current(Max) |
120 mA |
Storage capacity |
162 templates |
Work Temperature |
−20˚C - +50˚C |
Price |
Jordanian Dinar |
2.2.6. Lock
In this system, the electric solenoid lock (Figure 8) is chosen for locking mechanism. It is simple to install an electric door lock with only two weirs to be connected with the system. Table 4 shows the specifications of the electric solenoid lock.
Figure 8. Electric solenoid lock.
Table 4. Electric lock assembly solenoid specifications.
Size |
L 27 mm * W 29 mm * H 18 mm |
Voltage |
12 v DC |
Current |
745 mA |
Price |
JD |
2.2.7. Keypad (Figure 9)
A keypad is simply a group of buttons or switches, we can call them the keypad elements. Each element represents a single value assigned to it inside the program by the programmer. Keypads come in different dimensions, sizes, and shapes like 1 × 4, 3 × 4, etc. Figures below show the internal structure of 4 * 4 keypad sizes.
A 4 × 3 Keypad Membrane module (Figure 10) is chosen in this system in order to enter the code used to open the lock. Two buttons in this keypad are customized by programming to do specific things that will be discussed in next chapter (Table 5, Membrane Keypads Specifications).
2.2.8. Battery (Figure 11)
12 V 4000 mAh rechargeable LITHIUM-ION battery (Table 6 shows the specifications of LITHIUM-ION Battery) with charger is chosen to supply the solenoid lock circuit and the whole system in when main power failure.
Figure 9. 4 × 4 keypad internal structure.
Figure 10. 4 × 3 membrane keypad internal structure.
Table 5. Membrane keypads specification.
|
Rigid Keypads |
Operating current max. |
20 mA |
Buttons Feeling |
Hard button (Mechanical) |
Installation |
Has screw holes |
Operating temperature |
−20˚C - 60˚C |
Price |
Jordanian Dinar |
Figure 11. LITHIUM-ION battery.
Table 6. LITHIUM-ION battery specifications.
Voltage |
12.6 V |
Actual Capacity |
4000 mAh |
Rechargeable |
Yes |
Rechargeable Times |
More than 500 times |
Price |
20 Jordanian Dinar |
Output Current |
A |
2.2.9. 12 V DC OMRON Relay 8PIN DPDT (Figure 12)
This relay is used to switch from main power supply to the battery in case of main power failure. Table 7 shows the 12 V DC Omron relay 8PIN DPDT specification.
Figure 12. 12 V DC OMRON relay 8PIN DPDT.
Table 7. 12 V DC Omron relay 8PIN DPDT specifications.
Type |
MY2NJ |
Contact capacity |
AC 5 A 250 V DC 5 A 30 V |
Size |
27.6 × 21.5 × 34 mm |
Price |
Jordanian Dinar |
2.2.10. Adapter 12V 2A Power Supply AC/DC (Figure 13)
This adapter is used to power the whole system from the socket and Table 8 shows the specification of adapter 12 V 2 A Power Supply.
Figure 13. Adapter 12V 2A power supply.
Table 8. ADAPTER 12 V specifications.
Input |
100 to 240 V AC at 50 to 60 Hz |
Output |
12 V 2 A |
Rating |
24 W |
Price |
Jordanian Dinar |
1) Magnetic Switch (Figure 14):
This switch is used to indicate the status of the door if it’s closed or opened.
2) Push Button (Figure 15):
This button is used to open the lock from inside the door.
3) Voltage Regulator (Figure 16):
In order to operate the microcontrollers and components, we need this device to have 5 V for both ESP32-CAM and ESP32 for operation. This device makes the 12 V supplied from battery into 5-volt which is suitable for the needed devices (Table 9, 7805 Voltage Regulator Specification).
4) OLED Screen (Figure 17):
Used to show specific words during operation of the system (Table 10 shows OLED Screen Specifications).
Figure 14. Magnetic switch.
Figure 15. Push button.
Figure 16. 7805 voltage regulator.
Table 9. 7805 voltage regulator specification.
Input Voltage |
Min: 7 V DC Max: 25 V DC |
Current |
Operating Current: 5 mA Output Current Up to 1.5 A |
Price |
Ordanian Dinar |
Figure 17. OLED screen.
Table 10. OLED screen specifications.
Size |
0.91 inch |
Matrix Size: 128 × 32 (white, blue) |
128 × 32 (white, blue) |
Temperature: −40˚C to +70˚C |
Temperature: −40˚C to +70˚C |
Voltage |
3.3 V |
Price |
Jordanian Dinar |
5) Voltage Indicator (Figure 18):
Used to show the battery voltage as an indicator
6) Small Components to be used:
Figure 18. Voltage indicator.
Figure 19. Female DC power adapter.
Figure 20. Bread board jumper wires.
Figure 21. Heat sink.
Figure 22. LEDs.
Since design and product development are constantly looking for ways to improve human and consumer needs, the user may choose the cheapest and most functional product among competing products. The table below shows the estimated product prices; it can be said that 125 JOD is a reasonable price and competitive between other products mentioned before (Table 11 shows the prices of the total components).
Table 11. Prices of the components.
Component |
Price |
ESP32 |
11 JOD |
ESP32-CAM |
15 JOD |
Fingerprint Scanner |
45 JOD |
Battery |
20 JOD |
Keypad |
5 JOD |
Electrical Solenoid lock |
10 JOD |
12 V DC ORMON Relay |
6 JOD |
Adapter 12 V 2 A Power Supply |
4 JOD |
OLED Screen |
4 JOD |
Other components |
5 JOD |
Total cost |
125 JOD |
3. Operation and Analysis
Simulation programs with microcontrollers and components will be used to simulate some parts of the system and analyze the results using Proteus software and LT-spice. There will also be a discussion of power analysis and its simulations.
3.1. ESP32 Microcontrollers
As previously mentioned, the ESP32 microcontrollers are widely used in IoT applications. They are easy to use, powerful, with many new features that usual microcontrollers do not have.
3.2. System Modes
As the system starts in operation, first it will search for an internet connection, if there is an internet, it will complete its whole operation based on that, i.e., it will work on online mode, connection if not, it will work on offline mode. It should be noted that it will always search for internet connection.
3.2.1. Online Mode
If the system is connected to the Wi-Fi, first it will print on serial monitor that it’s connected to the Wi-Fi then connecting to Blynk application and a message will be shown on the application that is connected. [7]
Then the ESP32 will check if the system is closed, if it’s closed then also a message will be shown on Blynk “the door is closed. Then ESP32 will check if a push button is pushed (opened from inside) or not, if not the door will be closed. Other way it will be opened and a message will be shown on Blynk that it’s opened.
If the user placed his fingerprint on the scanner and matched the data that was already stored, the door will be opened and a message on app shows that it’s opened as well as the OLED screen will show (Door is Open)
If did not match, ESP32-Cam will take a photo and send it to the Telegram application and OLED shows (No Match!). It must be known that the default way of entering is by fingerprint.
If the user pressed the * key, its convert to the keypad, then a LED is turned on and a new PIN number is generated and sent to the Blynk (4 digits). The (#) is used to delete any entered digit, i.e., acts like backspace. If the PIN number is matched, the door is opened and OLED screen shows that also, if not a photo is taken and sent to Telegram application while OLED screen shows (Wrong Pin!).
In Online Mode, different commands can be done with Blynk application on the system operation like as shown in Figure 23: [8]
enroll_f: Adding a new fingerprint to the system.
del_f: Deleting a fingerprint.
list_f: List all the fingerprints.
unlock: Open the door.
keydef: To change the default key used in offline mode.
clc: To clear the terminal screen.
view_key: to show the default key for the user.
3.2.2. Offline Mode
If the system fails to connect to the Wi-Fi, it will be in offline mode operation, and it has less status.
Figure 23. Terminal commands.
If the user placed his fingerprint on the scanner and matched the data that was already stored, the door will be opened, and OLED screen will show (Door is Open).
If not, the OLED screen shows (No Match!), and the door is still locked.
If the user pressed the * key, it convert to the keypad, then a LED is turned on and the user should enter the default PIN which is (4 digits). While having the ability to delete the entered digit by (#) button. If the PIN number isn’t matched, (Wrong Pin!) is shown on the OLED screen.
3.2.3. Breaking Mode
If the door was opened by unusual way i.e., not by the mentioned ways, The system takes a photo and be sent to the application.
The ESP32 in this state, is in sleep mode while ESP32-CAM is on.
3.3. System Operation
The mentioned controllers are to heart of the system, and as long as the system has an internet access, ESP32 is connected with the Blynk application, while being ESP32-Cam connected with Telegram.
3.3.1. ESP32 Cam and Telegram
With the help of IoT and its advantages, and in the online modes, our system deals with sending the captured photos through the API and sent to a server from telegram which is specific to IoT applications. It should be noted that the picture is very fast when being taken and sent, not a considerable delay. Figure 24 below shows the captured images and being shown on the phone.
3.3.2. ESP32 and Blynk
As the whole system is being controlled by this microcontroller, it’s also being controlled with the phone application Blynk, as long as it’s s in online mode, and by meaning controlled, we can open the lock by using this application. In our system, we have a feature which is reports, the function of reports widget is to configure and customize data reports in CSV format, i.e., we have the ability to know when the door was opened in the mentioned format. i.e., shows the status history of the lock. It is also possible to be sent to your email and downloaded as an Excel sheet with time and date of the opening times as shown in Figure 25.
Figure 24. ESP32Cam and telegram.
Figure 25. Blynk reports.
3.4. System Physical Connection
Figure 26 below shows how would the Physical Connection be like, it can be seen that the ESP32 is connected to all the other components while having 3 connected lines with ESP32-CAM for different purposes:
1) ESP32 makes the ESP32-CAM take a photo and be sent to Telegram.
2) Face detection and recognition (Future Works).
3) The result of Face detection and recognition, if there is a matching, tells the ESP32 to open the lock.
Figure 26. Physical connection.
3.5. Simulation and Analysis
Proteus software is used to implement the architecture outlined in the earlier pages for more analysis. Figures below show the implementation of the two microcontrollers ESP32-Cam (Figure 27) & ESP32 in Proteus program (Figure 28). As mentioned before, child sheet method is used in Proteus due to missing library of microcontrollers.
Figure 29 below shows the overall system simulation using Proteus software. Unfortunately, ESP32 and ESP32-CAM do not have a library and a child sheet method is used in order to make them act like an Arduino. Also, the fingerprint as well as soloed lock are having the same problem and cannot be dealt with the previous method.
Figure 27. ESP32-Cam microcontroller in Proteus.
Figure 28. ESP32-Microcontroller in Proteus.
4. Conclusions
1) As IoT is becoming more permeate in all applications, home-automation is becoming a major aspect of IoT.
Figure 29. Overall system simulation.
2) The system presented in this project aims to provide a secure area for human beings and their belongings.
3) The proposed system has the ability to provide visual feedback in case of error entry attempts at a reasonable cost.
4) ESP32 family series is considered to be one of the best controllers in IoT applications due to their powerful features.
5) The searching of related work of our system gained us different ideas and yielded new concepts to be taken into consideration.
6) Many problems of Node MCU (ESP32 family series) in Proteus software are solved by using the child sheet method and labeling the wires while using the Arduino library.
7) Using the telegram application gives advantages to users such as access to information and the ability to communicate and receive data from the system.