My new laptop came with something my old one didn’t have: Windows Hello integration. In this case that’s an infrared (IR) camera with 2 IR emitters that can recognize your face and log you in automatically, even in the dark.
If you’re still not sure what Windows Hello is, listen to some enthusiastic clapping while Microsoft shows it to you:
My daily driver is Ubuntu though, and i could’t find any way of getting a similar thing working on there. A quick Google search shows I'm not the only one looking for this, but no actual active projects or even proof of concepts show up. Time to try this myself.
Accessing the IR camera and emitters
First of all I want to be able to open the IR camera. As it turns out, it’s incredibly easy in Linux.
The IR camera is recognized by Ubuntu as an USB camera, just like the normal webcam is. The emitters start working as soon as the camera is active, so even they aren’t a problem.
Using fswebcam we can capture and save frames from the device at /dev/video1 . Run the following command to make a snapshot:
fswebcam -d /dev/video1 photo.jpg
The resolution is either surprisingly small or wrongly reported by the underlying library: 340x340.
How do we reliably recognize faces?
Saving pictures of my face is great and all, but it would be nice to actually do something with it. For this I used an awesome python module called face_recognition, which is build on OpenCV and dlib.
This module can read a video stream directly from the webcam, which is ideal for this purpose. It can also dump an encoded version of the face of the user so we don’t have to store and process full images every time we use the script. In my tests it can process around 20 frames per second, which should be more than enough to quickly recognize me.
One small issue is that the brightness of the camera is very poor in the first 10 frames or so. Handing the algorithm a second version of my face model, calculated from a badly lit photo, gives it the power to flawlessly detect me.
PAM authentication on Linux
Linux has a lovely, file based, system to deal with all kinds of different ways to authenticate a user. It consists of a bunch of config files in /etc/pam.d which all represent different places where you could need to log in. For example, /etc/pam.d/sudo is responsible for the authentication process after you issue a sudo command.
One particular file is quite interesting for us. It’s called common-auth and is automatically called from all other files if authentication is needed. If we add a rule here it will get executed anytime we need to enter a user password.
Python headaches
The face_recognition module is written in python, so to be able to use that we need to write a PAM module. Good thing that there’s a script that works as a bridge between PAM and the python interpreter, descriptively called pam-python.
It brings with it a few really bad headaches though. First of all, the documentation of pam-python can be hard to read and is very centered around the idea that you already know everything there is to know about PAM. The best way i could find to learn it is to look though other scripts that use pam-python and to try and decipher what they are doing.
Secondly, it runs the python 2.7 interpreter. This would be no big issue normally, but the face_recognition module needs python 3.5. My not-so-elegant solution to this is to let a master python2 wrapper script start a child python3 script. The wrapper also manages the conversation with PAM.
And finally, because this script can be run in a system state where no user is authenticated (on the login screen for instance), the /home partition might not even be mounted and a lot of environment variables are not available. This complicates things a bit, but nothing a bit of trial, error and a lot of reboots can’t fix.
Letting the magic happen
After putting it all in a script it’s amazing how well it all works together. Errors in the script get passed to PAM and are shown to the user in a really friendly way, on the lock screen for example:
Just running sudo and not having to type a password, even in the dark, makes apt-geting a lot smoother. It’s the little things.
If you already have an IR camera with emitter already and want to try this out, the installation instructions can be found on github.
