Back in 2017, a close friend and I were fascinated by clips like this one:
Noah (whom I will refer to later) took the same selfie every day for 20 years and animated it so you could watch him
age and change over two decades in just a few minutes. We were intrigued by the idea of having such a video
for ourselves. So we agreed to try it out too and started taking a selfie every other day.
The beginning
I soon realized what many influencers had before me: 'Taking selfies ain't easy.'
Every photo would be one frame in the final sequence. For us human watchers to grasp the changes in a fast
GIF, we need the changes to be limited to certain areas and colors. If the lighting, rotation, background, facial
expression, facial covers, etc. change too much, the video will be chaotic.
Just look at this GIF:
A chaotic GIF of selfies with too many changes to grasp the similarities.
I suppose that is why Noah on YouTube looks like he's "on the verge of crying" (one of the top comments).
A stern expression might make for a better GIF since different
smiles disturb our perception when the frames change fast.
So, starting out, I didn't intend to look particularly happy either.
As you will see, years later I - somewhat - changed that.
What also caught my eye was the lighting. You normally
do not think about this but in most places the face is lit from exactly one direction,
casting a shadow on one part of the face while the other part is bright.
In his video, Noah chose one side to be dominantly lit while the other side stayed dark.
I wanted to
take the selfies everywhere and not always in the exact same setting, so I tried to
look straight into the main light source. Then again, I had to watch out that
my arms and the phone didn't cast a shadow on my face. Also, the difference between
warm, natural light and cold, artificial light can be irritating when the changes
in the GIF are frequent. I had to exclude lots of selfies because they didn't fit
their neighboring frames.
In the beginning, the process of daily selfie taking was not very rewarding but
neither was it too tedious. To keep ourselves motivated, my buddy and I made a bet:
whoever missed taking a selfie for two consecutive weeks would need to join on a festival trip
chosen by the other.
Some background: back then we were attending four or more festivals a year.
Yet, our taste in music differed sometimes, so it was worth a lot to finally being able to force a friend to
join a more niche kind of music event that fully fulfilled ones own musical taste.
The challenge
The months passed, and the selfies amassed in our galleries. Now, we were facing the
next challenge. How should we create such a video? I was just starting to program
more with Python back then but didn't even think about using it as a tool. Instead,
I manually tried to cut and resize all the images with the basic MacOS Preview app.
The thought of this exhausting process still makes my hands twitch.
Yet, after many hours, the result was not too bad (please keep in mind what
I said about the stern expression before judging my seemingly depressive mood):
The first GIF after one year (shortened).
The 'tool' I used to create the GIF from the individual photos was the MacOS built-in feature
to unite a selection of pictures as a GIF (I used a newer version of this feature
for the chaotic GIF at the top).
Not very cutting-edge but back then, it could include up to 50 pictures, which sufficed.
Yet, as the files were not compressed, the total file size grew to 400 MB later on! That is way too
much to include in a post like this but I could still show it to friends on my
laptop. By now, I create mp4 files instead of GIFs, which makes them at least 80 times smaller while
showing a way more detailed resolution at the same time.
Apart from alignment, handling the individual files was tiring as well. I even
messed up the order of the photos. What bothered me even more, is the jumping around of the face.
The facial markers are not aligned between the pictures (which is even more extreme
in Noah's video on YouTube). So I did some more manual cutting but I could not really
make it smooth in reasonable time.
AI to the rescue
Fast forward four years to 2022, I still took a selfie every now and then but missed many weeks. I even lost
the bet and owe joining one festival (pity me)! One day in my holidays,
the thought about the project came to me. By then, I had become a Python professional, dealing
with terabytes of data every day. So I thought: why not try to automate the full
process and tinker with the new tools at my hand.
So I decided to try out OpenCV with its AI models for facial recognition.
The simple idea was to let the AI recognize the location of the face in the images.
Then I could resize, cut and realign all the selfies, such that the face is always in the same location.
It took only some tinkering with a Jupyter Notebook to make it work. I was immensely
impressed by how fast I could automate a process that would have taken me hours and
days (by then there were more than 200 photos) to do manually. Here's how it works:
The code is pretty straightforward since all I do is recognize the face location
and then set a fixed horizontal and vertical ratio that the photos will be aligned
to.
A selfie with the box printed that OpenCV recognized as the face area.
Let's say, I decided the horizontal margin ratio is 0.75 and the
face in one photo was recognized to be 100px wide and has a left margin of 40px (= distance from the face to the left side) and a right margin
of 50px (i.e. the face is not exactly in the center but a bit to the left of the photo, like in the selfie above).
Then, the horizontal margin ratio, for me, means that I want 75px (= ratio * face width) both
on the left and right of the face, so the face ends up in the center. Since the left margin
is only 40px and I need 75px, I create a black left-padding of 35px. On the right,
I also want 75px but have 50px, so I add 25px here. If the margin was greater than
what I want, I cut from the side of the image. All this is fairly easy to do with
OpenCV.
The same selfie with left and right padding.
I rather cut than introduce lots of black padding in the photos. So I did a small analysis
of the ratios between the face and both the left and the right margin.
The result was this:
A scatter plot of left margin ratio to right margin ratio.
The margins roughly range from 0.4 to 1.4. Therefore, I set a horizontal margin ratio
of 0.75 so lots of photos are cut and fewer photos get a padding.
Interestingly, the ratios seem to align quite well to the center, but judging from the
top right corner of the plot, I tend to have a slightly higher margin on the right than left.
This is most likely because I always hold my phone with my right hand when taking a selfie.
I also apply this process of alignment to the vertical dimension (as in the example
above). Then, OpenCV also offers out-of-the-box mp4 creation, which makes it possible for me to create sequences of
hundreds of frames with small file sizes and higher resolution than GIF itself.
The video you see below has 229 frames in 36 seconds and 6.6MB. I removed many photos from this
post that I do not want to share publicly, but I ended up taking one photo about every
ten days on average.
The result
The final result, which you were hopefully looking forward to by now is here:
The current sequence after 7 years.
Looking at it, it's a personal journey through my life. Within these past seven years,
I changed my environment a lot: I lived abroad four times on four continents and moved
within Munich six times. This reminds me that the only constant is change. Exciting!
I admit that I am not yet 100% satisfied with my setup, since some frames still break
the visual experience too much and I still struggle to find suitable lighting in many
settings. Yet, I am happy to have so many photos already and I might apply
some filters and new models on them in the future.
Let me know what you think!