Project 3: Face Morphing (GitHub repo)
Overview
This project focuses on image morphing, where the shape and color of one image are transformed into another. The project is divided into four main parts:
- Morphing Methodology and Implementation: Covers the techniques and implementation details of the image morphing process.
- Annotation Tool: Describes the custom tool I developed for annotating and obtaining image correspondence points.
- Result: Shows the reuslt of a morhphing sequence (animation) and an experiment using morphing techniques to create a "mean face" from multiple images.
- Interactive Warping Tool: Demonstrates an interactive tool that allows users to warp images by dragging keypoints in real time.
Morphing Methodology and Implementation
Weighted Average of Pixels at the Same Coordinates
Morphing combines image warping (changing the shape) and cross-dissolving (changing the color). The simplest method for cross-dissolving is the weighted average method: I(t) = (1-a(t))xI1 + a(t)xI2, where I(t) is the output image at time t and I1 and I2 are the source and target images, respectively. The weight a(t) ranges from 0 to 1, such that when a(t) = 0, the output is the source image, and when a(t) = 1 , it is the target image.
The limitation of this method is that it only computes pixel values without considering the underlying semantics of the image, such as the structure or meaning of objects. Therefore, to improve results, we first need to find corresponding pixels in both images that share similar semantics.
Weighted Average of Pixels with Similar Sematics (Correspondence)
Given a pair of pixels/points (called correspondence) (p1,p2) that share similar sematics, but likely have different coordinates, the weighted average point is computed as p(t) = (1-a(t))xp1+a(t)xp2. Using this approach, we can compute a set of "weighted-average" correspondences from two sets of corresponding points.
Triangular Mesh and Warping
We can compute the trajectory of all correspondence points (keypoints), but we still need to handle the other pixels/points. One way to do this is by creating a triangular mesh using the keypoints to cover the entire image.
The triangle mesh will depends on the keypoints coordinate the triangulatino algorithm used. Theorecitally there's no restriction on keypoints coordiantes, we can use kepyoints on I1, I2 or any I(t) with a(t) within the rnage of[0,1]. But in practise, we decided to use the "mid-image" to genearete the triangular mesh (i.e. use keypoints on I(t)=0.5xI1+0.5xI2).
This gives us a set of "correspondence triangles." For each pair of correspondence triangles, we can compute an affine transformation matrix A such that [p1',p2',p3'] = A*[[p1,p2,p3],[1,1,1]], where p1,p2,p3 are the coordinates of the correspondence keypoints as 2x1 column vectors. Note that on the right side of the equation, "[[p1,p2,p3],[1,1,1]]" represents the homogeneous coordinates of the keypoints. Therefor A is a 2x3 matrix. This can be compactly written as P' = AxP, and A can be solved as A = P' x inverse(P).
Rather than computing A as P'=AxP, we can also compute it as P=A'xP', which is known as the inverse transofrmation. For image at time t (denoted as I(t)), if we want to find the corresponding coordinates of a pixel/point in "correspondence triangle", we can use this inverse transfomration to obtian the coordiante in the source image (in our case, I1 or I2).
In this implementaion, we use the "correspondence triangles", the
transformation matrix and
skimage.draw.polygon to obtain the correspondign pxiel coordiantes.
Weighted Average of Color (Cross-dissolving)
The final step to complete the morphing process is cross-dissolving. At time t, using the keypoints and the (inverse) transofmraiton matrix, we can determine the corresponding pixel in images I1 and I2 for each pixel of I(t). Now we simply perform a weighted average of the pixel color, whcih should be straightforward.
Annotation Tool
To perform face morphing, we need to obtain several pairs of correspondence points from the
images we want to operate on. For this purpose, an annotation tool was developed. The output of
the annotation tool is a Python built-in dictionary, where the key is a unique ID, and the value
is a custom class object (Correspondence). This data structure is then saved as a
pickle file. When used for morphing, the data structure is converted into two lists
containing the keypoints' coordinates (e.g., [[x1, y1], [x2, y2], ...] or [[col1, row1], [col2,
row2], ...]).
Result
"Mid-face"
As we mentioned in morphign methodoloy, we use the "mid-image" (or "mid-face") to genearete the triangular mesh (warping weight = 0.5). Also, cross-dissolving weight = 0.5 was used to compute the "mid-face" image of me and the target image (i.e.,averaging keypoitns and color).
"Average (Brazilian) face" and the "Brazilian Chinese"
I used the FEI Face Database to compute an avearge face. The FEI face database is a Brazilian face dataset that contains two categories: neutral faces and smiling faces, each has 200 images.The method involves computing a 'mid-face' image by cross-dissolving 200 images from each category. This process uses a template of 46 keypoints, as shown in the figure below.
Caricatures: Extrapolating from the mean
What we are doing so far can be thought of as interpolation: we interpolate points between two sets of points, using weights in the range of [0,1] for the weighted average. However, we can also create caricatures by extrapolating the points between images, allowing the warping weight to be less than 0 or greater than 1.
Interactive Warping Tool
An interactive warping tool was developed using morphing and warping techniques. It allows users to add and edit keypoints, as well as drag the keypoints to warp the image.
