If we think of images as function of x, y in a 2D grid with smallest step size 1, we can define the derivative of an image with respect to x as the difference between to pixels in the x direction with a fixed y due to the definition of derivative since the smallest, delta is 1. Similarly, partial derivative with respect to y is horizontal differences. This allows us to define D_x = [1 -1] D_y = [1 -1]^T as derivative or Finite Difference Operators and convolve with the image to get the gradient of the image, i.e. edge detection. In my code, I get the partial derivative along x through a function called partial_x, which convolves the image with the Finite Difference Operator in the x direction, same for y. To calculate the gradient, I simply compute np.sqrt(np.square(partial_x) + np.square(partial_y)) and then show the gradient magnitude of the image as you can see below.
![Partial derivative with respect to [1 -1]](project2_images/partial_x.png)
Partial derivative with respect to [1 -1]
![Partial derivative with respect to [1 -1]^T](project2_images/partial_y.png)
Partial derivative with respect to [1 -1]^T
Gradient magnitude plot of the photogropher
Binarized gradient magnitude plot, magnitude > 0.32 for the above image
As we can see from the above images, the result is noisy, which means that the variance of the 2d distribution of the image is rather high. We can regulate this by convolving the image with a Gaussian Filter. A Gaussian Filter is simply a Gaussian distribution on a 2D kernel, and by adjusting the variance of the Gaussian Filter with an acceptable kernel size, the result will be less sharp and noisy than the identity filter we are using right now. Thus, taking the partial derivatives of the "blurred" image will result in a less noisy gradient magnitude plot. The reason I quoted blurred is to signify the effect that a smooth distribution blurs the image while the identity filter act like dirac delta (basically, if the variance of a normal distribution is 0, it kind of looks and acts like a dirac delta function, which is the identity filter in this case, by adjusting the variance, we can have various smoothing or blurring effects on the image). The images obtained by first convolving with a Gaussian Filter, then taking the partial derivatives with the Finite Difference Operator is shown below
Derivative of Gaussian in the x direction
Derivative of Gaussian in the y direction
Blurred image of the photographer by convolving with a Gaussian kernel of size 7 and variance 1
First blurred with the above gaussian filter, then convolved with the finite difference operators to obtain the above magnitude
In the above image, the gaussian kernel is first convolved with the finite difference operators, then the image itself (grayscale) is convolved with the first convolution
As we can see through the images, the image of the photographer is blurred through the Gaussian Filter, and then taking the finite difference operators through that results in a smoothed (blurred) out image. Since convolution is associative, we can achieve the same result by convolving the Gaussian Filter with the Finite Difference Operators, and then convolving it with the original unblurred image.
With the image unsharp mask filter, to get the details of the image, we blur the image with a gaussian filter, and then subtract this filtered image from the original image to get the high frequency components or the "details". And then, to the original image, we add these "details" with a scalar in front. This gives us a "sharpened" image. The images sharpened with this filter are below.
The original image of Taj Mahal
Details of the above image obtained by normalizing the subtraction of the original image by the filtered image by a gaussian kernel of size 25 and standard deviation 4
Original image sharpened with a the unmask sharp filter with a standard deviation of 4, kernel size of 25, and scalar alpha by 1.5
Original image of the Chrysler Building
Details of the above image obtained by normalizing the subtraction of the original image by the filtered image by a gaussian kernel of size 25 and standard deviation 4
Original image sharpened with a the unmask sharp filter with a standard deviation of 4, kernel size of 25, and scalar alpha by 1.7
Pink Floyd's Atom Heart Mother's album cover
Details of the above image obtained by the same procedure as others
Album cover sharpened with a the unmask sharp filter with a standard deviation of 4, kernel size of 25, and scalar alpha by 1.5
Image of the presidential yacht of Turkey
Blurred with a gaussian kernel of size 13 and standard deviation 2
Details of the above image obtained by the described procedure with a gaussian kernel of sigma = 3 and size = 6 * sigma + 1
Sharpened MV Savarona with alpha = 3
As you can see above, with this unmask sharp filter adds in the fine details and makes the images sharper. However; as you can see above, first blurring then adding details doesn't recover the image. The reason behind this is that when we first blur an image, we effectively apply a lowpass filter on the image, not passing high frequency components through. This results in information loss regarding high frequency components of the image and the details get more blurred as well, which can't recover the original image because the high frequency componenents aren't available.
Hybrid images are a byproduct of how we view images. Our eyes have limited resolution and sensitivity and thus we can resolve finer details when viewing an image close up and from far away, we see a 'silhouette' of the image, ie a blurred version. Viewing from far is like downsampling and viewing close is like upsampling. If we view high frequency components of an image (ie edges) as finer details, then we can show a filtered version of the image that is highpass filtered, where only high frequencies are visible. In order to see this image, the viewer has to view the image from a close distance. Inversely, to be able to better see an image from far we may lowpass filter it (which is like downsampling the image) and viewing this from far would enable the viewer to better view the lowpass filtered image, and since our eyes are sensitive to finer details (ie higher frequencies), viewing the image from afar would obscure the highpass filtered image. This results in an hybrid image where when viewed from far, the viewer sees the low frequency component, and when viewed from a close, the viewer sees the high frequency component. To achieve this effect, I lowpass filter the image that I want it to be seen from far by convolving the image with a gaussian kernel with an associated standard deviation sigma_1, and 6 * sigma_1 + 1 for the kernel size. To highpass filter an image, I first blur the image with a gaussian kernel with an associated sigma_2, and then subtract this low frequency component from the image to get the high frequency components and return this highpas filtered image. I then simply add the low frequency and high frequency components to get the hybrid image. The FFT plots below demonstrate that I achieved what I wanted with my favorite hybrid image of a very complicated watch as lowpass filtered image, and the galaxy as the highpassed image. The very white FFT plot demonstrates that higher frequencies were passed whereas the lowest of the frequencies (the center of the image) are rejected on the log scale, whereas for the lowpass filtered image, we see a very sharp white in the center since they represent the lower frequencies. All of my images are colored, however; through experimentation, I saw that color is important for the lowpassed image and not so important for the highpassed image because incorporating color on an highpass filter doesn't really change how it looks colorwise (still kind of gray). It is because to obtain the highpassed image, we make subtraction operation of two colored images, one of them is the original and the other one is the blurred. This results in attenuated colors.
"DerekNutmeg", in the above image, derek is lowpassed with a gaussian kernel of standard deviation 7, and nutmeg highpassed with the procedure described above with a standard deviation of 6
"PatekGalaxy" lowpass and highpass with standard deviation 5
"GalaLion", emblem lowpassed with sigma_1 = 15, lion highpassed with sigma_2 = 10
"GibsonStradi", guitar lowpassed with sigma_1 = 3, violin highpassed with sigma_2 = 6
"ShostaPotter", Harry Potter lowpassed with sigma_1 = 5, Shostakovich highpassed with sigma_2 = 4
My least favorite image among all of them is the gibson-stradivarius image because the violin still has some color components attached and having a white background increases the effect of color of the violin, which I don't like. And for that reason I view that image and the galataLion images as failures.
I implemented the Gaussian Stack by repeatedly applying a gaussian filter on the original image (convolving with the original image). On each level, I multiply the standard deviation by two, so the kernel size gets bigger as well and this results in a blurred image. The effects of the gaussian stack is below:
Gaussian stack of apple with an initial sigma of 3, levels 1 through 5 shown above
Gaussian stack of apple with an initial sigma of 3, levels 1 through 5 shown above
I then implemented the laplacian stack by taking subtracting blurred images (from the gaussian stack, I subtract more refined image from the blurred image stack[i] - stack[i + 1] to capture the details). Due to the nature of this problem, for both stacks to have the same size I also append the lest level of blur of the Gaussian Stack to the Laplacian Stack for both of stacks to have the same size:
Laplacian stack of apple
Laplacian stack of orange
I then created a vertical mask and apply the gaussian stack on the mask and multiply on each level the element of the laplacian stack with the blurred mask at that level. This results in some blurring near the fusing points of both images, which results in a smoother transition. We then add all the elements in this new masked stack, which results in this fused image "oraple". I also incorporate color by doing this masking process on each color channel. Images below show stacks from level 1 to level 5 and all gaussian kernels use a sigma value of 3 and kernel size of 6 * sigma + 1:
Laplacian stack of masked apple
Laplacian stack of masked orange
Final laplacian stack of the combination of the masked laplacians normalized
Oraple obtained with a laplacian stack of level 5 and sigma = 3 for all gaussian kernels used throughout
The visualizations are made through normalizing the images with the minmax normalization. I used the same mask by blending Istanbul and New York below. I also created new irregular masks for all the other pictures below. In order to accomplish this, I used Adobe Photoshop 2024 and chose the region that I wanted to include in both images with only two pixels black and white. Also, for alignment purposes, I downscaled the bigger image according to its ratio, and then cropped if necessary. I also incorporated color by masking each color channel of the pictures:
Image obtained by fusing skylines of Istanbul and New York with a level 5 laplacian stack where the sigma = 2 for all the gaussian kernels used to generate the image, the mask is the same as the oraple
Image obtained by using the mask below and using sigma = 2 throughout all gaussian kernels and a laplacian stack of level 5
\
Mask used to obtain the above image
Image obtained by using the mask below and using sigma = 1 using the mask below, since this is my favorite image, detailed analysis is included below as well. In creating this image, I again used a 5 level laplacian stack, and all the associated gaussian kernels have variance 1 and kernel size 7.
Back to main page