Computer Vision Basics Geoff Hulten

Computer Vision Basics Geoff Hulten

Predictions in Computer Vision Classification Cat Eye Closed Dog Localization Segmentation Important Points Cat vs Not-Cat Important Points Eye vs Not Eye Eye Opened

Image Basics Greyscale Intensity 8-bpp 255 0 0 0 128 0 0 0 64 Encoding Normalize Red Channel 255 Green 255 255 Channel Color Encoding 255 0 Channel 0 0 Blue 255 0 255 00 0 255 00 0 0 128 255 0 0 0 64 0 0 0 0 0.5 0 0 0 .25 Intensity 0 255 1.0 To Grey Intensity 1.0 .21 .21 .72 .03 .21 .72 .72 .01

Indexing Image Data x 0 1 2 1.0 0 0 1 0 0.5 0 2 0 0 .25 y 0 X 0 Y 0 X 1 Y 0 X 2 Y 0 X 0 Y 1 X 1 Y 1 X 2 Y 1 X 0 Y 2 X 1 Y 2 X 2 Y 2 1.0 0 0 0 0.5 0 0 0 0.25 𝑖𝑛𝑑𝑒𝑥 (𝑦 𝑤𝑖𝑑𝑡h ) 𝑥 from PIL import Image from PIL import Image image Image.open( path ) image Image.open( path ) pixels image.load() intensity image.getpixel( ( 1, 1) ) / 255.0 intensity pixels[1, 1] / 255.0

Blink Image Pipeline Load Grey Normalize Size 24x24 Crop Region of Interest Localize 1.0 0 0 0 0.5 0 0 0 .25 24x24 Intensity Array 2.3 -.5 Not -.5 in.87 Homework -.5 -.5 Framework -.5 -.5 .16 24x24 Normalized Array

Very Basic Image Features Define the Region Select the Property Whole Image Intensity Grids Response to Regions of Interest Relative to Points of Interest Gradient Wavelets Select the Conversion Average Min/Max Histograms

Intensity Features Example Features: Region Whole Image Property Intensity 1.0 0 0 0 0.5 0 0 0 .25 Average: 0.194 Max: 1.0 Min: 0.0 Hist0-.2: .666 Hist.2-.4: .111 Hist.4-.6: .111 Hist.6-.8: 0 Hist.8-1:.111 Example Features: Region Middle Column Property Intensity 1.0 0 0 0 0.5 0 0 0 .25 Average: 0.166 Max: 0.5 Min: 0.0 Hist0-.2: .666 Hist.2-.4: 0 Hist.4-.6: .333 Hist.6-.8: 0 Hist.8-1: 0

Selecting Regions Whole Image Regular Grid 8 Intensity Features 32 Intensity Features Region of Interest 8 Intensity Features Using Localization 24 Intensity Features Combinations If using: Avg, Min, Max, 5 histograms 72 Intensity Features Feature selection: By region By feature By feature type

Gradients 𝐺𝑟𝑎𝑑𝑖𝑒𝑛𝑡 ( 𝑥 ) 𝐼𝑛𝑡𝑒𝑛𝑠𝑖𝑡𝑦 ( 𝑥 1 ) 𝐼𝑛𝑡𝑒𝑛𝑠𝑖𝑡𝑦 (𝑥 1) Pixel Intensity Image X-Gradient Y-Gradient 1 0.25 0.9 0.25 Gradient (on second axis) Pixel Intensity 0.25 0.25 0 0.8 0.7 0.6 0.5 -0.5 0.4 0.3 0.2 -1 0.1 0 0 1 2 3 4 5 6 7 8 X Location Image X-Gradient Y-Gradient

Features from Gradients Whole Image Regular Grid Region of Interest Using Localization 16 Gradient Features 64 Gradient Features 16 Gradient Features 48 Gradient Features Example Features: AverageX: 0.194 MaxX: 1.0 MinX: 0.0 HistX0-.2: .666 HistX.2-.4: .111 HistX.4-.6: .111 HistX.6-.8: 0 HistX.8-1:.111 AverageY: 0.23 MaxY: 0.4 MinY: 0.1 HistY0-.2: .2 HistY.2-.4: 0 HistY.4-.6: .2 HistY.6-.8: 0 HistY.8-1:.6 Combinations 144 Gradient Features

Convolutions 3x3 Filter 0 0 0 -1 0 1 0 0 0 Response Convolve Intensity Data 0 0 1 1 1 0 0 1 1 1 0 0 1 1 1 0 0 1 1 1 0 0 1 1 1 0 0 1 1 1 0 0 0 0 0 0 1 1 0 0 0 1 1 0 0 0 1 1 0 0 0 1 1 0 0 0 0 0 0 0

Sobel Edge Detection Simple X Gradient Sobel X Gradient 0 0 0 -1 0 1 -1 0 1 -2 0 2 0 0 0 -1 0 1 Simple Y Gradient Sobel Y Gradient 0 1 0 1 2 1 0 0 0 0 0 0 0 -1 0 -1 -2 -1

Wavelet Features Gabor Wavelets Haar Wavelets Neural Networks?

Summary of Basics of Computer Vision Basic Predictions Classification Localization Segmentation Construct Features With Region Intensity or response Statistics Preprocessing Pipeline Normalize: color, size Localize & crop Convert to intensity, normalize A lot of modern computer vision done with neural networks – we’ll get there