In Part 1 I put together a short python script to automate the stitching together of 16 pictures into a single composite. Now I want to take some of the effort out of selecting the 16 pictures, in order to speed up the process

Each month my wife and I choose 16 of the best pictures involving our son from the hundreds we’ve taken. These end up on our fridge in a 4×4 grid. Although we take a picture of the pictures on the fridge, it always looks a bit screwed up and it’s nice to have a good version stitched together on the computer.

How can I minimise the effort needed to recreate a physical collage from a photo?

The problem

The problem looks a bit like this:

As part of the selection, we have up to 100 printed out into actual photos, so we can isolate the image files for those, but the refinement to the final 16 is done by hand.

1. Read in the individual images, and a picture of the ‘raw’ composite from the fridge
2. Break the raw composite down into 16 tiles
3. Check each of the individual images against the tiles
4. Present the user with the original tile and an ordered list of the ‘candidate’ images which are the best match
5. Loop for all 16, then stitch the final image together

1. Read in the images

I’m going to use a combination of OpenCV and Pillow for my image processing, the first step will be in OpenCV so I’ll read the files in like that.

I’ve created some objects to hold the data around each image – my foray into C# a few years ago got me thinking in Objects, and although I’m not sure it’s Pythonic (dahling), it helps me order my thoughts, and write more debuggable code

class Composite:
    imageName: str
    rawImage: Image.Image
    pilOutputImage: Image.Image
    outputImageArr: list
    tiles: list
    candidates: list
    readyToProcess: bool
    done: bool
    imageXSize: int
    imageYSize: int
    date: datetime
    
    
    def __init__(self, _imageName, _rawImage, _imageX _imageYSize):
        self.imageName=_imageName
        self.rawImage=_rawImage
        self.tiles=[]
        self.candidates=[]
        self.outputImageArr=[]
        self.readyToProcess=False
        self.done=False
        self.imageX=_imageX        
        self.imageYSize=_imageYSize
        self.pilOutputImage = Image.new('RGB', (_imageX4, _imageYSize*4),(255,255,255))
        self.date=datetime.datetime.strptime(_imageName.split(".")[0],'%b%y')
        
        
    def incompleteTiles(self):
        return [t for t in self.tiles if t.done==False]

    def pilImage(self):
        return Image.fromarray(self.rawImage[:,:,::-1].astype('uint8'))
    
    def buildImg(self):
        for tile in self.outputImageArr:
            Image.Image.paste(self.pilOutputImage, tile.pilImage(), (self.imageXtile.xPos, self.imageYSize*tile.yPos))
            
    def addCompleteTile(self, tile):
        self.outputImageArr.append(tile)
        Image.Image.paste(self.pilOutputImage, tile.pilImage(), (self.imageXtile.xPos, self.imageYSize*tile.yPos))

This is an example of the ‘Composite’ object – there will be one of these for each of my Raw Composite files (the fridge images). This holds the rest of the information relating to the Tiles (the 16 sub images) and the Candidate images. It also has the logic to build the final output image.

class Tile:
    xPos: int 
    yPos: int
    rawImage: Image.Image
    candidates: list
    done: bool
    
    def __init__(self, _x, _y, _rawImage):
        self.xPos=_x
        self.yPos=_y
        self.rawImage=_rawImage
        self.candidates=[]
        self.done=False
    
    def pilImage(self):
        return Image.fromarray(self.rawImage[:,:,::-1].astype('uint8'))

This is a Tile. It knows what it looks like in a raw form, it’s position on the 4×4 grid, and the candidate images which could be it

class Img:
    imageName: str
    rawImage: Image.Image
    error: float
    exifDT: datetime
    
    def __init__(self, _imageName, _rawImage, _dt):
        self.imageName=_imageName
        self.rawImage=_rawImage
        self.exifDT = _dt
        
    def pilImage(self):
        return Image.fromarray(self.rawImage[:,:,::-1].astype('uint8'))

Finally this is an image, I use it to keep some of the important info relating to the image in a single place. It’s got the concept of ‘Error’ in there, which will come from the matching code – the lowest error is our best candidate

This is my rough data structure, now I need to populate it

def loadImages(dir, size, type):
    compositeDir="C:/Users/alexc/Source/Repos/picStitcher/photos/"+dir+"/"
    returnList=[]
    for image_file in os.listdir(compositeDir):
        image = cv2.imread(compositeDir+image_file)
        img = Image.open(compositeDir+image_file)
        try:
            dt=img.getexif()[36867]
            dt=datetime.strptime(dt, '%Y:%m:%d %H:%M:%S')
        except:
            dt=datetime(1900, 1, 1, 00, 00)
        if type=="Composite":
            outObj=Composite(image_file, cv2.resize(image, (size[0]*4, size[1]*4), interpolation = cv2.INTER_AREA), size[0], size[1])
        elif type=="Img":
            outObj=Img(image_file, cv2.resize(image, size, interpolation = cv2.INTER_AREA), dt )

        returnList.append(outObj)
  
    return returnList

This function loads raw images (in OpenCV format – a numpy array) and creates an appropriate object to hold them. Theres some extra data (like date, and image size) which is useful later on.

2. Breaking out the tiles

This is a bit of a fudge. Because the pictures taken from the fridge are imperfect, it’s difficult to cut out the exact image, but the hope is that the matching process will work ok with a rough target.

def detile(name, image, imageXSize, imageYSize):
    outArr=[]
    for x in range(0,4):
        for y in range(0,4):
            im=image[y*imageYSize:(y+1)*imageYSize,x*imageXSize:(x+1)*imageXSize]
            outObj=Tile(x, y, im)
            outArr.append(outObj)
    
    return outArr

So I just loop through a 4×4 grid, cut each image and make a new tile out of it. Once that’s done, it gets added to the Tiles list in the Composite object

3. Checking for matches

Let’s assume I have 100 candidate images for my 16 tiles. That’s 1,600 checks which need to be done, and that’s going to take some time. This seemed like a good excuse to have a play with Threading in python

I’ve tried threading before in C#, and it felt messy. I’ve never really had proper training, so I just google my way through it, and I wasn’t ready to handle the idea that information would be processed outside of the main thread of my program. It was really useful (e.g. for not having the UI lock-up when heavy processing was happening), but I didn’t properly understand the way I was supposed to handle communication between the threads, or how to ‘safely’ access common sources of info.

This time, things are a bit simpler (and I’ve been thinking about it for a few more years).

I need my threads to work on the checking algorithm

    def wrapper_func(tileQueue, rawImages):
        print("thread")
        while True:
            try:
                tile = tileQueue.get(False)
                for image in rawImages:
                    result = cv2.matchTemplate(tile.rawImage, image.rawImage, cv2.TM_SQDIFF_NORMED )
                    mn,mx,mnLoc,mxLoc = cv2.minMaxLoc(result)
                    output=Img(image.imageName,image.rawImage, 0)
                    output.error=mn
                    tile.candidates.append(output)
                tileQueue.task_done()
            except queue.Empty:
                pass

Here I have a queue of tiles, and my raw images. Each tile needs to be checked against all the images, but there’s no reason why they can’t be checked independently of each other. This ‘wrapper function’ – which still uses the function name from a StackOverflow answer I cribbed (this one, I think) – is run in 8 different threads, and checks each tile against all the raw images, appending a candidate to the tile each time.

I’m pretty sure this qualifies as thread-safe. The append operation just adds new stuff in, and I’m not reading from anywhere in the list.

    def check_matches(composite, rawImages):
        print("check_matches.. may take a while")
        tileQueue=queue.Queue()
        
        for tile in composite.tiles:
            tileQueue.put(tile)
        for _ in range(8):        
            threading.Thread(target=wrapper_func, args=(tileQueue, rawImages)).start()
        
        tileQueue.join() #wait until the whole queue is processed
        for tile in composite.tiles:
            tile.candidates.sort(key=lambda x:x.error)  #sort the candidates by error (the error of the match)
            
        composite.readyToProcess=True
        print(composite.imageName+" ready to process")

Stepping out a layer, this processes an entire composite. First splitting into a queue of tiles, then fire this into 8 threads. Once the queue is processed, sort the candidates, then set a ‘ready to process’ flag for the UI to respond to.

    def loop_composites(): 
        thread.join()
        print("looping composites")
        global readyState, rawComposites, rawImages
        for composite in rawComposites:
            print(composite.date)
            compMinDT = composite.date-timedelta(days=30)
            compMaxDT = composite.date+timedelta(days=30)
            rawImagesFiltered=[i for i in rawImages if (i.exifDT>=compMinDT and i.exifDT<=compMaxDT) or i.exifDT==datetime.datetime(1900, 1, 1, 00, 00)]
            matchThread = threading.Thread(target=check_matches, args=(composite, rawImagesFiltered))
            matchThread.start()
            matchThread.join()

This is the outer layer of the loop. I’ve added in some filtering logic which uses the exif data to say when the photo was taken. All fridge photos are from a 1 month period (aligned to the middle of the month), so I can use the composite date and check for a period either side. This should reduce the processing if I want to run many months at the same time.

4. The UI

I’ve decided that there’s no way this process will work without a human in the loop – the whole point is to have the correct image at the end!

I’m using Flask to serve a simple web front end which will present the user with the images, and allow them to choose the correct one

On the left you can see the original composite image – you can see that the overall images are a bit skewed, and there’s significant light distortion on the left hand of the image

In the middle is the 1st tile cut from the raw composite

Next is the first match candidate (it’s correct! yay!)

Then is a placeholder for the newly constructed image

5. Loop, complete, stitch and save

Clicking ‘correct’ adds the candidate to the output image array, and pastes the image into the right hand ‘progress’ image

6. How did we do

The original reason for creating this program, other than the fun of making it, was to reduce the time it took to create the composites. Let’s consider two metrics:

1. How many images did I need to click through to get to the final composite
2. How many times did the program get the right image first time

	Jun 21	Jul 21
Total Clicks	202	39
Right first time	11	10

With around 100 photos per month, I only needed to check 5-6 of them, and worst case I needed 40 clicks for each of the remaining photos. I think that’s pretty good.