Category Archives: Hobby Coding

Namesphere.co.uk – or – how I tried to use GenAI to retire early, and failed

A long time ago, I was sat in a corner of an office, and was involved in a conversation with a guy who made a small passive income from a website which contained some relatively trivial information, but for reasons I won’t describe, was highly searchable.

When LLMs came along, I started to think about how I could try and do something similar. I’m not sure when the idea for a website about people’s names came to me, but it seemed to fit the brief:

  • Highly searched (‘what does my baby’s name mean?’, ‘what are names like X?’)
  • Low impact of wrongness (who’s to say what Alex really means?)
  • Potential to generate both specific content about names, then click-baity articles about those names

All toegther, this seemed like a possible winner: Get the site up and running, pop some content in there, adsense, ???, profit!

Here’s the result, if you want to skip to the good bit: https://namesphere.co.uk

The architecture

I’d used HTML and PHP a lot as a ‘kid’, and even scored a job building a website for a schoolfriend’s family business. More recently I’d got something working in Angular, so that seemed a sensible start

I wanted to avoid any cost for this project, other than maybe a domain and some setup, so I came across Firebase, and it seemed to have decent documentation for Angular deployments. The content of the site would sit in Firestore, and be called dynamically into my Angular app.

This ended up biting me pretty severly, as although everything worked ok, the dynamic site rendering was only crawlable by Google.. more on that later.

The content

I’d played with some offline LLMs when they first hit the headlines, and the wrappers have only improved since.

koboldcpp – let me run any models I liked, with CUDA optimisations, in windows. Crucially it also came with a local API interface. I just wanted quick-start, so this was perfect

TheBloke – I can’t remember how I found this person’s work, probably a reddit post/tutorial, but I really liked the way they summarised the model quantisations they’d made. In the end, I used a mix of capybarahermes 2.5 mistral models for the gen.

I wanted to do two things:

Defintions – a page for each of the names, with a set amount of content, e.g. a defintion, some alternative names, then some other fillere

Articles – some text which pupported to be magazine content written by different authors for a website about names – this would be the dynamic content which might help funnel traffic.

Definitions

I got a list of names from the ONS, and then did an unnecessary amount of processing on it. First I wanted to remove male/female duplication, then I wanted to sort by a sensible measure of popularity – initially I might only work on the most popular names.

One of the biggest issues was getting the LLM response to neatly fit a website template. I iterated over a lot of different prompts, eventually settling on:

    <|im_start|>system
    You are a helpful staff writer and researcher for a website which specialises in peoples names, their origins and meanings. you reply truthfully. you always reply with html formatting<|im_end|>
    <|im_start|>user
    I'd like to know more about the name {name}. please complete the following sections, please wrap each section heading in an <h2> tag, and the remaining text in a <p> tag. The list should be in <ul> with <li> elements. do not use <strong> tags. you may use </ br> line breaks if needed.
    origin
    meaning
    current usage
    aspirational summary
    5 common nicknames
    <|im_end|>
    <|im_start|>assistant

Initially I’d gone for a more robust approach of each section separately, but with around 30k names, this was just taking too long. It was quicker to have each name return all the data.. but.. at the cost of less reliability in the output

In the end I applied a section ‘reviewer’ stage which checked for the basics, like number of html tags. Any names which failed were requeued for generation, and it usually worked second time around.

The Data

The records were stored as the HTML which would be injected into my angular template. This massively increased the potential for any janky formatting from the LLM to get into the site, but typically the output was very good.

And after a few days of batch running/cleansing, I had a lot of defintions

I haven’t checked even a fraction of the total, but every one i’ve seen looks like a reasonable output. Certainly enough to get some clicks!

Articles

This was a bit more fun, and I went overboard.

First I used ChatGPT to create a set of author personas – one of my key findings here is how much more reliable and fast ChatGPT is than anything I worked with offline.

Then I had a separate ‘Editor’ who prompted each writer with some ideas of what was wanted. This was utterly overkill, and I spent a lot of time just making this ‘office politics’ work properly.

The articles were then served up via a flask front end, so I could make tweaks, and eventually upload them to the site.

I saved a lot of the data from the early steps so I could re-run and tweak as needed

Namesphere

No idea why I called it that.

Now I’ll be the first to admit that I shouldn’t be allowed anywhere near a website editor, but, here it is:

My articles were attributed, and contained links to the appropriate detail:

And then the name defintions sat seperately:

The result

It actually worked.

I’d burned probably 24-36 hours of PC time, with my trusty nvidia 3070 doing a lot of the work, but I had a massive amount of content.

The output

One of the things which suprised me the most was how I never found a dodgy bit of information in the output. Somehow, coded in each of my 4-8gb models was a staggaring amount of detail about the naming practices of different cultures. Especially if you aren’t worried about being sued for getting it wrong

The formatting remained an issue. Particularly for the articles, some of my writers would go off the deep end, and produce something which couldn’t be easily returned to the formatting I wanted.. but.. I could re-run the whole thing in a few minutes, and it usually came good in the end.

My millions

I’m writing this from the Maldives, where my passive income has…

No.

Sadly not.

With everything up and running, I’ll admit I got very excited. I started getting into SEO articles, worrying about the way Angular’s client rendering handled meta tags, and was even arrogant enough to apply to adsense.

And with, kind of, good reason:

That’s my first 10 days since launch. I felt giddy.

Then it all went wrong. This is the full picture, to roughly today.

I’ll never know for sure if I got lucky at the start, or unlucky at the end (reddit tells endless stories about google’s algorithm changes). Honestly it’s probably for the best that my clearly AI generated content isn’t making it to the first page of the worlds biggest search engine.

My decision to use Angular and dynamic data also torpedoed my ability to list with other engines.

What did I learn?

  1. LLMs/GenAI are really cool, and for a certain set of circumstances, you can get amazing things from them
  2. Angular is a PITA. No tool should be releasing major version increments that often. It risks looking like Python’s package black-hole.
  3. There’s no such thing as a free lunch – Maybe my next idea will be a winner, but for now, I’m at least pleased my bill has been essentially £0

Perhaps when I have a few minutes I’ll create a static version of the site, and have a go at automating some social media engagement drivers. Watch this space!

‘Cheating’ at Tuble

Since Wordle took the world by storm, a large number of innovative clones have emerged, one which I was pointed to recently is Tuble, where the aim is to guess the correct Tube Station in as few attempts as possible.

Once the initial fun dies down, I’ve quite enjoyed looking at these games as analytical puzzles, and finding an algorithm to solve them.

The Problem

This belongs to Tuble.co.uk

As you make guesses, the game returns two data points to help you refine the next guess. The colour of the blob tells you how many zones are between the guess and the answer, and the number is how many station stops it would take to get there.

Tube map | Transport for London
This belongs to TFL

The Tube map is pretty well known, especially for Londoners, and the game only includes the core tube (no DLR, no TFL Rail, and also no Crossrail). Turning the map into data is a graph problem, and the biggest hurdle is turning the picture above into a collection of node and edge data.

The solution

  1. Get the station data into Python
  2. Determine the possible answers given guess(es) made
  3. Determine the next best guess

1. Getting the data into Python

A quick google pointed me to this blog post (thanks Mark Dunne!), Mark had already found this github repo (thanks Nicola!) which had pre-built csvs with the info needed to build a networkx graph representation of the tube.

connections = pd.read_csv('london.connections.csv')

#filter out DLR
connections=connections[connections['line']!=13] 
connections=connections[connections['line']!=5]

station1=connections["station1"]
station2=connections["station2"]
stationMasterList=list(set(pd.concat([station1, station2]).to_list()))

stations=stations[stations.index.isin(stationMasterList)]

This first step removes the DLR and ELL stations (which aren’t in the game)

Then I’ve shamelessly copy-pasted this code from Mark’s notebook

graph = nx.Graph()

for connection_id, connection in connections.iterrows():
    station1_name = stations.loc[connection['station1']]['name']
    station2_name = stations.loc[connection['station2']]['name']
    graph.add_edge(station1_name, station2_name, time = connection['time'])
    
#add the connection between Bank and Monument manually
graph.add_edge('Bank', 'Monument', time = 1)

I’ve now got a graph with all my stations (nodes) and lines (edges).

2. Determine the possible answers given a guess

I’m not sure if there’s a more efficient way to do this, but the first iteration is just a loop through all the stations in the graph, and adding them to a list if they meet the conditions of the last guess.

def getPossiblesList(startStation, maxDistance, zoneFlag): #zoneFlag takes values as: green=0, yellow=1, orange=2, red=3
    startZone=stations[stations["name"]==startStation]["zone"].item()
    returnList=[]
    for id, stat in stations['name'].iteritems():
        if len(nx.shortest_path(graph, startStation, stat))-1 == maxDistance:
            details = stations[stations["name"]==stat][["name", "zone", ]].to_records(index=False)
            name=details[0][0]
            zone=details[0][1]
            if zoneFlag==0:
                if startZone==zone:
                    returnList.append(name)
            if zoneFlag==1:
                if abs(zone-startZone)>=1 and abs(zone-startZone)<2:
                    returnList.append(name)
            if zoneFlag==2:
                if abs(zone-startZone)>=2 and abs(zone-startZone)<3:
                    returnList.append(name)
            if zoneFlag==3:
                if abs(zone-startZone)>=3:
                    returnList.append(name)
    return returnList

My first guess is 16 stops from the correct station, and 1 zone away (plus or minus), given that, here are the possible stations which meet the criteria

['Holloway Road', 'Bethnal Green', 'Finsbury Park', 'Chalk Farm', 'Kentish Town', 'Bermondsey']

This is probably something you could do in your head, and leaves a much reduced number of possible options

3. Determine the next best guess

It will almost always be best to guess one of the stations in the shortlist, but there may be a better guess which limits the possible options more

E.g. if the possible stations are Regent’s Park or Piccadilly Circus, a guess of Oxford Circus wouldn’t help atall. Green Park would be better as you’d know exactly which station was right based on the feedback

However, given there are only 2 options, it would be much more sensible to guess one of those options as you’ve got a 50% chance of ‘getting lucky’.

Gut feel is that there could be an edge case where guessing from the shortlist would risk you getting less information than a non-shortlist station, in which case you’d be gambling a possible win on the next turn, vs a definite win on a subsequent turn.

def tubleResult(startStation, endStation):
    path=len(nx.shortest_path(graph, startStation, endStation))-1 #minus one as both stations are included in the result
    startZone=stations[stations["name"]==startStation]["zone"].item()
    endZone=stations[stations["name"]==endStation]["zone"].item()

    if abs(endZone-startZone)<1.0:
        zoneFlag=0
    elif abs(endZone-startZone)>=1 and abs(endZone-startZone)<2:
        zoneFlag=1
    elif abs(endZone-startZone)>=2 and abs(endZone-startZone)<3:
        zoneFlag=2
    elif abs(endZone-startZone)>=3:
        zoneFlag=3
    return(path, zoneFlag) #zoneFlag takes values as: green=0, yellow=1, orange=2, red=3

This code returns the values which the game would give you if you guessed a ‘start station’ and the correct result was ‘end station’. Using this, I can compare the information I’d get from using different guesses and select the one with the least ambiguous outcome.

fullList=[] #a full list of the stations, for the case where there are no previous guesses
for id, stat in stations['name'].iteritems():
    fullList.append(stat)

guesslist=[]
guesslist.append(('Ealing Broadway', 16, 1)) #Station name, number of stops away, encoded colour of the tile

returnLists=[] #store all the possible answers, based on guesses
for guess in guesslist:
    returnLists.append(getPossiblesList(guess[0], guess[1], guess[2]))

remainderList=[] #this will store a unique/deduped list of stations
if len(returnLists)==0:
    remainderList=list(set(fullList)) #clumsy dedup
else:
    remainderList=list(set.intersection(*[set(list) for list in returnLists])) #funky bit of code which I hope gives me the intersection of all results
    print(remainderList)

In the absence of a proper UI, the user adds more lines to the ‘guessList’, and the code runs from that.

if len(remainderList)==1:
    print ("The answer is "+remainderList[0])
if len(remainderList)==2:
    print ("The answer is either "+remainderList[0] +" or "+remainderList[1])
else:
    #Remainder List Loop
    bestDiff=1000
    bestStation=""
    for guessStat in remainderList:
        outcomes=[]
        for remainderStat in remainderList:
            outcomes.append(tubleResult(guessStat, remainderStat))
        numOutcomes=len(outcomes)

        numUniqueOutcomes=len(set(outcomes))
        diff=numOutcomes-numUniqueOutcomes
        if diff<bestDiff:
            bestDiff=diff
            bestStation=guessStat

    #Full List Loop
    for guessStat in fullList:
        outcomes=[]
        for remainderStat in remainderList:
            outcomes.append(tubleResult(guessStat, remainderStat))
        numOutcomes=len(outcomes)

        numUniqueOutcomes=len(set(outcomes))
        diff=numOutcomes-numUniqueOutcomes
        if diff<bestDiff:
            bestDiff=diff
            bestStation=guessStat

    print('The best guess is ' +bestStation +" with a duplication of "+ str( bestDiff))

This block of code will return one of 3 outcomes, a single result, a pair, or the best guess. I loop the final step first through the remainder list, then through all stations. If no station gives more information than one on the remainder list, I’d rather pick from that list – just in case I get lucky!

The performance

It works!


					

How many rooms does a labour ward need?

Labour ward at Wexham Park Hospital

We recently welcomed our second son into the world, and while my wife waited for a labour room to become free, we discussed the complexity of deciding how many rooms any given hospital must need. I wondered about the information available, and how they can prepare for spikes.. and that got me thinking about how to model it.. so here goes

I’ll be including good and bad outcomes of the pregnancy journey – please consider if you’re comfortable reading that before you continue.

The problem

A human pregnancy lasts around 40 weeks. Most babies arrive between 38 and 42, but there are lots of reasons why a baby can arrive earlier than that. In the UK, mothers to be are offered a range of options for where they would like to have their baby, roughly: Home Birth, Midwife-lead units, Labour wards.

Some of the home births end up moving to a medical setting due to complications, but most women who give birth in hospital will head there when they’re in established labour. A small proportion (including my wife) will be induced for various reasons, and they’re admitted before labour starts.

There are a limited number of rooms available in any labour ward (more so since Covid, as hospitals have typically set up quarantine rooms form mums-to-be who have the virus). It’s not immediately clear how much pressure these rooms are under as the overlapping situations for each mother vary.

I want to start with an agent-based approach:

  1. Creating new Women at a rate which is consistent with UK stats
  2. Simulating the success rates for each as they progress
  3. Simulating a start of labour (again, consistent with the distributions in the real world)
  4. Simulating a total time needed in a room to give birth, and rest afterwards

I can then analyse these stats to see how many rooms would be needed.

Step 1. Women

Statistic: Conception rate in England and Wales in 2019, by age group (rate per 1,000 women)* | Statista
Find more statistics at Statista

There are lots of useful stats available for this project, starting with the above, which I can piece together. As a starting point, I’m going to choose a population size, then assign them ages which match the UK distribution.

I’ll then run the population over a period of time, by day, giving each a probability of falling pregnant, then for those who are, running the course of each pregnancy to get my outcome.

class Woman:
    ageYears: int
    ageDays: int
    pregnant:bool
    conceptionDt: datetime
    dueDt: datetime
    termWeeks: int
    termDays: int
    formerPregnancies: list
    labourList: list

    def __init__(self, _age):
        self.pregnant=False
        self.ageYears=_age
        self.ageDays=0
        self.formerPregnancies=[]
        self.labourList=[]

Part of the point of this is to get some stats at the end, so along with some obvious properties, I’ve got a couple of lists to store the results of previous pregnancies, and the details of each simulated labour.

2. Simulating success rates

Again, content warning here, I’m not being gruesome, but having kids isn’t always smooth sailing.

My simulation starts with an almost unknown commodity, the day of conception. Practically it’s almost impossible to know this, but it exists.

From the point of (simulated) pregnancy, there are a set of outcomes I’m allowing by probability.

  1. Miscarriage – loss of a pregnancy in the first 20 weeks
  2. Termination – decision to end the pregnancy within the first 26 weeks
  3. Birth – entering labour and delivering a baby

Much, much more depth could exist here, but I think this will be enough to give me the data I want.

3. Simulating start of labour

This is probably the most interesting bit for my problem, but I like the idea of the having more accuracy behind it. Modelling the ages, and age related outcomes gives me more nuance in the analysis at the end.

Are first babies more likely to be late? | by Allen Downey | Towards Data  Science
There are lots of charts like this on the web, the basic message being that due dates are meaningless

It’s this distribution which started me wondering about the way this would play out across a bigger population, and therefore how confident the NHS can be about the number of labour rooms.

4. Total time in labour

def labourStageCalcs(_ageYears, _termWeeks, _date):
    offset=randint(0,23) #pick a random hour of the day for things to kick-off
    DT=datetime.datetime.combine(_date, datetime.datetime.min.time())
    
    firstLatentStageStDT=DT+timedelta(hours=offset) #Start of contractions
    lenFirstLatent=gauss(12,4) #assume mean 12h, SD 4
    
    firstActiveStageStDT=firstLatentStageStDT+timedelta(hours=lenFirstLatent) #Start of active labour
    lenFirstActive=gauss(6,2)
    
    secondStageStDT=firstActiveStageStDT+timedelta(hours=lenFirstActive) #Start of 'pushing'
    lenSecond=gauss(1,0.5)
    
    thirdStageStDT=secondStageStDT+timedelta(hours=lenSecond) #Baby is born, the cleanup begins
    thirdStageEnDT=thirdStageStDT+timedelta(hours=6) #Give each women 6 hours in the room to recover

    return firstLatentStageStDT, firstActiveStageStDT, secondStageStDT, thirdStageStDT,thirdStageEnDT

At this stage, I just want something which works, no nuance by age/etc

The results

Assuming a female population of 200k, with no current pregnancies, run for 720 days, this show the ramp up of pregnancies

Around 9 months later, the population hit’s a steady state with around 2.3k women pregnant at any time.

This shows a daily view of the maximum number of beds occupied at any time

And here as a table. The average (mean and median) is 5, with only 5 days needing 10 or more, and only one needing 11 – the highest on any given day

This answers most of my original question. Given a known population, it’s not a big surprise that the number of pregnant women hits a fixed level – I’ve created that by adding a constant number (based on the UK population rate). It’s more interesting that the randomness surrounding the actual birth still results in a number of rooms which can be estimated ahead of time with reasonable confidence.

In future iterations, I’d like to look at the data which would be available to the hospital ahead of time, and maybe add some more variability based on things like age.

Wordle

Half hangman, half mastermind, Wordle is a fun new obsession for half of the internet. Fun as it is to try and remember 5 letter words, what if I got the computer to solve it?

The Problem

You get 6 guesses to find the word of the day. Each guess is assessed and you find out which letters you guessed correctly (i.e. appear in the word) and if any of those are in the right position in the word. It looks like this:

A four-row grid of white letters in colored square tiles, with 5 letters in each row, reading ARISE, ROUTE, RULES, REBUS. The A, I, O, T, and L are in gray squares; the R, S, and E of ARISE, U and E of ROUTE, and U and E of RULES are in yellow squares, and the R of ROUTE, R and S of RULES, and all letters of REBUS are in green squares.
Grey means the letter isn’t in the word
Yellow is in the word, but not in the right place
Green is correct

In this example, the first guess is pure chance.

There is one word each day.

The Solution

My first solution will run through 3 steps:
1. Read in a defined wordlist
2. Read in any previous guesses with the results
3. Strip out words from the wordlist based on the guesses
4. Generate a new guess

https://github.com/alexchatwin/wordleSolver

1. Getting a defined wordlist

I don’t think there’s any way to do this without starting with a list of possible words. without this we’re just guessing symbols, and part of the nature of the puzzle is the players knowledge of english words.

I’ve chosen to use https://www-cs-faculty.stanford.edu/~knuth/sgb-words.txt, but I gather the actual seedlist for the Wordle game is available in GitHub

2. Reading in previous guesses

Each guess generates information about the true word.

Grey – the letter is not in the word at all
Yellow – the letter is in the word, but not where the guess has put it
Green – the letter is in the word and is in the right place

There are some subtleties to the yellow and green options – in both cases there is the potential for a letter to appear twice, it’s easy as a novice player to incorrectly assume green means job-done for that letter. Equally yellow is as much about narrowing down valid letters as it is about restricting the possibilities for the square.

3. Strip out words based on the information from the guesses

I’m going to loop through all the remaining words on the wordlist and prune them in one of 3 ways:

  1. Remove any word containing any number of grey letters
  2. For yellow letters, remove words with that character in that position, and also words where that character isn’t in at least one other position
  3. For green letters, remove any words without that letter in that position

By gradually sifting through the words, I end up with a smaller list of possible words which fulfil the ‘rules’ as they evolve.

4. Generate a new guess

I want to use some tactics with each guess, it’s not helpful to have the algorithm only change a single letter at a time, or I’ll run out of guesses. The game also requires each guess to be a real word.

‘Splittiness’

I’m going to use an approach similar to a decision tree for v1. I’ll start by working out how many words each letter appears in, and then what fraction of the total words that represents:

S46%Y15%
E46%M14%
A38%H14%
R31%B12%
O29%G11%
I27%K10%
T25%F9%
L25%W9%
N21%V5%
D19%X2%
U19%Z2%
C16%J2%
P16%Q1%

Here S & E are the ‘winners’ – they’re each present in almost half of the words in the list. This is useful because guessing a first word with them will allow me to split the wordlist most evenly and remove/include a large number of possible words, reducing my search space considerably.

For later versions, I want to look at the overlap between them, e.g. if most words containing S also contain E, it might be that the first guess is better to skip E, and try the next letter down.

The best starting word, the word which will give me the best split between words I should include and remove contains S,E,A,R,O

Anagramming

SEARO is, unfortunately not a word, so the next step is to try the different permutations of the letters until I get something which is (more-so, something which is a word on my wordlist – so it’s still a viable guess).

First guess is easy enough, a simple iteration through the permutations of the letters comes up with – AROSE. I can use all the letters in a viable word, but I need a strategy to ensure I don’t get stuck further down the line. Even one letter down, EAROI doesn’t yield a valid 5 letter English word.

My solution is to add another level to my anagram solver, My preference is to have the 5 most splitty letters, but I’d take 1,2,3,4,6 if they didn’t work, and 1,2,3,4,7 after that. I now have a way to ensure that I’ll be able to sweep the full range of available letters, hoping I’m stepping ever nearer the right answer.

Rinse and repeat

Putting AROSE into the puzzle gives me

And then I repeat the process, putting in the results and asking for the next best word to try, until the board looks like this:

This is Wordle 215, from 20/1/22, and with only 4 guesses needed, 1 better than my personal attempt!

There’s a tangible sense of the algorithm building on what it learns each round.. but also a tedious inevitability that it will reach a valid answer at the end.

I suppose part of the challenge of wordle is the need for a human to remember all the possible words and to make efficient choices. This bit of coding has helped me understand the puzzle at a lower level.

Update – 21/1/22

Wordle 216 6/6

⬜🟩⬜⬜⬜
⬜🟩⬜🟨⬜
🟨🟩🟩⬜⬜
⬜🟩🟩⬜🟩
⬜🟩🟩🟩🟩
🟩🟩🟩🟩🟩

Today’s word was solved, but I learned 2 things:
1. My algorithm will happily guess double letters which it has no evidence exist in the word – I think this is sub-optimal
2. If you guess the same letter twice, and it’s in the word once, you’ll get two yellow blocks if they’re in the wrong places, but a green and a grey (not yellow) if one of them is correct

Updates to follow!

picStitcher2 – automatically creating a photo collage

In Part 1 I put together a short python script to automate the stitching together of 16 pictures into a single composite. Now I want to take some of the effort out of selecting the 16 pictures, in order to speed up the process

Each month my wife and I choose 16 of the best pictures involving our son from the hundreds we’ve taken. These end up on our fridge in a 4×4 grid. Although we take a picture of the pictures on the fridge, it always looks a bit screwed up and it’s nice to have a good version stitched together on the computer.

How can I minimise the effort needed to recreate a physical collage from a photo?

The problem

The problem looks a bit like this:

As part of the selection, we have up to 100 printed out into actual photos, so we can isolate the image files for those, but the refinement to the final 16 is done by hand.

  1. Read in the individual images, and a picture of the ‘raw’ composite from the fridge
  2. Break the raw composite down into 16 tiles
  3. Check each of the individual images against the tiles
  4. Present the user with the original tile and an ordered list of the ‘candidate’ images which are the best match
  5. Loop for all 16, then stitch the final image together

1. Read in the images

I’m going to use a combination of OpenCV and Pillow for my image processing, the first step will be in OpenCV so I’ll read the files in like that.

I’ve created some objects to hold the data around each image – my foray into C# a few years ago got me thinking in Objects, and although I’m not sure it’s Pythonic (dahling), it helps me order my thoughts, and write more debuggable code

class Composite:
    imageName: str
    rawImage: Image.Image
    pilOutputImage: Image.Image
    outputImageArr: list
    tiles: list
    candidates: list
    readyToProcess: bool
    done: bool
    imageXSize: int
    imageYSize: int
    date: datetime
    
    
    def __init__(self, _imageName, _rawImage, _imageX _imageYSize):
        self.imageName=_imageName
        self.rawImage=_rawImage
        self.tiles=[]
        self.candidates=[]
        self.outputImageArr=[]
        self.readyToProcess=False
        self.done=False
        self.imageX=_imageX        
        self.imageYSize=_imageYSize
        self.pilOutputImage = Image.new('RGB', (_imageX4, _imageYSize*4),(255,255,255))
        self.date=datetime.datetime.strptime(_imageName.split(".")[0],'%b%y')
        
        
    def incompleteTiles(self):
        return [t for t in self.tiles if t.done==False]

    def pilImage(self):
        return Image.fromarray(self.rawImage[:,:,::-1].astype('uint8'))
    
    def buildImg(self):
        for tile in self.outputImageArr:
            Image.Image.paste(self.pilOutputImage, tile.pilImage(), (self.imageXtile.xPos, self.imageYSize*tile.yPos))
            
    def addCompleteTile(self, tile):
        self.outputImageArr.append(tile)
        Image.Image.paste(self.pilOutputImage, tile.pilImage(), (self.imageXtile.xPos, self.imageYSize*tile.yPos))

This is an example of the ‘Composite’ object – there will be one of these for each of my Raw Composite files (the fridge images). This holds the rest of the information relating to the Tiles (the 16 sub images) and the Candidate images. It also has the logic to build the final output image.

class Tile:
    xPos: int 
    yPos: int
    rawImage: Image.Image
    candidates: list
    done: bool
    
    def __init__(self, _x, _y, _rawImage):
        self.xPos=_x
        self.yPos=_y
        self.rawImage=_rawImage
        self.candidates=[]
        self.done=False
    
    def pilImage(self):
        return Image.fromarray(self.rawImage[:,:,::-1].astype('uint8'))

This is a Tile. It knows what it looks like in a raw form, it’s position on the 4×4 grid, and the candidate images which could be it

class Img:
    imageName: str
    rawImage: Image.Image
    error: float
    exifDT: datetime
    
    def __init__(self, _imageName, _rawImage, _dt):
        self.imageName=_imageName
        self.rawImage=_rawImage
        self.exifDT = _dt
        
    def pilImage(self):
        return Image.fromarray(self.rawImage[:,:,::-1].astype('uint8'))

Finally this is an image, I use it to keep some of the important info relating to the image in a single place. It’s got the concept of ‘Error’ in there, which will come from the matching code – the lowest error is our best candidate

This is my rough data structure, now I need to populate it

def loadImages(dir, size, type):
    compositeDir="C:/Users/alexc/Source/Repos/picStitcher/photos/"+dir+"/"
    returnList=[]
    for image_file in os.listdir(compositeDir):
        image = cv2.imread(compositeDir+image_file)
        img = Image.open(compositeDir+image_file)
        try:
            dt=img.getexif()[36867]
            dt=datetime.strptime(dt, '%Y:%m:%d %H:%M:%S')
        except:
            dt=datetime(1900, 1, 1, 00, 00)
        if type=="Composite":
            outObj=Composite(image_file, cv2.resize(image, (size[0]*4, size[1]*4), interpolation = cv2.INTER_AREA), size[0], size[1])
        elif type=="Img":
            outObj=Img(image_file, cv2.resize(image, size, interpolation = cv2.INTER_AREA), dt )

        returnList.append(outObj)
  
    return returnList

This function loads raw images (in OpenCV format – a numpy array) and creates an appropriate object to hold them. Theres some extra data (like date, and image size) which is useful later on.

2. Breaking out the tiles

This is a bit of a fudge. Because the pictures taken from the fridge are imperfect, it’s difficult to cut out the exact image, but the hope is that the matching process will work ok with a rough target.

def detile(name, image, imageXSize, imageYSize):
    outArr=[]
    for x in range(0,4):
        for y in range(0,4):
            im=image[y*imageYSize:(y+1)*imageYSize,x*imageXSize:(x+1)*imageXSize]
            outObj=Tile(x, y, im)
            outArr.append(outObj)
    
    return outArr

So I just loop through a 4×4 grid, cut each image and make a new tile out of it. Once that’s done, it gets added to the Tiles list in the Composite object

3. Checking for matches

Let’s assume I have 100 candidate images for my 16 tiles. That’s 1,600 checks which need to be done, and that’s going to take some time. This seemed like a good excuse to have a play with Threading in python

I’ve tried threading before in C#, and it felt messy. I’ve never really had proper training, so I just google my way through it, and I wasn’t ready to handle the idea that information would be processed outside of the main thread of my program. It was really useful (e.g. for not having the UI lock-up when heavy processing was happening), but I didn’t properly understand the way I was supposed to handle communication between the threads, or how to ‘safely’ access common sources of info.

This time, things are a bit simpler (and I’ve been thinking about it for a few more years).

I need my threads to work on the checking algorithm

    def wrapper_func(tileQueue, rawImages):
        print("thread")
        while True:
            try:
                tile = tileQueue.get(False)
                for image in rawImages:
                    result = cv2.matchTemplate(tile.rawImage, image.rawImage, cv2.TM_SQDIFF_NORMED )
                    mn,mx,mnLoc,mxLoc = cv2.minMaxLoc(result)
                    output=Img(image.imageName,image.rawImage, 0)
                    output.error=mn
                    tile.candidates.append(output)
                tileQueue.task_done()
            except queue.Empty:
                pass

Here I have a queue of tiles, and my raw images. Each tile needs to be checked against all the images, but there’s no reason why they can’t be checked independently of each other. This ‘wrapper function’ – which still uses the function name from a StackOverflow answer I cribbed (this one, I think) – is run in 8 different threads, and checks each tile against all the raw images, appending a candidate to the tile each time.

I’m pretty sure this qualifies as thread-safe. The append operation just adds new stuff in, and I’m not reading from anywhere in the list.

    def check_matches(composite, rawImages):
        print("check_matches.. may take a while")
        tileQueue=queue.Queue()
        
        for tile in composite.tiles:
            tileQueue.put(tile)
        for _ in range(8):        
            threading.Thread(target=wrapper_func, args=(tileQueue, rawImages)).start()
        
        tileQueue.join() #wait until the whole queue is processed
        for tile in composite.tiles:
            tile.candidates.sort(key=lambda x:x.error)  #sort the candidates by error (the error of the match)
            
        composite.readyToProcess=True
        print(composite.imageName+" ready to process")

Stepping out a layer, this processes an entire composite. First splitting into a queue of tiles, then fire this into 8 threads. Once the queue is processed, sort the candidates, then set a ‘ready to process’ flag for the UI to respond to.

    def loop_composites(): 
        thread.join()
        print("looping composites")
        global readyState, rawComposites, rawImages
        for composite in rawComposites:
            print(composite.date)
            compMinDT = composite.date-timedelta(days=30)
            compMaxDT = composite.date+timedelta(days=30)
            rawImagesFiltered=[i for i in rawImages if (i.exifDT>=compMinDT and i.exifDT<=compMaxDT) or i.exifDT==datetime.datetime(1900, 1, 1, 00, 00)]
            matchThread = threading.Thread(target=check_matches, args=(composite, rawImagesFiltered))
            matchThread.start()
            matchThread.join()

This is the outer layer of the loop. I’ve added in some filtering logic which uses the exif data to say when the photo was taken. All fridge photos are from a 1 month period (aligned to the middle of the month), so I can use the composite date and check for a period either side. This should reduce the processing if I want to run many months at the same time.

4. The UI

I’ve decided that there’s no way this process will work without a human in the loop – the whole point is to have the correct image at the end!

I’m using Flask to serve a simple web front end which will present the user with the images, and allow them to choose the correct one

On the left you can see the original composite image – you can see that the overall images are a bit skewed, and there’s significant light distortion on the left hand of the image

In the middle is the 1st tile cut from the raw composite

Next is the first match candidate (it’s correct! yay!)

Then is a placeholder for the newly constructed image

5. Loop, complete, stitch and save

Clicking ‘correct’ adds the candidate to the output image array, and pastes the image into the right hand ‘progress’ image

And once all the tiles have a match, the image is saved

Success! Original ‘fridge’ picture on the left, and new composite on the right – blurred to protect the innocent

6. How did we do

The original reason for creating this program, other than the fun of making it, was to reduce the time it took to create the composites. Let’s consider two metrics:

  1. How many images did I need to click through to get to the final composite
  2. How many times did the program get the right image first time
Jun 21Jul 21
Total Clicks20239
Right first time1110

With around 100 photos per month, I only needed to check 5-6 of them, and worst case I needed 40 clicks for each of the remaining photos. I think that’s pretty good.