Fine tuning a folk music OMR

Canonicalisation & Evaluation

Beating Claude??

This is a report of some fine-tuning I did, so it’s a bit longer and more technical than my other posts

When I understood how compact the abc notation was for representing music, it immediately set my mind thinking about the possibilities.

What I liked was that in a relatively small amount of text you could represent a whole piece of folk music. Folk music needs to be shared, sometimes in places where there’s no internet, so I built trad.pub which encodes the entirety of the song into a QR code.

The natural next question after that was ‘how will the user get the music into the app for sharing’, and what I wanted the answer to that to be was that you could take a quick snap of some handwritten sheet music, and with a local LLM on the device, we would convert it to abc notation and load it into the trad.pub app for sharing.

I tried Claude and ChatGPT on this problem, and they were terrible. (I’ve since tried Fable and it was great, and even the more recent versions of Opus are getting fairly close, so things are obviously moving quickly!) and no small LLM was anything like up to the task of reading music out of the box, so I knew that to achieve this, I would have to fine-tune an existing model.

Data, Data, Data

Of course, the key requirement for fine-tuning is a good data set. In some ways, there was an embarrassment of riches - there are a number of abc datasets available, with many different folk tunes. I had a good discussion with the creator of folktunefinder.com, and the session has many tunes, fantastically they publish them all on github. I ended up using Nottingham Music Database, and Henrik Norbek’s collection a lot, but there were other sources too.

On the other hand, images of hand-written sheet music were in short supply, especially paired with their abc transcription.

So it became clear that the fine-tuning dataset would consist of a relatively small set of photos that I would hand transcribe, and a larger set that I would generate from the datasets available to me. At least generating from a dataset meant I had a known transcription.

Problems and Solutions

Much as I love abc notation, it has a number of features that make it convenient to write, but less convenient for training an LLM.

Many Forms

One of the big ones is that any piece of music can be represented in abc in many different ways - there’s not a single form that a piece of music has to take. This means that starting with an abc notation piece of music, rendering it, then scanning it and recognising it might produce very different results to the original, yet still be correct. This lack of a canonical form came from a number of main places:

• Default note-length (L:): to make it convenient for the writer, each piece can have a different default note length, and the length of each note in the piece is notated based on reference to the default. L:1/8 a2 ≡ L:1/4 a means exactly the same thing.
• Octaves: Octaves can be shown with any combination of ’ (up an octave) or , (down an octave), and even without that, upper-case is the octave starting the G below middle C and lower-case is the next octave up. This means there are many ways to indicate which octave a note is in - A' is the same as a.
• Broken rhythm: A>B indicates that half of the B’s timing has gone to the A instead, A<B indicates the opposite. Here’s the example from the standard itself:

Example: The following lines all mean the same thing (the third version is recommended):

L:1/16
a3b cd3 a2b2c2d2

L:1/8
a3/2b/2 c/2d3/2 abcd

L:1/8
a>b c<d abcd

• Whitespace / new lines: Most of the time, whitespace is just decorative in abc notation, however sometimes it’s used to indicate where the beams that join notes shorter than a quarter note should be split. This can be meaningful for how the piece should be played. Some new lines are decorative, some are meaningful.

This meant that the first step, before rendering anything, was to fix all these different ways of representing the music.

Canonicalisation

The solution to having source music with many different ways of notating, each potentially idosyncratic to the different transcriber, was to create a tool that would convert them all to a single form, that I could treat as canonical.

The steps were:

1. Normalise the default note duration. The question here is how? We could of course choose a particular standard length, but that would be very un-idiomatic for some kinds of music (fast reels might be L:1/16 while a slow air might be L:1/4). In the end I chose to select the most common of the note lengths in the piece to use for the default, and renotate everything to that default.
2. Formatting. Strip all extraneous new lines and whitespace that doesn’t affect the beaming of notes. Spotting which whitespace and new lines is meaningful and which isn’t was surprisingly tricky.
3. Removing real features that couldn’t be reasonably detected/distinguished from a rendering. For example, ABC notation supports specifying modes, a piece notated as in no key (K:none) or C major (K:C), or A minor (K:Am) or D dorian (K:Ddor) or G mixolydian (K:Gmix) etc, will all render with the same 0 sharps or flats, and while they might be musically meaningful, asking the LLM to infer which it should be recorded as, especially when the source material might not be consistent in how it is transcribed, would cause problems.

Checking the canonicalisation

In order to make sure we were doing what we wanted to do, we took both the original abc and the rewritten abc into music21, a totally different music toolkit and compared them to make sure that the note durations etc matched. Music21 didn’t capture every possible aspect, but it captured most of the most important ones, so this check on the whole corpus provided me with a lot of confidence that everything was working properly.

Generating the Training Data

For the data that I was generating, I took the source ABC, canonicalised it, then rendered it using a couple of different engines, at different angles and backgrounds.

Interim Results

I was originally very keen on the Gemma 4 models, which are extremely good for their size and vision capable but it turned out that with default settings Gemma wasn’t able to understand sheet-music detail at all. I had to explicitly unfreeze the vision layers, increase the vision token budget and stop the pre-resize to 512px. Despite all of that, even with the fine tuning, Gemma still struggled to understand the importance of notes on a staff.

This led me to change models, and I tried fine tuning Qwen-3-VL-8B. It performed much better. Without the fine-tuning, the model would hallucinate some of the text, and frequently went into a single note loop, repeated until it hit the token cap. Sometimes it would produce reasonable headers, but then never generate any actual notes. After the fine tune, we got a much better result - here’s the pitch accuracy measure:

In order to measure the difference cleanly, we had to exclude from the evaluation set not just images that were not in the training set, but also images of tunes that were in the training set in some other form - we didn’t want to give the fine-tuned model an unfair advantage.

The fine-tuning took us from an unusable 38% to a respectable 91%.

I published the fine-tune to huggingface.

Canonicalisation - the final form (IR)

The natural next step of canonicalisation was to actually change the tokenisation of the model - to create an intermediate, unambiguous representation that was still fairly close to abc notation, and then to add new tokens to the models vocabulary so that it could represent those tokens without any ambiguity of language creeping in to confuse the musical aspects.

I plan to write more about this approach later, but here’s the result:

We managed to squeeze another 3% by using this approach!

Next Steps?

So why isn’t this deployed and running in the browser on device yet? Well, even 94% isn’t that great, and I still have few more ideas I’d like to explore first, there are new models coming out the whole time, and I’ve also been a little disappointed by how long the model takes to do the transcription, it’s about 3 minutes running on modal to transcribe an image, which isn’t a great user experience.

I hope to squeeze some more performance out before starting to look at how to add it to trad.pub.