We recently ran a hackthon in NYC, kindly hosted by Larry Miller and his amazing team at NYU Steinhardt and the Sony Audio Institute. Their effort was enormous and we simply wouldn't have been able to do it without them.

The focus of the hackathon was to understand how teams went about untangling a particularly messy data room that we had conjured up ahead of time, which tried its best to be both light-hearted, but also representative of some of the challenges we see in our day-to-day of evaluating catalogs of music rights for valuation and ultimately acquisition.

There were 7 teams in total and each did an incredible job. Everyone was equipped with their LLM of choice and all took subtly different approaches to try and deduce what was there and what was not.

We gave the team some simple objectives:

• Meet peers with similar problems, but different experiences/approaches
• Take with you at least 1 approach or learning that you can use in your day job
• Have fun!

And asked them to focus on the following things:

• Understand what is for sale
• Set out the required set of documents to properly value the catalog for your fund/portfolio and to make an investment decision
• Perform a gap analysis based on what information has been provided against what your fund requires
• Create a process to collect the missing documents

The Hollow Verge

The data room represented a catalog for a ficitional band, Hollow Verge, represented by a fictional fund, Northbridge Catalog Partners.

It was littered with various challenges that, on the whole, the teams picked up on and identified as issues to overcome. We included some of the common challenges we see out in the field for the teams to tackle:

• Missing information; The most common issue is where information is missing or incomplete. A statement for a given period or payor is not available or hasn't been provided.
• Duplicate information; duplicate statements for the same periods have been added either in a different format (pdf, csv, xlsx etc) or simply the same file twice.
• Poorly filed information; filenames are no longer faithful to their originals, removing key information that is often shipped with the statements, or files have been categorised against the wrong payor or wrong period in the folder system (if indeed they have been put in folders at all).
• Summary documents don't match the detail; summary documents for certain payors contain extra information that the detailed statements don't, which need reasoning with.
• Exotic formats; photos converted to pdfs, handwritten scribbles, scans, faxes (!!), the whole nine yards.
• Contradictory documents; that exist but are incomplete or contradictory to other documents provided that need some detective work.

We added some more obscure challenges too, which you can check out in the full data room - give it a go and see where you get to.

How we do it

The obvious question that was asked on the day and afterwards, is how do we do it. So here it is an overview of the process we follow!

Save all the documents

We stick the documents somewhere safe, typically a blob store (think AWS s3, GCS etc.) where we can easily access them from a server and make sure they are properly versioned, backed up and resilient across regions where applicable (and indeed compliant with any local data laws).

Extract the documents to raw text

The teams on the day used LLMs, which are increasingly awesome, but passing them full documents with all of the file metadata and cruft humans need to see uses a lot of tokens without adding much value. So we extract the raw information from the documents and save it as raw text. That way it's easier to manipulate, takes less compute and less tokens.

Categorize the docs

Through a series of rules running from path names (file and folder), content structure, and content themes, we put the documents into categories that make sense for the task at hand, which can vary depending on whether we're doing financial analysis, legal analysis, etc. We do this with a coupling of traditional deterministic processes and LLM probabilistic processes.

Parse the docs

Once categorized, we run parsers across the docs and build new ones where we don't have any. The parsing gives us a level of granularity we need downstream to extract out the pertinent information. Through the parsing process we do a degree of normalization too, just to make things more machine readable.

Fan out the information

Once parsed there are several different paths to take, but broadly we will be:

• Extracting and matching entities (works, recordings and releases)
• Pulling out royalty lines and mapping multiple definitions to canonical ones
• Matching documents to statement periods and where there is any duplicate of information for the same date range, selecting a canonical document and surfacing the options for review
• Identifying and surfacing gaps in statements and summaries for the periods we need
• Preparing legal documents for summarization

Using human experience

Computers are increasingly good at reasoning with grey areas where there is material subjectivity surrounding a piece of information, however, we still believe humans with specialized experience are significantly faster and more accurate (for now). With that in mind, we focus on workflow. Giving analysts the tools, the information and the signals to support with their decision making to get the catalog to a place it needs to be. For example:

• Propose matches for song entities
• Show similar looking documents for the same periods (statements and summaries alike)
• Visualize gaps or uneven statement cadence

The whole approach is focused on the business process, wrapping technology around it and not the the other way around.

End to end audit log

For each step in the transformation, we retain a detailed audit log of exactly where the information came from and how it was transformed, so we can follow a number or an assumption right back to the document it came from. Data needs to be explainable, especially in financial services.

Many roads can lead to the same destination

As our talented teams illustrated on the day, there are many ways to unpick a messy data room and there isn't a single right answer. There are, however, many people working on the same problem and often in silos for no material gain, so as much as anything our aim for the hackathon was to bring people together to share their approaches and to make what can be an arduous process a little easier.

Stay tuned for the next one!