The catalogue behind Gradsy holds 1,196 universities across 23 destinations, and 107,853 individual programs. It is the least glamorous thing we have built and the reason everything else is possible.
Why we own the data
We could have used someone else's feed. We decided not to, for one reason: a student's decision is only as good as the requirements it was based on, and we wanted to be accountable for those requirements rather than pointing at a supplier.
Owning the pipeline means when something is wrong, it is our problem to fix, and we can fix it that day.
Programs are not rows
The tempting model is one table of universities and one of programs, joined on an ID. That model breaks the first time reality shows up.
A single program can have several intakes with different deadlines. Entry requirements vary by the applicant's country of origin. Tuition is quoted per year in some places, per credit in others, per full program in a few. The same degree appears under three different names depending on which page you land on.
Most of the work was not fetching data. It was deciding what a "program" is, precisely enough that 107,853 of them could be compared against each other without the comparison being meaningless.
Normalising without flattening
The hard trade-off is between comparability and truth.
Convert everything into one tidy shape and you get a catalogue that compares beautifully and describes nothing accurately. Keep every institution's own structure and you get a catalogue that is faithful and unsearchable.
We resolved it by separating the two. Records keep the institution's own figures — its currency, its wording, its structure. A derived layer sits on top for comparison, and every derived value can be traced back to the source it came from. When a student sees tuition in Gradsy, the local figure is the fact, and the estimate beside it is clearly labelled as derived.
Staying current
A catalogue is not a delivery, it is a commitment. Requirements change, intakes move, programs close.
Two things follow. The first is that ingestion has to be repeatable — a re-run that reconciles against what is already there rather than a one-off import. The second is that institutions need a direct way to correct their own listings, which is why Unihub for Institutions exists.
What we would tell anyone doing this
Three things, in the order we learned them.
- Decide what your unit is before you collect anything. Every ambiguity you leave in the definition becomes a thousand ambiguities in the data.
- Never overwrite the source figure. Derive as much as you like, but keep what the institution actually said, or you will not be able to explain yourself later.
- Assume it is already out of date. Build the correction path at the same time as the ingestion path, not afterwards.
None of this is visible in the product. A student who searches for a master's in Germany sees a list, and the list is right. That is the whole point.