To add to @barthuijbers idea, I do believe there must be another way to link addresses and buildings to parcels. For instance, our dutch colleagues regularly request Eigendomsinformatie (ownership information) from Kadaster which has a clear relationship between the Kadastrale objectidentificatie and BAG identificatie (address id or Verblijfsobject identificatie) as shown in this screenshot
Since this piece of information is already made by Kadaster, it seems they already have this linking information in their databases. Maybe they are already working on it to make it available.