This process is kind of finnicky, so I'm documenting it here so that I don't forget it.
Firstly, pair the device in Ubuntu.
Boot into Windows and pair the device there.
In an admin command prompt, launch
psexec -s -i regedit and store the value from
You'll need the Value field. Take a picture of it or write it to a disk accessible from Ubuntu.
Write the device key into a file in a path something like this
/var/lib/bluetooth/A0\:C5\:89\:13\:3E\:3F/04:69:F8:BA:47:B5/info in the
After a reboot you should have your keyboard working in Windows and Ubuntu.