Where is the mapping from character or block file to physical address

I've become quite interested in special character / block files in Linux. I have followed a tutorial to create a basic hello world character file, but one thing confuses me. Let's say the motherboard has many PCI slots. Now let's say we add a hard disk, cd-rom or floppy drive into the first slot. If we ignore any type of auto setup / detection, I believe the way to setup this hardware would be:

1. Obtain a driver
2. Use insmod to load the driver into the Kernel
3. Use mknod to create a special character / block file

Now we should be able to interact with the device from user space by reading and writing to this file. But here's my confusion, when I ran mknod I would have used a Major Number to select the correct driver, and a Minor Number to give it an instance. But how does the OS know that my hardware is in slot 1, and therefore has whatever physical address offset for interacting with that device? Is that encoded into the minor number or something?