As far as I understand it, like with anything on a Mac, you simply have to drag the driver files to the trash and empty the trash.

Search for the file sames in Finder to see where they're stored and then just trash them from there. I have seen people mention that you may have some preference files elsewhere related to the driver but they say it won't hurt at all to leave them if you don't know where they are.