Everything is doable: XenForo is software, it is neither compiled nor encrypted and it is using PHP, a language that is easy in comparison. So why should this not be doable? The two relevant questions are:
• is it a good idea?
• is it worth the effort?
I agree with [USER=266758]@stromb0li[/USER] here: It is neither a good idea nor worth the effort. For exactly the reasons he outlined. And I also agree with the solution he provided: Modify your login system to act as an OAuth-provider - problem solved w/o needing to modify the XF-codebase.
You did not give any reason why you would need to use your system instead of the one built in other than "I want it" (which is slightly different from "need"). This
sounds a bit weird. XF is a webbased application (including the authentification). As such it runs on basically any web environment and thus is also "compatible" with any Linux distribution. Also, I don't see why a login system for a forum should need to be part of an operating system. Sure, one can do that (again: it's software) - but why? When it comes to compatibility we are in the area of standards and this points (again) to oauth, as this is one of the compatibility standards in this field.
I'm not a developer but what one can read here in the forums is that developing plugins for XF seems to be very much straight forward and well organized - rather the opposite of complex.