In short, yes. The inflation pool wallet can be hacked just like any other crypto wallet can be hacked. You would need the private key. Thus, the security implications are the same and it is up to the owner for the community pool wallet (the inflation pool that all voters voted on) to take precautions just as if it was their own personal wallet.
First: all the Stellar inflation rewards is directly handled by stellar core, so it’s regulated by the SCP (the same mechanism that validate every transaction)
Second: the inflation reward is sent to the voted inflation address (aka “inflation pool”), any not to all addresses that voted for it
What an inflation pool usually does, is to redistribute the amount received to all who voted for them, but as you said, this is up to the pool developers.
They can decide to not redistribute anymore or they can decide to apply a fee (a known pool reserve for himself the 10% of the amount as “maintenance cost”
What happen if a pool will not redistribute the amount completely *? As happened less than a week ago, users just voted for another inflation address that “claims” to redistribute the full amount, and the original pool lost his votes and simply it will not receive rewards from SCP,
*This can happen to various reason like hacks, policy change, or whatever the inflation holder thinks
About “trusting a wallet”, any wallet can be hacked (even so called hardware wallet). There is always a low possibility (close to 0) that I luckily guess a secret key of an existing wallet.