I see that it is possible to add custom data, but it seems like there is no validation of its correctness on the blockchain side. So the question - is it possible to implement custom transaction types with custom state and validation logic (not necessarily linked to cryptocurrency) without forking Stellar core code?
No it's impossible without changing the protocol. Basic validation and smart contracts logic can be implemented on top of transaction preconditions, read more here. Use cases are somewhat limited, but CAP-21 proposal (hopefully we'll see it implemented by the end of the year) introduces more advanced preconditions for Stellar smart contracts.