Are there any known situations, although infrequent or difficult to meet, where the Stellar Consensus protocol could not work well or where the network can be attacked by malicious nodes?
A partial answer is here, but I'd like to know more: https://rubin.io/public/pdfs/oort.pdf
Two open problems in Stellar are the mechanism by which quorums are chosen (peer selection) and how new arguments may be proposed such that contention is low (i.e. avoid dueling proposers).
Stellar consensus can be extremely inefficient in terms of number of messages sent, especially with dueling proposers.
Is Stellar As Secure As You Think? from 2019 suggests some weaknesses:
According to our results, all nodes get stuck after only two nodes fail. This implies that, currently (Jan. 2019), the value of x in Stellar is approximately 4.5 (≈2/44×100), where the total number of active validators is 44 and f is 2. To make matters worse, the two nodes that cause the entire system to be stuck are run by the same organization, namely the Stellar foundation, making Stellar vulnerable to “single point of failures”. In summary, our contributions are as follows:
- We analyze FBA and prove that it is not superior to PBFT in terms of safety and liveness.
- We conduct a data analysis on the Stellar system, and show that the structure of quorum slices is highly centralized.
- We study cascading failures considering the current quorum slices. Our results imply that validators cannot achieve a consensus after deleting only two nodes run by the Stellar foundation.
A subsequent paper from the SDF, Fast and secure global payments with Stellar addresses these concerns:
However, nodes’ configurations were such that liveness (though not safety) depended on us, the Stellar Development Foundation (SDF); had SDF suddenly disappeared, other node operator swould have needed to intervene and manually remove us from quorum slices for the network to continue.
While we and many others want to reduce SDF’s systemic importance, this goal received increasing priority after researchers  quantified and publicized the network’s centralization without differentiating the risks to safety and liveness. A number of operators reacted with active configuration adjustments, primarily increasing the size of their quorum slices in an effort to dilute SDF’s importance; ironically this only increased the risk to liveness.