You may consider books with dysfunctional relationships those in which REPEATEDLY were happening damaging, hurtful actions between the main characters, either done only by one part to the other, or done to each other throughout the book.
It’s about all those actions that might make the relationship look unhealthy, that more than one time could or did endanger the physical, mental or emotional stability of the protagonists.
This list doesn’t exclude the books that aren’t targeted as romance books or love stories. Therefore the relationship could be purely sexual without involving any romantic feelings. The ending could be positive or not, the relationship could become functional or not.

The books you vote for or add to this list should contain more than one scene with the above description.
The repeated scenes between the protagonists that are qualifying here could include the following: dubious or non-consensual sexual content; BDSM that doesn’t comply with the consent, safety and the communication required; abuse of any nature; violence between the mc's; blackmailing; threats; psychological manipulation; bullying; verbal attacks; cheating; lying as an usual practice between the mc’s; substance abuse impacting the interactions of the mc’s; selfish attitude and behavior

Related lists:
M/M Books With Negative/Offensive/Hurtful/Aggressive/Damaging Actions Between Protagonists (possible spoilers)