This is almost an impossible question. I love them both so much!
I've read both books. Phantom is the quicker, easier read, so I've read it multiple times. I love the romance and mystery. Les Mis is a much more sweeping, epic tale of politics, revolution, and the lives caught in the middle. It certainly makes a grander statement than Phantom.
I've never seen either of the Broadway musicals, but I have attended local productions of Phantom and thoroughly enjoyed them. I've seen a few clips of Les Mis online and it also seems very good. The rendition I saw of Castle on a Cloud was particularly moving.
I've seen multiple movie adaptations of both. The original Lon Chaney version of Phantom is a classic. The 2004 version was visually stunning and I adore the soundtrack, but I was disappointed in the movie itself. It lacked heart in my opinion. I couldn't form an emotional connection to the characters. I thought the 2012 version of Les Mis was, quite frankly, magnificent.
Regarding soundtracks, as I said, the 2004 Phantom is one of my favorites. Despite my opinion of the movie, I can play the soundtrack and hear all the emotion I felt was lacking. Perhaps the lack of visuals lets me focus more on the nuances in the songs and inject my own interpretation? My husband and I used the 2004 rendition of All I Ask of You in our wedding, so there's definitely a special place for it in my heart. While the songs in Les Mis are good and work within the context of the story, outside of I Dreamed a Dream and Castle on a Cloud, I would likely never listen to them on their own. Those two songs, though, are heart-rending.
Overall, I think Les Mis is probably better from a technical standpoint, but Phantom has my heart.