If you think that Pearl would have participated in that, I would personally be down with her having been involved in taking him away. I imagine that pre-all of that Steven was probably fairly fond of Pearl conceptually because she seemed really cool, even if he didn't end up seeing her a lot. Kind of like how he kind of admired the CGs in the show when he was younger.

Pearl also would have probably been allowed in the same areas of the fae realm that Steven would have been kept, so she could have been exposed to him there. Though he was not having a good time pretty much ever.