He said: "I would've liked to have kept Paula. She was very gracious and I said that to her. She understands it's business; it's never personal."
With Nicole Scherzinger and show host Steve Jones also fired from 'The X Factor', only record company boss L.A. Reid has kept his job as a judge alongside Simon, and the 52-year-old star says they kept him because there is no one who could do better.
Simon told 'Extra': "There's not a record executive out there who is as good as him for the job."
Following Paula and Nicole's departure, it had been claimed Beyonce Knowles had been offered $100 million to join the show, but Simon insists the speculation is untrue.
He said: "I have no idea where this came from. It's complete nonsense."
While Simon would like to see Mariah Carey on the panel, he doesn't think it would happen because of her career commitments and responsibility to her family, husband Nick Cannon and their nine-month-old twins Moroccan and Monroe.
He said: "I adore her, love her. But you can't expect someone who has just had two kids and has a record career to make the kind of commitment we would need on this show."
Instead, Simon thinks Madonna would be a "great" judge, though she would be "expensive", while he is also keen on Katy Perry, saying the 'Firework' hitmaker "would be fun - and she's feisty."
