The 65-year-old Good Morning Britain presenter went head-to-head with former Saturdays singer Frankie Bridge to win the keys to Gwrych Castle in north Wales and secure a place in the main camp for their team.
Madeley lost to 32-year-old Loose Women presenter and landed his fellow Yellow Team members living on basic rations in The Clink.
Arriving on the show Madeley admitted: "Some people can't stand me, everyone's a bit of marmite aren't they...?"
But when presenters Ant McPartlin and Declan Donnelly revealed the public had voted for him and Bridge to take part in the first trial, Madeley admitted he was feeling, "taken aback."
He added: "They don't even know what we're like!"
Madeley then complained at being pitted against Bridge: "I'm twice her age!"
The former This Morning presenter and Bridge had to race up turrets by unlocking padlocks whilst being covered in live cockroaches, maggots, crickets and mealworms.
Bridge screamed: "Oh my God they stink! They're all down my back. I feel like they're biting."
Madeley confessed: "Something is biting very, very hard."
He told them: "I can only apologise, I did my best."
Madeley then further let his team down when he failed to light a fire.
After French footballer Ginola, 54, managed to get a camp fire going, Miller said: "I feel like Richard Madeley went for the edit, it's hit the post, David Ginonla's just rifled it in there!"
He added: "David Ginola is a bit of an icon to me. Still like so cool and good looking. I hope I look like him at his age because he is just fit isn't he?!"
Phillips revealed of the living conditions in The Clink: "Oh my, we are open to the elements, let's hope it doesn't rain.
"I'm about to sleep in probably the single worse bed and place that I have come across in all my 78 years."
Ginola was horrified by the cold shower and basic living, exclaiming: "There's no towels! It's horrible,"
Read more: Meet the I'm A Celebrity 2021 campmates
Bridge and the Red Team - BBC Breakfast presenter Louise Minchin, Paralympic athlete and Celebrity MasterChef winner Kadeena Cox, Olympic diver Matty Lee and Radio 1Xtra DJ Snoochie Shy - are living in the main camp.
Soap actor Danny Miller has been voted by the public to take part in the next Trial - an eating task titled 'The Dreaded Diner.
I'm A Celebrity...Get Me Out Of Here! continues at 9pm on ITV.
Watch: Who is I'm A Celeb's Richard Madeley