The promoted side took the lead through John McGinn after 20 minutes, and Arsenal's Ainsley Maitland-Niles was sent off for a second yellow four minutes before the interval.
Arsenal won a penalty after Bjorn Engels fouled Matteo Guendouzi after 59 minutes, allowing Nicolas Pepe to convert from the spot for his first goal since joining from Lille.
However, it took Villa only 90 seconds to regain the lead after Wesley met Jack Grealish's cross to make it 2-1.
Arsenal continued to chase the game, and two late goals from Calum Chambers and Pierre-Emerick Aubameyang sealed an unlikely victory.
Chambers found the back of the net after 81 minutes, and Aubameyang's direct free-kick won the match just three minutes later.
Journalist Jonny Gould applauded Villa's progress in the top-flight:
The Gunners did not cover themselves in glory during the first half, with their defence capitulating early on. Anwar El Ghazi whipped in a ball behind the Arsenal back line, and McGinn was allowed to flick the ball into the back of the net without a challenge.
The hosts attempted to immediately fight their way back into the game, but they laboured in midfield despite dominating possession. Arsenal struggled to test Villa goalkeeper Tom Heaton, and the first half ended with Unai Emery's side reduced to 10 men.
Maitland-Niles had picked up a caution after only 11 minutes, and a poor tackle on Neil Taylor saw the 22-year-old receive a second yellow.
Villa maintained their high standards at the start of the second half, with Grealish and McGinn in good form, but it was Arsenal who grabbed the equaliser shortly before the hour mark.
Guendouzi sprinted into the penalty area. Engels fouled the Arsenal man, and Pepe made it 1-1 from the spot.
Home fans had barely stopped celebrating their goal before Villa once again took the lead in the game.
Wesley was clinical with his finish at the near post after a superb dribble and cross from the influential Grealish.
Arsenal chased an equaliser once again, and David Luiz's free-kick was tipped over the crossbar by Heaton. Joe Willock and Aubameyang both went close as the home side put their foot on the gas.
The Gunners completed a famous comeback in the closing stages, with the 10 men scoring twice to snatch the points.
Chambers found the back of the net from close range with nine minutes remaining, and Aubameyang was the hero moments later with a free-kick that beat Heaton.
Villa had a late penalty appeal for handball denied by the video assistant referee, and the Gunners were relieved at the full-time whistle.
Arsenal and Villa both feature next in the Carabao Cup, with the Gunners hosting Nottingham Forest on Tuesday and Villa travelling to Brighton & Hove Albion on Wednesday.