The La Liga leaders dominated possession but could not find the vital breakthrough in the Catalan derby.
Oscar Melendo was the hero for the hosts as he netted with two minutes remaining.
There were victories for Sevilla and Valencia as they both beat their opponents 2-1, with Atletico Madrid and Alaves defeated, respectively.
Here are Wednesday's Copa del Rey results:
Espanyol 1-0 Barcelona
Atletico Madrid 1-2 Sevilla
Valencia 2-1 Alaves
Barca wasted an excellent chance of taking control of their tie against their local rivals, but Espanyol dug deep to earn a famous win.
Messi and Sergio Busquets both went close early on in the game, and it seemed the Blaugrana were going to have a simple day at the office.
However, the hosts stayed in touch as Barca played their usual game and tactics, although a shock didn't appear to be on the cards at half-time.
Sergi Roberto was fouled in the box just after the hour mark by Esteban Granero, earning Barca a penalty and a chance to take the lead.
Messi stepped up to take the spot-kick, but goalkeeper Diego Lopez was equal to the Argentina international's effort.
Barca remained favourites to grab a goal before the final whistle. However, home fans were ecstatic as Melendo found a shock winner after 88 minutes.
The substitute found the bottom corner, with Barca labouring through the final seconds of the first leg.
Football writer Andy Mitten reacted as Espanyol won the game:
A dramatic late show from Sevilla saw the visitors stun Atletico with two goals at the Wanda Metropolitano.
Diego Costa continued his hot streak since returning from the Premier League when he opened the scoring for Atletico in the 73rd minute.
Sevilla grabbed an equaliser with 10 minutes remaining through a deflected Jesus Navas effort, and the drama was to continue as Joaquin Correa grabbed the winner at the death.
Correa was a constant thorn in the side of the hosts, and his goal was a worthy reward as Atleti fluffed their lines.
Valencia mirrored Sevilla's achievements in their first leg at home to Alaves, scoring twice in the second half to gain a vital advantage.
Ruben Sobrino gave the visitors a surprise lead in the 66th minute, but the effort triggered a comeback—to the delight of the Los Che fans.
Goncalo Guedes equalised seven minutes later to set up a grandstand finish in which Rodrigo scored the winner.
The vital strike came after 82 minutes, and the hosts were relieved to be taking a slender lead to Alaves for the second leg on January 24.