The Roma defender curled the ball around the wall and into the top corner on 56 minutes, giving goalkeeper Keylor Navas no chance.
Opta showed how it is the third direct free-kick goal of the tournament:
Elsewhere there were a couple of surprise results as defending champions Germany slipped to defeat to Mexico, and Brazil were held to a 1-1 draw by Switzerland.
Here is a look at all Sunday's latest results, followed by a check on Monday's schedule.
Sunday's Latest Results
Costa Rica 0-1 Serbia (Group E)
Germany 0-1 Mexico (Group F)
Brazil 1-1 Switzerland (Group E)
3 p.m. local time/1 p.m. BST/8 a.m. ET, Sweden vs. South Korea (Group F)
6 p.m. local time/4 p.m. BST/11 a.m. ET, Belgium vs. Panama (Group G)
9 p.m. local time/7 p.m. BST/2 p.m. ET, Tunisia vs. England (Group G)
Serbia and Costa Rica traded chances in the first half. Aleksandar Mitrovic of the former headed wide in the opening minute. The action then switched to the other end, as Costa Rica's Marco Urena tested goalkeeper Vladimir Stojkovic with an angled shot.
Urena wasted another chance as he fired over when well placed, while Sergej Milinkovic-Savic's overhead kick was saved by Navas.
Kolarov made no mistake when he was presented with a free-kick after David Guzman stopped Mitrovic's driving run towards the penalty area just before the hour mark.
The Roma defender scored the only goal of the game with a superb strike from 25 yards. Former England goalkeeper David James offered his view of the effort:
It was a strike fit to win any match, and Costa Rica could not find a way back into the game. Football writer Kristan Heneage was not too impressed with Costa Rica:
The win gives Serbia's hopes of qualifying for the knockout stages a big boost, although there is still some way to go. They are in a tough group that also contains Brazil and Switzerland.
However, manager Mladen Krstajic's squad has plenty of quality, and highly rated midfielder Milinkovic-Savic was particularly impressive. Goal's Carlo Garganese was wowed:
Both teams are back in action on Friday, when Serbia take on Switzerland while Costa Rica face Brazil.
Germany must bounce back from an opening day defeat after being beaten by Mexico. Hirving Lozano scored the only goal of the match as El Tri secured victory.
Mexico struck on 35 minutes as Javier Hernandez raced clear and picked out Lozano on the left. The winger managed to cut inside before firing low past a diving Manuel Neuer in the Germany goal.
Kirk Semple of the New York Times showed how the strike made the earth move in Mexico City:
Germany almost equalised when Toni Kroos' free-kick hit the crossbar but that was the closest Die Mannschaft came to scoring.
Mexico had further chances to extend their lead as they caused Germany all sorts of problems in a vibrant attacking display.
Indy Football showed how Mexico fans celebrated the win:
It was a deserved and impressive win for El Tri, while Germany have plenty of work to do in their remaining group games against Sweden and South Korea.
Brazil also need to improve after being held by Switzerland on Sunday. Philippe Coutinho's trademark goal looked set to send the Selecao on their way to victory, but they were pegged back in the second half by Steven Zubar's header.
Selecao Brasileira highlighted where it went wrong for Brazil:
The result means that neither Brazil, Germany, Argentina or Spain have managed a win in their opening game and leaves the World Cup looking wide open.