A European intelligence agency is understood to have tipped off Vienna about the suspected mole.
If confirmed, the unmasking of a spy in the heart of Europe would be the latest blow for President Vladimir Putin's intelligence services.
But it would also demonstrate the extent of Russia's hostile intelligence operations, penetrating a country that is one of Moscow's closest allies in the European Union.
The development comes after Britain and its allies exposed a botched plan by Russian military intelligence to hack the international chemical weapons watchdog.
Two GRU officers were also named as the prime suspects in the nerve agent attack against former Russian spy Sergei Skripal and his daughter Yulia in Salisbury.
Austrian chancellor Sebastian Kurz repeatedly referred to the foiled plot against the Organisation for the Prohibition of Chemical Weapons when he spoke to reporters about Austria's suspected Russian agent.
"Based on the information we have, but also because of recent events for example in the Netherlands, we can very much assume at the moment that our suspicion will be confirmed," he said.
The alleged mole is a recently retired colonel who was believed to have spied for Moscow from the 1990s until this year, the Austrian leader said.
Weapons systems and migration into Europe were among the issues the suspected spy or his handlers were said to have been interested in.
"If the suspicion is confirmed, such cases... do not improve relations between Russia and the European Union," Mr Kurz said, without naming the suspect.
The case has been referred to prosecutors and the former colonel has been questioned, according to Reuters.
The move is particularly notable because Austria has close ties to Russia, which are understood to have caused concern for a number of countries, including the UK.
Mr Kurz governs in coalition with the far-right and pro-Moscow Freedom Party, while foreign minister Karin Kneissl danced with Mr Putin at her wedding in August.
Austria was in the minority of EU countries that did not expel any Russian diplomats over the poisoning of the Skripals in March and the chancellor has met twice with the Russian president since the attack.
More than 50,000 people have signed Sky News' petition backing televised leaders' debates for elections - have you?
In a possible shift in attitude, Austria's foreign minister has cancelled a planned trip to Moscow over the suspected spy case and summoned the Russian charge d'affaires since the ambassador is outside the country.
"For the moment we are demanding transparent information from the Russian side," Mr Kurz said.
Russia reacted with indignation to the allegation.
More from Austria
Putin on the Ritz: Russian president dances at foreign minister's wedding
Attacker shot dead outside Iran diplomat's Vienna residence
Skiers swing wildly on lift during storm at Austria resort
Skiers trapped as lift is buffeted by storm
Austria court legalises same-sex marriage from 2019
Austria election winner Sebastian Kurz to have coalition talks with far-right Freedom Party
Foreign minister Sergei Lavrov said he was "unpleasantly surprised" and claimed Moscow knew nothing about the retired officer, local news agencies reported.
Russia had in turn summoned the Austrian ambassador, reports said.