Technology has delivered more data to the oil industry than ever before, thanks to penetrating satellites and powerful computers. But oil markets may be getting less transparent as Western sanctions and other measures push more of the global trade into the shadows. Data on flows and inventories are crucial to how the market prices oil, and while there's no shortage of it available, reliable information is getting harder to come by. A combined 15% of all traded crude — volumes from sanctioned Russia, Iran and Venezuela — now takes place in the shadows. These producers are getting better at hiding their intentions and activities in oil markets, and the resulting loss of transparency is making it harder for forecasters to accurately assess supply and demand trends. Forecasters' job was already getting more difficult because of changing consumer habits in response to the pandemic, confusing macroeconomic signals, and a breakdown of established trends due to the accelerating energy transition. Growing reams of data never guaranteed a fully transparent oil market, but they help inform insights. Now the data is less reliable. Data analytics firms like Kayrros, Kpler and Vortexa produce huge volumes of raw numbers on storage tanks and tanker positions that analysts translate into relevant narratives. Satellites show fairly reliable information on crude inventories, which helps assess hard-to-calculate stocks in non-OECD countries that don’t report them — or perform checks on those that do. These firms also track tankers and their cargoes. But Iran and Venezuela have become savvy at evading satellite tracking in recent years. And Russia, which saw its exports banned in much of the OECD after its February 2022 invasion of Ukraine, is moving some of its volumes off radar, especially refined products.

Crucially, significant trade has moved from the more transparent Atlantic Basin into non-OECD Asia, traditionally a region with limited reliable data — even for above-board trades. Lacking fully accurate pricing and destination data for 6.5 million of barrels per day of crude and around 3 million b/d of refined products is a major setback for a market that technology had been shining a brighter light on. Even though the shadow trade has its own pricing and freight rituals, its volumes and discounts impact regular trades in the conventional oil market. China and India are buying the bulk of the more than 2.5 million b/d of Russian crude that used to go to OECD countries, including 2.1 million b/d to the EU. Tanker trackers can still see most of that Russian trade but lose track when volumes get comingled. Iranian crude exports of more than 1 million b/d and some 600,000 b/d of Venezuelan volumes have gone mostly dark and are thought to end up mostly in Asia as well. Asia imports 23 million b/d of crude, of which over 13 million b/d comes from the Middle East, mostly on term contracts. Much of the rest is bought in the spot market, which is less transparent with sanctioned crude clouding the picture. More Russian products are now moving into Asia also.

Commercial data collection assists government reporting, and together they are crucial for transparency, which helps with market stability. But a breakdown is occurring here also. Russia has started limiting the publication of detailed production data. It still provides headline production volumes as a member of Opec-plus to Opec’s secondary sources, including Energy Intelligence, but verifying that data has become harder. Producers and consumers have increasingly intensified cooperation over the past two decades in a quest for better data. But these improvements have not always yielded stability, as massive price swings since 2000 show — from negative values in 2020 to peaks over $140 per barrel in 2008 and $130/bbl in 2022. Organizations have continued efforts to broaden data collection. China and India became associate members of the Paris-based International Energy Agency. The International Energy Forum, based in Riyadh, has tried to become the non-partisan repository of OECD and non-OECD data with the Joint Organizations Data Initiative (Jodi) database. But even Jodi received Russia’s last update on production and refinery runs in February.

Of course, a lack of transparency is nothing new in oil markets, where information has always commanded a premium. Top commodity traders like Vitol, Glencore, Trafigura and Gunvor move millions of barrels every day largely outside direct view. The traditional Western-dominated system has tried to limit corruption payments, demand reputable insurance and encourage use of younger and safer tankers. The results have not been perfect, and the growing shadow market now promises greater challenges. The EU is already investigating reports of falsified documents of Russian trade, as officials suspect that Russian diesel is still entering the bloc via third countries.