How Big Data & Internet of Things contribute to optimize upstream operations of Oil & Gas industry


The Oil & Gas industry is extremely competitive and highly regulated environment. Against this uncertain environment characterized by the eternal necessity to renewal reserves of natural resources, fluctuating demand and price volatility, Oil & Gas companies need to increase production, optimize costs and reduce the impact of environmental risks.

Oil & Gas upstream sector is complex, data-driven business with data volumes growing exponentially. Upstream organizations work simultaneously with both structured and unstructured data. They must capture and manage more data than ever and are struggling to store, analyze and get useful information from these huge volumes of data. Under these conditions, the traditional analysis tools would fail but with the appropriate infrastructure and tools, Oil & Gas companies can get measurable value from these data.

Big Data Definition

The first definition of Big Data was developed by Meta Group (now part of Gartner) by describing their three characteristics called “3V”: Volume, Velocity and Variety. Based on data quality, IBM has added a fourth V called: Veracity. However, Oracle has added a fourth V called: Value, highlighting the added value of Big Data.

Big Data is defined by six characteristics called “6V”: Volume, Velocity, Variety, Variability Veracity and Value.

Fig. 1. The “6V” of Big Data

“Big Data technologies describe a new generation of technologies and architectures, designed to economically extract value from very large volumes of a wide variety of data, by enabling high velocity capture, discovery and/or analysis, while ensuring their veracity by an automatic quality control in order to obtain a big value”.

These technologies are essentially based on the Apache™ Hadoop® project that’s open-source software for reliable, scalable, distributed computing.

Big Data in the Oil & Gas Upstream Sector

Oil & Gas Upstream sector is no stranger to Big Data and Internet of Things (IoT). They often use thousands of sensors installed in subsurface wells and surface facilities to provide continuous data-collecting, real-time monitoring of assets.

The data volume is coming from sensors and various measuring devices. “Structured” data is handled with specific applications used to manage exploration planning, reservoir modeling, production and other upstream activities. But much of this data is “unstructured” or “semi-structured” such as logging and imaging files should be managed with appropriate tools for Big Data.

To support the real-time decision-making, Oil & Gas companies need tools that integrate various data sources into a unified whole. Being able to process Big Data makes it possible to derive insight from the relationships that will surface when all of these sources are processed as a whole. But to unlock this value, Oil & Gas companies need access to the appropriate technology, tools, and expertise.

Like generic Big Data, the Upstream Data is also characterized by the 6V:

Fig. 2. Upstream Big Data

Oil & Gas companies should take advantage of their Research & Development Centers (CRD) or through close collaboration with existing independent centers to explore the potential of Big Data to resolve problems and technical difficulties they encounter in daily operations and capitalize knowledge to improve performance. They must in this case, taking into account the requirements of a shared collaborative environment that supports the flow of total production of the working groups.

To solve problems, R&D workgroups should have permanent eye on the wealth of data contained in the patent databases to be inspired particularly as these databases are considered as Open Big Data.

The traditional step of data pre-processing (data cleaning step in the conventional information processing) must be banned when processing huge volume of geophysical and geological data. The reason is simple ! Often through our cognitive bias, we exclude all data that doesn’t fit into our established « scientific » framework. As a result, we miss the opportunity to discover new visualization of unexplained phenomena.

Big Data contributes significantly to reduce risk and optimize costs related to operations and Health, Safety and Environment:

  • Prevent undesired events while drilling like kicks,
  • Predict drill maintenance/downtime, optimize drill parameters and prevent blowout accidents,
  • Using weather or workforce scheduling data to avoid creating dangerous conditions for workers and mitigating environmental risks.


Leading Oil & Gas companies are already began projects to deploying Big Data technologies that can help them track new business opportunities, reduce costs and reorganize operations.

By recognizing the value of the unexploited data assets in supporting fact-based decision-making, Oil & Gas companies establish real cases of Big Data uses. Then they can create enhanced business value based on innovation and able to lead towards a sustainable competitive advantage.

Oil & Gas companies must first proceed to a gap analysis to determine the major requirements of technology and data-management expert staff. This allows a focused investment in mature and proven technologies as well as those who will face the exponential growth of data volumes.

Oil & Gas companies must create new strategies that will help them manipulate these data and use them to support experts in their business process and managers in their decision-making process.


  1. Baaziz A. & Quoniam L. (2014). How to use Big Data technologies to optimize operations in upstream petroleum industry. 21st World Petroleum Congress, Moscow. June 19, 2014.
  2. Baaziz, A. & Quoniam, L. (2013). How to use Big Data technologies to optimize operations in Upstream Petroleum Industry. International Journal of Innovation – IJI, 1(1), 19-25. doi:10.5585/iji.v1i1.4 .