• Big not always better when it comes to size of data sets.
• Researchers at MIT say they can predict best outcomes with greater precision.
• Define the data set, then solve.

Some of the most positive results in the practical deployments of technology can come from niche problems being solved using carefully chosen, highly-selective data sets, researchers claim.

A new method developed at the Massachusetts Institute of Technology can help identify the smallest possible data set that can solve a problem, and suggest the best solution once that data is assembled. In addition to the obvious advantage of being able to solve a problem, a smaller, more focused data set requires fewer measurements, less compute, and shorter data gathering processes.

A post on MIT’s website outlines an example problem of determining the least expensive path for a new subway line under a city. Each potential path would mean different costs, emanating from construction costs, planning requirements, complexities of routing around existing underground infrastructure, road closures, logistics, and a dozen more variables. Additionally, factors at play might mean that the cheapest route to construct may mean long travel times, or require large amounts of power to run the trains. Conversely, the fastest route for travellers may be the most expensive to build, and so on.

MIT’s method considers the underlying structure that underpins the problem (in this case, the city layout, power costs, construction limits, and so on) and identifies the smallest set of locations where field studies would be most effective. Its proposed method then identifies how to use the data collected from the field studies to find the most effective route.

“We’ve shown that with careful selection, you can guarantee optimal solutions with a small dataset, and we provide a method to identify exactly which data you need,” said Asu Ozdaglar, Mathworks Professor and head of the MIT Department of Electrical Engineering and Computer Science (EECS).

The team developed a mathematical characterisation of ‘optimality regions’ which define how various decisions could be optimal, based in this case on fastest travel time, construction expense, energy price, and so on. The algorithm then loops, asking “Is there any scenario that would change the optimal decision in a way my current data can’t detect?” If the answer is ‘yes’, it adds a the need for a measurement that describes that difference.

The resulting data set, defined in this way, is often surprisingly small, the researchers found. “When we say a dataset is sufficient, we mean that it contains exactly the information needed to solve the problem. You don’t need to estimate all the parameters accurately; you just need data that can discriminate between competing optimal solutions,” said Amine Bennouna, former MIT postdoc, now an assistant professor at Northwestern University.

After the data is collected, a further algorithm finds the best solution. “The algorithm guarantees that, for whatever scenario could occur within your uncertainty, you’ll identify the best decision,” said Omar Bennouna, an EECS graduate student and one of the team’s researchers.

The teams says there’s a misconception that smaller data sets mean less accurate results. “These are exact sufficiency results with mathematical proofs […] not probably, but with certainty,” Bennouna said.

The team want to engage in deeper research with more complex situations, and study the effects of ‘noisy’ observations on datasets.

The current research [PDF] is to be presented at the Conference on Neural Information Processing Systems in San Diego, early December.

(Image source: “2007.08 – ‘View over the long construction site of the new metro tunnel’, under the city-center of Amsterdam, for the future Noord-Zuidlijn, location Vijzelgracht, old center; Dutch urban city photo + geotag, Fons Heijnsbroek, The Netherlands” by Amsterdam free photos & pictures of the Dutch city is marked with CC0 1.0.)

Want to experience the full spectrum of enterprise technology innovation? Join TechEx in Amsterdam, California, and London. Covering AI, Big Data, Cyber Security, IoT, Digital Transformation, Intelligent Automation, Edge Computing, and Data Centres, TechEx brings together global leaders to share real-world use cases and in-depth insights. Click here for more information.

TechHQ is powered by TechForge Media. Explore other upcoming enterprise technology events and webinars here.