Do you recall that children’s puzzle called Where’s Waldo or Where’s Wally? It’s the one where you have to find a character in red and white stripes and blue pants. You know how hard it can be to find him, right? That is because there is a lot going on in the picture!

Like in Waldo's puzzle, a lot is happening in a supercomputer where multiple scientific applications are running. Scientists use these machines to run experiments to understand the world we live in, from atoms to stars. Those experiments often take hours, days, or even weeks to complete, generating massive amounts of data, and most of the time, not in an efficient way.

For instance, when running a regional-scale earthquake simulation, we can get 75 TB of data from a single experiment. That is roughly the same amount of data if you would watch Netflix in 4K resolution for over one and a half years. That is a huge amount of data, and data movement performance must be efficient! Because of the complexity of today's supercomputers, it is harder to pinpoint the data movement performance problems than to find Waldo! 

Here at Berkeley Lab, in the Scientific Data Division, we are working towards enabling faster science so scientists can get the data they require as fast as possible without having to worry about performance issues. We do that by using metrics comprising all aspects of a supercomputer complexity and applying different techniques from statistical analysis, data mining, and machine learning to identify those issues and map those to a subset of viable solutions so applications can run faster. It’s like devising an efficient way to find Waldo instead of randomly looking at the picture.

Because we work at the system level, we have a unique opportunity to interact with distinct science domains where applications can benefit from our solutions. For instance, in our initial results, an astrophysics application got 4 times faster and a linear-algebra application almost 200 times faster. All of this by intelligently detecting data movement problems and mapping to solutions based on the unique characteristics of each application and domain!

We strongly believe our research can help find and fix data movement performance issues automatically so scientists can focus on what matters the most: the science they do. You can think of it as a very smart, fast,  and automatic strategy to find Waldo in the puzzle.