How we made SEG-Y slicing fast

But before we get to that, how does an industry-standard format become a bottleneck this early?

SEG-Y was built in the 1970s to store seismic data on magnetic tape, and fifty years later it’s still the format the whole industry runs on. It simply wasn’t built for fast reading, but now, when speed matters most, it becomes a hurdle.

The problem is in how it lays out the data: traces are written back to back, ordered by inline. Reading a full inline means pulling one contiguous run of bytes off the disk, which is fast. Crosslines are slower: each needs one trace from every inline, scattered across the volume. But depth/time slices are the worst case: in order to be read, each one of them needs a single sample from every trace in the cube, requiring millions of random-access operations across the entire file — far slower than any inline or crossline read.

With the format came the libraries to read it: the best-known is segyio, the standard open-source reader; faster ones, like segfast, exist too. They parse the headers, map every trace to its byte offset, and read only the traces you ask for. But the file’s layout is unchanged, so the slow directions remain relatively slow.

There’s a different approach, ditching SEG-Y altogether by converting the cube into a quicker format, like VDS or ZGY. But since almost all the field data still comes in SEG-Y, the price is the conversion itself: it takes time, leaves you maintaining a second copy of the data (often larger than the original), and with ZGY, ties you to specific software.

With SegAir, our Rust-based engine for SEG-Y slicing, we wanted to take on that bottleneck directly without any conversions. It reads the scattered traces in parallel across all cores, decodes IBM floats (the format most field SEG-Y uses to store its samples) with SIMD kernels, and pulls slices about twice as fast as segfast.

SEG-Y is here to stay, and while new formats will appear for specific tasks, they won’t replace it. So we’ll keep making it faster.

Has a format ever turned into a bottleneck for you?