feat: (perf) allow spawning multiple tasks per read#2156
feat: (perf) allow spawning multiple tasks per read#2156tafia wants to merge 2 commits intoapache:mainfrom
Conversation
Scanning of all files is both cpu and io intensive. While we can control the io parallelism via concurrency_limit* arguments, all the work is effectively done on the same tokio task, thus the same cpu. This situation is one of the main reason why iceberg-rust is much slower than pyiceberg while reading large files (my test involved a 10G file). This PR proposes to split scans into chunks which can be spawned independently to allow cpu parallelism. In my tests (I have yet to find how to benchmark it in this project directly), reading a 10G file: - before: 38s - after: 16s - pyiceberg: 15s
|
The error seems unrelated to the PR (python) or wrong (need to get iterator ownership not just elements) |
|
I'll take a look. In theory there's nothing stopping from generating FileScanTasks that span pieces of the files now (this is what Comet does with the existing reader). I've still been running into mismatches in parallelism though and trying to get more CPU utilization out of the Iceberg scan stages of Comet jobs, even when we've properly dispatched a bunch of I/O requests. I suspect you could be onto something here. Thanks! I'll take a pass this week. |
Yes, this is far from an optimal solution but at least it is a simple move in the right direction. Fyi, I've also added some row-group and columns parallelism but the changes are more complex and not ready to be merged. |
blackmwk
left a comment
blackmwk
left a comment
There was a problem hiding this comment.
Thanks @tafia for this pr. Do you mind to try datafusion integration rather than using arrow reader directly? I'm declining to make the to_arrow method more complicated. If you want a high performance local query engine, using datafusion is the right direction.
3 participants
Scanning of all files is both cpu and io intensive. While we can control the io parallelism via concurrency_limit* arguments, all the work is effectively done on the same tokio task, thus the same cpu.
This situation is one of the main reason why iceberg-rust is much slower than pyiceberg while reading large files (my test involved a 10G file).
This PR proposes to split scans into chunks which can be spawned independently to allow cpu parallelism.
In my tests (I have yet to find how to benchmark it in this project directly), reading a 10G file:
Which issue does this PR close?
I haven't found any particular issue but several comments are referring to cpu bounded processing.
What changes are included in this PR?
This PR proposes to split scans into chunks which can be spawned independently to allow cpu parallelism.
Are these changes tested?
I have added a test to show that the change doesn't affect the output. I have yet to find a good benchmark to prove my claim about the performance. Any tip on how I could do would be welcomed!