Skip to content

feat: support Spark format_string expression#3588

Open
n0r0shi wants to merge 1 commit intoapache:mainfrom
n0r0shi:format-string-func
Open

feat: support Spark format_string expression#3588
n0r0shi wants to merge 1 commit intoapache:mainfrom
n0r0shi:format-string-func

Conversation

@n0r0shi
Copy link

@n0r0shi n0r0shi commented Feb 25, 2026

Summary

  • Wire format_string from the datafusion-spark crate (FormatStringFunc) to Comet
  • Register in jni_api.rs and add serde mapping via CometScalarFunction in QueryPlanSerde.scala

Test plan

  • New test in CometStringExpressionSuite — passes
  • Spotless formatting check passes
  • Rust native build passes

Register datafusion-spark's FormatStringFunc UDF and add serde
mapping in stringExpressions.
Copy link
Member

@andygrove andygrove left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean serde wiring. The CometScalarFunction approach and UDF registration look correct.

For the tests, it would be a good fit to use the SQL file framework instead of CometStringExpressionSuite. There are already many string expression SQL files in spark/src/test/resources/sql-tests/expressions/string/. A format_string.sql file there would automatically compare Spark and Comet results.

It might also be worth adding a test with %f for floating-point formatting, since Spark uses java.util.Formatter with US locale and there could be subtle differences in how Rust formats floats. Something like format_string('%.2f', 3.14159) would exercise that path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants