Skip to content

feat: support make_dt_interval and make_interval expressions#3591

Open
n0r0shi wants to merge 1 commit intoapache:mainfrom
n0r0shi:make-interval-funcs
Open

feat: support make_dt_interval and make_interval expressions#3591
n0r0shi wants to merge 1 commit intoapache:mainfrom
n0r0shi:make-interval-funcs

Conversation

@n0r0shi
Copy link

@n0r0shi n0r0shi commented Feb 25, 2026

Summary

  • Add serde support for make_dt_interval and make_interval expressions
  • Both functions already exist in the datafusion-spark crate, so this is serde-only wiring

Test plan

  • Added tests in CometTemporalExpressionSuite

Register datafusion-spark's SparkMakeDtInterval and SparkMakeInterval
UDFs and add serde mappings in temporalExpressions.
@andygrove
Copy link
Member

Thanks for wiring up these two expressions. The serde and registration changes look correct to me.

I have a couple of suggestions on the test side.

The project has been moving expression tests to the SQL file-based framework (CometSqlFileTestSuite) rather than adding to Scala test suites. You can see examples in spark/src/test/resources/sql-tests/expressions/datetime/ (like make_date.sql). Would you be open to moving these tests to SQL files instead? The framework automatically runs each query through both Spark and Comet and compares results, so you don't need the checkSparkAnswerAndOperator calls. There's a guide at https://datafusion.apache.org/comet/contributor-guide/sql-file-tests.html if that helps.

It would also be good to add some null handling coverage. Both expressions are null-intolerant per the Spark spec, so testing with NULL column values (not just non-null data) would help catch any issues. Something like inserting a row with NULLs in the test table would do it.

@andygrove
Copy link
Member

Test failure:

- make_dt_interval *** FAILED *** (145 milliseconds)
  org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 1190.0 failed 1 times, most recent failure: Lost task 1.0 in stage 1190.0 (TID 3164) (localhost executor driver): java.lang.UnsupportedOperationException: Unsupported data type: Duration(MICROSECOND)
	at org.apache.spark.sql.comet.util.Utils$.fromArrowType(Utils.scala:109)
	at org.apache.spark.sql.comet.util.Utils$.fromArrowField(Utils.scala:82)
	at org.apache.spark.sql.comet.util.Utils.fromArrowField(Utils.scala)
	at org.apache.comet.vector.CometDecodedVector.<init>(CometDecodedVector.java:49)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants