How to check null on scala.math.BigDecimal?


The following code raises the NullPointerException. Even there is Option(x._1.F2).isDefined && Option(x._2.F2).isDefined to prevent the null values?

case class Cols (F1: String, F2: BigDecimal, F3: Int, F4: Date, ...) def readTable() : DataSet[Cols] = { import sqlContext.sparkSession.implicits._ sqlContext.read.format("jdbc").options(Map( "driver" -> "com.microsoft.sqlserver.jdbc.SQLServerDriver", "url" -> jdbcSqlConn, "dbtable" -> s"..." )).load() .select("F1", "F2", "F3", "F4") .as[Cols] } import org.apache.spark.sql.{functions => func} val j = readTable().joinWith(readTable(), func.lit(true)) readTable().filter(x => (if (Option(x._1.F2).isDefined && Option(x._2.F2).isDefined && (x._1.F2- x._2.F2< 1)) 1 else 0) //line 51 + ..... > 100)

I tried !(x._1.F2== null || x._2.F2== null) and it still gets the exception.

The exception is

java.lang.NullPointerException at scala.math.BigDecimal.$minus(BigDecimal.scala:563) at MappingPoint$$anonfun$compare$1.apply(MappingPoint.scala:51) at MappingPoint$$anonfun$compare$1.apply(MappingPoint.scala:44) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:395) at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:234) at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:228) at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827) at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:108) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source)

<strong>Update:</strong> I tried the following expression and the execution still hit the line x._1.F2- x._2.F2. Is it a way to check if BigDecimal is null?

(if (!(Option(x._1.F2).isDefined && Option(x._2.F2).isDefined && x._1.F2!= null && x._2.F2!= null)) 0 else (if (x._1.F2- x._2.F2< 1) 1 else 0))

Update 2

<strong><em>The exception is gone after I wrapped the minus into (math.abs((l.F2 - r.F2).toDouble).</em></strong> Why?


Try adding this this to your if statement:

&& (x._1.F2 && x._2.F2) != null

I've had a similar issue in Java and that's what has worked for me.


Looking at the source code for BigDecimal, on line 563: <a href="https://github.com/scala/scala/blob/v2.11.8/src/library/scala/math/BigDecimal.scala#L563" rel="nofollow">https://github.com/scala/scala/blob/v2.11.8/src/library/scala/math/BigDecimal.scala#L563</a>

It may be possible that x._1.F2.bigDecimal or x._2.F2.bigDecimal is null, though I'm not really sure how that would happen, given the constructor checks for that. But maybe check for null there and see if that solves the problem?

BTW, you should really avoid all the ._1, ._2s... You should be able to do something like:

val (l: Cols, r: Cols) = x

To extract the tuple values.


  • Utilising Yii2.0 checkboxlist for Parent-child relation models
  • TSQL: Trigger on Commit
  • Yii 2 ActiveDataProvider query with ->all() gives “Call to a member function andFilterWhere() on
  • disk storage of arrays etc
  • the specified deps.json does not exist angular2 asp.net core application.
  • Embedding SQLServer CE in an installer
  • SQL CLR Trigger - get Target / Table name
  • Increasing the storage of an existing SQL Server RDS instance on AWS
  • Perl Dbi and stored procedures
  • concatenating results from SQL query and NULL columns
  • Can I combine several select queries in .net so they are returned in 1 batch?
  • Rows to Columns using Sql Query
  • new spark.sql.shuffle.partitions value not used after checkpointing
  • Hibernate reverse engineering - mapping a table to Java enum
  • detecting connection lost in spark streaming
  • Glassfish - java.lang.NoClassDefFoundError
  • File extension of zlib zipped html page?
  • iOS Cordova first plugin - plugin.xml to inject a feature
  • Group list of tuples by item
  • IE11 throwing “SCRIPT1014: invalid character” where all other browsers work
  • pyodbc doesn't report sql server error
  • Parsing a CSV string while ignoring commas inside the individual columns
  • Problem deserializing objects from cache on MyBatis 3/Java
  • jQuery .attr() and value
  • Spark fat jar to run multiple versions on YARN
  • ImageMagick, replace semi-transparent white with opaque white
  • PHPUnit_Framework_TestCase class is not available. Fix… - Makegood , Eclipse
  • Cannot connect to cassandra from Spark
  • Update CALayer sublayers immediately
  • PHP - How to update data to MySQL when click a radio button
  • Counter field in MS Access, how to generate?
  • Cross-Platform Protobuf Serialization
  • Statically linking a C++ library to a C# process using CLI or any other way
  • Alternatives to the OPTIONAL fallback SPARQL pattern?
  • Why winpcap requires both .lib and .dll to run?
  • Comma separated Values
  • How do you join a server to an Active Directory (domain)?
  • How does Linux kernel interrupt the application?
  • apache spark aggregate function using min value
  • XCode 8, some methods disappeared ? ex: layoutAttributesClass() -> AnyClass