Features and enhancements done by Yash0215 · Pull Request #299 · awslabs/deequ

Yash0215 · 2020-09-29T15:11:24Z

New Features:
1.
A Date Time Distribution analyzer for analyzing the distribution of the records based on 'DateType' or 'TimestampType' feature within fixed time intervals.
files changed/created:
DateTimeDistribution.scala
DateTimeAggregation.scala
DeequFunctions.scala
...

6 new Constraints added covering more use cases for DateTime quality checks:
files changed/created:
Check.scala
Constraint.scala

Constraint 'isContainedIn' is now supports more Scala Numeric Types
files changed/created:
Check.scala

Enahancements:

Issue: Timestamp support Timestamp support #47
New State and Metric is implemented for this enhancement since previous analyzer only support Double Metric and Standard analyzer. a new abstract analyzer for timestamp analysis is implemented.
files changed/created:
MinimumDateTime.scala, MaximumDateTime.scala (for new analyzer implementation)
Analyzer.scala
Analyzer for Precision and Scale of BigDecimals Analyzer for Precision and Scale of BigDecimals #46
New State and Metric is implemented. Also new analyzers that provides precision and scale of Spark's 'DecimalType'.
files changed/created:
Minimum.scala, Maximum.scala, Sum.scala, Mean.scala (for new analyzer implementation)
Analyzer.scala

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Features: > Support Spark's DateType and TimestampType > min and max analysis on date time fields

> New constraints for Spark's DateType and Timestamp Types

> BigDecimal State and Metric Support in Min, Max, Sum and Mean Analyzers

> 'isContained' is made available for all Numeric Types and Char

> Analyzer - DateTimeDistribution for analyzing distributions metrics (count, ratio) over some time intervals

sscdotopen · 2020-10-07T08:38:13Z

Thanks! Could you look at the CI failures and fix them?

lucene · 2020-10-15T09:04:31Z

Hi, is this feature now available?

Yash0215 · 2020-10-15T09:24:02Z

Thanks! Could you look at the CI failures and fix them?

sorry for the late reply. Ill work on it.

lucene · 2020-10-15T11:59:14Z

This is valuable feature you've added @Yash0215. awesome. Looking forward to using it.

lucene · 2020-10-15T13:13:10Z

Can I assist in anyway @yash021? I'm looking to utilise this functionality asap.

Yash0215 · 2020-10-26T06:07:43Z

please recommend if any changes are needed.

aviatesk · 2020-10-30T08:28:06Z

So the two enhancements implemented in this PR are essentially orthogonal, right ? If so I'd recommend we split them into separate PRs and focus on each.

sscdotopen · 2020-10-30T09:10:05Z

Yes, that would make a lot of sense.

rounakdatta · 2021-02-26T17:06:14Z

Hello @Yash0215 @sscdotopen, were there furthur developments on this work? If not, I can volunteer to take it forward.

aviatesk · 2021-02-26T17:10:47Z

FWIW, I forked this project and am actively developing it afterwards.
See this changelog to see the enhancements/bugfixes.

I'm also thinking of announcing it as an active fork of this project somewhere (maybe as an issue for this repo) in the near future.

twollnik · 2021-07-20T14:57:24Z

Hi, thanks so much for introducing all these changes. Unfortunately, we currently don't have availability to give this a proper review. Will keep this PR in the backlog for now. If you have the opportunity to submit a couple of smaller reviews that would be great. It's hard to find the time to do big reviews and a few smaller PRs could help us understand the main ideas and make progress on this.

twollnik · 2021-10-19T07:48:13Z

@Yash0215 Please get back to us on this if you get the chance. We are considering closing this PR soon.

RunnX · 2022-06-14T01:18:21Z

@twollnik @Yash0215 - Will this PR be merged to improve deequ to handle timestamp/date support?

suadhika · 2023-02-20T11:26:05Z

Any update on this? to improve deequ for handling of timestamp/date support?

zeotuan · 2024-04-18T11:41:31Z

This PR is quite big with multiple unrelated changes making review hard. Chunk this into multiple smaller PR should be a good start. I am interested in timestamp/date support

github-actions

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: 38d3f74c) — may not be fully accurate. Reply if this doesn't help.

github-actions · 2026-05-06T21:52:04Z

  def metricValue(): Double
 }

+/** A state which produces a BigDecimalValued metric  */


The BigDecimalValuedState trait's metricValue() returns java.math.BigDecimal which can be null (as seen in BigDecimalMeanState.metricValue() when count is 0). This is unsafe — callers like BigDecimalScanShareableAnalyzer.computeMetricFrom wrap it in Success(theState.metricValue()), which would produce Success(null) rather than a proper Failure. Consider returning Option[BigDecimal] or handling the null/zero-count case before wrapping in Success.

github-actions · 2026-05-06T21:52:04Z

@@ -52,6 +54,11 @@ trait DoubleValuedState[S <: DoubleValuedState[S]] extends State[S] {
  def metricValue(): Double


The Analyzer trait has lost its extends Serializable declaration in this diff. The original code has trait Analyzer[S <: State[_], +M <: Metric[_]] extends Serializable, but the new code shows just trait Analyzer[S <: State[_], +M <: Metric[_]]. This will break serialization for Spark jobs that serialize analyzers (e.g., for broadcast or closure capture).

github-actions · 2026-05-06T21:52:04Z

+    Seq.empty
+  }
+}
+


NumMatchesAndCount has lost its override val fullColumn: Option[Column] = None field and the FullColumn trait mixin compared to the existing code. This will break row-level results for Completeness, Compliance, PatternMatch, and other analyzers that rely on NumMatchesAndCount carrying a fullColumn.

github-actions · 2026-05-06T21:52:04Z

+  override def aggregationFunctions(): Seq[Column] = {
+    sum(conditionalSelection(column, where)) ::
+      count(conditionalSelection(column, where)).cast(LongType) :: Nil
+  }


BigDecimalMeanState.metricValue() returns null when count == 0L. This will result in Success(null) being stored in the metric, which will cause NPEs downstream (e.g., when flattening the metric or running assertions). Should return a failure or handle this case in computeMetricFrom.

github-actions · 2026-05-06T21:52:04Z


  override def filterCondition: Option[String] = where
 }
+


The refactored Maximum analyzer has lost the FullColumn trait on MaxState, the analyzerOptions parameter, the augmented outcome logic (criterion with conditionalSelectionWithAugmentedOutcome), and the computeMetricFrom override for the WHERE-filters-all-rows case. This is a regression that breaks row-level results and filtered-row handling for hasMax checks.

github-actions · 2026-05-06T21:52:04Z


  override def filterCondition: Option[String] = where
 }
+


Same regression as Maximum: the refactored Minimum analyzer has lost the FullColumn trait on MinState, the analyzerOptions parameter, the augmented outcome logic, and the computeMetricFrom override. This breaks row-level results and filtered-row handling for hasMin checks.

github-actions · 2026-05-06T21:52:04Z

      column: String,
-      allowedValues: Array[String],
+      allowedValues: Array[T],
      assertion: Double => Boolean,


The isContainedIn signature has been changed from Array[String] to generic Array[T], but the existing overloads with Array[String] that take different parameter combinations still exist. After erasure, Array[T] and Array[String] both become Array[Object], which will cause compilation errors due to ambiguous overloaded methods. The generic version should be a separate method name or the existing overloads need to be removed/consolidated.

github-actions · 2026-05-06T21:52:04Z



-import com.amazon.deequ.analyzers.KLLSketch
+import com.amazon.deequ.analyzers.catalyst.DateTimeAggregation


The import import com.amazon.deequ.analyzers.catalyst.DateTimeAggregation was added but the original import import com.amazon.deequ.analyzers.KLLSketch was removed. However, KLLSketch is not used in this file (it's referenced via StatefulKLLSketch), so this removal is fine. But note that DateTimeAggregation is in the same package (com.amazon.deequ.analyzers.catalyst) — this import is unnecessary since the file is in org.apache.spark.sql package, not com.amazon.deequ.analyzers.catalyst. Actually, looking more carefully, the file declares package org.apache.spark.sql so the import IS needed. No issue here.

github-actions · 2026-05-06T21:52:04Z

+    ifNoNullsIn(result, offset) { _ =>
+      DateTimeDistributionState(
+        DateTimeDistributionState.computeStateFromResult(Map.empty[Long, Long] ++ result.getMap(0),
+        DateTimeDistribution.getDateTimeAggIntervalValue(interval)))


DateTimeDistribution extends ScanShareableAnalyzer but does not extend Analyzer with Serializable. Since the base Analyzer trait lost extends Serializable in this PR's changes to Analyzer.scala, this class won't be serializable, which will cause failures when Spark tries to serialize it for distributed execution.

github-actions · 2026-05-06T21:52:04Z

+    val data = ordersAsDataframe(session,
+      Order(1, new BigDecimal(213.2132), Timestamp.valueOf("2020-02-15 07:15:00")),
+      Order(2, new BigDecimal(43.21324432876), Timestamp.valueOf("2020-02-15 07:45:00.999")),
+      Order(3, new BigDecimal(56.8881238823888), Timestamp.valueOf("2020-02-15 08:15:49.786")),


new BigDecimal(213.2132) uses the BigDecimal(double) constructor which is well-known to produce unexpected results due to floating-point representation (e.g., new BigDecimal(0.1) gives 0.1000000000000000055511151231257827021181583404541015625). Use new BigDecimal("213.2132") or BigDecimal.valueOf(213.2132) instead.

sudsali · 2026-05-18T19:20:18Z

@Yash0215 This appears to overlap with #568 (DateTimeMetric) which has a more complete implementation. Closing this one in favor of that PR. Thank you for the contribution!

Yash0215 and others added 9 commits September 23, 2020 12:46

Entity Enums name correction

8924c92

Timestamp analysis support -

0efb7a8

Features: > Support Spark's DateType and TimestampType > min and max analysis on date time fields

Feature Added:

3f6e27e

> New constraints for Spark's DateType and Timestamp Types

Feature Added:

927ed98

> BigDecimal State and Metric Support in Min, Max, Sum and Mean Analyzers

Feature Added:

c0e0707

> 'isContained' is made available for all Numeric Types and Char

Feature Added:

692ce76

> Analyzer - DateTimeDistribution for analyzing distributions metrics (count, ratio) over some time intervals

Fixes

e65a03d

Commit

7ac28fb

Commit

6799a21

Yash0215 added 5 commits October 22, 2020 19:48

Adding Licence

5415856

Formatting

66aebaf

Formatting

4303404

Formatting

172be49

Formatting

4c9df65

ets mentioned this pull request Mar 26, 2021

Considering a fork of this repo to get it usable again #354

Closed

eapframework approved these changes Aug 30, 2021

View reviewed changes

shehzad-qureshi added the enhancement New feature or request label Jan 31, 2023

shehzad-qureshi added the help wanted Extra attention is needed label Jan 31, 2023

zeotuan mentioned this pull request May 16, 2024

Add DateTimeMetric, Analyzer and Example #568

Open

github-actions Bot requested changes May 6, 2026

View reviewed changes

sudsali closed this May 18, 2026

		@@ -52,6 +54,11 @@ trait DoubleValuedState[S <: DoubleValuedState[S]] extends State[S] {
		def metricValue(): Double



		import com.amazon.deequ.analyzers.KLLSketch
		import com.amazon.deequ.analyzers.catalyst.DateTimeAggregation

Uh oh!

Conversation

Yash0215 commented Sep 29, 2020

Uh oh!

sscdotopen commented Oct 7, 2020

Uh oh!

lucene commented Oct 15, 2020

Uh oh!

Yash0215 commented Oct 15, 2020

Uh oh!

lucene commented Oct 15, 2020

Uh oh!

lucene commented Oct 15, 2020

Uh oh!

Yash0215 commented Oct 26, 2020

Uh oh!

aviatesk commented Oct 30, 2020

Uh oh!

sscdotopen commented Oct 30, 2020

Uh oh!

rounakdatta commented Feb 26, 2021

Uh oh!

aviatesk commented Feb 26, 2021

Uh oh!

twollnik commented Jul 20, 2021

Uh oh!

twollnik commented Oct 19, 2021

Uh oh!

RunnX commented Jun 14, 2022

Uh oh!

suadhika commented Feb 20, 2023

Uh oh!

zeotuan commented Apr 18, 2024

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 6, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 6, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 6, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 6, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 6, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 6, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 6, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 6, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 6, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 6, 2026

Choose a reason for hiding this comment

Uh oh!

sudsali commented May 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants