Skip to content

Commit bfafc9c

Browse files
phernandezclaude
andcommitted
fix(core): refuse ISO points finer than a microsecond
`_INSTANT_BOUND` caps a fractional second at six digits and refuses a longer one rather than truncating it, because dropping digits would store a different instant than the author wrote. That refusal only ever governed the strict path. An over-precise point never matches `_INSTANT_BOUND`, so `_read_iso_day` fell through to the flexible reader, which truncated the fraction and answered with a time of day on the very day the ISO head names. Every check that path makes then passed: `@occurred:2026-01-01T10:00:00.1234567` and its quoted form both indexed as `2026-01-01T10:00:00.123456Z`, re-derived identically on every reindex, with nothing said to the author. The day check is what guards the flexible reading, and a truncated fraction sails straight through it -- the digits it drops were never in the answer to be checked. Judged on the author's text in `_classify_authored_point` instead, so both readers refuse the same token for the same reason, and for the same reason the calendar width rule already exists: a digit run wider than the syntax allows is a typo, not a shorthand. Six digits and fewer are untouched, so `14:00:00.5` and `.123456` still read; the range-literal path already refused these correctly. This also corrects an overstated claim in the classifier's own comment. It said none of the three ISO variants can reach the flexible reader, which is false: an `_IsoDay`'s trailing text is deliberately read by it, and that is what reads `2026-06-10 10:00 AM`. What the classifier actually settles for good is the calendar -- a head naming no date dies there and is never re-guessed. The trailing is fenced by two rules, one on what the reader returned and one on what the author wrote, and the comment now says so. A sweep of 216 ISO-time spellings (T/t/space separators, minute and second precision, Z/z and offset zones with and without a colon, fractions of 0/1/6/7/9 and 30 digits) reported 54 tokens whose stored instant disagreed with a literal reading of the text. All 54 were this one defect; after the fix the sweep reports none. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014pmKq6bqCi6Zp6BTHuZjrp Signed-off-by: phernandez <paul@basicmachines.co>
1 parent 06cce5f commit bfafc9c

3 files changed

Lines changed: 160 additions & 8 deletions

File tree

src/basic_memory/temporal.py

Lines changed: 45 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -580,8 +580,20 @@ def _calendar_span(lower: date, upper: date | None) -> TemporalRange:
580580
# *once* is the whole design. Four review rounds went the other way: each added a shape test
581581
# whose failure meant "not my business", so a token that failed the test fell through to the
582582
# flexible reader and the next round found another shape that failed it. Here the classifier
583-
# is total -- a token that opens with ISO syntax is an `_IsoDay`, an `_IsoMonth` or a
584-
# `_MalformedIso`, and none of the three can reach the flexible reader.
583+
# is total: a token that opens with ISO syntax is an `_IsoDay`, an `_IsoMonth` or a
584+
# `_MalformedIso`, and there is no fourth answer to fall through on.
585+
#
586+
# What that buys is narrower than "ISO-shaped text never reaches the flexible reader", and
587+
# stating it precisely matters, because the loose version is false. An `_IsoDay`'s *trailing*
588+
# text is still read by the flexible reader -- that is what reads `2026-06-10 10:00 AM`, and
589+
# no grammar of clock spellings could. What the classifier settles for good is the *calendar*:
590+
# a head that names no date dies here, and a real one is carried on the variant so the reading
591+
# below can be held to it. The trailing is fenced by two rules instead, and dateparser's answer
592+
# is believed only when both hold. It must come back as a time of day on the day the head names
593+
# -- checked in `_read_iso_day`, against what it *returned*, since a suffix's looks do not say
594+
# what it will do with it. And the text must not spell precision a canonical instant cannot
595+
# carry -- checked here, on the text, because that is the one defect the returned-value check
596+
# cannot see: a truncated fraction still lands on the right day.
585597

586598
# The ISO calendar components a point *opens* with: `YYYY-MM` and an optional `-DD`. A date
587599
# carrying a time (`2026-06-10T14:00`, `2026-06-10 10:00 AM`) is matched on its date part
@@ -595,6 +607,18 @@ def _calendar_span(lower: date, upper: date | None) -> TemporalRange:
595607
# reader, which is the one outcome ISO syntax must never have.
596608
_ISO_CALENDAR_HEAD = re.compile(r"^(\d{4})-(\d+)(?:-(\d+))?")
597609

610+
# A fractional-second run too wide for a canonical instant to carry. `_INSTANT_BOUND` caps the
611+
# fraction at six digits and *refuses* a longer one rather than truncating it, because dropping
612+
# digits would store a different instant than the author wrote -- but that refusal only ever
613+
# governed the strict path. The flexible reader has no such scruple: it truncates
614+
# `2026-01-01T10:00:00.1234567` to `...123456Z` and reports a time on the right day, so every
615+
# check `_read_iso_day` makes passes and the authored instant is quietly rewritten on each
616+
# reindex. Judged on the text so both paths refuse the same token for the same reason, and it
617+
# is the same reason the calendar width rule exists: a digit run wider than the syntax allows
618+
# is a typo, not a shorthand. Six digits and fewer are untouched -- `14:00:00.5` is precision
619+
# a canonical instant holds exactly, so it still reads.
620+
_OVER_PRECISE_FRACTION = re.compile(r"\.\d{7,}")
621+
598622

599623
def _named_calendar_date(year: str, month: str, day: str | None) -> date | None:
600624
"""The date ISO-shaped calendar components name, or None when they name none.
@@ -645,11 +669,12 @@ class _IsoMonth:
645669

646670
@dataclass(frozen=True, slots=True)
647671
class _MalformedIso:
648-
"""A point written in ISO syntax that names nothing on the calendar.
672+
"""A point written in ISO syntax that cannot be read as written.
649673
650-
`2026-13-01`, `2026-01-0100`, `2026-06 10:00`. The author reached for a machine date
651-
and missed, so there is no reading to fall back on -- only a guess, which is what this
652-
variant exists to make unreachable.
674+
`2026-13-01`, `2026-01-0100`, `2026-06 10:00`, `2026-01-01T10:00:00.1234567`. Either the
675+
components name nothing on the calendar, or they name a moment finer than a canonical
676+
instant records. The author reached for a machine date and missed, so there is no reading
677+
to fall back on -- only a guess, which is what this variant exists to make unreachable.
653678
"""
654679

655680

@@ -690,6 +715,16 @@ def _classify_authored_point(point: str) -> _AuthoredPoint:
690715
# Outcome: a bare month denotes its own period; a month with anything after it is
691716
# malformed.
692717
return _MALFORMED_ISO if trailing else _IsoMonth(int(year), int(month))
718+
719+
# Trigger: the text after the date spells a fraction of a second wider than six digits.
720+
# Why: no reader here can store it, and the two that try disagree -- `_canonical_instant`
721+
# refuses it, while the flexible reader truncates it and still answers with a time on
722+
# the head's day, which is precisely what `_read_iso_day`'s returned-value check cannot
723+
# catch. A guard that asks what came back cannot see digits that never made it in.
724+
# Outcome: refused as malformed, so the strict and flexible paths give the same answer to
725+
# the same text and the token stays observation content rather than a rounded instant.
726+
if _OVER_PRECISE_FRACTION.search(trailing):
727+
return _MALFORMED_ISO
693728
return _IsoDay(named, trailing)
694729

695730

@@ -794,8 +829,10 @@ def parse_authored_point(
794829
Non-ISO spellings are read leniently, because guessing at `June 10, 2026` is the
795830
whole point of this reader. A token that *is* ISO-shaped is held to its own text
796831
instead: its calendar components must name a real date, and anything trailing them
797-
must be a time of day on that date. `2026-06-10 10:00 AM` reads; `2026-01-01T` does
798-
not, because the author reached for an instant and no instant is there.
832+
must be a time of day on that date, written to a precision this module can store.
833+
`2026-06-10 10:00 AM` reads; `2026-01-01T` does not, because the author reached for an
834+
instant and no instant is there; `2026-01-01T10:00:00.1234567` does not either,
835+
because storing it would mean dropping the digits that made it worth writing.
799836
800837
Returns None when the text names no date. That is not an error -- the caller leaves
801838
such a token as ordinary observation content.

tests/markdown/test_temporal_qualifier.py

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -437,6 +437,54 @@ def test_quoted_point_reads_a_multi_word_date(qualifier: str, literal: str, kind
437437
assert str(observation) == line
438438

439439

440+
@pytest.mark.parametrize(
441+
"point",
442+
[
443+
"2026-01-01T10:00:00.1234567",
444+
"2026-01-01T10:00:00.1234567Z",
445+
"2026-01-01T10:00:00.1234567+02:00",
446+
"2026-01-01T10:00:00." + "1" * 30,
447+
],
448+
)
449+
def test_a_point_finer_than_a_microsecond_stays_content_in_both_forms(point: str):
450+
"""An over-precise instant is refused whichever form carries it to the reader.
451+
452+
Both forms filed `[2026-01-01T10:00:00.123456Z,)` -- the authored instant with its
453+
last digits dropped, and no sign to the author that anything was lost. The quoted form
454+
reached it by a different route than the bare one: quoting suppresses the truncation
455+
guards, on the reasoning that a delimited value cannot be a truncated *token*. That is
456+
still true, and beside the point here -- the loss is inside the value, so only refusing
457+
the point itself covers both. Pinned together so a fix to one form cannot miss the
458+
other.
459+
"""
460+
for qualifier in (f"@occurred:{point}", f'@occurred:"{point}"'):
461+
line = f"- [decision] {qualifier} The cutover ran."
462+
463+
observation = _observation(line)
464+
465+
assert observation.temporal == []
466+
# Refused, not reported: how someone spelled a date is not a diagnostic this
467+
# feature issues. The line keeps every character and stays full-text searchable.
468+
assert observation.temporal_error is None
469+
assert observation.content == f"{qualifier} The cutover ran."
470+
assert str(observation) == line
471+
472+
473+
def test_a_point_at_exactly_microsecond_precision_is_still_filed():
474+
"""The boundary the refusal above stops at, end to end through the parser."""
475+
for qualifier in (
476+
"@occurred:2026-01-01T10:00:00.123456",
477+
'@occurred:"2026-01-01T10:00:00.123456"',
478+
):
479+
observation = _observation(f"- [decision] {qualifier} The cutover ran.")
480+
481+
[assertion] = observation.temporal
482+
assert assertion.time_kind is TimeKind.OCCURRED
483+
assert str(assertion.valid_during) == "[2026-01-01T10:00:00.123456Z,)"
484+
assert assertion.valid_during.axis is TemporalRangeAxis.INSTANT
485+
assert observation.content == "The cutover ran."
486+
487+
440488
def test_a_quoted_relative_date_is_read_where_its_unquoted_form_is_not():
441489
"""`2 days ago` always read fine; only the token rule kept it out."""
442490
quoted = _observation('- [decision] @occurred:"2 days ago" The cutover ran.')

tests/test_temporal.py

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -533,6 +533,73 @@ def test_an_iso_date_whose_suffix_re_guesses_it_is_unread(written: str):
533533
assert parse_authored_point(written) is None
534534

535535

536+
@pytest.mark.parametrize(
537+
"written",
538+
[
539+
# The reported shape: one digit more than a canonical instant carries. The lenient
540+
# reader truncated it to `...123456Z`, on the very day the head names, so every
541+
# check the reading makes passed and the index recorded an instant 100ns off the
542+
# one the author wrote -- re-derived identically by every reindex.
543+
"2026-01-01T10:00:00.1234567",
544+
# The same defect wearing every spelling of the syntax around it. None of these is
545+
# distinguishable by what the reader *returned* -- each truncates and each lands on
546+
# the right day -- which is why this one is judged on the text instead.
547+
"2026-01-01t10:00:00.1234567",
548+
"2026-01-01 10:00:00.1234567",
549+
"2026-01-01T10:00:00.1234567Z",
550+
"2026-01-01T10:00:00.1234567z",
551+
"2026-01-01T10:00:00.1234567+02:00",
552+
"2026-01-01T10:00:00.1234567-05:00",
553+
"2026-01-01T10:00:00.1234567+0200",
554+
# Precision far past anything a clock emits, truncated just as quietly: a 20- and a
555+
# 30-digit fraction both stored six digits and discarded the rest without a word.
556+
"2026-01-01T10:00:00.12345678901234567890",
557+
"2026-01-01T10:00:00." + "1" * 30,
558+
],
559+
)
560+
def test_an_iso_point_finer_than_a_microsecond_is_unread(written: str):
561+
"""Over-precision is refused on the lenient path too, not silently rounded.
562+
563+
`canonical_bound` has always refused these -- dropping digits would store a different
564+
instant than the author wrote -- but that refusal only governed the strict path. A
565+
point one digit too precise never matched `_INSTANT_BOUND`, so it fell to the lenient
566+
reader, which truncated it and answered with a time on the correct day. The day check
567+
is what guards that path, and a truncated fraction sails straight through it: the
568+
digits it drops were never in the answer to be checked. Judged on the author's text
569+
instead, so both readers refuse the same token for the same reason.
570+
"""
571+
assert parse_authored_point(written) is None
572+
573+
574+
@pytest.mark.parametrize(
575+
("written", "lower"),
576+
[
577+
# Exactly six digits: the widest fraction a canonical instant carries, so it is
578+
# stored whole and nothing is dropped. The refusal above must stop precisely here.
579+
("2026-01-01T10:00:00.123456", "2026-01-01T10:00:00.123456Z"),
580+
("2026-01-01 10:00:00.123456", "2026-01-01T10:00:00.123456Z"),
581+
("2026-01-01T10:00:00.123456Z", "2026-01-01T10:00:00.123456Z"),
582+
("2026-01-01T10:00:00.123456+02:00", "2026-01-01T08:00:00.123456Z"),
583+
# Narrower fractions were never in question, and are pinned so a future widening
584+
# of the rule cannot quietly take them.
585+
("2026-01-01T10:00:00.1", "2026-01-01T10:00:00.100000Z"),
586+
("2026-06-10 14:00:00.5", "2026-06-10T14:00:00.500000Z"),
587+
],
588+
)
589+
def test_a_fraction_a_canonical_instant_can_hold_still_reads(written: str, lower: str):
590+
"""Refusing over-precision must cost nothing that stores losslessly.
591+
592+
Six digits is the boundary, not "any fraction is suspicious": these name a moment the
593+
canonical form records exactly, so there is no truncation to prevent and no reason to
594+
withhold the assertion.
595+
"""
596+
span = parse_authored_point(written)
597+
598+
assert span is not None
599+
assert span.axis is INSTANT
600+
assert span.lower == lower
601+
602+
536603
@pytest.mark.parametrize("today", ["2026-03-07", "2026-09-01"])
537604
def test_a_clock_reading_on_a_month_is_not_completed_from_the_indexing_date(today: str):
538605
"""`2026-13`'s disease in the suffix: a time of day needs a day to fall on.

0 commit comments

Comments
 (0)