fix(core): key observation identity and note type to the owning note

Two ways a valid-time query could silently return nothing, both from derived
state answering a question about the note it came from.

**An authored assertion could become permanently unqueryable.** A temporal
qualifier is peeled off an observation before the content is stored, so two
lines that differ only in their qualifier persist identical content and derive
identical synthetic permalinks. The search index is unique on (permalink,
project_id), so `index_entity_markdown` skipped the second as a duplicate --
while its temporal row went on addressing an observation with no search
projection. Querying the second window returned nothing, and every reindex
reproduced the omission from the same markdown:

    - [decision] @effective[2026-06-10,2026-07-27) The cache layer will use Redis.
    - [decision] @effective[2027-06-10,2027-07-27) The cache layer will use Redis.

Reproduced before the fix: two observation rows, one search row, two
memory_time_index rows, and the 2027 query returning zero results.

The duplicate check is not what is wrong -- it guards a real unique index. Its
input is. Identity is derived after the peel, so it is derived from a string the
note does not consider distinguishing. Nothing on the observation row separates
these two, so no row-local rule can: category, content, context and tags are all
equal, and the qualifier lives in its own projection by an explicit design
decision. A relationship to that projection is not usable either -- all three
readers of `Observation.permalink` read it on *detached* instances after their
session has closed, so a lazy load would raise rather than resolve.

So the ordinal is stored, exactly as `note_section.duplicate_index` already does
for duplicate headings. `replace_observations_for_generation` is the one place
that sees a note's whole observation set in document order, and it counts the
ordinal over `observation_permalink_tail` -- shared with `Observation.permalink`
so the count is taken over exactly the identity the address is built from,
rather than rebuilt inline and drifting (#929). Keying on the generated tail
rather than raw values also closes slug aliasing, where `Foo Bar` and `foo-bar`
are different content that generate one permalink.

The ordinal is 0 for the first observation of any identity, so every permalink
that resolves today is byte-identical afterwards; only later twins gain a
suffix. It fixes the same collision for two observations differing only in
`(context)`, which had the same defect for the same reason.

**A valid-time query combined with `note_types` could not match anything.** A
note's type lives in its frontmatter, so only its entity row carries
`metadata.note_type`; observation rows carry tags and relation rows carry
nothing. Both backends read the type off each row, which asks "is this row an
entity of type X?" when the question was "does this row belong to a note of type
X?". Valid time selects observation rows, so the two predicates were never true
of the same row and the conjunction was unsatisfiable.

Resolved through the owning note instead, in one shared builder both backends
call -- only the JSON accessor differs, and that is all each supplies. Every
search row already carries `entity_id` and an entity row's own `id` equals it,
so one non-correlated membership test covers all three row kinds. Non-correlated
for the reason `temporal_filters` documents: SQLite's `search_index` is an FTS5
virtual table and a correlated EXISTS beside a MATCH is refused outright.

This makes `note_types` return observation and relation rows of matching notes,
where it previously collapsed to entity rows. That is the fix, not a side
effect: restricting which *kind* of row may match is `entity_types`' job, the
two axes are independent, and a query setting neither returns all three kinds.
`test_search_type` asserted the old entity-only shape and is updated -- it
recorded what the defect allowed, not an intention.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014pmKq6bqCi6Zp6BTHuZjrp
Signed-off-by: phernandez <paul@basicmachines.co>

Author: phernandez

src/basic_memory/alembic/versions/v5o6b7s8d9e0_add_observation_duplicate_index.py

@@ -0,0 +1,49 @@
+"""Add the duplicate ordinal that keeps same-identity observations addressable.
+
+An observation's synthetic permalink is built from its category and content, both of
+which survive the peel that strips a temporal qualifier and a (context) off the authored
+line. Two observations differing only in one of those therefore shared one address, and
+the permalink-keyed search index kept only the first -- so the second note's authored
+valid time addressed an observation with no search row, and queries for its interval
+found nothing (SPEC-82).
+
+This ordinal separates such twins. It defaults to 0, which is the value every existing
+row takes and the value the first observation of any identity keeps, so no permalink that
+resolves today changes. Search rows are derived state and are rebuilt from markdown, so
+the second twin becomes addressable on the next index pass for that note.
+
+Revision ID: v5o6b7s8d9e0
+Revises: u4t5e6m7p8o9
+Create Date: 2026-09-02 10:00:00.000000
+
+"""
+
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+revision: str = "v5o6b7s8d9e0"
+down_revision: Union[str, None] = "u4t5e6m7p8o9"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Add observation.duplicate_index, defaulting every existing row to 0."""
+    with op.batch_alter_table("observation", schema=None) as batch_op:
+        batch_op.add_column(
+            sa.Column(
+                "duplicate_index",
+                sa.Integer(),
+                server_default=sa.text("0"),
+                nullable=False,
+            )
+        )
+
+
+def downgrade() -> None:
+    """Remove the duplicate ordinal."""
+    with op.batch_alter_table("observation", schema=None) as batch_op:
+        batch_op.drop_column("duplicate_index")

src/basic_memory/models/knowledge.py

@@ -339,6 +339,45 @@ def __repr__(self) -> str:  # pragma: no cover
         )
 
 
+def observation_permalink_tail(category: str | None, content: str) -> str:
+    """The part of an observation's permalink that distinguishes it within its note.
+
+    This is the single definition of what makes two observations of one note share an
+    address, and it is deliberately shared with the writer that assigns
+    `Observation.duplicate_index`: an ordinal only disambiguates if it is counted over
+    exactly the identity the permalink is built from. Computing the two separately is the
+    defect this function exists to prevent -- rebuilding the permalink format inline is
+    what diverged from the search index for long observations (#929).
+
+    Note what is *not* here. A qualifier (`@effective[...]`) and a `(context)` are peeled
+    off the line before the observation is stored, so neither reaches this string, and two
+    observations that differ only in one of them arrive identical. That is not an oversight
+    to correct by stuffing them back in: the peel is the feature, and valid time is its own
+    projection rather than an observation column. The note still distinguishes such lines,
+    so the *address* must too, which is what the ordinal counted over this tail supplies.
+
+    Slug aliasing is why the count keys on this generated text rather than on the raw
+    values: `Foo Bar` and `foo-bar` are different content that generate one permalink, so
+    an ordinal counted over raw content would leave them colliding.
+
+    Content is truncated to 200 chars to stay under PostgreSQL's btree index limit of
+    2704 bytes.
+    """
+    if len(content) > 200:
+        # Trigger: content exceeds the 200-char budget imposed by PostgreSQL's
+        # 2704-byte btree index row limit, so the permalink can only carry a prefix.
+        # Why: two distinct observations with the same category and an identical
+        # 200-char prefix would collide on the same synthetic permalink, and the
+        # search index (permalink-keyed upsert) silently drops the second one.
+        # Outcome: a short stable digest of the FULL content disambiguates
+        # truncated permalinks while staying well under the index limit.
+        digest = hashlib.sha256(content.encode("utf-8")).hexdigest()[:12]
+        content_for_permalink = f"{content[:200]}-{digest}"
+    else:
+        content_for_permalink = content
+    return generate_permalink(f"observations/{category}/{content_for_permalink}")
+
+
 class Observation(Base):
     """An observation about an entity.
 
@@ -360,6 +399,11 @@ class Observation(Base):
     tags: Mapped[Optional[list[str]]] = mapped_column(
         JSON, nullable=True, default=list, server_default="[]"
     )
+    # Which of the note's same-identity observations this one is, in document order.
+    # See `observation_permalink_tail` for why an ordinal is needed at all and why it is
+    # stored rather than derived: `permalink` is read on *detached* instances, long after
+    # the session that could have looked at this row's siblings has closed.
+    duplicate_index: Mapped[int] = mapped_column(Integer, default=0, server_default=text("0"))
 
     # Relationships
     entity = relationship("Entity", back_populates="observations")
@@ -371,24 +415,17 @@ def permalink(self) -> str:
         We can construct these because observations are always defined in
         and owned by a single entity.
 
-        Content is truncated to 200 chars to stay under PostgreSQL's
-        btree index limit of 2704 bytes.
+        `duplicate_index` is what keeps the address faithful when one note says the same
+        thing twice. It is 0 for the first observation carrying a given identity, so the
+        overwhelming majority of permalinks are byte-identical to what they have always
+        been; only the second and later twins gain a trailing ordinal.
         """
-        if len(self.content) > 200:
-            # Trigger: content exceeds the 200-char budget imposed by PostgreSQL's
-            # 2704-byte btree index row limit, so the permalink can only carry a prefix.
-            # Why: two distinct observations with the same category and an identical
-            # 200-char prefix would collide on the same synthetic permalink, and the
-            # search index (permalink-keyed upsert) silently drops the second one.
-            # Outcome: a short stable digest of the FULL content disambiguates
-            # truncated permalinks while staying well under the index limit.
-            digest = hashlib.sha256(self.content.encode("utf-8")).hexdigest()[:12]
-            content_for_permalink = f"{self.content[:200]}-{digest}"
-        else:
-            content_for_permalink = self.content
-        return generate_permalink(
-            f"{self.entity.permalink}/observations/{self.category}/{content_for_permalink}"
+        base = generate_permalink(
+            f"{self.entity.permalink}/{observation_permalink_tail(self.category, self.content)}"
         )
+        if not self.duplicate_index:
+            return base
+        return f"{base}/{self.duplicate_index}"
 
     @override
     def __repr__(self) -> str:  # pragma: no cover

src/basic_memory/repository/note_type_filters.py

@@ -0,0 +1,75 @@
+"""SQL for the note-type search predicate.
+
+A note type is a property of the *note*, not of the individual rows projected from it.
+One markdown file becomes several search rows -- the entity itself, one per observation,
+one per outgoing relation -- and only the entity row carries `metadata.note_type`, because
+that is where the frontmatter lives. An observation row carries `metadata.tags`; a relation
+row carries no metadata at all.
+
+Reading the type off each row therefore answers "is this row an entity of type X?" when the
+question asked was "does this row belong to a note of type X?". Those coincide for entity
+rows and for nothing else, which is invisible until a filter selects non-entity rows -- as a
+valid-time filter does, since authored time lives on observations (SPEC-82). The conjunction
+of the two was unsatisfiable: every row admitted by the temporal predicate was excluded by
+the note-type one.
+
+Resolving through the owning note fixes that at the source. Every search row already carries
+`entity_id`, and an entity row's own `id` equals it, so one membership test covers all three
+row kinds without special-casing any of them and without copying the type onto rows that
+would then have to be kept in step with the note's frontmatter.
+
+The predicate is a *non-correlated* subquery for the reason `temporal_filters` documents at
+length: SQLite's `search_index` is an FTS5 virtual table, and a correlated `EXISTS` beside a
+`MATCH` makes SQLite refuse the statement outright. A non-correlated `IN` is evaluated once,
+independently, and composes with every FTS shape in this repository while leaving bm25
+ranking intact.
+
+One builder serves both dialects. Only the JSON accessor differs, so that is the single
+thing a backend supplies -- the rule itself lives here rather than being written out once
+per backend and drifting.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Sequence
+
+from basic_memory.schemas.search import SearchItemType
+
+SEARCH_TABLE = "search_index"
+
+# The alias the owning note's row carries inside the subquery.
+_OWNER = "note_type_owner"
+
+# Each dialect's expression for the owning note's frontmatter type.
+SQLITE_NOTE_TYPE_VALUE = f"json_extract({_OWNER}.metadata, '$.note_type')"
+POSTGRES_NOTE_TYPE_VALUE = f"{_OWNER}.metadata->>'note_type'"
+
+
+def build_note_type_predicate(
+    note_types: Sequence[str],
+    params: dict[str, Any],
+    *,
+    note_type_value: str,
+) -> str:
+    """Build the WHERE-clause fragment restricting rows to notes of the given types.
+
+    The stored type keeps the frontmatter's own casing (`Chapter`), while the filter is
+    documented case-insensitive, so both sides are folded to lowercase.
+
+    Binds are added to `params` in place, following the convention the surrounding FTS
+    query builders already use. `project_id` is bound by the caller for the whole query.
+    """
+    placeholders = []
+    for index, note_type in enumerate(note_types):
+        name = f"note_type_{index}"
+        params[name] = note_type.lower()
+        placeholders.append(f":{name}")
+
+    return (
+        f"{SEARCH_TABLE}.entity_id IN (\n"
+        f"  SELECT {_OWNER}.id\n"
+        f"    FROM {SEARCH_TABLE} AS {_OWNER}\n"
+        f"   WHERE {_OWNER}.type = '{SearchItemType.ENTITY.value}'\n"
+        f"     AND {_OWNER}.project_id = :project_id\n"
+        f"     AND LOWER({note_type_value}) IN ({', '.join(placeholders)}))"
+    )

src/basic_memory/repository/observation_repository.py

@@ -9,6 +9,7 @@
 from sqlalchemy.orm.interfaces import LoaderOption
 
 from basic_memory.models import Observation
+from basic_memory.models.knowledge import observation_permalink_tail
 from basic_memory.repository.relation_repository import current_relation_generation_statement
 from basic_memory.repository.repository import Repository
 from basic_memory.temporal import TemporalAssertion
@@ -142,17 +143,33 @@ async def replace_observations_for_generation(
             return ObservationGenerationWriteResult(generation_is_current=False)
 
         await self.delete_by_fields(session, entity_id=entity_id)
-        rows = [
-            Observation(
-                project_id=self.project_id,
-                entity_id=entity_id,
-                content=obs.content,
-                category=obs.category,
-                context=obs.context,
-                tags=obs.tags,
+        # A note may say the same thing twice and mean two different things -- most
+        # sharply when a temporal qualifier or a (context) is what separates them, since
+        # both are peeled off before the content reaches this row. The permalink is built
+        # from what survives that peel, so those twins would address one row, and the
+        # permalink-keyed search index would keep only the first (SPEC-82).
+        #
+        # This is the one place that sees a note's whole observation set in document
+        # order, so it is where the ordinal that separates them can be counted at all.
+        # `observation_permalink_tail` is shared with `Observation.permalink` so the
+        # count is taken over exactly the identity the address is built from.
+        duplicates_seen: dict[str, int] = {}
+        rows = []
+        for obs in observations:
+            identity = observation_permalink_tail(obs.category, obs.content)
+            duplicate_index = duplicates_seen.get(identity, 0)
+            duplicates_seen[identity] = duplicate_index + 1
+            rows.append(
+                Observation(
+                    project_id=self.project_id,
+                    entity_id=entity_id,
+                    content=obs.content,
+                    category=obs.category,
+                    context=obs.context,
+                    tags=obs.tags,
+                    duplicate_index=duplicate_index,
+                )
             )
-            for obs in observations
-        ]
         await self.add_all_no_return(session, rows)
         # add_all_no_return flushes, so every row now carries its database id.
         # Reading them here, inside the same transaction, is what lets the temporal

src/basic_memory/repository/postgres_search_repository.py

@@ -35,6 +35,10 @@
     build_fts_page_stage,
 )
 from basic_memory.repository.metadata_filters import parse_metadata_filters
+from basic_memory.repository.note_type_filters import (
+    POSTGRES_NOTE_TYPE_VALUE,
+    build_note_type_predicate,
+)
 from basic_memory.repository.temporal_filters import build_temporal_predicate
 from basic_memory.repository.semantic_errors import SemanticDependenciesMissingError
 from basic_memory.repository.semantic_vector_index import SemanticVectorIndex
@@ -1169,19 +1173,18 @@ async def _build_fts_query_parts(
 
         # Handle note type filter (frontmatter type field, parameterized).
         # Trigger: caller passed `note_types` to scope by the frontmatter `type` field.
-        # Why: the stored note_type preserves the frontmatter casing (e.g. `Chapter`),
-        #      but the filter is documented case-insensitive. JSONB `@>` containment is
-        #      exact-match, so capitalized types were unfindable.
-        # Outcome: compare LOWER(metadata->>'note_type') against lowercased filter
-        #          values so `note_types=["Chapter"]` matches a stored `Chapter`.
+        # Why: the type belongs to the note, but only its entity row carries the
+        #      frontmatter; observation and relation rows do not. Reading it off each row
+        #      silently excluded every non-entity row, which made `note_types` combined
+        #      with a valid-time filter unsatisfiable.
+        # Outcome: resolved through the owning note in one shared builder, so both
+        #          backends ask the same question and observation rows of a matching note
+        #          are admitted.
         if note_types:
-            type_placeholders = []
-            for idx, note_type in enumerate(note_types):
-                param_name = f"note_type_{idx}"
-                params[param_name] = note_type.lower()
-                type_placeholders.append(f":{param_name}")
             conditions.append(
-                f"LOWER(search_index.metadata->>'note_type') IN ({', '.join(type_placeholders)})"
+                build_note_type_predicate(
+                    note_types, params, note_type_value=POSTGRES_NOTE_TYPE_VALUE
+                )
             )
 
         # Handle date filter

src/basic_memory/repository/sqlite_search_repository.py

@@ -40,6 +40,10 @@
     build_fts_page_stage,
 )
 from basic_memory.repository.metadata_filters import parse_metadata_filters, build_sqlite_json_path
+from basic_memory.repository.note_type_filters import (
+    SQLITE_NOTE_TYPE_VALUE,
+    build_note_type_predicate,
+)
 from basic_memory.repository.temporal_filters import build_temporal_predicate
 from basic_memory.repository.semantic_errors import SemanticDependenciesMissingError
 from basic_memory.repository.semantic_vector_index import SemanticVectorIndex
@@ -909,20 +913,18 @@ async def _build_fts_query_parts(
 
         # Handle note type filter (frontmatter type field, parameterized).
         # Trigger: caller passed `note_types` to scope by the frontmatter `type` field.
-        # Why: the stored note_type preserves the frontmatter casing (e.g. `Chapter`),
-        #      but the filter is documented case-insensitive; comparing raw values
-        #      would miss capitalized types.
-        # Outcome: fold both sides to lowercase so `note_types=["Chapter"]` matches a
-        #          stored `Chapter`, `chapter`, etc.
+        # Why: the type belongs to the note, but only its entity row carries the
+        #      frontmatter; observation and relation rows do not. Reading it off each row
+        #      silently excluded every non-entity row, which made `note_types` combined
+        #      with a valid-time filter unsatisfiable.
+        # Outcome: resolved through the owning note in one shared builder, so both
+        #          backends ask the same question and observation rows of a matching note
+        #          are admitted.
         if note_types:
-            type_placeholders = []
-            for idx, t in enumerate(note_types):
-                param_name = f"note_type_{idx}"
-                params[param_name] = t.lower()
-                type_placeholders.append(f":{param_name}")
             conditions.append(
-                "LOWER(json_extract(search_index.metadata, '$.note_type')) "
-                f"IN ({', '.join(type_placeholders)})"
+                build_note_type_predicate(
+                    note_types, params, note_type_value=SQLITE_NOTE_TYPE_VALUE
+                )
             )
 
         # Handle date filter using datetime() for proper comparison

tests/repository/test_postgres_search_repository.py

@@ -142,6 +142,10 @@ async def test_postgres_search_repository_index_and_search(session_maker, test_p
         permalink="docs/coffee-brewing",
         file_path="docs/coffee-brewing.md",
         type="entity",
+        # An entity row addresses itself: every indexing path sets entity_id on all three
+        # row kinds, and note_type is resolved through it, so a row built by hand here
+        # must carry it too or it belongs to no note at all.
+        entity_id=1,
         metadata={"note_type": "note"},
         created_at=now,
         updated_at=now,