Client parent_tasks/child_tasks resolve wrong tasks when a foreach index is a string prefix of another

[winklemad]

Problem

LocalMetadataProvider.filter_tasks_by_metadata matches metadata values with re.match, which anchors at the start but not the end:

# metaflow/plugins/metadata_providers/local.py:264
regex = re.compile(pattern)
...
and regex.match(meta.get("value", ""))

Task.parent_tasks / child_tasks build the pattern from foreach-execution-path, so for a foreach with 11 or more items the pattern middle:1 also matches the values middle:10 and middle:11.

Reproduction

Local metadata, 12 item foreach, start -> middle -> tail -> join -> end:

class ForeachPrefixFlow(FlowSpec):
    @step
    def start(self):
        self.items = list(range(12))
        self.next(self.middle, foreach="items")
    # middle -> tail -> join -> end

python flow.py run --max-workers 4

Then through the Client API:

WRONG PARENTS  ForeachPrefixFlow/<run>/tail/14  fep=middle:1  -> ['middle:1', 'middle:10', 'middle:11']
1 of 12 tail tasks have the wrong parent count

Only the task whose path is a string prefix of another is affected. Every other index resolves to exactly one parent.

The spin CLI turns this into a hard failure:

$ python flow.py spin ForeachPrefixFlow/<run>/tail/14
[.../tail/0] Internal error:
[.../tail/0] Step tail is not a join step but it gets multiple inputs.
[.../tail/0] Task failed.

Control, same run, a non colliding task:

$ python flow.py spin ForeachPrefixFlow/<run>/tail/15     # fep=middle:0
Task finished successfully.

Possible fix

regex.fullmatch instead of regex.match on that line looks like it covers it. The two deliberate pattern shapes seem unaffected:

• ".*" still matches everything, and it also short circuits earlier in the function
• the foreach join pattern "<path>,.*" still matches deeper paths such as middle:1,inner:0

Applying that one change locally takes the run above from 1 of 12 tail tasks have the wrong parent count to 0 of 12, makes spin .../tail/14 finish successfully, and leaves the control task passing.

ServiceMetadataProvider forwards the pattern to /filtered_tasks?pattern=... rather than matching in process, so this is only about the local provider. Whether the same anchoring question applies on the service side is not something I can see from here.

Before I open a PR

plugins/metadata_providers/ is Core Runtime, and CONTRIBUTING asks for an acknowledged issue and an agreed approach first, so I would rather check than guess.

Is fullmatch the direction you want, or would you prefer the anchoring live in the pattern construction in client/core.py instead? Happy to write it either way. I have the reproduction flow and a unit test covering the prefix collision plus the .* and "<path>,.*" cases ready to go.

[alinaqvi1129]

can a maintainer assign this issue to me

[talsperre]

@winklemad please go ahead with contributing a fix.

[winklemad]

Thanks @talsperre, will do.

Quick lay of the land so we don't step on #3342: nileshpatil6's PR already has the correct local-provider fix there (regex.match → regex.fullmatch) plus a prefix-matching test, and I checked it's right for all three pattern shapes client/core.py builds — current_path, the bare .*, and the {current_path},.* descendant form. The piece that's still open is the service provider: service.py forwards the raw pattern to the metadata service (/filtered_tasks?pattern=...) and lets the server match, so on the service backend middle:1 can still pull in middle:10/middle:11 and the two providers disagree.

I can put up a PR that anchors the forwarded pattern (^(?:pattern)$) so it collapses to an exact match regardless of whether the service applies it with match/search/fullmatch, alongside the local fullmatch change, with tests for both — crediting #3342 for the local part. The .* short-circuit stays untouched, and the descendant {path},.* form still matches its children. Or if you'd rather just land #3342 for the local fix, I'll scope mine to only the service side. Which do you prefer?

[winklemad]

Put up the fix in #3357. It's the complete version: regex.fullmatch in the local provider (crediting #3342 for that approach) plus anchoring the pattern the service provider forwards (^(?:pattern)$) so the service backend matches in full too, with unit tests for both that fail on master and pass on the branch. If you'd rather land #3342 for the local fix and keep mine to just the service side, happy to rescope — let me know.