Skip to content

Streaming operator modules need similar begin hook #393

Open

Description

@bjester

❌ This issue is not open for contribution. Visit Contributing guidelines to learn about the contributing process and how to find suitable issues.

Observed

During implementing streaming deserialization, it was noticed that a Transform like AppModelValidate could use a hook to reset state, similar to a Source's method begin.

💡 Tactical Suggestion (Optional): The docstring says this transform "Must be instantiated per pipeline run". StoreModelSource.begin() already resets fk_cache, so the transform's own state could reset at the same point. Then reusing the instance can't leak IDs from an earlier run:

def __call__(self, items):
    self.accepted_ids = set()
    self.failed_ids = set()
    self.deleted_ids = {}
    return super().__call__(items)

If you prefer, a begin()-style hook on Transform would do the same thing.

Originally posted by @bjester in #390 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions