Skip to content

Processing cleanups v0 #105

Description

@RBendias

Cross-cutting

  • tests withCUDA across the whole test/processing suite
  • reconsider whether input is a good name (shadows the Python builtin) - Don't use input as var name #274
  • reorganize package
  • why do we need buffers and should they be in v0?
  • define nan strategy for all processing → global strategy (including documenting)
  • check for necessary early validation
  • batch support
  • Align processor docs - come up with processor docs rule
  • Check that repr is aligned across processors
  • Set generator within base model
  • Rename processors (SoftmaxTemperatur -> Softmax, potentially ShuffleCategories instead of CategoryShuffle)
  • Check any processor code and align with Agents.md
  • Add target inverse as processor
  • Update Recipe instruction

Power

  • Needs to be fixed: run on regression label. Entered **only when the inverse Yeo-Johnson produces ±inf which is the case when model outputs inf which is unrealistic -> done in quantile PR
  • Add AGENTS.md #6 — vectorize: replace the per-column transform/inverse loops with a vectorized torch.where form (validated bit-identical) for torch.compile-friendliness including torch.compile smoke test

Quantile (added from PR #96 review)

  • revisit the n_quantiles cap int(subsample * 0.2) (reviewer-flagged; align with sklearn min(n_samples, subsample) or document)
  • memory-efficient deterministic subsampling — avoid materializing randperm(n_samples)
  • include Quantile in the vectorization / torch.compile work (per-column _transform_col loop)

TabICL processors (PR #106 / #107 review)

  • SigmaClip: guard NaN std (clamp to min_std, or fall back to stage-1 stats / ±inf bounds) so all-NaN or all-outlier columns don't corrupt forward; add an all-NaN SigmaClip test; document the soft-bound NaN policy (inherited from TabICL OutlierRemover)
  • SoftmaxTemperature: reject non-finite temperature (nan/inf), not just <= 0
  • MeanImpute: pre-mask ±inf before nanmean so infinities don't leak into fitted means

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions