Repository navigation
Validity of wide pointer metadata #166
Description
Activity
But for dyn Trait pointers, what should we require about the vtable?
Is there consensus about the validity invariant of the vtable pointer ? E.g. if the vtable pointer can be null, then there doesn't need to be a vtable at all. If the vtable is a reference, then whether there needs to be a valid vtable at all would depend on whether validity of references is transitive. Or do we want to be more strict about vtables ?
Nit: The issue title talks about references, but the text (and
Weak::newdiscussion) about pointers more generally, which is it?Food for thought example: creating a
Weak<[T]>.As long as the nightly-only
Weak::as_raw(),Weak::into_raw()etc. exist we have to actually initialize the pointer metadata to something; we can't just "initialize" it with the equivalent ofMaybeUninit. So we need some kind ofWeak::new_unsized(metadata: <T as Pointee>::Metadata). This would also allowWeak::new()to to work with unsized types if the pointer metadata implementsDefault:impl<T: ?Sized> Weak<T> { pub fn new() -> Weak<T> where <T as Pointee>::Metadata: Default { unimplemented!() } }
(obvious disadvantage: unnecessary complexity confusing beginners in 99% of use-cases)
Equally, it might be reasonable to eventually (with arbitrary self types) change the slice API so that
len()takes a pointer:impl<T> [T] { pub const len(self: *const Self) -> usize { unimplemented!() } }
...which in this case could actually be accessed via
Weak::as_raw:let w: Weak<[u8]> = Weak::new_unsized(42); assert_eq!(w.as_raw().len(), 42);
Heck, you could even imagine a
DerefPtrtype trait. But I suspect the use-cases of it would be so narrow as to be unnecessary complexity.As for how this would interact with trait objects, the compiler can probably implement
Defaultfor the vtable type automatically. And again, it'd be plausible to allow the calling of trait methods that take pointers.- changed the title
[-]Validity of references: fat pointer metadata[/-][+]Validity of fat pointer metadata[/+]on Jul 16, 2019 Nit: The issue title talks about references, but the text (and Weak::new discussion) about pointers more generally, which is it?
Good point! That actually explains part of my confusion. I updated text and title.
I just realized that with the new title, this discussion would now also apply to fat raw pointers. But IIRC we agreed that fat raw pointers do not have to have valid metadata.
So actually maybe this thread should be about references?
Weakinternally usesNonNullwhich is a raw pointer, and IIRC there was agreement that raw pointers do not need to have valid metadata.Similar to #72 (comment), I wonder if it would be worth to write up a (small) RFC that says that fat raw pointers (including
NonNull) do not assume validity of their metadata, i.e., the validity invariant of their metadata is the same as that ofusize? (It is already the case that a thin raw pointer itself has the same validity invariant asusize, as the two can be cast to each other in safe code.)- changed the title
[-]Validity of fat pointer metadata[/-][+]Validity of wide pointer metadata[/+]on Jul 28, 2019 @RalfJung So like
usizethey're assumed to not beundefbits, but beyond that there are no further assumptions?Do we assume the metadata for all wide pointers is always safely castable to
usize? Eg could there be a future wide pointer type where the metadata has a subset of valid bit representations, like, say,bool?Eg could there be a future wide pointer type where the metadata has a subset of valid bit representations, like, say, bool?
Do you have a concrete application in mind?
So like usize they're assumed to not be undef bits, but beyond that there are no further assumptions?
Exactly. And if we decide we are fine with uninitialized integers, then so would be wide raw pointer metadata.
This also matches*const T/*mut T, which have the exact same validity invariant asusize-- after all, they can be safely converted back and forth.Do we assume the metadata for all wide pointers is always safely castable to usize? Eg could there be a future wide pointer type where the metadata has a subset of valid bit representations, like, say, bool?
I guess we should clarify that everything we are saying right now is only for wide pointer types that already exist (slices and trait objects). I do not want to unnecessarily constrain custom DST designs.
Eg could there be a future wide pointer type where the metadata has a subset of valid bit representations, like, say, bool?
Do you have a concrete application in mind?
Having a hard time coming up with one, other than trait objects where it's plausible you'd want to call a method on the trait for a potentially invalid pointer. Basically, that'd allow a call like this to work:
fn foo(self: *const dyn Foo)Dereferencing the raw pointer will require the vtable to be valid. The question here is about invariants that are maintained even when the pointer is just "passed around".
16 remaining items
if you can't construct "invalid" pointers and you can't do pointer arithmetic
You can have "invalid" pointers as far as the data ptr is concerned; just the vtable has to be valid. There's still plenty of use-cases for that. In particular this means a general
&mut TwithT: ?Sizedcan be turned into*mut Tand used without any aliasing restrictions. The vtable is in immutable memory, there are no aliasing problems with that.So, "defeating the borrow checker" as you called it, works fine with raw trait object pointers.
IIRC wide pointers as in slices do have a stable memory layout, (struct Slice {ptr: *const T, len: usize}).
Am I wrong?You are wrong.
slice::from_raw_parts(_mut)as well asslice::as(_mut)_ptrandslice::lenare the only supported ways to (de)construct slice references and raw pointers.transmuteing wide slice pointers/references is not supported.Generally, in terms of data layout, the only things you can rely on are things that are explicitly documented. There is no implicit stabilization of layout.
the UCG docs currently document that such a layout is the real layout, but that is an unstable implementation detail and subject to change etc etc.
You can create a null trait object by first creating a null pointer to a sized type, then casting or coercing to the unsized pointer.
In this case the vtable pointer would be valid and non-null, though.
You can create a null trait object by first creating a null pointer to a sized type, then casting or coercing to the unsized pointer.
In this case the vtable pointer would be valid and non-null, though.
Yes, I meant that in reply to "how can I create a null/dangling pointer with a valid metadata?"
the UCG docs currently document that such a layout is the real layout, but that is an unstable implementation detail and subject to chamge etc etc.
When the UCGs get RFC'ed, this layout becomes guaranteed for the cases mentioned in the document at least (e.g. no multi-trait objects).
Reacted by LokathorAccepted RFC#2580 Pointer Metadata is currently written such that all
*const dyn Traitmust have avalidusable vtable pointer (at least for getting layout) as a safety invariant. (Pointers having a safety invariant is... weird, but I suppose acceptable?)Specifically, it provides the API
pub fn metadata<T: ?Sized>(ptr: *const T) -> <T as Pointee>::Metadata; pub struct DynMetadata<DynTrait: ?Sized> { ... } impl<DynTrait> DynMetadata<DynTrait> { pub fn size(self) -> usize; pub fn align(self) -> usize; pub fn layout(self) -> alloc::Layout; }
This API does only mandate a safety invariant on pointer metadata, but it is at least interesting in the context of this discussion to note that it applies to raw pointers.
That sounds like it could still make a bunch of existing code unsound since "raw pointers do not have a safety invariant" seems like a reasonable assumption people would make.
Status update: the safety invariant (not validity invariant) for wide raw pointer metadata are pretty much fixed at this point:
- slices: can be any usize (we have safe
ptr::slice_from_raw_partsbut unsafesize_of_val_raw) - dyn: must be a valid vtable (so that we can have safe dyn trait upcasting)
However the validity invariants can of course be more liberal than that.
- slices: can be any usize (we have safe
With rust-lang/rust#124220, Miri now enforces the validity invariant on dyn trait pointers and references as: must point to a vtable for the right trait.
My current inclination is that we should use very liberal validity invariants for metadata of wide raw pointers, which matches the pointer itself: it has to be initialized, and that's it. This means we have to get rid of the niche in wide raw ptr vtable pointers.
Meanwhile, for references we should go as strong as possible: for dyn Trait, require the vtable to be for the right trait; for slices, require the full size of the type (unsized slice tail plus the statically sized prefix) to fit into
isize::MAX.
The discussions at #76 and #77 are about the validity invariant that all references have to maintain in general; this issue here is specifically about the validity invariant of the metadata component of any fat pointer -- which do not have to be references!
Rc<[u8]>is also a fat pointer. The expected metadata depends on the type of the "unsized tail".For slices, it seems clear that the invariant will be the same as that for
usize. But fordyn Traitpointers, what should we require about the vtable? My guess would have been at least as much as an&[usize; 3](a vtable has at least 3 slots: size, alignment and drop function, not necessarily in that order). But in this forum thread, the valid question came up whyWeak::newdoes not work fordyn Trait, and indeed it could if we could "fake" the fat pointer component (though doing that generically seems hard).And then there is the question how this interacts with eventual custom/user-defined DSTs.