Refactor: Use consistent URL representation for all storage paths (including file://) #1326

Activity

1. dimitri-yatsenko

   added

   enhancementIndicates new improvementsIndicates new improvements

   on Jan 9, 2026
2. github-actions

   added

   triageIndicates issues, pull requests, or discussions need to be reviewed for the first timeIndicates issues, pull requests, or discussions need to be reviewed for the first time

   on Jan 9, 2026
3. dimitri-yatsenko

   assigned

   dimitri-yatsenko

   and

   d-v-b

   on Jan 9, 2026
4. dimitri-yatsenko

   added this to the DataJoint 2.0 milestone on Jan 9, 2026
5. dimitri-yatsenko

   added a commit that references this issue on Jan 9, 2026

   feat: Add consistent URL representation for all storage paths (#1326)

   ...

   7ca9b11
6. dimitri-yatsenko

   removed

   triageIndicates issues, pull requests, or discussions need to be reviewed for the first timeIndicates issues, pull requests, or discussions need to be reviewed for the first time

   on Jan 9, 2026
7. dimitri-yatsenko commented on Jan 9, 2026

   

   dimitri-yatsenko

   on Jan 9, 2026

   Last edited by dimitri-yatsenko

   MemberAuthor

   More actions

   Implementation Complete

   Changes to src/datajoint/storage.py

   New constants and functions:

   # All supported URL protocols
   URL_PROTOCOLS = ("file://", "s3://", "gs://", "gcs://", "az://", "abfs://", "http://", "https://")
   
   def is_url(path: str) -> bool:
       """Check if a path is a URL (including file://)."""
       return path.lower().startswith(URL_PROTOCOLS)
   
   def normalize_to_url(path: str) -> str:
       """
       Normalize a path to URL form.
       Converts local filesystem paths to file:// URLs. URLs are returned unchanged.
       
       Examples:
           normalize_to_url("/data/file.dat") -> "file:///data/file.dat"
           normalize_to_url("s3://bucket/key") -> "s3://bucket/key"
       """
   
   def parse_url(url: str) -> tuple[str, str]:
       """
       Parse a URL into (protocol, path).
       Now handles file:// in addition to remote protocols.
       
       Examples:
           parse_url("s3://bucket/key") -> ("s3", "bucket/key")
           parse_url("file:///data/file.dat") -> ("file", "/data/file.dat")
       """

   New StorageBackend.get_url() method:

   def get_url(self, path: str) -> str:
       """
       Get the full URL for a path in storage.
       Returns consistent URL representation for any backend.
       
       Examples:
           # Local storage
           backend.get_url("schema/table/file.dat") 
           -> "file:///data/schema/table/file.dat"
           
           # S3 storage
           backend.get_url("schema/table/file.dat")
           -> "s3://mybucket/schema/table/file.dat"
       """

   Removed (no backward compatibility needed for pre-release):

   • REMOTE_PROTOCOLS constant
   • is_remote_url() function
   • parse_remote_url() function

   Tests

   Added tests/unit/test_storage_urls.py with 23 unit tests covering:

   • URL protocol constants
   • is_url() for all protocols and edge cases
   • normalize_to_url() path conversion
   • parse_url() for all supported protocols

   Documentation

   Updated in datajoint-docs:

   • src/how-to/configure-storage.md - Added "URL Representation" section
   • src/reference/specs/type-system.md - Added URL representation table
8. dimitri-yatsenko

   added a commit that references this issue on Jan 9, 2026

   feat: Add consistent URL representation for all storage paths (#1326)

   ...

   abebba9
9. dimitri-yatsenko commented on Jan 9, 2026

   

   dimitri-yatsenko

   on Jan 9, 2026

   MemberAuthor

   More actions

   Fix implemented in #1328
10. dimitri-yatsenko

    added a commit that references this issue on Jan 9, 2026

    feat: Add URL representation for storage paths and remove orphaned do…

    ...

    1a3e3d1
11. dimitri-yatsenko commented on Jan 9, 2026

    

    dimitri-yatsenko

    on Jan 9, 2026

    MemberAuthor

    More actions

    Closed by #1328
12. dimitri-yatsenko

    closed this as completedon Jan 9, 2026