Repository navigation

978 Migrate Local Users, Datasets, Folders and Files - #1058

Open

tcnichol wants to merge 99 commits into

mainfrom

978-migrate-users

Open

978 Migrate Local Users, Datasets, Folders and Files#1058
tcnichol wants to merge 99 commits into
mainfrom
978-migrate-users

Commits on Mar 28, 2024
- adding new dependency and class
  tcnichol
  committed
  511923d
- starting a user migration
  tcnichol
  committed
  f763cb4
- adding script for migrate metadata definitions
  tcnichol
  committed
  84d43bc
- adding option of temporary password (for use with users who are being migrated from old clowder instance
  tcnichol
  committed
  e82e023
- reverting changes
  tcnichol
  committed
  dcdbea1
- temporary - for users that are migrated
  tcnichol
  committed
  79e56b4
Commits on Apr 23, 2024
- creates users
  tcnichol
  committed
  33f1cab
Commits on Apr 25, 2024
- does not actually create the dataset
  tcnichol
  committed
  93509e7
- using beanie might not work here due to async issues
  tcnichol
  committed
  aba43a3
- using nest asyncio to fix problem with even loop closing
  tcnichol
  committed
  727685f
Commits on Apr 26, 2024
- entry from right table for file bytes
  tcnichol
  committed
  d4e9526
Commits on May 20, 2024
- dependencies for migration, adding file entry and indexing files
  tcnichol
  committed
  8c7d6bd
- wrong api key
  tcnichol
  committed
  2ba7df5
Commits on May 21, 2024
- api key works for user,
  tcnichol
  committed
  f624784
- file upload works, sloppy needs fixed
  tcnichol
  committed
  adc6c63
- we are only getting partial files, not sure how to fix yet
  tcnichol
  committed
  171b2b3
Commits on May 26, 2024
- files upload right now
  tcnichol
  committed
  1f6a2ed
- delete file
  tcnichol
  committed
  2bf100c
Commits on May 27, 2024
- adding folders
  tcnichol
  committed
  4eccae1
Commits on May 30, 2024
- folder hierarchy - this should work
  tcnichol
  committed
  c01ce96
Commits on May 31, 2024
- works for now, folders need to be fixed
  tcnichol
  committed
  c2ab289
Commits on Jun 1, 2024
- previous is no longer selectable when moving to my datasets. we start on the first page
  tcnichol
  committed
  2d7cdce
- Merge branch 'main' into 978-migrate-users
  tcnichol
  committed
  65599bb
- new folder hierarchy method works
  tcnichol
  committed
  5be97bf
Commits on Jun 2, 2024
- need a license for datasets
  tcnichol
  committed
  f479284
- method for getting all the folders
  tcnichol
  committed
  cb4b835
Commits on Jun 3, 2024
- need to remove logging, but folders are created and files uploaded to correct folders
  tcnichol
  committed
  e885a2a
- should work now
  tcnichol
  committed
  b4db0f4
- fixing typos
  tcnichol
  committed
  72e0845
- formatting
  tcnichol
  committed
  a6cd25a
- sample .env
  tcnichol
  committed
  54747ca
- formatting
  tcnichol
  committed
  55ff21a
Commits on Jul 13, 2024
- Merge branch 'main' into 978-migrate-users
  tcnichol
  committed
  e216405
- refactoring migrate_user
  tcnichol
  committed
  851503e
Commits on Jul 15, 2024
- formatting
  tcnichol
  committed
  074154e
Commits on Aug 5, 2024
- Merge branch 'main' into 978-migrate-users
  tcnichol
  committed
  909b8f6
Commits on Aug 6, 2024
- Merge branch 'release/v2.0-beta-3' into 978-migrate-users
  tcnichol
  committed
  b309023
Commits on Aug 14, 2024
- Merge remote-tracking branch 'origin/main' into 978-migrate-users
  tcnichol
  committed
  58ada46
Commits on Aug 22, 2024
- rename
  tcnichol
  committed
  62d5b02
- Merge branch 'release/v2.0-beta-3' into 978-migrate-users
  longshuicy
  committed
  8772325
Commits on Aug 26, 2024
- migrate metadata definition; modify migration script (#1184)
  
  longshuicy
  and
  tcnichol
  authored
  ec56aa4
Commits on Aug 27, 2024
- fix clowder user api key creation issue
  longshuicy
  committed
  8050460
- fix user api key if user already exists
  tcnichol
  committed
  c936760
- Merge remote-tracking branch 'origin/978-migrate-users' into 978-migrate-users
  tcnichol
  committed
  5547715
- switch headers and files
  tcnichol
  committed
  86c8493
Commits on Aug 29, 2024
- Migrate metadata (#1192)
  longshuicy
  authored
  2a52c7c
Commits on Sep 3, 2024
- 1187 migrate collections and spaces as v2 metadata. (#1190)
  
  authored
  4e68f21
Commits on Aug 16, 2025
- Merge branch 'main' into 978-migrate-users
  tcnichol
  committed
  4d5281a
- removed - already processed
  tcnichol
  committed
  70ea8af
Commits on Aug 17, 2025
- new readme, using tomli and a config file to limit or set options for migration
  tcnichol
  committed
  92f4f2c
- adding example for config.toml
  tcnichol
  committed
  80d41b0
- exclude users - users we do not migrate (or their files)
  tcnichol
  committed
  bc67fc7
- adding example env
  tcnichol
  committed
  d86a756
- proper path to .env file
  tcnichol
  committed
  a453068
- excluding items
  tcnichol
  committed
  c64be06
- check to see if we should migrate a dataset
  tcnichol
  committed
  c7e218c
- using the excluded space ids
  tcnichol
  committed
  c4d3cdb
- fix - if license absent in v1 assign a default license for v2
  tcnichol
  committed
  ca80dbd
- fix type in license
  tcnichol
  committed
  f98edda
- adding TODO items
  tcnichol
  committed
  5a84349
- working on this, need a better way to check what collections a dataset is contained in
  tcnichol
  committed
  1932362
Commits on Aug 18, 2025
- sloppy hack - keeping for commit since it's got the descendant collections right - can check from there
  tcnichol
  committed
  21629c7
- new classes (to be consolidated later)
  tcnichol
  committed
  b600cc9
- this seems to be working
  tcnichol
  committed
  734c9f1
- instructions for now (rough draft)
  tcnichol
  committed
  87324ec
- metadata posts
  tcnichol
  committed
  55a54bb
Commits on Aug 19, 2025
- log metadata def - errors are when there are duplicates
  tcnichol
  committed
  dbe3016
Commits on Aug 21, 2025
- get top level collections
  tcnichol
  committed
  73609d2
Commits on Aug 22, 2025
- method stubs for new way of processing
  tcnichol
  committed
  093bfb0
- more method stubs
  tcnichol
  committed
  8477cba
Commits on Aug 23, 2025
Commits on Aug 24, 2025
- remove unused methods
  tcnichol
  committed
  a5dd8ea
- too many folders created
  tcnichol
  committed
  f534160
Commits on Aug 25, 2025
- progress for dataset folders and files
  tcnichol
  committed
  a00e600
Commits on Aug 26, 2025
- the folder hierarchy part is not working for v2
  tcnichol
  committed
  2fb33c4
Commits on Aug 27, 2025
- new route should return all folders in a dataset
  tcnichol
  committed
  10f2026
Commits on Aug 28, 2025
Commits on Aug 29, 2025
- new method should work for uploading files to either a dataset or a folder, most of these will be uploaded to folders following the 'collection becomes dataset' model
  tcnichol
  committed
  1a5e2b7
Commits on Sep 1, 2025
- partial fix?
  tcnichol
  committed
  04fc4b2
Commits on Sep 2, 2025
- messy, but works, can post the data
  tcnichol
  committed
  c76e75e
Commits on Sep 3, 2025
- worked for one collection with datasets and folders
  tcnichol
  committed
  c45d8f0
- other collection also works now
  tcnichol
  committed
  9ca8850
- get list of datasets in collections, we need this for migration
  tcnichol
  committed
  e2be15b
- this should work for datasets now as well
  tcnichol
  committed
  c94dd6d
- single dataset works, except metadata
  tcnichol
  committed
  88e2534
- removing unused methods
  tcnichol
  committed
  7e139cb
- adding todo with dataset metadata to a folder
  tcnichol
  committed
  5b2562d
Commits on Sep 8, 2025
- migrate now adds a metadata file to a folder
  tcnichol
  committed
  0a8e764
- running pipenv run black
  tcnichol
  committed
  19b0f02
- running pipenv run black for app
  tcnichol
  committed
  0fa7465
- fix typo that got in
  tcnichol
  committed
  80ae7cf
- ran codegen?
  tcnichol
  committed
  a034ca5
- ran codegen?
  tcnichol
  committed
  1f3e5ed
Commits on Sep 9, 2025
- more readme instructions
  tcnichol
  committed
  34c868b
Commits on Nov 5, 2025
- fixing lint
  tcnichol
  committed
  27a1355

Commits

Commits on Mar 28, 2024

Commits on Apr 23, 2024

Commits on Apr 25, 2024

Commits on Apr 26, 2024

Commits on May 20, 2024

Commits on May 21, 2024

Commits on May 26, 2024

Commits on May 27, 2024

Commits on May 30, 2024

Commits on May 31, 2024

Commits on Jun 1, 2024

Commits on Jun 2, 2024

Commits on Jun 3, 2024

Commits on Jul 13, 2024

Commits on Jul 15, 2024

Commits on Aug 5, 2024

Commits on Aug 6, 2024

Commits on Aug 14, 2024

Commits on Aug 22, 2024

Commits on Aug 26, 2024

Commits on Aug 27, 2024

Commits on Aug 29, 2024

Commits on Sep 3, 2024

Commits on Aug 16, 2025

Commits on Aug 17, 2025

Commits on Aug 18, 2025

Commits on Aug 19, 2025

Commits on Aug 21, 2025

Commits on Aug 22, 2025

Commits on Aug 23, 2025

Commits on Aug 24, 2025

Commits on Aug 25, 2025

Commits on Aug 26, 2025

Commits on Aug 27, 2025

Commits on Aug 28, 2025

Commits on Aug 29, 2025

Commits on Sep 1, 2025

Commits on Sep 2, 2025

Commits on Sep 3, 2025

Commits on Sep 8, 2025

Commits on Sep 9, 2025

Commits on Nov 5, 2025