Repository navigation
Fix CI, audit all scripts, add new utilities and incremental GitHub backups - #53
Merged
Merged
Conversation
- run_audiocraft.sh / run_videogen.sh: remove blank lines before the shebang (SC1128), mark executable, reformat with 4-space indents - install_signal.sh: reformat to repo indent style - backup.sh: collapse multi-line subshells that beautysh mis-indents
Go through every script in src/ and fix bugs, misleading help text and leftover cruft while keeping each script's interface and style. Notable fixes: - random_password.sh always failed with no output; squash_branch.sh, system_info.sh, dead_code.sh and fetch_github_repos_names.sh were broken outright - resize_to_a4.sh overwrote originals by default (now writes ./resized) - strip_python_comments.sh mangled code and strings; strip_digits.sh deleted whole words instead of digits - web_block.sh --clear removed localhost entries; all actions now stay inside the managed section of the hosts file - backup.sh: no plaintext archive left on encryption failure, rsync errors propagate, passphrase no longer visible in ps, retention is scoped per host - extract.sh wrote .gz output over the directory path; convert_to_mp4.sh could overwrite its own input - in-place editors (last_line_empty, remove_*_whitespace, beautify, ...) now preserve file permissions and skip .git and binary files - beautify_script.sh --check now fails on beautysh parse errors - disk_usage.sh, clear_cache.sh, purge_and_reinstall_nodejs.sh, cpu_temp.sh, network_bandwidth.sh: option handling and locale fixes - leading-zero numbers no longer parsed as octal across math scripts - eval on user input removed (clear_cache.sh, empty_trash.sh, sum_args.sh) - errors go to stderr with non-zero exit codes; missing dependencies are reported up front - restore the executable bit on scripts that had lost it
- rename workflow to ci.yml, use actions/checkout@v4 and setup-python, apt-get update before installing, pin beautysh, cancel stale runs - add a smoke-test job: every script is checked for shebang, executable bit and syntax, and pure scripts are run against known outputs - hooks/_run_all.sh works from any directory, also checks tests/, and prints a summary of failing hooks - add .editorconfig matching the repo's formatting rules
New scripts: - retry.sh: retry a command with exponential backoff - wait_for_port.sh: wait for a TCP port, then optionally run a command - find_duplicate_files.sh: find identical files and report wasted space - find_large_files.sh: list the largest files or directories - ssl_cert_expiry.sh: check TLS certificate expiry for hosts (cron friendly) - git_cleanup_branches.sh: delete merged or upstream-gone local branches README: list the new scripts and the previously undocumented run_audiocraft.sh, run_videogen.sh and setup_obs_background_blur.sh, correct descriptions that no longer matched behavior, document how to run the checks locally, and normalize repository link casing.
Incremental backups: - keep backups in DEST/repos and DEST/archives instead of a fresh runs/<timestamp>/ copy on every run (runs/ now holds only the summary and checksums) - write DEST/backup-manifest.json with each repository's HEAD commit, remote refs fingerprint, settings fingerprint, output path, size, archive SHA-256 and timestamps; paths are stored relative to DEST so the backup directory can be moved - check remote refs with git ls-remote before each backup and skip repositories whose refs and settings are unchanged and whose backup still exists - download archives at the exact commit and name them owner__repo@<sha>.tar.gz; add --keep-archives N to prune old ones - add --force to back up everything and --backup-manifest FILE to relocate the record; summaries count unchanged repositories Authentication: - add --auth auto|token|gh|public for discover and backup; gh uses the GitHub CLI login via "gh auth token" - disable the user's git credential helpers for HTTPS so public mode is really anonymous and token/gh modes use exactly the chosen token - send the token to git as basic auth (x-access-token), which GitHub's git-over-HTTPS endpoint accepts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Gets CI green again, fixes bugs found by auditing every script in
src/, adds new utilities and a smoke-test suite, and makesdownload_all_github_repos.shback up only repositories that changed.CI fix
run_audiocraft.sh/run_videogen.shhad blank lines before the shebang (SC1128)backup.sh,install_signal.shand the two scripts abovelast_line_empty.shbeautify_script.shnow restores them, and exec bits are restored on all scriptsScript audit (~110 scripts)
Highlights:
resize_to_a4.shoverwrote originals by default (now writes./resized);strip_python_comments.shmangled code/strings;strip_digits.shdeleted whole words;web_block.sh --clearremovedlocalhostfrom/etc/hosts;extract.shwrote.gzoutput over the directory path;backup.shcould leave an unencrypted archive when encryption failedrandom_password.sh,system_info.sh,squash_branch.sh,dead_code.sh,fetch_github_repos_names.sh.git/ binary filesevalon user input removed; errors go to stderr with non-zero exit codes; leading-zero numbers no longer parsed as octal; missing dependencies reported up frontBehavior changes worth noting:
strip_python_comments.shkeeps docstrings,are_anagrams.shexits 1 for non-anagrams,system_info.shwithout options shows everything,month_to_number.shaccepts full month names.download_all_github_repos.sh
DEST/backup-manifest.jsonrecords each repo's HEAD commit, refs fingerprint, settings, output path, size and timestamps.git ls-remote(no API quota) is checked first and unchanged repositories are skipped. Backups live inDEST/repos/DEST/archivesinstead of a newruns/<timestamp>/copy each run; archives are named by commit, with--keep-archives N,--forceand--backup-manifest FILE.--auth auto|token|gh|publicfor discover and backup;ghuses the GitHub CLI login. Global git credential helpers are disabled for these git calls sopublicis truly anonymous.runs/are not reused, so the first run after this change downloads everything once.New scripts
retry.sh,wait_for_port.sh,find_duplicate_files.sh,find_large_files.sh,ssl_cert_expiry.sh,git_cleanup_branches.shCI and tooling
ci.yml:actions/checkout@v4, setup-python,apt-get update, pinned beautysh, concurrency canceltests/smoke_test.shjob: shebang/exec-bit/syntax checks for every script plus behavior tests for the pure oneshooks/_run_all.shworks from any directory, also checkstests/, and summarizes failures.editorconfig; README lists all scripts with corrected descriptionsTesting
./hooks/_run_all.shand./tests/smoke_test.sh(400 checks) pass locallydownload_all_github_repos.shtested offline against local bare repos (mirror/clone/sparse/archive, change detection, relocation, pruning) and against GitHub with--auth gh,tokenandpublicon a private repositoryKnown issues left as-is
fetch_github_repos_names.shtakes the token on the command line (visible inps)purge_and_reinstall_nodejs.sh --all-usersrunsrm -rfas root under other users' homesserver_health_monitor.shre-sends the alert email every interval