Handle rapid/bulk YouTube uploads by enqueueing all unseen uploads by Copilot · Pull Request #222 · GalvinPython/feedr

Copilot · 2026-07-22T08:04:42Z

Feedr was only comparing and processing the current head upload per channel, so when multiple videos were published between polling intervals, earlier items in the burst were skipped. This affected both regular uploads and Shorts.

Root cause
- Polling logic treated latestAllId drift as a single-event update and only queued one video ID (the newest), dropping intermediate uploads.
Behavior change
- Added unseen-upload backfill for each channel’s uploads playlist (UU...): fetch playlist items until the stored latestAllId is reached (or pagination ends), collect all unseen IDs, then process them in chronological order (oldest → newest).
- Preserves existing per-video classification flow (Video / Short / Stream) and existing guild subscription filtering before enqueueing Discord notifications.
Implementation notes
- Introduced fetchUnseenUploadVideoIds(channelId, latestKnownVideoId) in fetchLatestUploads.ts.
- Reused the existing update + notification path for each recovered video ID to keep behavior consistent with single-upload handling.

const unseenVideoIds = await fetchUnseenUploadVideoIds(channelId, latestKnownVideoId);

// Maintain expected notification order for burst uploads
const videosToProcess =
  unseenVideoIds.length > 0 ? unseenVideoIds.reverse() : [latestVideoId];

for (const videoId of videosToProcess) {
  // existing content-type detection + DB update + guild filtering + enqueue
}

GalvinPython · 2026-07-22T09:09:23Z

+        const pageTokenParam = pageToken ? `&pageToken=${pageToken}` : "";
+        const res = await fetch(
+            `https://youtube.googleapis.com/youtube/v3/playlistItems?part=snippet&maxResults=50&playlistId=${uploadPlaylistId}&key=${env.youtubeApiKey}${pageTokenParam}`,
+        );


we do not need to have page tokens imo - 50 videos is plenty enough to check

GalvinPython · 2026-07-22T10:23:31Z

+                for (const videoId of videosToProcess) {
+                    // Use duration-based detection to reduce API quota usage
+                    // and avoid UULF which is currently lagging.
+                    // YouTube Shorts are currently limited to 3 minutes (180s).
+                    // Videos at exactly 180s could still be shorts, so we use
+                    // a strict greater-than check.
+                    const durationSeconds = await fetchVideoDuration(videoId);
+                    const SHORTS_DURATION = 180;
+
+                    let contentType: PlaylistType | null = null;
+
+                    if (durationSeconds > SHORTS_DURATION) {
+                        // Over the shorts limit: cannot be a short, check only if it's a stream
+                        const streamVideoId =
+                            await getSinglePlaylistAndReturnVideoData(
+                                channelId,
+                                PlaylistType.Stream,
+                            );
+
+                        if (videoId === streamVideoId.videoId) {
+                            contentType = PlaylistType.Stream;
+                        } else {
+                            // Not a stream and over 3 min; must be a regular video
+                            contentType = PlaylistType.Video;
+                        }
+                    } else {
+                        // Under 3 minutes: could be a short or a video, check UUSH and UULV
+                        const [shortVideoId, streamVideoId] = await Promise.all([
+                            getSinglePlaylistAndReturnVideoData(
+                                channelId,
+                                PlaylistType.Short,
+                            ),
+                            getSinglePlaylistAndReturnVideoData(
+                                channelId,
+                                PlaylistType.Stream,
+                            ),
+                        ]);
+
+                        if (videoId === shortVideoId.videoId) {
+                            contentType = PlaylistType.Short;
+                        } else if (videoId === streamVideoId.videoId) {
+                            contentType = PlaylistType.Stream;
+                        } else {
+                            // Not in shorts or streams playlist → regular video
+                            contentType = PlaylistType.Video;
+                        }
+                    }
+
+                    console.log(
+                        "Determined content type:",
+                        contentType,
+                        `(duration: ${durationSeconds}s)`,
+                    );
+
+                    if (contentType) {
+                        console.log(
+                            `Updating ${contentType} video ID for channel`,
+                            channelId,
+                            "to",
+                            videoId,
+                        );
+                    } else {
+                        console.error(
+                            "No valid video ID found for channel",
+                            channelId,
+                            "with video ID",
+                            videoId,
+                        );
+                        continue;
+                    }
+
+                    const updateSuccess = await youtubeUpdateVideoId(
+                        channelId,
+                        videoId,
+                        contentType,
+
+                        // Temporarily using current date for update time
+                        new Date(),
+                    );
+
+                    if (!updateSuccess.success) {
+                        console.error(
+                            "Error updating video ID in fetchLatestUploads",
+                        );
+
+                        return;
+                    }
+
+                    console.info(`Filtered guilds for channel ID ${channelId}:`, {
+                        count: discordGuildsToUpdate.data.filter(
+                            (
+                                guild,
+                            ): guild is typeof dbGuildYouTubeSubscriptionsTable.$inferSelect =>
+                                "youtubeChannelId" in guild &&
+                                "trackVideos" in guild &&
+                                "trackShorts" in guild &&
+                                "trackStreams" in guild,
+                        ).length,
+                    });
+
+                    updates.set(videoId, {
+                        channelInfo,
+                        discordGuildsToUpdate: discordGuildsToUpdate.data.filter(
+                            (
+                                guild,
+                            ): guild is typeof dbGuildYouTubeSubscriptionsTable.$inferSelect =>
+                                "youtubeChannelId" in guild &&
+                                ((contentType === PlaylistType.Video &&
+                                    guild.trackVideos) ||
+                                    (contentType === PlaylistType.Short &&
+                                        guild.trackShorts) ||
+                                    (contentType === PlaylistType.Stream &&
+                                        guild.trackStreams)),
+                        ),
+                    });
+                }


this algorithm is checking the playlist for each video, however the playlist function will only return the latest video in that playlist. so if there's two shorts, it will be categorised as one video (default) and one short. we should check each playlist until each video id has been associated with a category to further reduce api calls

GalvinPython

@copilot make changes from comments

Initial plan

a1f1da0

Copilot AI assigned Copilot and GalvinPython Jul 22, 2026

github-project-automation Bot added this to Feedr Jul 22, 2026

Copilot AI linked an issue Jul 22, 2026 that may be closed by this pull request

Discord Bot Missing Notifications for Rapid/Bulk YouTube Uploads #221

Open

Copilot started work on behalf of GalvinPython July 22, 2026 08:05 View session

fix: queue all unseen rapid YouTube uploads

ede7b91

Copilot AI changed the title ~~[WIP] Fix missing notifications for rapid bulk YouTube uploads~~ Handle rapid/bulk YouTube uploads by enqueueing all unseen uploads Jul 22, 2026

Copilot finished work on behalf of GalvinPython July 22, 2026 08:09

Copilot AI requested a review from GalvinPython July 22, 2026 08:09

GalvinPython reviewed Jul 22, 2026

View reviewed changes

This comment was marked as outdated.

Sign in to view

GalvinPython self-requested a review July 22, 2026 10:24

GalvinPython requested changes Jul 22, 2026

View reviewed changes

Copilot started work on behalf of GalvinPython July 22, 2026 10:25 View session

Apply remaining changes

5354ef4

Copilot finished work on behalf of GalvinPython July 22, 2026 10:27

Copilot AI requested a review from GalvinPython July 22, 2026 10:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle rapid/bulk YouTube uploads by enqueueing all unseen uploads - #222

Handle rapid/bulk YouTube uploads by enqueueing all unseen uploads#222
GalvinPython with Copilot wants to merge 3 commits into
mainfrom
copilot/fix-missing-notifications-youtube-uploads

Copilot AI commented Jul 22, 2026 •

edited

Loading

Uh oh!

GalvinPython Jul 22, 2026

Uh oh!

GalvinPython Jul 22, 2026

Uh oh!

This comment was marked as outdated.

Uh oh!

GalvinPython left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Copilot AI commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GalvinPython Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

GalvinPython Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

GalvinPython left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented Jul 22, 2026 •

edited

Loading