Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[dashboard][server] Make Project overview page load faster by pre-fetching and caching Git provider data (branch details) #7610

Merged
merged 1 commit into from
Jan 31, 2022
Merged

[dashboard][server] Make Project overview page load faster by pre-fetching and caching Git provider data (branch details) #7610

merged 1 commit into from
Jan 31, 2022

Conversation

jankeromnes
Copy link
Contributor

@jankeromnes jankeromnes commented Jan 14, 2022 •
edited
Loading

Description

Make Project Branches page load faster by pre-fetching and caching GitHub/GitLab/Bitbucket data (branch details).

Related Issue(s)

Fixes #7536

How to test

  1. Open any Branches page on https://gitpod.io -- notice it sometimes takes 30-40s to load 🙄
  2. Add Projects in the preview environment and open its Branches page -- should be fast ⚡

Release Notes

[dashboard][server] Make Project overview page load faster by pre-fetching and caching Git provider data (branch details)

Documentation

  • /werft with-clean-slate-deployment

/uncc

@codecov

This comment has been minimized.

Sign in to view
@jankeromnes

This comment has been minimized.

Sign in to view
@jankeromnes jankeromnes force-pushed the jx/faster-project-overview branch 3 times, most recently from 3f07860 to 9b4074d Compare January 18, 2022 11:21
@jankeromnes

This comment has been minimized.

Sign in to view
@jankeromnes jankeromnes force-pushed the jx/faster-project-overview branch from 95ff25f to 40a6848 Compare January 18, 2022 14:22
@jankeromnes

This comment has been minimized.

Sign in to view
@jankeromnes

This comment has been minimized.

Sign in to view
@jankeromnes jankeromnes force-pushed the jx/faster-project-overview branch from 40a6848 to 4bf716d Compare January 18, 2022 15:06
@jankeromnes jankeromnes force-pushed the jx/faster-project-overview branch from 4bf716d to feabf72 Compare January 19, 2022 11:20
jankeromnes
jankeromnes commented Jan 19, 2022
View reviewed changes
components/gitpod-db/src/typeorm/migration/1642497869312-ProjectInfo.ts
export class ProjectInfo1642497869312 implements MigrationInterface {

public async up(queryRunner: QueryRunner): Promise<void> {
await queryRunner.query("CREATE TABLE IF NOT EXISTS `d_b_project_info` ( `projectId` char(36) NOT NULL, `overview` longtext NOT NULL, `deleted` tinyint(4) NOT NULL DEFAULT '0', `_lastModified` timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6), PRIMARY KEY (`projectId`), KEY `ind_dbsync` (`_lastModified`)) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;");
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: I'm using longtext for overview instead of just text because the (enormous) Project Overview of the GitLab repository (my stress test) wouldn't fit into a simple text.

We could also look into trimming unnecessary information from Project Overviews, but I'd leave this as a potential future optimization.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should look into this issue. I assume it's because of commit message length.

@jankeromnes jankeromnes marked this pull request as ready for review January 19, 2022 11:24
@jankeromnes
Copy link
Contributor Author

jankeromnes commented Jan 19, 2022 •
edited
Loading

Hi @AlexTugarev 👋 if you like, could you please take a look at this change when you have time? 👀

It basically changes the behavior of projectsService.getProjectOverview() from:

  • directly fetch all branches from GitHub/GitLab/Bitbucket API and wait for the results (can take many seconds or even minutes)

to:

  • return a cached version of all branches if available (fast!)
  • but at the same time also fetch all branches from GitHub/GitLab/Bitbucket API in the background, and update the cache once the data arrives (same behavior as before when there is no cache)

I think this is a good trade-off to make the Branches page load very fast. ⚡

@AlexTugarev
Copy link
Member

@jankeromnes, I'm sorry have missed this PR. Happy to look into it and test ride the next thing.

@jankeromnes jankeromnes requested a review from a team January 31, 2022 08:20
AlexTugarev
AlexTugarev reviewed Jan 31, 2022
View reviewed changes
components/gitpod-db/src/typeorm/project-db-impl.ts
await projectInfoRepo.save({
projectId,
overview,
creationTime: new Date().toISOString(),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not using for now, but let's keep it. we can add more optimization based on it.

@AlexTugarev
Copy link
Member

/lgtm

Changes look good! Thanks!

@roboquat roboquat added the lgtm label Jan 31, 2022
@roboquat
Copy link
Contributor

LGTM label has been added.

Git tree hash: 6da2feca088f465eb87d1f38d2a8ad6bd0780d35

@roboquat
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: AlexTugarev

Associated issue: #7536

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@roboquat roboquat merged commit 0736b3b into main Jan 31, 2022
@roboquat roboquat deleted the jx/faster-project-overview branch January 31, 2022 10:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AlexTugarev AlexTugarev AlexTugarev left review comments

Assignees

@AlexTugarev AlexTugarev

Labels
approved release-note size/L team: webapp Issue belongs to the WebApp team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Make the Project "Branches" page load faster
3 participants
@jankeromnes @AlexTugarev @roboquat