Compare commits

...
2 Commits
Author SHA1 Message Date
lobo 03469f9eb5 perf(library): add dashboard-games and stored game detail
Build, Test & Push Core Service / docker (push) Successful in 1m32s
Build, Test & Push Core Service / quality-check (push) Successful in 2m26s
Add GET /api/library/dashboard-games for the top and playing tiles
in one call. It has no count query.
GET /api/library/games/{appid}?refresh=false answers from stored
data and sets achievementsStale. It does not wait for Steam.
entryPlatforms reads the view once in a materialized CTE:
planning ~33 ms -> 15 ms.
Test CORS Access-Control-Max-Age 3600 on preflight.

Refs #47
2026-10-12 09:25:49 +02:00
lobo b1aaec0615 fix(speedrun): pause queue on 429/420 instead of dropping it
speedrun.com returns 429 or 420 when we ask too fast. These were
counted as errors: 3 in a row dropped both queues. Now the thread
pauses and retries the same game. Rate: 1 request per 2 s. Refs #48
2026-10-12 08:25:43 +02:00
20 changed files with 704 additions and 49 deletions

No files matched your search

+7 -4
View File
@@ -72,7 +72,9 @@ To Be Played is split into three repositories:
least one verified run. An edition that finds nothing tries its IGDB parent game with the parent's year, else its
name without an edition suffix. Several same-name games and no year to decide: no link. The game detail returns
the speedrun.com link as `speedrunUrl`. speedrun.com has no Steam or IGDB link and no key, so one queue works for
all users, with up to 3 requests per game and 1 request per second; a result is kept 30 days. Migration V47 asks
all users, with up to 3 requests per game and 1 request every 2 seconds; a result is kept 30 days. A rate limit answer
(429 or 420) pauses the queue (its Retry-After, else 1 min doubling up to 30 min) and asks the same game again; the
queue is never dropped for it. Migration V47 asks
every stored match again once at boot, behind user requests; links set by an auditor are never touched.
- **Achievements**: after every library sync a background job fetches the achievement schema of each owned game
and the user's unlocks of each game whose playtime changed. One Steam call per second, so user requests keep half
@@ -198,7 +200,7 @@ The token comes from Auth's `tbp_access` cookie or a `Bearer` header, whichever
(`USER < AUDITOR < PLATFORM_ADMIN`). A token without `aud` is refused. The claims are pinned in
`src/test/resources/contract/auth-core-contract.json`, the same file as in Auth; the tests build tokens only from it.
`POST`, `PUT` and `DELETE` need an `X-TBP-Client` or `Authorization` header (CSRF guard), and CORS allows only the
frontend origin.
frontend origin; preflight answers carry `Access-Control-Max-Age: 3600` (pinned by `ClientHeaderFilterTest`).
A token for another audience (for example a profile ticket) is rejected on normal endpoints; the profile ticket is
accepted only by `GET /api/public/profiles/library` in the `X-Profile-Ticket` header.
Every error answer of a controller has the body `{code, message}` (a 401 or 403 of the security filters has none); a refused request is `400 VALIDATION_FAILED` with
@@ -234,6 +236,7 @@ ArchUnit role rules (`RoleAnnotationRules`) and `src/test/resources/contract/aut
| `GET` | `/api/rewards/progress/{track}/games?page=`, `/api/rewards/history/{kind}/{ref}/games?page=` (the games behind a track or a `MONTH` / `STREAK` bonus; `400 NOT_A_GAME_LIST`) | JWT |
| `PUT` | `/api/rewards/equipped/{slot}` (`{item}`, null = take off; `200` wallet, `409 NOT_OWNED`, `400 WRONG_SLOT`) | JWT |
| `GET` | `/api/library/games` (one row per set, the counted edition, with `platforms[]` of all its editions: Steam, PlayStation, then manual, and `totalPlaytimeMinutes` (all platforms; `playtimeMinutes` stays the counted edition's own); `name` matches any edition; `source=STEAM,PSN` keeps rows with a platform of that source, a PlayStation title on a Steam row counts as `PSN`; ignored with `hidden=true`, which lists every hidden row; `400` on an unknown source; an old `hideDuplicates` param is ignored), `/api/library/games/{appid}` | JWT |
| `GET` | `/api/library/dashboard-games?top=10&playing=8` (`{top, playing}`: the Dashboard tiles in one call, listed entries by set playtime, most first; `playing` only status PLAYING; same rows and order as `/games` with `sort=playtimeMinutes,desc` / `status=PLAYING`, no count query; limits clamped to 1..20; calls `syncIfStale` like `/games`) | JWT |
| `PATCH` | `/api/library/games/{appid}/backlog` (only the sent fields; `null` clears one) | JWT |
| `PUT`, `DELETE` | `/api/library/games/{appid}/corrections/{field}` (`{value}`; `name`, `image`, `speedrun`, `moods`, `endless`, `genres`, `time_to_beat` = `{normallyMinutes, completelyMinutes}`; `204`, `400 INVALID_CORRECTION`) | JWT, AUDITOR |
| `GET` | `/api/library/catalog?q=&page=&size=` (every game some user still has, by name part, `PageDto` of `{appid, name, imageUrl, platforms}`; `q` at most 100 characters; size 20, 40 or 80) | JWT, AUDITOR |
@@ -261,7 +264,7 @@ ArchUnit role rules (`RoleAnnotationRules`) and `src/test/resources/contract/aut
| `GET`, `DELETE` | `/api/library/psn-account` (`{enabled, linked, linkedAt, expiresAt, reviewImportId}`: the PSN import waiting for review, else null), unlink (`204`, deletes the stored token) | JWT |
| `GET`, `DELETE` | `/api/library/imports/{id}/undo-check` (`{games, changed}`), `/api/library/imports/{id}/games` (undo; `204`, `409 IMPORT_NOT_SAVED`) | JWT |
| `POST` | `/api/library/sync` | JWT |
| `GET` | `/api/library/games/{appid}` (detail; slice 3: `achievements[]` also hold trophies with `kind` `STEAM`\|`PSN`, `setId`, `trophyType`, `group`, `rarity`; new `achievementSets[]` = `{id, kind, platform, name, total, unlocked, perfect, visible, listed}`, Steam first, then PS5 before PS4; `hasAchievements` = any set; the top-level counts stay Steam only; `platforms[]` = `{source, platform, appid, playtimeMinutes, lastPlayedAt, achievementsUnlocked, achievementsTotal, perfect}` one line per platform of the entry, Steam, PS5, PS4, manual, each with its own progress, `appid` = the edition whose detail lists them; `totalPlaytimeMinutes` = their sum) | JWT |
| `GET` | `/api/library/games/{appid}` (detail; `refresh=false` answers from stored data and sets `achievementsStale` when a due Steam refresh was skipped (default `refresh=true` refreshes, as before); slice 3: `achievements[]` also hold trophies with `kind` `STEAM`\|`PSN`, `setId`, `trophyType`, `group`, `rarity`; new `achievementSets[]` = `{id, kind, platform, name, total, unlocked, perfect, visible, listed}`, Steam first, then PS5 before PS4; `hasAchievements` = any set; the top-level counts stay Steam only; `platforms[]` = `{source, platform, appid, playtimeMinutes, lastPlayedAt, achievementsUnlocked, achievementsTotal, perfect}` one line per platform of the entry, Steam, PS5, PS4, manual, each with its own progress, `appid` = the edition whose detail lists them; `totalPlaytimeMinutes` = their sum) | JWT |
| `POST` | `/api/library/broken-images` (`{appid, kind}`, kind `COVER` or `ACHIEVEMENT_ICONS`; `204`) | JWT |
| `GET` | `/api/library/backlog?limit=&released=` (`{games}` in the user's order; `limit` 1-20, none = all) | JWT |
| `PUT` | `/api/library/backlog` (`{appids}` = the whole order; `204`) | JWT |
@@ -389,7 +392,7 @@ Inside a module: `api` (controllers, DTOs), `service` (business logic), `domain`
### Sync flow
1. `GET /api/library/games`, `/games-count` and `/picker` call `LibrarySyncService.syncIfStale`. It starts a sync
1. `GET /api/library/games`, `/dashboard-games`, `/games-count` and `/picker` call `LibrarySyncService.syncIfStale`. It starts a sync
when the last one is older than 24 hours, older than 15 minutes after a failure, or was for another Steam ID.
`POST /api/library/sync` starts one at once, but at most every 60 seconds (else `429` with `Retry-After`).
2. A nightly job (23:55, Europe/Berlin) syncs every account used in the last 14 days, one after the other. A
@@ -14,6 +14,7 @@ import core_service.modules.library.api.dto.CatalogHitDto;
import core_service.modules.library.api.dto.CorrectionMarkDto;
import core_service.modules.library.api.dto.GameDetailDto;
import core_service.modules.library.api.dto.CorrectionDto;
import core_service.modules.library.api.dto.DashboardGamesDto;
import core_service.modules.library.api.dto.FinishedHintDismissal;
import core_service.modules.library.api.dto.GameDto;
import core_service.modules.library.api.dto.HiddenUpdateDto;
@@ -99,6 +100,7 @@ public class LibraryController {
private static final int MAX_ACTIVITY_DAYS = 90;
private static final int MAX_PICKER_EXCLUDES = 100;
private static final int MAX_BACKLOG_GAMES = 20;
private static final int MAX_DASHBOARD_GAMES = 20;
private static final Set<String> SORTABLE_FIELDS = Set.of("playtimeMinutes", "firstSeenAt", "lastSeenAt", "appid", "rating", "priority");
private final LibrarySyncService syncService;
@@ -148,6 +150,16 @@ public class LibraryController {
return ResponseEntity.ok(PageDto.of(dtoPage));
}
@GetMapping("/dashboard-games")
public ResponseEntity<DashboardGamesDto> getDashboardGames(@AuthenticationPrincipal UserPrincipal principal,
@RequestParam(defaultValue = "10") int top,
@RequestParam(defaultValue = "8") int playing) {
Long userId = principal.getIdAsLong();
syncService.syncIfStale(userId, principal.steamIdText().orElse(null));
return ResponseEntity.ok(queryService.listDashboardGames(userId, Math.clamp(top, 1, MAX_DASHBOARD_GAMES),
Math.clamp(playing, 1, MAX_DASHBOARD_GAMES)));
}
private Sort parseSort(String sort) {
Sort fallback = Sort.by(Sort.Direction.DESC, "playtimeMinutes");
if (sort == null || sort.isBlank()) return fallback;
@@ -179,9 +191,10 @@ public class LibraryController {
@GetMapping("/games/{appid}")
public ResponseEntity<?> getGameDetail(@AuthenticationPrincipal UserPrincipal principal,
@PathVariable("appid") Long appid) {
@PathVariable("appid") Long appid,
@RequestParam(defaultValue = "true") boolean refresh) {
Long userId = principal.getIdAsLong();
return gameDetailService.getGameDetail(userId, principal.steamIdText().orElse(null), appid)
return gameDetailService.getGameDetail(userId, principal.steamIdText().orElse(null), appid, refresh)
.<ResponseEntity<?>>map(ResponseEntity::ok)
.orElseGet(() -> ResponseEntity.status(HttpStatus.NOT_FOUND)
.body(GAME_NOT_FOUND));
@@ -0,0 +1,8 @@
package core_service.modules.library.api.dto;
import java.util.List;
/**
* The MOST_PLAYED and PLAYING tiles: listed entries by set playtime, most first; a game can be in both.
*/
public record DashboardGamesDto(List<GameDto> top, List<GameDto> playing) {}
@@ -61,7 +61,9 @@ public record GameDetailDto(
// one line per platform of the entry, Steam first; empty for a hidden game
List<PlatformPlayDto> platforms,
// the sum of the platform lines; the counts above and playtimeMinutes stay this edition's own
int totalPlaytimeMinutes
int totalPlaytimeMinutes,
// true when this answer skipped a due Steam refresh (refresh=false); the same call without it refreshes
boolean achievementsStale
) {
/**
@@ -309,7 +309,8 @@ public class GameLinkRepository {
/**
* @return per given entry key, the platforms of its editions (see V43): Steam, PlayStation, then manual, by
* platform name; a PlayStation platform without a name is left out when the entry has a named PlayStation one, or its own member row has a manual platform
* platform name; a PlayStation platform without a name is left out when the entry has a named PlayStation one, or its own member row has a manual platform.
* The view is read once: each reference costs ~16 ms of planning (nested views), more than the query runs.
*/
public Map<Long, List<EntryPlatform>> entryPlatforms(long userId, Collection<Long> appids) {
Map<Long, List<EntryPlatform>> platforms = new HashMap<>();
@@ -317,12 +318,14 @@ public class GameLinkRepository {
return platforms;
}
jdbc.query("""
with p as materialized (
select appid, member_appid, source, platform from user_game_entry_platform
where user_id = :user and appid in (:appids))
select appid, source, platform
from user_game_entry_platform p
where user_id = :user and appid in (:appids)
and not (source = 'PSN' and platform = '' and exists (
select 1 from user_game_entry_platform n
where n.user_id = p.user_id and n.appid = p.appid and n.platform <> ''
from p
where not (source = 'PSN' and platform = '' and exists (
select 1 from p n
where n.appid = p.appid and n.platform <> ''
and (n.source = 'PSN' or (n.source = 'MANUAL' and n.member_appid = p.member_appid))))
group by appid, source, platform
order by appid, case source when 'STEAM' then 0 when 'PSN' then 1 else 2 end, platform
@@ -134,12 +134,26 @@ public class AchievementSyncService {
}
public void refreshSchemaIfStale(Long appid) {
Optional<GameAchievementSchemaMeta> existing = schemaMetaRepository.findById(appid);
if (existing.isEmpty() || isStale(existing.get())) {
if (schemaStale(appid)) {
refreshSchema(appid, false, false);
}
}
/**
* @return true when the schema meta of the game is missing or older than the schema TTL
*/
public boolean schemaStale(Long appid) {
return schemaMetaRepository.findById(appid).map(this::isStale).orElse(true);
}
/**
* @return true when the user's achievement meta of the game is missing or older than ttl
*/
public boolean userStale(Long userId, Long appid, Duration ttl) {
return userAchievementMetaRepository.findByUserIdAndAppid(userId, appid)
.map(meta -> meta.getRefreshedAt().isBefore(Instant.now().minus(ttl))).orElse(true);
}
/**
* Sets FINISHED when the user unlocked the ending achievement (see {@link EndingAchievement}) of a Steam game whose
* status they never chose. Uses stored rows only.
@@ -168,8 +182,7 @@ public class AchievementSyncService {
}
public void refreshUserIfStale(Long userId, String steamId, Long appid, int playtimeMinutes, Duration ttl) {
Optional<UserGameAchievementMeta> existing = userAchievementMetaRepository.findByUserIdAndAppid(userId, appid);
if (existing.isEmpty() || existing.get().getRefreshedAt().isBefore(Instant.now().minus(ttl))) {
if (userStale(userId, appid, ttl)) {
refreshUser(userId, steamId, appid, playtimeMinutes);
}
}
@@ -98,6 +98,15 @@ public class GameDetailService {
* @param steamId null when the user has none; then the stored achievements are shown without a refresh
*/
public Optional<GameDetailDto> getGameDetail(Long userId, String steamId, Long askedAppid) {
return getGameDetail(userId, steamId, askedAppid, true);
}
/**
* @param refreshAchievements false answers from stored data and only says in {@code achievementsStale} that the
* Steam refresh is due
*/
public Optional<GameDetailDto> getGameDetail(Long userId, String steamId, Long askedAppid,
boolean refreshAchievements) {
Long appid = askedAppid;
Optional<UserGame> userGameOpt = userGameRepository.findById(new UserGameId(userId, appid));
if (userGameOpt.isEmpty()) {
@@ -112,17 +121,18 @@ public class GameDetailService {
return Optional.empty();
}
SteamGame meta = steamGameRepository.findById(appid).orElse(null);
Refreshed refreshed = refresh(userGameOpt.get(), steamId);
return Optional.of(assemble(refreshed.userGame(), meta, refreshed.hasSchema()));
Refreshed refreshed = refresh(userGameOpt.get(), steamId, refreshAchievements);
return Optional.of(assemble(refreshed.userGame(), meta, refreshed.hasSchema(), refreshed.achievementsStale()));
}
/**
* @param userGame the game after the refresh, re-read when the finish rule changed its status
* @param achievementsStale a due Steam refresh was skipped
*/
private record Refreshed(UserGame userGame, boolean hasSchema) {}
private record Refreshed(UserGame userGame, boolean hasSchema, boolean achievementsStale) {}
// May call Steam, IGDB and speedrun.com and write.
private Refreshed refresh(UserGame userGame, String steamId) {
private Refreshed refresh(UserGame userGame, String steamId, boolean refreshAchievements) {
Long userId = userGame.getUserId();
Long appid = userGame.getAppid();
// A hidden game gets no calls to Steam, IGDB or speedrun.com; it shows what is stored.
@@ -138,42 +148,64 @@ public class GameDetailService {
timeToBeatService.refreshInBackground(List.of(appid));
moodService.refreshInBackground(List.of(appid));
releaseDateService.refreshInBackground(List.of(appid));
return new Refreshed(userGame, false);
return new Refreshed(userGame, false, false);
}
if (!hidden) {
try {
achievementSyncService.refreshSchemaIfStale(appid);
} catch (Exception e) {
log.warn("Achievement schema of appid={} not refreshed, showing stored data: {}", appid, e.getMessage());
}
}
boolean stale = !hidden && refreshSchemaIfAsked(appid, refreshAchievements);
boolean hasSchema = schemaMetaRepository.findById(appid).map(GameAchievementSchemaMeta::isHasSchema).orElse(false);
if (!hasSchema) {
return new Refreshed(userGame, false);
return new Refreshed(userGame, false, stale);
}
if (steamId != null && !hidden) {
try {
achievementSyncService.refreshUserIfStale(userId, steamId, appid, userGame.getPlaytimeMinutes(),
Duration.ofHours(achievementTtlHours));
} catch (Exception e) {
log.warn("Achievements of user={} appid={} not refreshed, showing stored data: {}", userId, appid, e.getMessage());
}
stale |= refreshUserIfAsked(userGame, steamId, refreshAchievements);
}
if (!hidden && !userGame.isStatusSetByUser() && achievementSyncService.finishIfEndingUnlocked(userId, appid)) {
userGame = userGameRepository.findById(new UserGameId(userId, appid)).orElse(userGame);
}
return new Refreshed(userGame, true);
return new Refreshed(userGame, true, stale);
}
/**
* @return true when the refresh was not asked and is due
*/
private boolean refreshSchemaIfAsked(Long appid, boolean refreshAchievements) {
if (!refreshAchievements) {
return achievementSyncService.schemaStale(appid);
}
try {
achievementSyncService.refreshSchemaIfStale(appid);
} catch (Exception e) {
log.warn("Achievement schema of appid={} not refreshed, showing stored data: {}", appid, e.getMessage());
}
return false;
}
/**
* @return true when the refresh was not asked and is due
*/
private boolean refreshUserIfAsked(UserGame userGame, String steamId, boolean refreshAchievements) {
Long userId = userGame.getUserId();
Long appid = userGame.getAppid();
Duration ttl = Duration.ofHours(achievementTtlHours);
if (!refreshAchievements) {
return achievementSyncService.userStale(userId, appid, ttl);
}
try {
achievementSyncService.refreshUserIfStale(userId, steamId, appid, userGame.getPlaytimeMinutes(), ttl);
} catch (Exception e) {
log.warn("Achievements of user={} appid={} not refreshed, showing stored data: {}", userId, appid, e.getMessage());
}
return false;
}
// Reads only.
private GameDetailDto assemble(UserGame userGame, SteamGame meta, boolean hasSchema) {
private GameDetailDto assemble(UserGame userGame, SteamGame meta, boolean hasSchema, boolean achievementsStale) {
List<AchievementSetDto> psnSets = psnSets(userGame);
List<AchievementDto> psnTrophies = psnSets.isEmpty() ? List.of() : psnTrophies(userGame);
if (!hasSchema) {
return buildDto(userGame, meta, true, 0, 0, List.of(), List.of(), psnSets, psnTrophies);
return buildDto(userGame, meta, true, 0, 0, List.of(), List.of(), psnSets, psnTrophies, achievementsStale);
}
Long userId = userGame.getUserId();
Long appid = userGame.getAppid();
@@ -206,7 +238,8 @@ public class GameDetailService {
.toList();
return buildDto(userGame, meta, statsVisible, schemaList.size(),
AchievementProgress.capped(unlocked, schemaList.size()), achievementDtos, schemaList, psnSets, psnTrophies);
AchievementProgress.capped(unlocked, schemaList.size()), achievementDtos, schemaList, psnSets, psnTrophies,
achievementsStale);
}
private List<AchievementSetDto> psnSets(UserGame userGame) {
@@ -233,7 +266,8 @@ public class GameDetailService {
private GameDetailDto buildDto(UserGame userGame, SteamGame meta, boolean statsVisible, int total, int unlocked,
List<AchievementDto> achievements, List<GameAchievementSchema> schema,
List<AchievementSetDto> psnSets, List<AchievementDto> psnTrophies) {
List<AchievementSetDto> psnSets, List<AchievementDto> psnTrophies,
boolean achievementsStale) {
Optional<GameTimeToBeat> timeToBeat = gameTimeToBeatRepository.findById(userGame.getAppid());
Set<String> unlockedNames = achievements.stream().filter(AchievementDto::unlocked).map(AchievementDto::apiName)
.collect(Collectors.toSet());
@@ -299,7 +333,8 @@ public class GameDetailService {
coming.map(ComingGame::getReleaseDate).orElse(null),
sets,
plays.stream().map(line -> toDto(line, userGame, total, unlocked, statsVisible)).toList(),
PlatformPlay.total(plays)
PlatformPlay.total(plays),
achievementsStale
);
}
@@ -3,6 +3,7 @@ package core_service.modules.library.service;
import core_service.modules.library.api.dto.AchievementSummaryDto;
import core_service.modules.library.api.dto.ActivityDto;
import core_service.modules.library.api.dto.BacklogOrderDto;
import core_service.modules.library.api.dto.DashboardGamesDto;
import core_service.modules.library.api.dto.GameDto;
import core_service.modules.library.api.dto.LibrarySummaryDto;
import core_service.modules.library.api.dto.PickReasonDto;
@@ -134,6 +135,20 @@ public class LibraryQueryService {
return new PageImpl<>(games, page.getPageable(), page.getTotalElements());
}
/**
* The same rows and order as {@link #listOwnedGames} without a filter, sorted {@code playtimeMinutes,desc} (top)
* and with {@code status=PLAYING} (playing), but without the count query; one enrichment for both lists.
*/
public DashboardGamesDto listDashboardGames(Long userId, int topLimit, int playingLimit) {
Sort byPlaytime = UserGameRepository.querySort(Sort.by(Sort.Direction.DESC, "playtimeMinutes"));
List<UserGame> top = userGameRepository.findListedByStatusIn(userId, EnumSet.allOf(GameStatus.class),
PageRequest.of(0, topLimit, byPlaytime));
List<UserGame> playing = userGameRepository.findListedByStatusIn(userId, List.of(GameStatus.PLAYING),
PageRequest.of(0, playingLimit, byPlaytime));
List<GameDto> dtos = toDtos(Stream.concat(top.stream(), playing.stream()).toList());
return new DashboardGamesDto(dtos.subList(0, top.size()), dtos.subList(top.size(), dtos.size()));
}
/**
* One row per {@link PickSource} (next up, next in series, because you played, same studio, friends played,
* longest waiting, mood), in the order of its constants, each with up to three backlog games. Budget and exclude
@@ -10,6 +10,7 @@ import core_service.modules.library.repo.GameSpeedrunRepository;
import core_service.modules.library.repo.SteamGameRepository;
import core_service.modules.library.repo.UserGameRepository;
import core_service.modules.speedrun_adapter.service.SpeedrunApiService;
import core_service.modules.speedrun_adapter.service.SpeedrunThrottledException;
import core_service.shared.background.BackgroundPools;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
@@ -42,6 +43,9 @@ import java.util.concurrent.atomic.AtomicBoolean;
* year of its IGDB parent game, a game without a parent by its name without an edition suffix. A sure hit without
* verified runs ends the lookup. See {@link SpeedrunMatch} for "sure".
* <p>
* A rate limit answer (429, 420) pauses the thread (Retry-After, else 1 min doubling up to 30 min) and asks the same
* game again; it never drops the queues.
* <p>
* {@link #recheckMarked} asks the rows a migration marked (checked_at = epoch) again. Those games wait behind every
* game a user asked for.
*/
@@ -55,6 +59,8 @@ public class SpeedrunService {
static final int MAX_ERRORS_IN_A_ROW = 3;
// A small batch, so a user request waits for a few games and not for 50.
static final int RECHECK_BATCH = 10;
static final Duration FIRST_PAUSE = Duration.ofMinutes(1);
static final Duration MAX_PAUSE = Duration.ofMinutes(30);
private final SpeedrunApiService speedrunApiService;
private final IgdbApiService igdbApiService;
@@ -75,7 +81,7 @@ public class SpeedrunService {
/**
* Queues the games without a fresh row and returns; the speedrun thread works the queue off. A game takes up to 3
* requests, and there is 1 request per second.
* requests, and there is 1 request every 2 seconds.
*/
public void refresh(Collection<Long> appids) {
if (!speedrunApiService.isEnabled() || appids.isEmpty()) return;
@@ -107,7 +113,7 @@ public class SpeedrunService {
running.set(false);
}
// Games can be queued after the last poll and before the flag was cleared.
if (pending.isEmpty() && recheck.isEmpty()) return;
if (Thread.currentThread().isInterrupted() || (pending.isEmpty() && recheck.isEmpty())) return;
}
}
@@ -136,6 +142,7 @@ public class SpeedrunService {
private void drain() {
int errorsInARow = 0;
int throttlesInARow = 0;
while (!pending.isEmpty() || !recheck.isEmpty()) {
boolean forUsers = !pending.isEmpty();
List<Long> batch = forUsers ? poll(pending, BATCH) : poll(recheck, RECHECK_BATCH);
@@ -165,12 +172,25 @@ public class SpeedrunService {
boolean stored;
try {
stored = checkAndStore(appid, name, years.get().getOrDefault(appid, Set.of()), parents.get().get(appid));
} catch (SpeedrunThrottledException e) {
Duration pause = pauseFor(e.getRetryAfter(), ++throttlesInARow);
log.warn("speedrun.com rate limit, pausing {} ({} games queued)", pause,
pending.size() + recheck.size() + batch.size() - i);
try {
pause(pause);
} catch (InterruptedException stop) {
Thread.currentThread().interrupt();
return;
}
i--; // same game again, after the user check at the loop head
continue;
} catch (Exception e) {
log.warn("Speedrun game of appid={} not stored: {}", appid, e.getMessage());
stored = false;
}
if (stored) {
errorsInARow = 0;
throttlesInARow = 0;
} else if (++errorsInARow >= MAX_ERRORS_IN_A_ROW) {
log.warn("Speedrun check failed {} times in a row, {} queued games dropped", errorsInARow, pending.size());
pending.clear();
@@ -181,6 +201,22 @@ public class SpeedrunService {
}
}
/**
* @param retryAfter the pause speedrun.com asked for; null when it did not say
* @param throttlesInARow throttled answers since the last stored game, at least 1
*/
static Duration pauseFor(Duration retryAfter, int throttlesInARow) {
if (retryAfter != null) {
return Duration.ofSeconds(Math.clamp(retryAfter.toSeconds(), 1, MAX_PAUSE.toSeconds()));
}
Duration backoff = FIRST_PAUSE.multipliedBy(1L << Math.min(throttlesInARow - 1, 5));
return backoff.compareTo(MAX_PAUSE) > 0 ? MAX_PAUSE : backoff;
}
void pause(Duration pause) throws InterruptedException {
Thread.sleep(pause);
}
private static List<Long> poll(ConcurrentSkipListSet<Long> queue, int size) {
List<Long> batch = new ArrayList<>();
Long next;
@@ -7,9 +7,12 @@ import lombok.Getter;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.http.HttpHeaders;
import org.springframework.stereotype.Service;
import org.springframework.web.client.HttpStatusCodeException;
import org.springframework.web.client.RestClient;
import java.time.Duration;
import java.util.List;
import java.util.Optional;
@@ -45,6 +48,8 @@ public class SpeedrunApiService {
/**
* Games whose name looks like the given one, in speedrun.com's order, without mods and category extensions.
* Empty when the adapter is off or the request failed.
*
* @throws SpeedrunThrottledException on a rate limit answer (429, 420)
*/
public Optional<List<Game>> findGames(String name) {
if (!enabled || name == null || name.isBlank()) {
@@ -67,6 +72,7 @@ public class SpeedrunApiService {
.map(game -> new Game(game.id(), game.names().international().trim(), game.released(), game.weblink()))
.toList());
} catch (Exception e) {
throwIfThrottle(e);
log.warn("speedrun.com search for \"{}\" failed: {}", name, e.getMessage());
return Optional.empty();
}
@@ -74,6 +80,8 @@ public class SpeedrunApiService {
/**
* Empty when the adapter is off or the request failed.
*
* @throws SpeedrunThrottledException on a rate limit answer (429, 420)
*/
public Optional<Boolean> hasVerifiedRuns(String gameId) {
if (!enabled) {
@@ -90,8 +98,26 @@ public class SpeedrunApiService {
}
return Optional.of(!runs.data().isEmpty());
} catch (Exception e) {
throwIfThrottle(e);
log.warn("speedrun.com runs of game {} failed: {}", gameId, e.getMessage());
return Optional.empty();
}
}
private static void throwIfThrottle(Exception e) {
if (!(e instanceof HttpStatusCodeException http)) return;
int status = http.getStatusCode().value();
if (status != 429 && status != 420) return;
Duration retryAfter = null;
HttpHeaders headers = http.getResponseHeaders();
String header = headers == null ? null : headers.getFirst(HttpHeaders.RETRY_AFTER);
if (header != null) {
try {
retryAfter = Duration.ofSeconds(Long.parseLong(header.trim()));
} catch (NumberFormatException ignored) {
// an HTTP-date: the caller backs off on its own
}
}
throw new SpeedrunThrottledException(retryAfter);
}
}
@@ -0,0 +1,22 @@
package core_service.modules.speedrun_adapter.service;
import lombok.Getter;
import java.time.Duration;
/**
* speedrun.com asked us to slow down: 429 (its Cloudflare) or 420 (the documented API limit).
*/
@Getter
public class SpeedrunThrottledException extends RuntimeException {
/**
* From the Retry-After header; null when it is missing or not a number of seconds.
*/
private final Duration retryAfter;
public SpeedrunThrottledException(Duration retryAfter) {
super("speedrun.com rate limit");
this.retryAfter = retryAfter;
}
}
+2 -2
View File
@@ -135,9 +135,9 @@ resilience4j:
limit-refresh-period: 1s
timeout-duration: 5s
speedrunApi:
# speedrun.com allows 100 requests/min per IP.
# speedrun.com documents 100 requests/min per IP, but its Cloudflare answered 429 (1015) at 60/min: 30/min.
limit-for-period: 1
limit-refresh-period: 1s
limit-refresh-period: 2s
timeout-duration: 30s
igdbApi:
# IGDB allows 4 requests/s. Taken per HTTP request.
@@ -46,7 +46,8 @@ class ClientHeaderFilterTest extends AbstractMockedAdaptersTest {
.header("Access-Control-Request-Headers", "x-tbp-client"))
.andExpect(status().isOk())
.andExpect(header().string("Access-Control-Allow-Origin", FRONTEND))
.andExpect(header().string("Access-Control-Allow-Credentials", "true"));
.andExpect(header().string("Access-Control-Allow-Credentials", "true"))
.andExpect(header().string("Access-Control-Max-Age", "3600"));
mockMvc.perform(options("/api/library/games/220/backlog")
.header("Origin", FRONTEND)
@@ -0,0 +1,130 @@
package core_service.modules.library.api;
import core_service.modules.library.domain.SteamGame;
import core_service.modules.library.domain.UserGame;
import core_service.modules.library.repo.SteamGameRepository;
import core_service.modules.library.repo.UserGameRepository;
import core_service.shared.test.AbstractIntegrationTest;
import core_service.shared.test.ContractTokens;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.webmvc.test.autoconfigure.AutoConfigureMockMvc;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.test.context.ActiveProfiles;
import org.springframework.test.web.servlet.MockMvc;
import org.springframework.test.web.servlet.request.MockHttpServletRequestBuilder;
import tools.jackson.databind.JsonNode;
import tools.jackson.databind.json.JsonMapper;
import java.time.Instant;
import java.util.List;
import java.util.stream.Stream;
import static org.assertj.core.api.Assertions.assertThat;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.jsonPath;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;
@AutoConfigureMockMvc
@ActiveProfiles(value = "prod", inheritProfiles = false)
class LibraryControllerDashboardGamesTest extends AbstractIntegrationTest {
private static final long USER = 931_101L;
private static final long OTHER = 931_102L;
private static final JsonMapper MAPPER = JsonMapper.builder().build();
@Autowired
private MockMvc mockMvc;
@Autowired
private SteamGameRepository steamGameRepository;
@Autowired
private UserGameRepository userGameRepository;
@Autowired
private JdbcTemplate jdbc;
@BeforeEach
void seed() {
userGameRepository.deleteAll();
// 25 games, playtime falls with the appid; 1, 3, 5 and 20 play (20 is outside the top 10)
for (long appid = 1; appid <= 25; appid++) {
steamGameRepository.save(new SteamGame(appid, "Game " + appid, null, Instant.now()));
userGameRepository.save(new UserGame(USER, appid, (int) (1000 - appid * 10), Instant.now(), Instant.now()));
}
jdbc.update("update user_game set status = 'BACKLOG' where user_id = ?", USER);
jdbc.update("update user_game set status = 'PLAYING' where user_id = ? and appid in (1, 3, 5, 20)", USER);
steamGameRepository.save(new SteamGame(900L, "Other", null, Instant.now()));
userGameRepository.save(new UserGame(OTHER, 900L, 99_999, Instant.now(), Instant.now()));
jdbc.update("update user_game set status = 'PLAYING' where user_id = ?", OTHER);
}
private static MockHttpServletRequestBuilder as(MockHttpServletRequestBuilder request, long userId) {
return request.header("Authorization", "Bearer " + ContractTokens.loginToken(userId));
}
private JsonNode json(MockHttpServletRequestBuilder request) throws Exception {
return MAPPER.readTree(mockMvc.perform(as(request, USER)).andExpect(status().isOk()).andReturn()
.getResponse().getContentAsString());
}
private static List<Long> appids(JsonNode games) {
return Stream.iterate(0, i -> i + 1).limit(games.size()).map(i -> games.get(i).get("appid").asLong()).toList();
}
@Test
void getDashboardGames_ReturnsBothTilesInTheOrderOfGames() throws Exception {
JsonNode dashboard = json(get("/api/library/dashboard-games").param("top", "3").param("playing", "3"));
JsonNode topGames = json(get("/api/library/games").param("size", "20").param("sort", "playtimeMinutes,desc"));
JsonNode playingGames = json(get("/api/library/games").param("size", "20").param("status", "PLAYING"));
assertThat(appids(dashboard.get("top"))).containsExactly(1L, 2L, 3L)
.isEqualTo(appids(topGames.get("content")).subList(0, 3));
assertThat(appids(dashboard.get("playing"))).containsExactly(1L, 3L, 5L)
.isEqualTo(appids(playingGames.get("content")).subList(0, 3));
assertThat(dashboard.get("top").get(0)).isEqualTo(topGames.get("content").get(0));
assertThat(dashboard.get("playing").get(2)).isEqualTo(playingGames.get("content").get(2));
}
@Test
void getDashboardGames_ClampsTheLimits() throws Exception {
JsonNode low = json(get("/api/library/dashboard-games").param("top", "0").param("playing", "-4"));
JsonNode high = json(get("/api/library/dashboard-games").param("top", "500").param("playing", "500"));
assertThat(low.get("top")).hasSize(1);
assertThat(low.get("playing")).hasSize(1);
assertThat(high.get("top")).hasSize(20);
assertThat(high.get("playing")).hasSize(4);
}
@Test
void getDashboardGames_DefaultsAreTenAndEight() throws Exception {
JsonNode dashboard = json(get("/api/library/dashboard-games"));
assertThat(dashboard.get("top")).hasSize(10);
assertThat(dashboard.get("playing")).hasSize(4);
}
@Test
void getDashboardGames_OnlyTheCallersGames() throws Exception {
JsonNode dashboard = json(get("/api/library/dashboard-games"));
assertThat(appids(dashboard.get("top"))).doesNotContain(900L);
assertThat(appids(dashboard.get("playing"))).doesNotContain(900L);
}
@Test
void getDashboardGames_WithoutLogin_401() throws Exception {
mockMvc.perform(get("/api/library/dashboard-games")).andExpect(status().isUnauthorized());
}
@Test
void getGameDetail_RefreshFalse_HasTheStaleFlag() throws Exception {
mockMvc.perform(as(get("/api/library/games/1").param("refresh", "false"), USER))
.andExpect(status().isOk())
.andExpect(jsonPath("$.appid").value(1))
.andExpect(jsonPath("$.achievementsStale").value(true));
}
}
@@ -0,0 +1,104 @@
package core_service.modules.library.repo;
import core_service.modules.library.domain.GameSource;
import core_service.modules.library.domain.GameStatus;
import core_service.modules.library.domain.SteamGame;
import core_service.modules.library.domain.UserGame;
import core_service.shared.test.AbstractPersistenceTest;
import jakarta.persistence.EntityManager;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.domain.PageRequest;
import org.springframework.data.domain.Sort;
import org.springframework.jdbc.core.JdbcTemplate;
import java.time.Instant;
import java.util.ArrayList;
import java.util.EnumSet;
import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
/**
* The Dashboard tiles read {@code findListedByStatusIn}; they must show the rows and the order of the Library list
* ({@code findOwnedGames}, no filter, playtime descending) for the same user.
*/
class DashboardGamesQueryGuardTest extends AbstractPersistenceTest {
private static final long USER_ID = 1L;
private static final long OTHER_USER_ID = 2L;
@Autowired
private UserGameRepository userGameRepository;
@Autowired
private SteamGameRepository steamGameRepository;
@Autowired
private JdbcTemplate jdbc;
@Autowired
private EntityManager entityManager;
@BeforeEach
void seed() {
List<SteamGame> catalog = new ArrayList<>();
for (long appid = 100; appid < 112; appid++) catalog.add(new SteamGame(appid, "Game " + appid, null, Instant.now()));
steamGameRepository.saveAll(catalog);
Instant now = Instant.now();
int[] playtime = {300, 120, 500, 500, 60, 800, 90, 400, 700, 10, 250, 250};
for (long user : new long[]{USER_ID, OTHER_USER_ID}) {
for (int i = 0; i < playtime.length; i++) {
userGameRepository.save(new UserGame(user, 100L + i, playtime[i], now, now));
}
}
// 100 and 101 are one set; 101 plays 120 but the set total (420) beats 102's single row for the user
jdbc.update("delete from game_link");
jdbc.update("insert into game_link (appid, set_id) values (100, 1), (101, 1), (108, 2), (109, 2)");
for (long user : new long[]{USER_ID, OTHER_USER_ID}) {
update(user, 100, "status = 'PLAYING'");
update(user, 102, "status = 'PLAYING'");
update(user, 103, "status = 'PLAYING'");
update(user, 105, "source = 'MANUAL', status = 'PLAYING'");
update(user, 106, "source = 'PSN', status = 'PLAYING'");
update(user, 107, "hidden_at = now(), status = 'PLAYING'");
update(user, 104, "removed_at = now(), status = 'PLAYING'");
update(user, 110, "status = 'FINISHED'");
update(user, 111, "status = 'PLAYING'");
jdbc.update("insert into user_psn_title (user_id, psn_key, state, appid, platform, playtime_minutes) "
+ "values (?, 'c1', 'STEAM_OWNED', 102, 'PS5', 700)", user);
}
// The users differ, so a query without the user filter returns other rows
update(OTHER_USER_ID, 100, "status = 'BACKLOG'");
update(OTHER_USER_ID, 108, "removed_at = now()");
update(OTHER_USER_ID, 109, "status = 'PLAYING'");
entityManager.flush();
entityManager.clear();
}
private void update(long user, long appid, String set) {
jdbc.update("update user_game set " + set + " where user_id = ? and appid = ?", user, appid);
}
@Test
void findListedByStatusIn_ReturnsTheRowsAndOrderOfTheLibraryList() {
Sort sort = UserGameRepository.querySort(Sort.by(Sort.Direction.DESC, "playtimeMinutes"));
for (long user : new long[]{USER_ID, OTHER_USER_ID}) {
for (int size : new int[]{8, 10, 20}) {
for (GameStatus status : new GameStatus[]{null, GameStatus.PLAYING}) {
PageRequest page = PageRequest.of(0, size, sort);
List<GameStatus> statuses = status == null ? List.copyOf(EnumSet.allOf(GameStatus.class)) : List.of(status);
List<Long> tile = userGameRepository.findListedByStatusIn(user, statuses, page).stream()
.map(UserGame::getAppid).toList();
List<Long> library = userGameRepository.findOwnedGames(user, false, status, null, null,
EnumSet.allOf(GameSource.class), page).stream().map(UserGame::getAppid).toList();
assertThat(tile).as("user=" + user + " size=" + size + " status=" + status).isEqualTo(library);
}
}
}
}
}
@@ -376,6 +376,22 @@ class GameLinkRepositoryTest extends AbstractPersistenceTest {
.extracting(p -> p.source() + ":" + p.platform()).containsExactly("MANUAL:PS5");
}
@Test
void entryPlatforms_AnotherUsersPlayStationRowsDoNotChangeMine() {
repository.link(100, -7);
ownOn(100, "STEAM", null);
jdbc.update("insert into user_psn_title (user_id, psn_key, state, appid) values (?, 'c1', 'STEAM_OWNED', 100)", USER);
jdbc.update("insert into user_game (user_id, appid, status, source, platform) values (?, 100, 'BACKLOG', 'STEAM', null)", OTHER_USER);
jdbc.update("insert into user_game (user_id, appid, status, source, platform) values (?, -7, 'BACKLOG', 'MANUAL', 'PS5')", OTHER_USER);
jdbc.update("insert into user_psn_title (user_id, psn_key, state, appid) values (?, 'c2', 'LINKED', -7)", OTHER_USER);
jdbc.update("update user_game set playtime_minutes = 500 where appid = 100");
assertThat(repository.entryPlatforms(USER, List.of(100L)).get(100L))
.extracting(p -> p.source() + ":" + p.platform()).containsExactly("STEAM:null", "PSN:null");
assertThat(repository.entryPlatforms(OTHER_USER, List.of(100L)).get(100L))
.extracting(p -> p.source() + ":" + p.platform()).containsExactly("STEAM:null", "MANUAL:PS5");
}
@Test
void editionCounts_GivesTwoForBothKeysAndNothingForASingleOwnedKey() {
repository.link(100, 200);
@@ -116,6 +116,33 @@ class AchievementSyncServiceTest extends AbstractUnitTest {
return saved.getValue();
}
@Test
void schemaStale_MissingOldAndFresh() {
when(schemaMetaRepository.findById(1L)).thenReturn(Optional.empty());
when(schemaMetaRepository.findById(2L)).thenReturn(Optional.of(
new GameAchievementSchemaMeta(2L, 5, true, Instant.now().minus(Duration.ofDays(31)))));
when(schemaMetaRepository.findById(3L)).thenReturn(Optional.of(schema(3L, 5)));
assertTrue(service.schemaStale(1L));
assertTrue(service.schemaStale(2L));
assertFalse(service.schemaStale(3L));
verifyNoInteractions(steamApiService);
}
@Test
void userStale_MissingOldAndFresh() {
Duration ttl = Duration.ofHours(1);
when(userAchievementMetaRepository.findByUserIdAndAppid(USER_ID, 1L)).thenReturn(Optional.empty());
when(userAchievementMetaRepository.findByUserIdAndAppid(USER_ID, 2L)).thenReturn(Optional.of(
new UserGameAchievementMeta(USER_ID, 2L, Instant.now().minus(Duration.ofHours(2)), true, 3, 60)));
when(userAchievementMetaRepository.findByUserIdAndAppid(USER_ID, 3L)).thenReturn(Optional.of(unlocks(3L, 3, 60)));
assertTrue(service.userStale(USER_ID, 1L, ttl));
assertTrue(service.userStale(USER_ID, 2L, ttl));
assertFalse(service.userStale(USER_ID, 3L, ttl));
verifyNoInteractions(steamApiService);
}
@Test
void syncLibrary_UnchangedPlaytime_AsksSteamNothing() {
library(List.of(game(620L, 120)), List.of(schema(620L, 10)), List.of(unlocks(620L, 3, 120)));
@@ -45,7 +45,13 @@ import java.util.Set;
import java.util.Map;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.anyLong;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.verifyNoInteractions;
import static org.mockito.Mockito.when;
/**
@@ -194,6 +200,57 @@ class GameDetailServiceTest extends AbstractMockedAdaptersTest {
assertEquals(5, userAchievementMetaRepository.findByUserIdAndAppid(USER_ID, 620L).orElseThrow().getAchievementsUnlocked());
assertEquals(5, detail.achievementsUnlocked());
assertFalse(detail.achievementsStale());
}
@Test
void getGameDetail_NoRefresh_StaleUserMeta_AsksSteamNothingAndSaysStale() {
schemaMetaRepository.save(new GameAchievementSchemaMeta(620L, 5, true, Instant.now()));
for (String name : List.of("A", "B", "C", "D", "E")) {
schemaRepository.save(new GameAchievementSchema(620L, name, name, null, null, null, false));
}
userAchievementMetaRepository.save(new UserGameAchievementMeta(USER_ID, 620L,
Instant.now().minus(Duration.ofHours(2)), true, 3, 60));
GameDetailDto detail = gameDetailService.getGameDetail(USER_ID, STEAM_ID, 620L, false).orElseThrow();
assertEquals(3, detail.achievementsUnlocked());
assertTrue(detail.achievementsStale());
verify(steamApiService, never()).getPlayerAchievements(anyString(), anyLong());
verify(steamApiService, never()).getSchemaForGame(anyLong());
}
@Test
void getGameDetail_NoRefresh_MissingSchema_AsksSteamNothingAndSaysStale() {
GameDetailDto detail = gameDetailService.getGameDetail(USER_ID, STEAM_ID, 620L, false).orElseThrow();
assertFalse(detail.hasAchievements());
assertTrue(detail.achievementsStale());
verify(steamApiService, never()).getSchemaForGame(anyLong());
verify(steamApiService, never()).getPlayerAchievements(anyString(), anyLong());
}
@Test
void getGameDetail_NoRefresh_FreshData_IsNotStale() {
schemaMetaRepository.save(new GameAchievementSchemaMeta(620L, 5, true, Instant.now()));
userAchievementMetaRepository.save(new UserGameAchievementMeta(USER_ID, 620L, Instant.now(), true, 3, 60));
GameDetailDto detail = gameDetailService.getGameDetail(USER_ID, STEAM_ID, 620L, false).orElseThrow();
assertFalse(detail.achievementsStale());
verifyNoInteractions(steamApiService);
}
@Test
void getGameDetail_NoRefresh_HiddenGame_IsNotStale() {
UserGame hidden = userGameRepository.findById(new UserGameId(USER_ID, 620L)).orElseThrow();
hidden.setHiddenAt(Instant.now());
userGameRepository.save(hidden);
GameDetailDto detail = gameDetailService.getGameDetail(USER_ID, STEAM_ID, 620L, false).orElseThrow();
assertFalse(detail.achievementsStale());
verifyNoInteractions(steamApiService);
}
@Test
@@ -8,9 +8,11 @@ import core_service.modules.library.repo.GameSpeedrunRepository;
import core_service.modules.library.repo.SteamGameRepository;
import core_service.modules.library.repo.UserGameRepository;
import core_service.modules.speedrun_adapter.service.SpeedrunApiService;
import core_service.modules.speedrun_adapter.service.SpeedrunThrottledException;
import core_service.shared.test.AbstractUnitTest;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.Timeout;
import org.mockito.InOrder;
import org.mockito.Mock;
import org.mockito.Spy;
@@ -18,14 +20,17 @@ import org.springframework.core.task.SyncTaskExecutor;
import org.springframework.core.task.TaskExecutor;
import org.springframework.dao.DataAccessResourceFailureException;
import java.time.Duration;
import java.time.Instant;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
import java.util.concurrent.TimeUnit;
import java.util.stream.LongStream;
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.eq;
@@ -59,14 +64,26 @@ class SpeedrunServiceTest extends AbstractUnitTest {
@Spy
private TaskExecutor speedrunExecutor = new SyncTaskExecutor();
private SpeedrunService service;
private final List<Duration> pauses = new ArrayList<>();
private PauseHook onPause = () -> {};
private interface PauseHook {
void run() throws InterruptedException;
}
private static final String LINK = "https://www.speedrun.com/prey2017";
@BeforeEach
void setUp() {
service = new SpeedrunService(speedrunApiService, igdbApiService, gameSpeedrunRepository, gameEditionRepository,
steamGameRepository, userGameRepository, syncFollowUpExecutor, speedrunExecutor);
when(speedrunApiService.isEnabled()).thenReturn(true);
steamGameRepository, userGameRepository, syncFollowUpExecutor, speedrunExecutor) {
@Override
void pause(Duration pause) throws InterruptedException {
pauses.add(pause);
onPause.run();
}
};
lenient().when(speedrunApiService.isEnabled()).thenReturn(true);
}
private void names(long... appids) {
@@ -282,6 +299,105 @@ class SpeedrunServiceTest extends AbstractUnitTest {
verify(speedrunApiService, times(SpeedrunService.MAX_ERRORS_IN_A_ROW)).findGames("Prey");
}
private static final Optional<List<SpeedrunApiService.Game>> NO_HITS = Optional.of(List.of());
@Test
void refresh_Throttled_PausesAndRetriesTheSameGame() {
names(620);
when(speedrunApiService.findGames("Prey")).thenThrow(new SpeedrunThrottledException(null)).thenReturn(NO_HITS);
service.refresh(List.of(620L));
assertThat(pauses).containsExactly(Duration.ofMinutes(1));
verify(speedrunApiService, times(2)).findGames("Prey");
verify(gameSpeedrunRepository).upsert(eq(620L), isNull(), isNull(), any());
}
@Test
void refresh_Throttled_KeepsTheQueue() {
names(1, 2, 3, 4, 5);
when(speedrunApiService.findGames("Prey")).thenThrow(new SpeedrunThrottledException(null),
new SpeedrunThrottledException(null), new SpeedrunThrottledException(null),
new SpeedrunThrottledException(null)).thenReturn(NO_HITS);
service.refresh(List.of(1L, 2L, 3L, 4L, 5L));
verify(gameSpeedrunRepository, times(5)).upsert(any(), isNull(), isNull(), any());
}
@Test
void refresh_ThrottledWithRetryAfter_PausesThatLong() {
names(620);
when(speedrunApiService.findGames("Prey")).thenThrow(new SpeedrunThrottledException(Duration.ofSeconds(90)))
.thenReturn(NO_HITS);
service.refresh(List.of(620L));
assertThat(pauses).containsExactly(Duration.ofSeconds(90));
}
@Test
void refresh_ThrottledAgain_PauseGrowsUpToTheCap() {
names(620);
SpeedrunThrottledException throttled = new SpeedrunThrottledException(null);
when(speedrunApiService.findGames("Prey")).thenThrow(throttled, throttled, throttled, throttled, throttled,
throttled, throttled).thenReturn(NO_HITS);
service.refresh(List.of(620L));
assertThat(pauses).containsExactly(Duration.ofMinutes(1), Duration.ofMinutes(2), Duration.ofMinutes(4),
Duration.ofMinutes(8), Duration.ofMinutes(16), Duration.ofMinutes(30), Duration.ofMinutes(30));
}
@Test
void refresh_StoredGame_ResetsTheBackoff() {
names(1, 2);
SpeedrunThrottledException throttled = new SpeedrunThrottledException(null);
when(speedrunApiService.findGames("Prey")).thenThrow(throttled).thenReturn(NO_HITS)
.thenThrow(throttled).thenReturn(NO_HITS);
service.refresh(List.of(1L, 2L));
assertThat(pauses).containsExactly(Duration.ofMinutes(1), Duration.ofMinutes(1));
}
@Test
void refresh_ThrottledReCheck_UserRequestQueuedDuringThePause_IsServedFirst() {
names(1, 2, 999);
when(gameSpeedrunRepository.findAppidsToRecheck(Instant.EPOCH)).thenReturn(List.of(1L, 2L));
when(speedrunApiService.findGames("Prey")).thenThrow(new SpeedrunThrottledException(null)).thenReturn(NO_HITS);
onPause = () -> service.refresh(List.of(999L));
service.recheckMarked();
InOrder order = inOrder(gameSpeedrunRepository);
order.verify(gameSpeedrunRepository).upsert(eq(999L), isNull(), isNull(), any());
order.verify(gameSpeedrunRepository).upsert(eq(1L), isNull(), isNull(), any());
order.verify(gameSpeedrunRepository).upsert(eq(2L), isNull(), isNull(), any());
}
@Test
@Timeout(value = 5, unit = TimeUnit.SECONDS, threadMode = Timeout.ThreadMode.SEPARATE_THREAD)
void refresh_InterruptedDuringPause_StopsAndKeepsInterruptFlag() {
names(1, 2, 999);
when(speedrunApiService.findGames("Prey")).thenThrow(new SpeedrunThrottledException(null));
onPause = () -> {
service.refresh(List.of(999L)); // a user asks during the pause
throw new InterruptedException();
};
service.refresh(List.of(1L, 2L));
assertThat(Thread.interrupted()).isTrue();
verify(speedrunApiService, times(1)).findGames("Prey");
}
@Test
void pauseFor_RetryAfterIsClampedToTheCap() {
assertThat(SpeedrunService.pauseFor(Duration.ofHours(2), 1)).isEqualTo(SpeedrunService.MAX_PAUSE);
assertThat(SpeedrunService.pauseFor(Duration.ZERO, 1)).isEqualTo(Duration.ofSeconds(1));
}
@Test
void refresh_CalledDuringARun_QueuesTheGameForTheRunningThread() {
names(620, 220);
@@ -4,18 +4,23 @@ import core_service.shared.test.AbstractUnitTest;
import io.github.resilience4j.ratelimiter.RateLimiterRegistry;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.http.HttpStatusCode;
import org.springframework.http.MediaType;
import org.springframework.test.web.client.MockRestServiceServer;
import org.springframework.web.client.RestClient;
import java.time.Duration;
import java.util.List;
import java.util.Optional;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.hamcrest.CoreMatchers.containsString;
import static org.springframework.test.web.client.match.MockRestRequestMatchers.queryParam;
import static org.springframework.test.web.client.match.MockRestRequestMatchers.requestTo;
import static org.springframework.test.web.client.response.MockRestResponseCreators.withServerError;
import static org.springframework.test.web.client.response.MockRestResponseCreators.withStatus;
import static org.springframework.test.web.client.response.MockRestResponseCreators.withTooManyRequests;
import static org.springframework.test.web.client.response.MockRestResponseCreators.withSuccess;
class SpeedrunApiServiceTest extends AbstractUnitTest {
@@ -66,6 +71,29 @@ class SpeedrunApiServiceTest extends AbstractUnitTest {
assertThat(service.findGames("Celeste")).isEmpty();
}
@Test
void findGames_TooManyRequests_ThrowsThrottledWithRetryAfter() {
server.expect(requestTo(containsString("/games"))).andRespond(withTooManyRequests(120));
assertThatThrownBy(() -> service.findGames("Celeste")).isInstanceOfSatisfying(SpeedrunThrottledException.class,
e -> assertThat(e.getRetryAfter()).isEqualTo(Duration.ofSeconds(120)));
}
@Test
void findGames_TooManyRequestsWithoutRetryAfter_ThrowsThrottledWithoutDelay() {
server.expect(requestTo(containsString("/games"))).andRespond(withTooManyRequests());
assertThatThrownBy(() -> service.findGames("Celeste")).isInstanceOfSatisfying(SpeedrunThrottledException.class,
e -> assertThat(e.getRetryAfter()).isNull());
}
@Test
void hasVerifiedRuns_Status420_ThrowsThrottled() {
server.expect(requestTo(containsString("/runs"))).andRespond(withStatus(HttpStatusCode.valueOf(420)));
assertThatThrownBy(() -> service.hasVerifiedRuns("o1y9j9v6")).isInstanceOf(SpeedrunThrottledException.class);
}
@Test
void findGames_Disabled_CallsNothing() {
service = create(false);