Compare commits

...
3 Commits
Author SHA1 Message Date
lobo 73cfcb58e2 refactor(api): one error body, token audiences, contract fixture
Build, Test & Push Core Service / docker (push) Successful in 1m23s
Build, Test & Push Core Service / quality-check (push) Successful in 1m7s
2026-09-29 14:48:28 +02:00
lobo 2360beeb81 fix(sync): short sync transactions, real Steam retries, no N+1 2026-09-29 13:27:31 +02:00
lobo 2a5fcc1c00 feat(import): import many games with background matching, review and undo 2026-09-29 10:20:54 +02:00
56 changed files with 2727 additions and 351 deletions

No files matched your search

+62 -7
View File
@@ -14,7 +14,11 @@ To Be Played is split into three repositories:
## Features
- **Steam sync**: owned games, playtime and achievements through the Steam Web API. Manual sync plus a nightly
sync. Resilience4j retries and rate-limits every Steam call; Caffeine caches game counts and player summaries.
sync. Resilience4j rate-limits every Steam call and tries the owned games up to three times (never again on a 4xx;
the wishlist and the calls a page waits for once); Caffeine caches game counts and player summaries. A sync commits its claim (`RUNNING`, with
`sync_started_at`), its writes and its finish apart, and calls Steam and IGDB outside any transaction: a second
sync sees `RUNNING` and skips, a failed one stores `FAILED`. A run older than `app.library.running-stale-minutes`
can be taken over; its own finish then changes nothing.
- **Backlog**: status (`BACKLOG`, `PLAYING`, `FINISHED`, `COMPLETED` = 100 %, `DROPPED`), rating 1–5, notes, priority,
filters and sorting. "Pile of Shame" summary with a random suggestion of what to play.
- **Picker**: up to 10 random backlog games for a time budget (IGDB normal time to beat) and a mood, HIGH priority
@@ -73,14 +77,20 @@ To Be Played is split into three repositories:
Core never issues tokens. It verifies Auth's EdDSA (Ed25519) JWTs with public keys only. It loads them from Auth's
JWKS by the token's `kid` and loads again when a new `kid` shows up (at most every 10 seconds). Loaded keys stay
until Core restarts. `JWT_PUBLIC_KEY` is used only until the first JWKS load works. The token comes from Auth's
`tbp_access` cookie or a `Bearer` header, whichever verifies first. Core reads `userId`, `steamId` and the active
`role` (missing or unknown = USER) from the claims and grants that role and every lower one
(`USER < AUDITOR < PLATFORM_ADMIN`).
`tbp_access` cookie or a `Bearer` header, whichever verifies first. Every token names its kind in `aud`
(`tbp-login`, `tbp-public-profile`, `tbp-internal`, with `iss=tbp-auth`); a login's `sub` is the user id. Core reads
`steamId` and the active `role` (missing or unknown = USER) from the claims and grants that role and every lower one
(`USER < AUDITOR < PLATFORM_ADMIN`). A token without `aud` comes from an Auth from before item 26 (kind in `typ`,
user id in `userId`) and is still accepted; item 29 removes that path. The claims are pinned in
`src/test/resources/contract/auth-core-contract.json`, the same file as in Auth; the tests build tokens only from it.
`POST`, `PUT` and `DELETE` need an `X-TBP-Client` or `Authorization` header (CSRF guard), and CORS allows only the
frontend origin.
A token with a `typ` claim (for example a profile ticket) is rejected on normal endpoints; the profile ticket is
A token for another audience (for example a profile ticket) is rejected on normal endpoints; the profile ticket is
accepted only by `GET /api/public/profiles/library` in the `X-Profile-Ticket` header.
`/internal/**` has its own filter chain: it accepts only a service token of Auth (`typ=service`, with `exp`, sent
Every error answer of a controller has the body `{code, message}` (a 401 or 403 of the security filters has none); a refused request is `400 VALIDATION_FAILED` with
`fields: [{field, message}]`, a duplicate key `409 CONFLICT`, anything unexpected `500 INTERNAL_ERROR` without details
(`shared/api/GlobalExceptionController`).
`/internal/**` has its own filter chain: it accepts only a service token of Auth (`aud=tbp-internal`, with `exp`, sent
as `Bearer`), never a login, also not in the dev profile. Traefik exposes the path, so the signed token is its
only guard.
@@ -103,6 +113,11 @@ method calling another service's method that needs more.
| `PUT` | `/api/library/games/{appid}/status` (status only, keeps rating, notes, priority) | JWT |
| `PUT` | `/api/library/games/{appid}/finished-hint` (`{dismissed: ENDING\|PLAYTIME}`, "Not yet"; `204`) | JWT |
| `PUT` | `/api/library/games/{appid}/hidden` (`{hidden}`; `409 MANUAL_GAME` for a game added by hand) | JWT |
| `POST` | `/api/library/imports` (`{source: TEXT\|CSV\|SEARCH, rows: [{title, platform, playtimeMinutes, status, igdbId}]}`, 1–1000 rows; `202 {id}` in `QUEUED`, `409 IMPORT_RUNNING`, `503 IGDB_OFF`) | JWT |
| `GET` | `/api/library/imports` (`{imports}`, the last 20), `/api/library/imports/{id}` (status, progress counts, `ahead` while it waits in a queue, `attempts`; the rows with their candidates while `READY`) | JWT |
| `POST` | `/api/library/imports/{id}/save` (`{rows: [{lineNo, igdbId, platform, status, playtimeMinutes}]}`; `202`, then poll until `SAVED`; `409 IMPORT_NOT_READY`) | JWT |
| `POST` | `/api/library/imports/{id}/retry` (a `FAILED` import; `202`, `409 IMPORT_NOT_FAILED` / `IMPORT_RUNNING`) | JWT |
| `GET`, `DELETE` | `/api/library/imports/{id}/undo-check` (`{games, changed}`), `/api/library/imports/{id}/games` (undo; `204`, `409 IMPORT_NOT_SAVED`) | JWT |
| `POST` | `/api/library/sync` | JWT |
| `POST` | `/api/library/broken-images` (`{appid, kind}`, kind `COVER` or `ACHIEVEMENT_ICONS`; `204`) | JWT |
| `GET` | `/api/library/picker?maxMinutes=&mood=&count=&exclude=` | JWT |
@@ -163,7 +178,7 @@ flowchart LR
| Module | Content |
|---|---|
| `library` | Sync, backlog, manual games, playtime history, wishlist, achievements, public profile data |
| `library` | Sync, backlog, manual games, imports, playtime history, wishlist, achievements, public profile data |
| `stats` | Public numbers for the landing page |
| `version` | `GET /api/version`: commit and build time of the running image |
| `account` | Deleted accounts, driven by Auth: tombstones in `deleted_user`, purge and orphan prune of the per-user rows |
@@ -265,6 +280,46 @@ id for negative keys. Search and add check both keys of a game. A game added und
a second row when IGDB learns its Steam appid later and the user owns it on Steam; an auditor then merges the two
(see "Merging duplicates").
### Imports
Many games at once (V22, `GameImportService`): a list, a CSV or several picks of the game search become one
`game_import` with a `game_import_row` per line. `game_import` is also the queue of `GameImportWorker`, which has
two queues, each on one thread for all users:
- **matching**: IGDB searches with typo help. One thread, so an import never takes the whole IGDB budget from the
interactive search; other users' searches may wait for the limiter meanwhile. Up to 5 candidates per line are
stored whole (JSONB), so the save needs no IGDB call, and `exact` marks a best one with the line's name.
- **saving**: adds the chosen lines the way the manual add does (same keys and duplicate check) with
`user_game.import_id` in the same transaction, then loads their time to beat and moods. A save never waits
behind another user's big matching.
Each line is stored when done, so a stopped try goes on where it stopped. A worker sets `progress_at` when it takes
an import and after every line. A watchdog (every minute) stops a try without progress for 5 minutes
(`app.library.import-stall-minutes`): the import goes to the end of its queue and the queue gets a new thread. A
restart counts as a stopped try too. After 3 tries the import is `FAILED`; the user can try it again. One import
per user can be in the matching queue (a unique partial index).
```mermaid
stateDiagram-v2
[*] --> QUEUED: POST /imports
QUEUED --> MATCHING: matching worker takes it
MATCHING --> READY: every line matched
READY --> SAVING: POST /save (the review)
SAVING --> SAVED: games added, time to beat and moods loaded
SAVED --> UNDONE: DELETE /games (undo)
MATCHING --> QUEUED: no progress for 5 min or restart
SAVING --> SAVING: no progress for 5 min or restart (end of the queue)
MATCHING --> FAILED: 3rd stopped try
SAVING --> FAILED: 3rd stopped try
FAILED --> QUEUED: POST /retry (stopped while matching)
FAILED --> SAVING: POST /retry (stopped while saving)
FAILED --> UNDONE: DELETE /games (the games added before it failed)
```
The undo deletes the games of the import that are still `MANUAL` (a game the Steam sync took over stays). A removed
row the import brings back gets no `import_id`: it holds rating, notes and history from before, so the undo keeps it;
`undo-check` counts those with a rating, notes, priority or a status change after the save.
## Run locally
### Local development without keys (profile `local`)
@@ -65,6 +65,6 @@ public class DevJwtAuthenticationFilter extends OncePerRequestFilter {
private UserPrincipal dummyPrincipal() {
log.trace("Dev mode: no JWT present, injecting dummy principal");
return new UserPrincipal("1", "dev@example.com", 76561198000000001L);
return new UserPrincipal("1", 76561198000000001L);
}
}
@@ -10,17 +10,26 @@ import io.jsonwebtoken.Jwts;
import org.springframework.stereotype.Service;
import java.util.List;
import java.util.Objects;
import java.util.Optional;
import java.util.Set;
import java.util.stream.Collectors;
/**
* Verifies JWTs issued by tbp-auth-service. Verify-only: {@link JwksKeyLocator} finds the Ed25519 public key.
* <p>
* A token names its kind in {@code aud}, and {@code sub} is the user id of a login. A token without {@code aud} comes
* from an Auth from before item 26: its kind is in {@code typ} and the user id in {@code userId}. Item 29 removes
* that path.
*/
@Service
public class JwtService implements ProfileTicketVerifier {
// Only non-login tokens carry this claim; parseLoginToken rejects them.
static final String ISSUER = "tbp-auth";
static final String LOGIN_AUDIENCE = "tbp-login";
static final String PROFILE_TICKET_AUDIENCE = "tbp-public-profile";
static final String SERVICE_AUDIENCE = "tbp-internal";
// The kind of a token before item 26; item 29 removes it.
private static final String TYPE_CLAIM = "typ";
private static final String PROFILE_TICKET_TYPE = "public-profile";
private static final String SERVICE_TOKEN_TYPE = "service";
@@ -36,6 +45,20 @@ public class JwtService implements ProfileTicketVerifier {
return parser.parseSignedClaims(token).getPayload();
}
private static boolean hasAudience(Claims claims) {
return claims.getAudience() != null && !claims.getAudience().isEmpty();
}
/**
* @param legacyType the {@code typ} of this kind before item 26, null for a login
*/
private static boolean isKind(Claims claims, String audience, String legacyType) {
if (hasAudience(claims)) {
return claims.getAudience().contains(audience) && ISSUER.equals(claims.getIssuer());
}
return Objects.equals(legacyType, claims.get(TYPE_CLAIM, String.class));
}
/**
* The user of a login token with a valid signature. Empty for anything else, also for a profile ticket: it is
* signed with the same key, but is no login.
@@ -43,11 +66,15 @@ public class JwtService implements ProfileTicketVerifier {
public Optional<UserPrincipal> parseLoginToken(String token) {
try {
Claims claims = claims(token);
String userId = claims.get("userId", String.class);
if (claims.get(TYPE_CLAIM) != null || claims.getSubject() == null || userId == null) {
if (!isKind(claims, LOGIN_AUDIENCE, null)) {
return Optional.empty();
}
return Optional.of(new UserPrincipal(userId, claims.getSubject(), claims.get("steamId", Long.class),
String userId = hasAudience(claims) ? claims.getSubject()
: claims.getSubject() == null ? null : claims.get("userId", String.class);
if (userId == null) {
return Optional.empty();
}
return Optional.of(new UserPrincipal(userId, claims.get("steamId", Long.class),
Role.parse(claims.get(ROLE_CLAIM) instanceof String role ? role : null)));
} catch (Exception e) {
return Optional.empty();
@@ -60,7 +87,7 @@ public class JwtService implements ProfileTicketVerifier {
public boolean isServiceToken(String token) {
try {
Claims claims = claims(token);
return SERVICE_TOKEN_TYPE.equals(claims.get(TYPE_CLAIM, String.class)) && claims.getExpiration() != null;
return isKind(claims, SERVICE_AUDIENCE, SERVICE_TOKEN_TYPE) && claims.getExpiration() != null;
} catch (Exception e) {
return false;
}
@@ -74,7 +101,7 @@ public class JwtService implements ProfileTicketVerifier {
public Optional<ProfileTicket> verifyProfileTicket(String ticket) {
try {
Claims claims = claims(ticket);
if (!PROFILE_TICKET_TYPE.equals(claims.get(TYPE_CLAIM, String.class))) {
if (!isKind(claims, PROFILE_TICKET_AUDIENCE, PROFILE_TICKET_TYPE)) {
return Optional.empty();
}
long userId = Long.parseLong(claims.get("profileUserId", String.class));
@@ -19,13 +19,13 @@ import java.util.Map;
public class AccountCleanupRepository {
// Children before user_game (user_game_status_change would also go by its cascade), the tombstones last.
private static final List<String> TABLES = List.of("user_game_status_change", "user_game_achievement",
static final List<String> TABLES = List.of("user_game_status_change", "user_game_achievement",
"user_game_achievement_meta", "user_game_playtime_day", "wishlist_game", "backlog_snapshot",
"user_steam_sync", "user_game", "deleted_user");
"user_steam_sync", "user_game", "game_import", "deleted_user");
// Every user with data has a row in one of these; deleteRows then removes the child rows too.
private static final List<String> ORPHAN_SOURCES = List.of("user_game", "user_steam_sync", "wishlist_game",
"deleted_user");
"game_import", "deleted_user");
private final NamedParameterJdbcTemplate jdbc;
@@ -0,0 +1,100 @@
package core_service.modules.library.api;
import core_service.shared.api.ErrorResponse;
import core_service.modules.library.api.dto.ImportHistoryDto;
import core_service.modules.library.api.dto.ImportRequest;
import core_service.modules.library.api.dto.ImportSaveRequest;
import core_service.modules.library.service.GameImportService;
import core_service.shared.security.UserPrincipal;
import jakarta.validation.Valid;
import lombok.RequiredArgsConstructor;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.security.core.annotation.AuthenticationPrincipal;
import org.springframework.web.bind.annotation.DeleteMapping;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import java.util.Map;
/**
* Imports of many games at once, see GameImport.
*/
@RestController
@RequestMapping("/api/library/imports")
@RequiredArgsConstructor
public class GameImportController {
private static final ErrorResponse NOT_FOUND = new ErrorResponse("IMPORT_NOT_FOUND", "No such import");
private static final ResponseEntity<?> IMPORT_RUNNING = ResponseEntity.status(HttpStatus.CONFLICT)
.body(new ErrorResponse("IMPORT_RUNNING", "Another import is still being matched"));
private final GameImportService importService;
@PostMapping
public ResponseEntity<?> create(@AuthenticationPrincipal UserPrincipal principal,
@Valid @RequestBody ImportRequest request) {
return switch (importService.create(principal.getIdAsLong(), request.source(), request.rows())) {
case GameImportService.CreateResult.Created created -> ResponseEntity.status(HttpStatus.ACCEPTED)
.body(Map.of("id", created.id()));
case GameImportService.CreateResult.AlreadyMatching ignored -> IMPORT_RUNNING;
case GameImportService.CreateResult.SearchUnavailable ignored -> ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE)
.body(new ErrorResponse("IGDB_OFF", "Importing games needs the IGDB app"));
};
}
@GetMapping
public ImportHistoryDto list(@AuthenticationPrincipal UserPrincipal principal) {
return new ImportHistoryDto(importService.list(principal.getIdAsLong()));
}
@GetMapping("/{id}")
public ResponseEntity<?> get(@AuthenticationPrincipal UserPrincipal principal, @PathVariable("id") long id) {
return importService.get(principal.getIdAsLong(), id)
.<ResponseEntity<?>>map(ResponseEntity::ok)
.orElseGet(() -> ResponseEntity.status(HttpStatus.NOT_FOUND).body(NOT_FOUND));
}
@PostMapping("/{id}/save")
public ResponseEntity<?> save(@AuthenticationPrincipal UserPrincipal principal, @PathVariable("id") long id,
@Valid @RequestBody ImportSaveRequest request) {
return switch (importService.save(principal.getIdAsLong(), id, request.rows())) {
case DONE -> ResponseEntity.accepted().build();
case NOT_FOUND -> ResponseEntity.status(HttpStatus.NOT_FOUND).body(NOT_FOUND);
case WRONG_STATUS, ALREADY_MATCHING -> ResponseEntity.status(HttpStatus.CONFLICT)
.body(new ErrorResponse("IMPORT_NOT_READY", "The import is still matching or saved already"));
};
}
@PostMapping("/{id}/retry")
public ResponseEntity<?> retry(@AuthenticationPrincipal UserPrincipal principal, @PathVariable("id") long id) {
return switch (importService.retry(principal.getIdAsLong(), id)) {
case DONE -> ResponseEntity.accepted().build();
case NOT_FOUND -> ResponseEntity.status(HttpStatus.NOT_FOUND).body(NOT_FOUND);
case WRONG_STATUS -> ResponseEntity.status(HttpStatus.CONFLICT)
.body(new ErrorResponse("IMPORT_NOT_FAILED", "Only a failed import can be tried again"));
case ALREADY_MATCHING -> IMPORT_RUNNING;
};
}
@GetMapping("/{id}/undo-check")
public ResponseEntity<?> undoCheck(@AuthenticationPrincipal UserPrincipal principal, @PathVariable("id") long id) {
return importService.undoCheck(principal.getIdAsLong(), id)
.<ResponseEntity<?>>map(ResponseEntity::ok)
.orElseGet(() -> ResponseEntity.status(HttpStatus.NOT_FOUND).body(NOT_FOUND));
}
@DeleteMapping("/{id}/games")
public ResponseEntity<?> undo(@AuthenticationPrincipal UserPrincipal principal, @PathVariable("id") long id) {
return switch (importService.undo(principal.getIdAsLong(), id)) {
case DONE -> ResponseEntity.noContent().build();
case NOT_FOUND -> ResponseEntity.status(HttpStatus.NOT_FOUND).body(NOT_FOUND);
case WRONG_STATUS, ALREADY_MATCHING -> ResponseEntity.status(HttpStatus.CONFLICT)
.body(new ErrorResponse("IMPORT_NOT_SAVED", "Only a saved import can be undone"));
};
}
}
@@ -1,5 +1,6 @@
package core_service.modules.library.api;
import core_service.shared.api.ErrorResponse;
import core_service.modules.library.api.dto.AchievementSummaryDto;
import core_service.modules.library.api.dto.ActivityDto;
import core_service.modules.library.api.dto.BacklogEntryDto;
@@ -62,6 +63,7 @@ import java.util.Set;
@RequiredArgsConstructor
public class LibraryController {
private static final ErrorResponse GAME_NOT_FOUND = new ErrorResponse("GAME_NOT_FOUND", "Game not in user library");
private static final Set<Integer> ALLOWED_SIZES = Set.of(20, 40, 80);
private static final int DEFAULT_SIZE = 20;
private static final int MIN_ACTIVITY_DAYS = 7;
@@ -129,7 +131,7 @@ public class LibraryController {
@PostMapping("/sync")
public ResponseEntity<?> forceSync(@AuthenticationPrincipal UserPrincipal principal) {
if (principal.steamId() == null) {
return ResponseEntity.badRequest().body(Map.of("code", "NO_STEAM_ID", "message", "User has no Steam ID"));
return ResponseEntity.badRequest().body(new ErrorResponse("NO_STEAM_ID", "User has no Steam ID"));
}
try {
syncService.forceSync(principal.getIdAsLong(), String.valueOf(principal.steamId()));
@@ -152,7 +154,7 @@ public class LibraryController {
return gameDetailService.getGameDetail(userId, steamId, appid)
.<ResponseEntity<?>>map(ResponseEntity::ok)
.orElseGet(() -> ResponseEntity.status(HttpStatus.NOT_FOUND)
.body(Map.of("code", "GAME_NOT_FOUND", "message", "Game not in user library")));
.body(GAME_NOT_FOUND));
}
/**
@@ -201,9 +203,9 @@ public class LibraryController {
case GameMergeService.Result.Merged merged -> ResponseEntity.ok(new MergedGameDto(merged.appid()));
case GameMergeService.Result.NotFound _ -> unknownGame();
case GameMergeService.Result.SameGame _ -> ResponseEntity.badRequest()
.body(Map.of("code", "SAME_GAME", "message", "A game cannot be merged with itself"));
.body(new ErrorResponse("SAME_GAME", "A game cannot be merged with itself"));
case GameMergeService.Result.SteamGames _ -> ResponseEntity.badRequest()
.body(Map.of("code", "STEAM_GAMES", "message", "Two Steam games cannot be merged"));
.body(new ErrorResponse("STEAM_GAMES", "Two Steam games cannot be merged"));
};
}
@@ -212,11 +214,11 @@ public class LibraryController {
}
private static ResponseEntity<?> unknownGame() {
return ResponseEntity.status(HttpStatus.NOT_FOUND).body(Map.of("code", "GAME_NOT_FOUND", "message", "Unknown game"));
return ResponseEntity.status(HttpStatus.NOT_FOUND).body(new ErrorResponse("GAME_NOT_FOUND", "Unknown game"));
}
private static ResponseEntity<?> invalidCorrection(String message) {
return ResponseEntity.badRequest().body(Map.of("code", "INVALID_CORRECTION", "message", message));
return ResponseEntity.badRequest().body(new ErrorResponse("INVALID_CORRECTION", message));
}
@PutMapping("/games/{appid}/backlog")
@@ -226,7 +228,7 @@ public class LibraryController {
return backlogService.update(principal.getIdAsLong(), appid, entry)
.<ResponseEntity<?>>map(ResponseEntity::ok)
.orElseGet(() -> ResponseEntity.status(HttpStatus.NOT_FOUND)
.body(Map.of("code", "GAME_NOT_FOUND", "message", "Game not in user library")));
.body(GAME_NOT_FOUND));
}
@PutMapping("/games/{appid}/hidden")
@@ -236,9 +238,9 @@ public class LibraryController {
return switch (backlogService.setHidden(principal.getIdAsLong(), appid, update.hidden())) {
case DONE -> ResponseEntity.ok(update);
case NOT_FOUND -> ResponseEntity.status(HttpStatus.NOT_FOUND)
.body(Map.of("code", "GAME_NOT_FOUND", "message", "Game not in user library"));
.body(GAME_NOT_FOUND);
case MANUAL_GAME -> ResponseEntity.status(HttpStatus.CONFLICT)
.body(Map.of("code", "MANUAL_GAME", "message", "A game added by hand can be deleted, not hidden"));
.body(new ErrorResponse("MANUAL_GAME", "A game added by hand can be deleted, not hidden"));
};
}
@@ -248,7 +250,7 @@ public class LibraryController {
@Valid @RequestBody FinishedHintDismissal dismissal) {
if (!backlogService.dismissFinishedHint(principal.getIdAsLong(), appid, dismissal.dismissed())) {
return ResponseEntity.status(HttpStatus.NOT_FOUND)
.body(Map.of("code", "GAME_NOT_FOUND", "message", "Game not in user library"));
.body(GAME_NOT_FOUND);
}
return ResponseEntity.noContent().build();
}
@@ -260,7 +262,7 @@ public class LibraryController {
return backlogService.updateStatus(principal.getIdAsLong(), appid, update.status())
.<ResponseEntity<?>>map(ResponseEntity::ok)
.orElseGet(() -> ResponseEntity.status(HttpStatus.NOT_FOUND)
.body(Map.of("code", "GAME_NOT_FOUND", "message", "Game not in user library")));
.body(GAME_NOT_FOUND));
}
@GetMapping("/activity")
@@ -1,5 +1,6 @@
package core_service.modules.library.api;
import core_service.shared.api.ErrorResponse;
import core_service.modules.library.api.dto.AddGameRequest;
import core_service.modules.library.api.dto.AddedGameDto;
import core_service.modules.library.api.dto.GameSearchDto;
@@ -23,7 +24,6 @@ import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import java.net.URI;
import java.util.Map;
/**
* Games added by hand from an IGDB search, next to the Steam library.
@@ -33,9 +33,8 @@ import java.util.Map;
@RequiredArgsConstructor
public class ManualGameController {
private static final Map<String, String> NOT_FOUND = Map.of("code", "GAME_NOT_FOUND", "message", "Game not in user library");
private static final Map<String, String> STEAM_GAME = Map.of("code", "STEAM_GAME",
"message", "Steam games come from the sync and cannot be changed here");
private static final ErrorResponse NOT_FOUND = new ErrorResponse("GAME_NOT_FOUND", "Game not in user library");
private static final ErrorResponse STEAM_GAME = new ErrorResponse("STEAM_GAME", "Steam games come from the sync and cannot be changed here");
private final ManualGameService manualGameService;
@@ -45,24 +44,24 @@ public class ManualGameController {
return manualGameService.search(principal.getIdAsLong(), query)
.<ResponseEntity<?>>map(ResponseEntity::ok)
.orElseGet(() -> ResponseEntity.status(HttpStatus.BAD_GATEWAY)
.body(Map.of("code", "IGDB_FAILED", "message", "IGDB did not answer")));
.body(new ErrorResponse("IGDB_FAILED", "IGDB did not answer")));
}
@PostMapping("/games")
public ResponseEntity<?> add(@AuthenticationPrincipal UserPrincipal principal,
@Valid @RequestBody AddGameRequest request) {
int minutes = request.playtimeMinutes() != null ? request.playtimeMinutes() : 0;
return switch (manualGameService.add(principal.getIdAsLong(), request.igdbId(), request.platform().trim(),
return switch (manualGameService.add(principal.getIdAsLong(), request.igdbId(), request.platform(),
request.status(), minutes)) {
case ManualGameService.AddResult.Added added -> ResponseEntity
.created(URI.create("/api/library/games/" + added.appid()))
.body(new AddedGameDto(added.appid()));
case ManualGameService.AddResult.AlreadyInLibrary ignored -> ResponseEntity.status(HttpStatus.CONFLICT)
.body(Map.of("code", "ALREADY_IN_LIBRARY", "message", "The game is already in the library"));
.body(new ErrorResponse("ALREADY_IN_LIBRARY", "The game is already in the library"));
case ManualGameService.AddResult.UnknownGame ignored -> ResponseEntity.status(HttpStatus.BAD_REQUEST)
.body(Map.of("code", "UNKNOWN_GAME", "message", "IGDB has no such game or did not answer"));
.body(new ErrorResponse("UNKNOWN_GAME", "IGDB has no such game or did not answer"));
case ManualGameService.AddResult.SearchUnavailable ignored -> ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE)
.body(Map.of("code", "IGDB_OFF", "message", "Adding games needs the IGDB app"));
.body(new ErrorResponse("IGDB_OFF", "Adding games needs the IGDB app"));
};
}
@@ -1,5 +1,6 @@
package core_service.modules.library.api;
import core_service.shared.api.ErrorResponse;
import core_service.modules.library.service.LibraryQueryService;
import core_service.shared.security.ProfileTicket;
import core_service.shared.security.ProfileTicketVerifier;
@@ -11,7 +12,6 @@ import org.springframework.web.bind.annotation.RequestHeader;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import java.util.Map;
import java.util.Optional;
/**
@@ -31,6 +31,6 @@ public class PublicProfileController {
return (ticket == null ? Optional.<ProfileTicket>empty() : profileTicketVerifier.verifyProfileTicket(ticket))
.<ResponseEntity<?>>map(t -> ResponseEntity.ok(queryService.getPublicLibrary(t.userId(), t.profileStats(), t.sections())))
.orElseGet(() -> ResponseEntity.status(HttpStatus.UNAUTHORIZED)
.body(Map.of("code", "INVALID_TICKET", "message", "Missing, invalid or expired profile ticket")));
.body(new ErrorResponse("INVALID_TICKET", "Missing, invalid or expired profile ticket")));
}
}
@@ -0,0 +1,32 @@
package core_service.modules.library.api.dto;
import core_service.modules.library.domain.GameImport;
import core_service.modules.library.domain.GameStatus;
import java.time.Instant;
import java.util.List;
/**
* An import and its progress, see {@link GameImport}.
*
* @param ahead while it waits in a queue (QUEUED, or SAVING before its save started): the imports of any user before
* it; else null
* @param attempts tries a watchdog or a restart stopped; FAILED after 3
* @param rows only for an import in READY that was asked for by its id; else null
*/
public record GameImportDto(long id, GameImport.Source source, GameImport.Status status, int total, int matched,
int chosen, int saved, int added, Integer ahead, GameImport.Step step, int attempts,
Instant createdAt, Instant savedAt, List<Row> rows) {
/**
* @param exact the first candidate has the line's name; see the review's preselection
*/
public record Row(int lineNo, String title, String platform, Integer playtimeMinutes, GameStatus status,
boolean searchFailed, boolean exact, List<Candidate> candidates) {}
/**
* @param inLibrary the user has this game already, from Steam or by hand
*/
public record Candidate(long igdbId, String name, Integer year, String imageUrl, List<String> platforms,
boolean inLibrary) {}
}
@@ -0,0 +1,8 @@
package core_service.modules.library.api.dto;
import java.util.List;
/**
* @param imports the user's last imports, newest first
*/
public record ImportHistoryDto(List<GameImportDto> imports) {}
@@ -0,0 +1,38 @@
package core_service.modules.library.api.dto;
import core_service.modules.library.domain.GameImport;
import core_service.modules.library.domain.GameStatus;
import jakarta.validation.Valid;
import jakarta.validation.constraints.AssertTrue;
import jakarta.validation.constraints.Max;
import jakarta.validation.constraints.Min;
import jakarta.validation.constraints.NotEmpty;
import jakarta.validation.constraints.NotNull;
import jakarta.validation.constraints.Size;
import java.util.List;
public record ImportRequest(
@NotNull GameImport.Source source,
@NotEmpty @Size(max = MAX_ROWS) List<@NotNull @Valid Row> rows
) {
public static final int MAX_ROWS = 1000;
/**
* @param igdbId the game already picked in the search; then the title is not searched
*/
public record Row(
@Size(max = 100) String title,
@Size(max = ManualGameUpdate.MAX_PLATFORM_LENGTH) String platform,
@Min(0) @Max(ManualGameUpdate.MAX_MINUTES) Integer playtimeMinutes,
GameStatus status,
@Min(1) Long igdbId
) {
@AssertTrue(message = "title or igdbId is needed")
public boolean isTitleOrIgdbId() {
return igdbId != null || (title != null && !title.isBlank());
}
}
}
@@ -0,0 +1,28 @@
package core_service.modules.library.api.dto;
import core_service.modules.library.domain.GameStatus;
import jakarta.validation.Valid;
import jakarta.validation.constraints.Max;
import jakarta.validation.constraints.Min;
import jakarta.validation.constraints.NotBlank;
import jakarta.validation.constraints.NotNull;
import jakarta.validation.constraints.Size;
import java.util.List;
/**
* The lines the user checked in the review, each with the game picked.
*/
public record ImportSaveRequest(@NotNull @Size(max = ImportRequest.MAX_ROWS) List<@NotNull @Valid Row> rows) {
/**
* @param igdbId a candidate of the line, or a game from a new search
*/
public record Row(
@Min(1) int lineNo,
@NotNull @Min(1) Long igdbId,
@NotBlank @Size(max = ManualGameUpdate.MAX_PLATFORM_LENGTH) String platform,
@NotNull GameStatus status,
@Min(0) @Max(ManualGameUpdate.MAX_MINUTES) Integer playtimeMinutes
) {}
}
@@ -0,0 +1,7 @@
package core_service.modules.library.api.dto;
/**
* @param games the import's games an undo would remove
* @param changed of these, the games with a rating, notes, priority or a new status since the save
*/
public record UndoCheckDto(int games, int changed) {}
@@ -0,0 +1,55 @@
package core_service.modules.library.domain;
import core_service.modules.igdb_adapter.service.IgdbApiService;
import java.time.Instant;
import java.util.List;
/**
* Many games at once (V22): the lines of a list or a CSV, matched with IGDB in the background, reviewed by the user
* and then saved in the background as games added by hand. The games keep the import's id, so the import can be
* undone.
*
* @param matched lines matched so far, of total
* @param chosen lines the user checked in the review
* @param saved of chosen, the lines the save has done
* @param added of saved, the lines that became a game in the library
* @param running a worker runs the import now; false while it waits in its queue
*/
public record GameImport(long id, long userId, Source source, Status status, int total, int matched, int chosen,
int saved, int added, boolean running, int attempts, Step step, Instant createdAt,
Instant queuedAt, Instant savedAt) {
public enum Source { TEXT, CSV, SEARCH }
public enum Status { QUEUED, MATCHING, READY, SAVING, SAVED, UNDONE, FAILED }
/**
* Of a save: null while the games are added, then DETAILS while their time to beat and moods load.
*/
public enum Step { DETAILS }
public enum SaveResult { ADDED, SKIPPED }
/**
* One input line; every field but lineNo may be missing.
*
* @param igdbId the game the user already picked in the search (source SEARCH); then title is not searched
*/
public record Line(int lineNo, String title, String platform, Integer playtimeMinutes, GameStatus status,
Long igdbId) {}
/**
* A line after the matching.
*
* @param searchFailed IGDB did not answer for this line
* @param exact the first candidate has the line's name (case and punctuation ignored)
* @param candidates best first, at most five
*/
public record Row(Line line, boolean searchFailed, boolean exact, List<IgdbApiService.Game> candidates) {}
/**
* The user's pick for a line in the review.
*/
public record Choice(int lineNo, long igdbId, String platform, GameStatus status, int playtimeMinutes) {}
}
@@ -87,6 +87,10 @@ public class UserGame {
@Column(name = "platform", length = 32)
private String platform;
/** The import that added this game (V22), so the import can be undone. Null for any other add. */
@Column(name = "import_id")
private Long importId;
public UserGame(Long userId, Long appid, int playtimeMinutes, Instant firstSeenAt, Instant lastSeenAt) {
this.userId = userId;
this.appid = appid;
@@ -36,6 +36,9 @@ public class UserSteamSync {
@Column(name = "last_sync_status", length = 16)
private Status lastSyncStatus;
@Column(name = "sync_started_at")
private Instant syncStartedAt;
@Column(name = "history_steam_id", length = 32)
private String historySteamId;
@@ -0,0 +1,283 @@
package core_service.modules.library.repo;
import core_service.modules.igdb_adapter.service.IgdbApiService;
import core_service.modules.library.domain.GameImport;
import core_service.modules.library.domain.GameStatus;
import lombok.RequiredArgsConstructor;
import org.springframework.jdbc.core.RowMapper;
import org.springframework.jdbc.core.namedparam.MapSqlParameterSource;
import org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate;
import org.springframework.jdbc.support.GeneratedKeyHolder;
import org.springframework.stereotype.Repository;
import tools.jackson.databind.json.JsonMapper;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Timestamp;
import java.time.Instant;
import java.util.Collection;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
/**
* Imports and their lines (V22), the queues of the import workers, and the games an import added.
*/
@Repository
@RequiredArgsConstructor
public class GameImportRepository {
private static final String SELECT_IMPORT = """
select i.*, r.* from game_import i cross join lateral (
select count(*) filter (where matched_at is not null) as matched,
count(*) filter (where chosen_igdb_id is not null) as chosen,
count(*) filter (where save_result is not null) as saved,
count(*) filter (where save_result = 'ADDED') as added
from game_import_row where import_id = i.id) r
""";
private static final RowMapper<GameImport> IMPORT = (rs, row) -> new GameImport(rs.getLong("id"),
rs.getLong("user_id"), GameImport.Source.valueOf(rs.getString("source")),
GameImport.Status.valueOf(rs.getString("status")), rs.getInt("total"), rs.getInt("matched"),
rs.getInt("chosen"), rs.getInt("saved"), rs.getInt("added"), rs.getTimestamp("progress_at") != null,
rs.getInt("attempts"), rs.getString("step") != null ? GameImport.Step.valueOf(rs.getString("step")) : null,
rs.getTimestamp("created_at").toInstant(), rs.getTimestamp("queued_at").toInstant(),
instant(rs.getTimestamp("saved_at")));
private final NamedParameterJdbcTemplate jdbc;
private final JsonMapper jsonMapper;
/**
* Puts the import in the matching queue. The V22 index refuses a second import there per user with a
* DuplicateKeyException.
*/
public long create(long userId, GameImport.Source source, List<GameImport.Line> lines) {
GeneratedKeyHolder key = new GeneratedKeyHolder();
jdbc.update("insert into game_import (user_id, source, status, total) values (:user, :source, 'QUEUED', :total)",
new MapSqlParameterSource("user", userId).addValue("source", source.name()).addValue("total", lines.size()),
key, new String[]{"id"});
long id = key.getKeyAs(Long.class);
jdbc.batchUpdate("insert into game_import_row (import_id, line_no, title, platform, playtime_minutes, status, "
+ "igdb_id) values (:id, :line, :title, :platform, :minutes, :status, :igdb)",
lines.stream().map(line -> new MapSqlParameterSource("id", id).addValue("line", line.lineNo())
.addValue("title", line.title()).addValue("platform", line.platform())
.addValue("minutes", line.playtimeMinutes())
.addValue("status", line.status() != null ? line.status().name() : null)
.addValue("igdb", line.igdbId())).toArray(MapSqlParameterSource[]::new));
return id;
}
public Optional<GameImport> find(long userId, long id) {
return jdbc.query(SELECT_IMPORT + "where i.id = :id and i.user_id = :user",
new MapSqlParameterSource("id", id).addValue("user", userId), IMPORT).stream().findFirst();
}
public List<GameImport> findByUser(long userId, int limit) {
return jdbc.query(SELECT_IMPORT + "where i.user_id = :user order by i.id desc limit :limit",
new MapSqlParameterSource("user", userId).addValue("limit", limit), IMPORT);
}
/**
* The imports before this one in its queue (of any user).
*/
public int countAhead(GameImport waiting, Collection<GameImport.Status> queue) {
return jdbc.queryForObject("select count(*) from game_import where status in (:queue) "
+ "and (queued_at, id) < (:queuedAt, :id)",
new MapSqlParameterSource("queue", queue.stream().map(Enum::name).toList())
.addValue("queuedAt", Timestamp.from(waiting.queuedAt())).addValue("id", waiting.id()),
Integer.class);
}
/**
* The oldest import of a queue.
*/
public Optional<Long> findNext(Collection<GameImport.Status> queue) {
return jdbc.queryForList("select id from game_import where status in (:queue) order by queued_at, id limit 1",
new MapSqlParameterSource("queue", queue.stream().map(Enum::name).toList()), Long.class)
.stream().findFirst();
}
/**
* Imports a worker runs whose last progress is before {@code before}, with their status.
*/
public Map<Long, GameImport.Status> findStalled(Instant before) {
Map<Long, GameImport.Status> stalled = new LinkedHashMap<>();
jdbc.query("select id, status from game_import where status in ('MATCHING', 'SAVING') "
+ "and progress_at < :before order by id",
new MapSqlParameterSource("before", Timestamp.from(before)),
rs -> { stalled.put(rs.getLong("id"), GameImport.Status.valueOf(rs.getString("status"))); });
return stalled;
}
public long findUserId(long id) {
return jdbc.queryForObject("select user_id from game_import where id = :id", new MapSqlParameterSource("id", id),
Long.class);
}
/**
* Marks the import as run by a worker, from now on watched by the watchdog.
*/
public void start(long id, GameImport.Status status) {
jdbc.update("update game_import set status = :status, progress_at = now() where id = :id",
new MapSqlParameterSource("id", id).addValue("status", status.name()));
}
public void setStep(long id, GameImport.Step step) {
jdbc.update("update game_import set step = :step, progress_at = now() where id = :id",
new MapSqlParameterSource("id", id).addValue("step", step.name()));
}
public void markReady(long id) {
jdbc.update("update game_import set status = 'READY', progress_at = null where id = :id",
new MapSqlParameterSource("id", id));
}
/**
* After the games are added: a status change after saved_at is the user's.
*/
public void markSaved(long id) {
jdbc.update("update game_import set status = 'SAVED', saved_at = now(), step = null, progress_at = null "
+ "where id = :id", new MapSqlParameterSource("id", id));
}
/**
* Counts a stopped try: back to the end of the queue of its phase, or FAILED after {@code maxAttempts} tries.
*/
public void retryOrFail(long id, int maxAttempts) {
jdbc.update("""
update game_import set attempts = attempts + 1, queued_at = now(), progress_at = null, step = null,
status = case when attempts + 1 >= :max then 'FAILED'
when status in ('QUEUED', 'MATCHING') then 'QUEUED' else 'SAVING' end
where id = :id and status in ('QUEUED', 'MATCHING', 'SAVING')
""", new MapSqlParameterSource("id", id).addValue("max", maxAttempts));
}
/**
* A FAILED import goes back to the queue of the phase it stopped in, with new tries. In the matching queue the
* V22 index refuses a second import per user with a DuplicateKeyException.
*
* @return false when the import did not fail
*/
public boolean retry(long userId, long id) {
return jdbc.update("""
update game_import i set attempts = 0, queued_at = now(), progress_at = null,
status = case when exists (select 1 from game_import_row r
where r.import_id = i.id and r.chosen_igdb_id is not null)
then 'SAVING' else 'QUEUED' end
where id = :id and user_id = :user and status = 'FAILED'
""", new MapSqlParameterSource("id", id).addValue("user", userId)) == 1;
}
/**
* @return false when the import was in none of {@code from} (a second save, or an undo of an import not saved)
*/
public boolean changeStatus(long userId, long id, Collection<GameImport.Status> from, GameImport.Status to) {
return jdbc.update("update game_import set status = :to, queued_at = now() "
+ "where id = :id and user_id = :user and status in (:from)",
new MapSqlParameterSource("id", id).addValue("user", userId)
.addValue("from", from.stream().map(Enum::name).toList()).addValue("to", to.name())) == 1;
}
public List<GameImport.Line> findUnmatchedLines(long importId) {
return jdbc.query("select * from game_import_row where import_id = :id and matched_at is null order by line_no",
new MapSqlParameterSource("id", importId), (rs, row) -> line(rs));
}
public List<GameImport.Row> findRows(long importId) {
return jdbc.query("select * from game_import_row where import_id = :id order by line_no",
new MapSqlParameterSource("id", importId), (rs, row) -> new GameImport.Row(line(rs),
rs.getBoolean("search_failed"), rs.getBoolean("exact"),
jsonMapper.readValue(rs.getString("candidates"),
jsonMapper.getTypeFactory().constructCollectionType(List.class, IgdbApiService.Game.class))));
}
public void saveMatch(long importId, int lineNo, boolean searchFailed, boolean exact,
List<IgdbApiService.Game> candidates) {
jdbc.update("update game_import_row set matched_at = now(), search_failed = :failed, exact = :exact, "
+ "candidates = cast(:candidates as jsonb) where import_id = :id and line_no = :line",
new MapSqlParameterSource("id", importId).addValue("line", lineNo).addValue("failed", searchFailed)
.addValue("exact", exact).addValue("candidates", jsonMapper.writeValueAsString(candidates)));
touch(importId);
}
/**
* Stores the review; a line number the import does not have is left out.
*/
public void saveChoices(long importId, List<GameImport.Choice> choices) {
jdbc.batchUpdate("update game_import_row set chosen_igdb_id = :igdb, chosen_platform = :platform, "
+ "chosen_status = :status, chosen_minutes = :minutes where import_id = :id and line_no = :line",
choices.stream().map(choice -> new MapSqlParameterSource("id", importId)
.addValue("line", choice.lineNo()).addValue("igdb", choice.igdbId())
.addValue("platform", choice.platform()).addValue("status", choice.status().name())
.addValue("minutes", choice.playtimeMinutes())).toArray(MapSqlParameterSource[]::new));
}
/**
* The chosen lines the save has not done yet, in line order.
*/
public List<GameImport.Choice> findUnsavedChoices(long importId) {
return jdbc.query("select * from game_import_row where import_id = :id and chosen_igdb_id is not null "
+ "and save_result is null order by line_no",
new MapSqlParameterSource("id", importId), (rs, row) -> new GameImport.Choice(rs.getInt("line_no"),
rs.getLong("chosen_igdb_id"), rs.getString("chosen_platform"),
GameStatus.valueOf(rs.getString("chosen_status")), rs.getInt("chosen_minutes")));
}
public void saveResult(long importId, int lineNo, GameImport.SaveResult result) {
jdbc.update("update game_import_row set save_result = :result where import_id = :id and line_no = :line",
new MapSqlParameterSource("id", importId).addValue("line", lineNo).addValue("result", result.name()));
touch(importId);
}
/**
* The import's games still added by hand; a game the Steam sync took over stays in the library.
*/
public int countGames(long userId, long importId) {
return jdbc.queryForObject("select count(*) from user_game where user_id = :user and import_id = :import "
+ "and source = 'MANUAL'", new MapSqlParameterSource("user", userId).addValue("import", importId),
Integer.class);
}
public List<Long> findAddedAppids(long importId) {
return jdbc.queryForList("select appid from user_game where import_id = :import",
new MapSqlParameterSource("import", importId), Long.class);
}
/**
* Of {@link #countGames}: those with a rating, notes, priority or a status change after the save.
*/
public int countChangedGames(long userId, long importId) {
return jdbc.queryForObject("""
select count(*) from user_game g join game_import i on i.id = g.import_id
where g.user_id = :user and g.import_id = :import and g.source = 'MANUAL'
and (g.rating is not null or g.notes is not null or g.priority is not null
or exists (select 1 from user_game_status_change c where c.user_id = g.user_id
and c.appid = g.appid and c.changed_at > i.saved_at))
""", new MapSqlParameterSource("user", userId).addValue("import", importId), Integer.class);
}
/**
* Deletes {@link #countGames} the way the manual remove does: with status, rating, notes and history.
*/
public int deleteGames(long userId, long importId) {
return jdbc.update("delete from user_game where user_id = :user and import_id = :import and source = 'MANUAL'",
new MapSqlParameterSource("user", userId).addValue("import", importId));
}
// The watchdog's sign of life.
private void touch(long importId) {
jdbc.update("update game_import set progress_at = now() where id = :id", new MapSqlParameterSource("id", importId));
}
private static GameImport.Line line(ResultSet rs) throws SQLException {
String status = rs.getString("status");
return new GameImport.Line(rs.getInt("line_no"), rs.getString("title"), rs.getString("platform"),
rs.getObject("playtime_minutes", Integer.class), status != null ? GameStatus.valueOf(status) : null,
rs.getObject("igdb_id", Long.class));
}
private static Instant instant(Timestamp timestamp) {
return timestamp != null ? timestamp.toInstant() : null;
}
}
@@ -41,6 +41,8 @@ public interface UserGameRepository extends JpaRepository<UserGame, UserGameId>
List<UserGame> findByUserIdAndRemovedAtIsNull(Long userId);
List<UserGame> findByUserId(Long userId);
@Query("select ug.appid from UserGame ug where ug.userId = :userId and ug.removedAt is null and ug.appid in :appids")
List<Long> findActiveAppids(@Param("userId") Long userId, @Param("appids") Collection<Long> appids);
@@ -1,7 +1,9 @@
package core_service.modules.library.repo;
import core_service.modules.library.domain.UserSteamSync;
import jakarta.persistence.LockModeType;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.data.jpa.repository.Lock;
import org.springframework.data.jpa.repository.Modifying;
import org.springframework.data.jpa.repository.Query;
import org.springframework.data.repository.query.Param;
@@ -9,38 +11,56 @@ import org.springframework.transaction.annotation.Transactional;
import java.time.Instant;
import java.util.List;
import java.util.Optional;
public interface UserSteamSyncRepository extends JpaRepository<UserSteamSync, Long> {
/**
* The row before a claim, locked until the claim commits: a run that claims after another one finished sees
* that run's Steam ID, not an older one.
*/
@Lock(LockModeType.PESSIMISTIC_WRITE)
@Query("select s from UserSteamSync s where s.userId = :userId")
Optional<UserSteamSync> findForClaim(@Param("userId") Long userId);
/**
* Atomic claim-run upsert. Returns 1 if this caller successfully claimed the run
* (row created, or existing row was not RUNNING / RUNNING marker stale).
* Returns 0 if another sync is already in flight, or the user is deleted (table deleted_user) — caller must skip.
*
* @param startedAt the run's key for {@link #markFinished}, in microseconds as Postgres stores it
*/
@Modifying
@Query(value = """
INSERT INTO user_steam_sync (user_id, steam_id, last_sync_status)
SELECT :userId, :steamId, 'RUNNING'
INSERT INTO user_steam_sync (user_id, steam_id, last_sync_status, sync_started_at)
SELECT :userId, :steamId, 'RUNNING', :startedAt
WHERE NOT EXISTS (SELECT 1 FROM deleted_user WHERE user_id = :userId)
ON CONFLICT (user_id) DO UPDATE
SET steam_id = EXCLUDED.steam_id,
last_sync_status = 'RUNNING'
last_sync_status = 'RUNNING',
sync_started_at = EXCLUDED.sync_started_at
WHERE user_steam_sync.last_sync_status <> 'RUNNING'
OR user_steam_sync.last_synced_at IS NULL
OR user_steam_sync.last_synced_at < :runningStaleBefore
OR user_steam_sync.sync_started_at IS NULL
OR user_steam_sync.sync_started_at < :runningStaleBefore
""", nativeQuery = true)
int tryClaimRun(@Param("userId") Long userId,
@Param("steamId") String steamId,
@Param("startedAt") Instant startedAt,
@Param("runningStaleBefore") Instant runningStaleBefore);
/**
* @return 0 when another run took over the stale claim; that run's RUNNING stays
*/
@Modifying
@Query(value = """
UPDATE user_steam_sync
SET last_synced_at = :now,
last_sync_status = :status
WHERE user_id = :userId
AND sync_started_at = :startedAt
""", nativeQuery = true)
void markFinished(@Param("userId") Long userId,
int markFinished(@Param("userId") Long userId,
@Param("startedAt") Instant startedAt,
@Param("status") String status,
@Param("now") Instant now);
@@ -0,0 +1,173 @@
package core_service.modules.library.service;
import core_service.modules.igdb_adapter.service.IgdbApiService;
import core_service.modules.library.api.dto.GameImportDto;
import core_service.modules.library.api.dto.ImportRequest;
import core_service.modules.library.api.dto.ImportSaveRequest;
import core_service.modules.library.api.dto.UndoCheckDto;
import core_service.modules.library.domain.GameImport;
import core_service.modules.library.repo.GameImportRepository;
import core_service.shared.security.RequiresRole;
import core_service.shared.security.Role;
import org.springframework.dao.DuplicateKeyException;
import org.springframework.stereotype.Service;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.support.TransactionTemplate;
import java.util.List;
import java.util.Optional;
import java.util.Set;
import java.util.stream.IntStream;
/**
* Imports of many games at once, see {@link GameImport}. The matching and the save run in {@link GameImportWorker};
* the save adds each chosen line the way {@link ManualGameService#add} does.
*/
@Service
@RequiresRole(Role.USER)
public class GameImportService {
public sealed interface CreateResult {
record Created(long id) implements CreateResult {}
record AlreadyMatching() implements CreateResult {}
record SearchUnavailable() implements CreateResult {}
}
public enum ChangeResult { DONE, NOT_FOUND, WRONG_STATUS, ALREADY_MATCHING }
private static final int HISTORY_SIZE = 20;
private final GameImportRepository importRepository;
private final GameImportWorker worker;
private final ManualGameService manualGameService;
private final IgdbApiService igdbApiService;
private final TransactionTemplate transaction;
public GameImportService(GameImportRepository importRepository, GameImportWorker worker,
ManualGameService manualGameService, IgdbApiService igdbApiService,
PlatformTransactionManager transactionManager) {
this.importRepository = importRepository;
this.worker = worker;
this.manualGameService = manualGameService;
this.igdbApiService = igdbApiService;
this.transaction = new TransactionTemplate(transactionManager);
}
public CreateResult create(Long userId, GameImport.Source source, List<ImportRequest.Row> rows) {
if (!igdbApiService.isEnabled()) {
return new CreateResult.SearchUnavailable();
}
List<GameImport.Line> lines = IntStream.range(0, rows.size()).mapToObj(index -> {
ImportRequest.Row row = rows.get(index);
return new GameImport.Line(index + 1, strippedOrNull(row.title()), strippedOrNull(row.platform()),
row.playtimeMinutes(), row.status(), row.igdbId());
}).toList();
long id;
try {
id = transaction.execute(tx -> importRepository.create(userId, source, lines));
} catch (DuplicateKeyException e) {
return new CreateResult.AlreadyMatching();
}
worker.wakeMatching();
return new CreateResult.Created(id);
}
public List<GameImportDto> list(Long userId) {
return importRepository.findByUser(userId, HISTORY_SIZE).stream().map(found -> dto(found, null)).toList();
}
public Optional<GameImportDto> get(Long userId, long id) {
return importRepository.find(userId, id).map(found -> dto(found,
found.status() == GameImport.Status.READY ? rows(userId, importRepository.findRows(id)) : null));
}
/**
* Once per import: stores the review and puts the import in the saving queue.
*/
public ChangeResult save(Long userId, long id, List<ImportSaveRequest.Row> chosen) {
if (importRepository.find(userId, id).isEmpty()) {
return ChangeResult.NOT_FOUND;
}
boolean queued = transaction.execute(tx -> {
if (!importRepository.changeStatus(userId, id, Set.of(GameImport.Status.READY), GameImport.Status.SAVING)) {
return false;
}
importRepository.saveChoices(id, chosen.stream().map(row -> new GameImport.Choice(row.lineNo(),
row.igdbId(), row.platform(), row.status(),
row.playtimeMinutes() != null ? row.playtimeMinutes() : 0)).toList());
return true;
});
if (!queued) {
return ChangeResult.WRONG_STATUS;
}
worker.wakeSaving();
return ChangeResult.DONE;
}
public ChangeResult retry(Long userId, long id) {
if (importRepository.find(userId, id).isEmpty()) {
return ChangeResult.NOT_FOUND;
}
try {
if (!importRepository.retry(userId, id)) {
return ChangeResult.WRONG_STATUS;
}
} catch (DuplicateKeyException e) {
return ChangeResult.ALREADY_MATCHING;
}
worker.wakeMatching();
worker.wakeSaving();
return ChangeResult.DONE;
}
public Optional<UndoCheckDto> undoCheck(Long userId, long id) {
return importRepository.find(userId, id).map(found -> new UndoCheckDto(
importRepository.countGames(userId, id), importRepository.countChangedGames(userId, id)));
}
/**
* Removes the import's games that are still added by hand, like {@link ManualGameService#remove}. Also for an
* import that FAILED while saving.
*/
public ChangeResult undo(Long userId, long id) {
if (importRepository.find(userId, id).isEmpty()) {
return ChangeResult.NOT_FOUND;
}
return transaction.execute(tx -> {
if (!importRepository.changeStatus(userId, id, Set.of(GameImport.Status.SAVED, GameImport.Status.FAILED),
GameImport.Status.UNDONE)) {
return ChangeResult.WRONG_STATUS;
}
importRepository.deleteGames(userId, id);
return ChangeResult.DONE;
});
}
private List<GameImportDto.Row> rows(Long userId, List<GameImport.Row> rows) {
Set<Long> owned = manualGameService.ownedIgdbIds(userId,
rows.stream().flatMap(row -> row.candidates().stream()).toList());
return rows.stream().map(row -> new GameImportDto.Row(row.line().lineNo(), row.line().title(),
row.line().platform(), row.line().playtimeMinutes(), row.line().status(), row.searchFailed(),
row.exact(), row.candidates().stream().map(game -> new GameImportDto.Candidate(game.igdbId(),
game.name(), game.year(), game.imageUrl(), ManualGameService.fittingPlatforms(game),
owned.contains(game.igdbId()))).toList())).toList();
}
private GameImportDto dto(GameImport gameImport, List<GameImportDto.Row> rows) {
Integer ahead = switch (gameImport.status()) {
case QUEUED -> importRepository.countAhead(gameImport, GameImportWorker.MATCHING_QUEUE);
case SAVING -> gameImport.running() ? null
: importRepository.countAhead(gameImport, GameImportWorker.SAVING_QUEUE);
default -> null;
};
return new GameImportDto(gameImport.id(), gameImport.source(), gameImport.status(), gameImport.total(),
gameImport.matched(), gameImport.chosen(), gameImport.saved(), gameImport.added(), ahead,
gameImport.step(), gameImport.attempts(), gameImport.createdAt(), gameImport.savedAt(), rows);
}
private static String strippedOrNull(String text) {
return text == null || text.isBlank() ? null : text.strip();
}
}
@@ -0,0 +1,267 @@
package core_service.modules.library.service;
import core_service.modules.igdb_adapter.service.IgdbApiService;
import core_service.modules.library.domain.GameImport;
import core_service.modules.library.repo.GameImportRepository;
import jakarta.annotation.PreDestroy;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.context.event.ApplicationReadyEvent;
import org.springframework.context.event.EventListener;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;
import java.time.Duration;
import java.time.Instant;
import java.util.Comparator;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.function.LongConsumer;
import java.util.stream.Collectors;
/**
* Runs imports in two queues, each on one thread for all users, with game_import as the queue:
* <ul>
* <li>matching: IGDB searches, so an import of 1000 lines never takes the whole IGDB budget from the interactive
* search;</li>
* <li>saving: adds the chosen games, then loads their time to beat and moods. A save never waits behind another
* user's big matching.</li>
* </ul>
* Each line is stored when done, so a stopped try goes on where it stopped. A watchdog stops a try without progress
* for {@code import-stall-minutes}; a restart counts as a stopped try. After {@link #MAX_ATTEMPTS} tries the import
* is FAILED.
*/
@Service
@Slf4j
public class GameImportWorker {
static final int MAX_CANDIDATES = 5;
static final int MAX_ATTEMPTS = 3;
private static final int RESTART_WAIT_SECONDS = 30;
static final Set<GameImport.Status> MATCHING_QUEUE = Set.of(GameImport.Status.QUEUED, GameImport.Status.MATCHING);
static final Set<GameImport.Status> SAVING_QUEUE = Set.of(GameImport.Status.SAVING);
private final GameImportRepository importRepository;
private final ManualGameService manualGameService;
private final IgdbApiService igdbApiService;
private final TimeToBeatService timeToBeatService;
private final MoodService moodService;
private final Duration stallTime;
// Only a try that started before this app is one a restart stopped.
private final Instant startedAt = Instant.now();
private final Lane matching;
private final Lane saving;
public GameImportWorker(GameImportRepository importRepository, ManualGameService manualGameService,
IgdbApiService igdbApiService, TimeToBeatService timeToBeatService, MoodService moodService,
@Value("${app.library.import-stall-minutes:5}") long stallMinutes) {
this.importRepository = importRepository;
this.manualGameService = manualGameService;
this.igdbApiService = igdbApiService;
this.timeToBeatService = timeToBeatService;
this.moodService = moodService;
this.stallTime = Duration.ofMinutes(stallMinutes);
this.matching = new Lane("game-import-match", MATCHING_QUEUE, this::match);
this.saving = new Lane("game-import-save", SAVING_QUEUE, this::save);
}
public void wakeMatching() {
matching.wake();
}
public void wakeSaving() {
saving.wake();
}
/**
* Imports a worker ran when the app stopped count as a stopped try; then both queues start.
*/
@EventListener(ApplicationReadyEvent.class)
public void resume() {
importRepository.findStalled(startedAt).keySet().forEach(id -> importRepository.retryOrFail(id, MAX_ATTEMPTS));
matching.wake();
saving.wake();
}
/**
* Stops a try without progress: the import goes back to the end of its queue (or FAILED), and its queue gets a
* new thread. The old thread is interrupted and writes nothing more once it sees that. Also wakes both queues:
* a run that died on a database error left its queue idle.
*/
@Scheduled(fixedDelayString = "${app.library.import-watchdog-ms:60000}")
public void watchdog() {
Map<Long, GameImport.Status> stalled = importRepository.findStalled(Instant.now().minus(stallTime));
if (!stalled.isEmpty()) {
log.warn("Imports {} made no progress for {}, trying again", stalled, stallTime);
stalled.keySet().forEach(id -> importRepository.retryOrFail(id, MAX_ATTEMPTS));
}
if (stalled.containsValue(GameImport.Status.MATCHING)) {
matching.restart();
} else {
matching.wake();
}
if (stalled.containsValue(GameImport.Status.SAVING)) {
saving.restart();
} else {
saving.wake();
}
}
@PreDestroy
void stop() {
matching.stop();
saving.stop();
}
private void match(long importId) {
importRepository.start(importId, GameImport.Status.MATCHING);
for (GameImport.Line line : importRepository.findUnmatchedLines(importId)) {
Match found = match(line);
// An IGDB call cut by the watchdog or the shutdown is no failed search.
if (Thread.currentThread().isInterrupted()) {
return;
}
importRepository.saveMatch(importId, line.lineNo(), found.searchFailed(), found.exact(), found.candidates());
}
importRepository.markReady(importId);
}
private void save(long importId) {
importRepository.start(importId, GameImport.Status.SAVING);
List<GameImport.Choice> choices = importRepository.findUnsavedChoices(importId);
if (!choices.isEmpty()) {
long userId = importRepository.findUserId(importId);
Map<Integer, GameImport.Row> rows = importRepository.findRows(importId).stream()
.collect(Collectors.toMap(row -> row.line().lineNo(), row -> row));
for (GameImport.Choice choice : choices) {
IgdbApiService.Game game = rows.get(choice.lineNo()).candidates().stream()
.filter(candidate -> candidate.igdbId() == choice.igdbId()).findFirst()
// A game from a new search in the review. IGDB knew it minutes ago, so no answer is an IGDB
// failure: the try stops and the line waits for the next one.
.or(() -> igdbApiService.getGame(choice.igdbId()))
.orElseThrow(() -> new IllegalStateException("IGDB did not answer for game " + choice.igdbId()));
if (Thread.currentThread().isInterrupted()) {
return;
}
boolean added = manualGameService.addGame(userId, game, choice.platform(), choice.status(),
choice.playtimeMinutes(), importId) instanceof ManualGameService.AddResult.Added;
importRepository.saveResult(importId, choice.lineNo(),
added ? GameImport.SaveResult.ADDED : GameImport.SaveResult.SKIPPED);
}
}
importRepository.setStep(importId, GameImport.Step.DETAILS);
List<Long> appids = importRepository.findAddedAppids(importId);
try {
timeToBeatService.refresh(appids);
moodService.refresh(appids);
} catch (RuntimeException e) {
// The next sync or page view asks again.
log.warn("Time to beat or moods of import {} not loaded: {}", importId, e.getMessage());
}
if (Thread.currentThread().isInterrupted()) {
return;
}
importRepository.markSaved(importId);
}
private record Match(boolean searchFailed, boolean exact, List<IgdbApiService.Game> candidates) {}
private Match match(GameImport.Line line) {
if (line.igdbId() != null) {
Optional<IgdbApiService.Game> game = igdbApiService.getGame(line.igdbId());
return new Match(game.isEmpty(), game.isPresent(), game.stream().toList());
}
return manualGameService.findGames(line.title())
.map(found -> rank(line, found))
.map(candidates -> new Match(false,
!candidates.isEmpty() && sameName(candidates.getFirst().name(), line.title()), candidates))
.orElseGet(() -> new Match(true, false, List.of()));
}
// Same name first, among them one on the line's platform; else IGDB's order.
static List<IgdbApiService.Game> rank(GameImport.Line line, List<IgdbApiService.Game> found) {
Comparator<IgdbApiService.Game> order = Comparator
.comparing((IgdbApiService.Game game) -> !sameName(game.name(), line.title()))
.thenComparing(game -> line.platform() == null
|| game.platforms().stream().noneMatch(line.platform()::equalsIgnoreCase));
return found.stream().sorted(order).limit(MAX_CANDIDATES).toList();
}
/**
* Case, accents, punctuation and marks like ™ ignored, as in the speedrun.com match.
*/
static boolean sameName(String a, String b) {
return SpeedrunMatch.normalize(a).equals(SpeedrunMatch.normalize(b));
}
/**
* One thread that runs the imports of a queue, oldest first, until the queue is empty. A wake while it runs
* leaves one more run waiting, so an import queued meanwhile is not missed.
*/
private final class Lane {
private final String name;
private final Set<GameImport.Status> queue;
private final LongConsumer work;
private volatile ExecutorService executor;
Lane(String name, Set<GameImport.Status> queue, LongConsumer work) {
this.name = name;
this.queue = queue;
this.work = work;
this.executor = newExecutor();
}
void wake() {
executor.execute(this::drain);
}
// Waits a while for the old thread, which stops at its next check, so two threads do not run one import.
synchronized void restart() {
ExecutorService old = executor;
old.shutdownNow();
try {
if (!old.awaitTermination(RESTART_WAIT_SECONDS, TimeUnit.SECONDS)) {
log.warn("Thread {} did not stop; a new one starts anyway", name);
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
executor = newExecutor();
wake();
}
void stop() {
executor.shutdownNow();
}
private void drain() {
Optional<Long> next;
while (!Thread.currentThread().isInterrupted() && (next = importRepository.findNext(queue)).isPresent()) {
long id = next.get();
try {
work.accept(id);
} catch (RuntimeException e) {
if (Thread.currentThread().isInterrupted()) {
return;
}
// To the end of the queue, so the next import is not blocked by this one.
log.error("Import {} stopped", id, e);
importRepository.retryOrFail(id, MAX_ATTEMPTS);
}
}
}
private ExecutorService newExecutor() {
// One run and at most one waiting run; more wakes add nothing.
return new ThreadPoolExecutor(1, 1, 0, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<>(1),
Thread.ofPlatform().name(name).daemon().factory(), new ThreadPoolExecutor.DiscardPolicy());
}
}
}
@@ -145,7 +145,7 @@ public class LibrarySyncService {
private boolean isAlreadyRunning(UserSteamSync sync) {
if (sync.getLastSyncStatus() != UserSteamSync.Status.RUNNING) return false;
if (sync.getLastSyncedAt() == null) return true;
return sync.getLastSyncedAt().isAfter(Instant.now().minus(Duration.ofMinutes(runningStaleMinutes)));
return sync.getSyncStartedAt() != null
&& sync.getSyncStartedAt().isAfter(Instant.now().minus(Duration.ofMinutes(runningStaleMinutes)));
}
}
@@ -4,7 +4,6 @@ import core_service.modules.library.domain.CorrectionField;
import core_service.modules.library.domain.GameSource;
import core_service.modules.library.domain.SteamGame;
import core_service.modules.library.domain.UserGame;
import core_service.modules.library.domain.UserGameId;
import core_service.modules.library.domain.UserSteamSync;
import core_service.modules.library.domain.WishlistGame;
import core_service.modules.library.repo.GameCorrectionRepository;
@@ -21,17 +20,18 @@ import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.ApplicationEventPublisher;
import org.springframework.scheduling.annotation.Async;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.transaction.support.TransactionTemplate;
import java.time.Duration;
import java.time.Instant;
import java.time.LocalDate;
import java.time.ZoneId;
import java.time.temporal.ChronoUnit;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import java.util.Set;
import java.util.stream.Collectors;
@@ -39,9 +39,11 @@ import java.util.stream.Collectors;
import core_service.modules.library.domain.UserSteamSync.Status;
/**
* Worker bean. Runs the actual Steam fetch + DB upserts.
* Separate bean so Spring proxy applies @Async and @Transactional
* (self-invocation from LibrarySyncService would bypass both).
* Worker bean. Runs the actual Steam fetch + DB upserts. Separate bean so the Spring proxy applies @Async
* (self-invocation from LibrarySyncService would bypass it).
* <p>
* The claim, the writes and the finish each commit on their own; the Steam and IGDB calls run outside any
* transaction. So other requests see RUNNING, and FAILED is stored also after a database error.
*/
@Service
@RequiredArgsConstructor
@@ -58,6 +60,7 @@ public class LibrarySyncWorker {
private final TimeToBeatService timeToBeatService;
private final MoodService moodService;
private final ApplicationEventPublisher eventPublisher;
private final TransactionTemplate transaction;
@Value("${app.library.running-stale-minutes:5}")
private long runningStaleMinutes;
@@ -66,7 +69,6 @@ public class LibrarySyncWorker {
private ZoneId zone;
@Async
@Transactional
public void runSync(Long userId, String steamId) {
sync(userId, steamId);
}
@@ -75,101 +77,129 @@ public class LibrarySyncWorker {
* Same as {@link #runSync}, but on the caller's thread. The nightly job uses it to sync one account after
* the other instead of starting all at once against the Steam rate limit and the connection pool.
*/
@Transactional
public void runSyncNow(Long userId, String steamId) {
sync(userId, steamId);
}
private record Claim(String historySteamId) {}
private void sync(Long userId, String steamId) {
Instant runningStaleBefore = Instant.now().minus(Duration.ofMinutes(runningStaleMinutes));
UserSteamSync before = userSteamSyncRepository.findById(userId).orElse(null);
Instant syncStartedAt = Instant.now().truncatedTo(ChronoUnit.MICROS);
Claim claim = transaction.execute(tx -> {
UserSteamSync before = userSteamSyncRepository.findForClaim(userId).orElse(null);
if (userSteamSyncRepository.tryClaimRun(userId, steamId, syncStartedAt, runningStaleBefore) == 0) {
return null;
}
String previousSteamId = before != null ? before.getSteamId() : null;
String historySteamId = before != null ? before.getHistorySteamId() : null;
int claimed = userSteamSyncRepository.tryClaimRun(userId, steamId, runningStaleBefore);
if (claimed == 0) {
if (previousSteamId != null && !previousSteamId.equals(steamId)) {
// The old games belong to the previous Steam account, also when Steam fails now. The upserts
// restore the games this account also owns.
userGameRepository.softDeleteMissingSince(userId, syncStartedAt, Instant.now());
wishlistGameRepository.deleteByUserId(userId);
}
return new Claim(before != null ? before.getHistorySteamId() : null);
});
if (claim == null) {
log.info("Sync skipped for user {} — another run already in flight, or the account is deleted", userId);
return;
}
Instant syncStartedAt = Instant.now();
Status finalStatus = Status.FAILED;
try {
boolean steamIdChanged = previousSteamId != null && !previousSteamId.equals(steamId);
// Growth needs a successful earlier sync of this account; else all its playtime would count as today.
boolean recordPlaytime = steamId.equals(historySteamId);
LocalDate today = LocalDate.ofInstant(syncStartedAt, zone);
if (steamIdChanged) {
// The old games belong to the previous Steam account. The upserts below restore the games this account also owns.
userGameRepository.softDeleteMissingSince(userId, syncStartedAt, Instant.now());
wishlistGameRepository.deleteByUserId(userId);
}
List<SteamOwnedGamesResponse.Game> games = steamApiService.getOwnedGames(steamId);
LocalDate playedOn = steamId.equals(claim.historySteamId()) ? LocalDate.ofInstant(syncStartedAt, zone) : null;
// Distinct: the maps in saveGames are loaded once, so a second line of a game would be a new game again.
List<SteamOwnedGamesResponse.Game> games = steamApiService.getOwnedGames(steamId).stream()
.filter(g -> g.appid() != null)
.collect(Collectors.toMap(SteamOwnedGamesResponse.Game::appid, g -> g, (first, second) -> first,
LinkedHashMap::new))
.values().stream().toList();
log.info("Sync user={} fetched {} games from Steam", userId, games.size());
// Before the empty check: an account without games can still have a public wishlist.
syncWishlist(userId, steamId, games);
Optional<Wishlist> wishlist = fetchWishlist(steamId, games);
boolean saved = transaction.execute(tx -> {
wishlist.ifPresentOrElse(found -> saveWishlist(userId, found),
() -> log.warn("Sync user={} kept the old wishlist, Steam did not answer", userId));
if (games.isEmpty()) {
// Steam errors and a private "Game details" setting also end here. A soft-delete would wipe the library.
// A private "Game details" setting ends here. A soft-delete would wipe the library.
log.warn("Sync user={} got no games for SteamID {}", userId, steamId);
return false;
}
saveGames(userId, steamId, games, syncStartedAt, playedOn);
return true;
});
if (!saved) {
return;
}
// Also the games added by hand: here a failed refresh from the add is asked again.
List<Long> igdbAppids = new ArrayList<>(games.stream().map(g -> g.appid().longValue()).toList());
igdbAppids.addAll(userGameRepository.findActiveManualAppids(userId));
igdbAppids.removeAll(new HashSet<>(userGameRepository.findHiddenAppids(userId)));
timeToBeatService.refresh(igdbAppids);
moodService.refresh(igdbAppids);
finalStatus = Status.OK;
} catch (Exception e) {
log.error("Sync failed for user {}", userId, e);
} finally {
Status status = finalStatus;
transaction.executeWithoutResult(tx -> {
if (userSteamSyncRepository.markFinished(userId, syncStartedAt, status.name(), Instant.now()) == 0) {
log.warn("Sync user={} ran past running-stale-minutes; the run that took over sets the status", userId);
} else if (status == Status.OK) {
// The listeners run after this commit.
eventPublisher.publishEvent(new LibrarySyncedEvent(userId, steamId));
}
});
}
}
private void saveGames(Long userId, String steamId, List<SteamOwnedGamesResponse.Game> games, Instant now,
LocalDate playedOn) {
List<Long> appids = games.stream().map(g -> g.appid().longValue()).toList();
// setSourceName keeps them anyway; skipping saves an update and a flush per corrected game and sync.
Set<Long> correctedNames = gameCorrectionRepository.findCorrectedAppids(CorrectionField.NAME,
games.stream().map(SteamOwnedGamesResponse.Game::appid).filter(Objects::nonNull).map(Integer::longValue).toList());
Set<Long> correctedNames = gameCorrectionRepository.findCorrectedAppids(CorrectionField.NAME, appids);
Map<Long, SteamGame> knownGames = steamGameRepository.findAllById(appids).stream()
.collect(Collectors.toMap(SteamGame::getAppid, game -> game));
Map<Long, UserGame> userGames = userGameRepository.findByUserId(userId).stream()
.collect(Collectors.toMap(UserGame::getAppid, game -> game));
List<Long> played = new ArrayList<>();
for (SteamOwnedGamesResponse.Game g : games) {
if (g.appid() == null) continue;
upsertGame(g, syncStartedAt, correctedNames);
if (upsertUserGame(userId, g, syncStartedAt, recordPlaytime ? today : null)) {
played.add(g.appid().longValue());
Long appid = g.appid().longValue();
upsertGame(g, knownGames.get(appid), now, correctedNames);
if (upsertUserGame(userId, g, userGames.get(appid), now, playedOn)) {
played.add(appid);
}
}
if (!played.isEmpty()) {
userGameRepository.playFromPileIfUntouched(userId, played);
}
int removed = userGameRepository.softDeleteMissingSince(userId, syncStartedAt, Instant.now());
int removed = userGameRepository.softDeleteMissingSince(userId, now, Instant.now());
if (removed > 0) {
log.info("Sync user={} soft-deleted {} games no longer owned", userId, removed);
}
userSteamSyncRepository.markHistoryBaseline(userId, steamId);
// Also the games added by hand: here a failed refresh from the add is asked again.
List<Long> igdbAppids = new ArrayList<>(games.stream().filter(g -> g.appid() != null).map(g -> g.appid().longValue()).toList());
igdbAppids.addAll(userGameRepository.findActiveManualAppids(userId));
igdbAppids.removeAll(new HashSet<>(userGameRepository.findHiddenAppids(userId)));
timeToBeatService.refresh(igdbAppids);
moodService.refresh(igdbAppids);
eventPublisher.publishEvent(new LibrarySyncedEvent(userId, steamId));
finalStatus = Status.OK;
} catch (Exception e) {
log.error("Sync failed for user {}", userId, e);
} finally {
userSteamSyncRepository.markFinished(userId, finalStatus.name(), Instant.now());
}
}
private void syncWishlist(Long userId, String steamId, List<SteamOwnedGamesResponse.Game> ownedGames) {
Optional<List<SteamApiService.WishlistItem>> wishlist = steamApiService.getWishlist(steamId);
if (wishlist.isEmpty()) {
log.warn("Sync user={} kept the old wishlist, Steam did not answer", userId);
return;
}
Set<Long> owned = ownedGames.stream()
.filter(g -> g.appid() != null)
.map(g -> g.appid().longValue())
.collect(Collectors.toSet());
List<SteamApiService.WishlistItem> wanted = wishlist.get().stream()
.filter(item -> !owned.contains(item.appid()))
.toList();
Map<Long, SteamApiService.StoreItem> store = wanted.isEmpty()
private record Wishlist(List<SteamApiService.WishlistItem> wanted, Map<Long, SteamApiService.StoreItem> store) {}
private Optional<Wishlist> fetchWishlist(String steamId, List<SteamOwnedGamesResponse.Game> ownedGames) {
Set<Long> owned = ownedGames.stream().map(g -> g.appid().longValue()).collect(Collectors.toSet());
return steamApiService.getWishlist(steamId).map(items -> {
List<SteamApiService.WishlistItem> wanted = items.stream().filter(item -> !owned.contains(item.appid())).toList();
return new Wishlist(wanted, wanted.isEmpty()
? Map.of()
: steamApiService.getStoreItems(wanted.stream().map(SteamApiService.WishlistItem::appid).toList());
: steamApiService.getStoreItems(wanted.stream().map(SteamApiService.WishlistItem::appid).toList()));
});
}
private void saveWishlist(Long userId, Wishlist wishlist) {
// A failed store batch leaves apps out of `store`; they keep the name and price of the last sync.
Map<Long, WishlistGame> previous = wishlistGameRepository.findForUser(userId).stream()
.collect(Collectors.toMap(WishlistGame::getAppid, w -> w));
wishlistGameRepository.deleteByUserId(userId);
wishlistGameRepository.saveAll(wanted.stream().map(item -> {
SteamApiService.StoreItem info = store.get(item.appid());
wishlistGameRepository.saveAll(wishlist.wanted().stream().map(item -> {
SteamApiService.StoreItem info = wishlist.store().get(item.appid());
if (info != null) {
return new WishlistGame(userId, item.appid(), info.name(), item.addedAt(),
info.price(), info.originalPrice(), info.discountPct());
@@ -182,9 +212,8 @@ public class LibrarySyncWorker {
}).toList());
}
private void upsertGame(SteamOwnedGamesResponse.Game g, Instant now, Set<Long> correctedNames) {
private void upsertGame(SteamOwnedGamesResponse.Game g, SteamGame existing, Instant now, Set<Long> correctedNames) {
Long appid = g.appid().longValue();
SteamGame existing = steamGameRepository.findById(appid).orElse(null);
// An unvetted app can come without a name, and the name column is NOT NULL.
boolean hasName = g.name() != null && !g.name().isBlank();
if (existing == null) {
@@ -202,9 +231,9 @@ public class LibrarySyncWorker {
* @param playedOn the day that gets the playtime growth, null to record none
* @return true when the game was played since the last sync
*/
private boolean upsertUserGame(Long userId, SteamOwnedGamesResponse.Game g, Instant now, LocalDate playedOn) {
private boolean upsertUserGame(Long userId, SteamOwnedGamesResponse.Game g, UserGame existing, Instant now,
LocalDate playedOn) {
Long appid = g.appid().longValue();
UserGame existing = userGameRepository.findById(new UserGameId(userId, appid)).orElse(null);
int playtime = g.playtimeForever() != null ? g.playtimeForever() : 0;
// Hours typed by hand are no Steam playtime, so the first Steam value of a game added by hand is a baseline.
boolean takesOverManualGame = existing != null && existing.getSource() == GameSource.MANUAL;
@@ -23,6 +23,7 @@ import org.springframework.transaction.support.TransactionTemplate;
import java.time.Instant;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Comparator;
import java.util.HashSet;
import java.util.List;
@@ -86,8 +87,7 @@ public class ManualGameService {
if (!igdbApiService.isEnabled()) {
return Optional.of(new GameSearchDto(false, List.of()));
}
return igdbApiService.searchGames(query).map(found -> withTypoHelp(query, found))
.map(games -> new GameSearchDto(true, hits(userId, query, games)));
return findGames(query).map(games -> new GameSearchDto(true, hits(userId, query, games)));
}
// Games already in ToBePlayed first, with their catalog name and image (an auditor's correction included); then
@@ -166,19 +166,7 @@ public class ManualGameService {
if (found.isEmpty()) {
return new AddResult.UnknownGame();
}
IgdbApiService.Game game = found.get();
AddResult result;
try {
result = addInTransaction(userId, game, platform, status, playtimeMinutes);
} catch (DataIntegrityViolationException e) {
// A parallel add, sync or merge changed the rows after the check. The second try sees it.
log.info("Add of igdbId={} for user={} raced another write, trying again", igdbId, userId);
try {
result = addInTransaction(userId, game, platform, status, playtimeMinutes);
} catch (DataIntegrityViolationException again) {
return new AddResult.AlreadyInLibrary();
}
}
AddResult result = addGame(userId, found.get(), platform, status, playtimeMinutes, null);
if (result instanceof AddResult.Added added) {
timeToBeatService.refreshInBackground(List.of(added.appid()));
moodService.refreshInBackground(List.of(added.appid()));
@@ -186,8 +174,56 @@ public class ManualGameService {
return result;
}
/**
* {@link #add} of a game already loaded from IGDB, without the background refresh; an import adds many games
* and loads their time to beat and moods once.
*
* @param importId the import that adds the game, so its undo can remove it; null for a single add. A removed
* row that comes back keeps no import id.
*/
public AddResult addGame(Long userId, IgdbApiService.Game game, String platform, GameStatus status,
int playtimeMinutes, Long importId) {
AddResult result;
try {
result = addInTransaction(userId, game, platform.strip(), status, playtimeMinutes, importId);
} catch (DataIntegrityViolationException e) {
// A parallel add, sync or merge changed the rows after the check. The second try sees it.
log.info("Add of igdbId={} for user={} raced another write, trying again", game.igdbId(), userId);
try {
result = addInTransaction(userId, game, platform.strip(), status, playtimeMinutes, importId);
} catch (DataIntegrityViolationException again) {
return new AddResult.AlreadyInLibrary();
}
}
return result;
}
/**
* IGDB's hits for a title, with the same typo help as {@link #search}; empty when IGDB failed or is off.
*/
public Optional<List<IgdbApiService.Game>> findGames(String query) {
if (!igdbApiService.isEnabled()) {
return Optional.empty();
}
return igdbApiService.searchGames(query).map(found -> withTypoHelp(query, found));
}
/**
* The IGDB ids of the games the user already has, under any of their keys (a merge included).
*/
public Set<Long> ownedIgdbIds(Long userId, Collection<IgdbApiService.Game> games) {
if (games.isEmpty()) {
return Set.of();
}
Map<Long, Long> targets = mergeRepository.findTargets(games.stream().flatMap(game -> ownKeys(game).stream()).toList());
Set<Long> owned = new HashSet<>(userGameRepository.findActiveAppids(userId,
games.stream().flatMap(game -> possibleKeys(game, targets).stream()).distinct().toList()));
return games.stream().filter(game -> possibleKeys(game, targets).stream().anyMatch(owned::contains))
.map(IgdbApiService.Game::igdbId).collect(Collectors.toSet());
}
private AddResult addInTransaction(Long userId, IgdbApiService.Game game, String platform, GameStatus status,
int playtimeMinutes) {
int playtimeMinutes, Long importId) {
return transaction.execute(tx -> {
Instant now = Instant.now();
Map<Long, Long> targets = mergeRepository.findTargets(ownKeys(game));
@@ -210,8 +246,12 @@ public class ManualGameService {
revived.setRemovedAt(null);
// Only a Steam game can be hidden.
revived.setHiddenAt(null);
// The row holds rating, notes and history from before, so an undo of the import must not delete it.
revived.setImportId(null);
} else {
userGameRepository.save(UserGame.manual(userId, appid, platform, status, playtimeMinutes, now));
UserGame added = UserGame.manual(userId, appid, platform, status, playtimeMinutes, now);
added.setImportId(importId);
userGameRepository.save(added);
}
return new AddResult.Added(appid);
});
@@ -263,7 +303,7 @@ public class ManualGameService {
}
// A longer name would not fit the column; with no platform left, the user types one.
private static List<String> fittingPlatforms(IgdbApiService.Game game) {
static List<String> fittingPlatforms(IgdbApiService.Game game) {
return game.platforms().stream().filter(platform -> platform.length() <= ManualGameUpdate.MAX_PLATFORM_LENGTH).toList();
}
@@ -1,9 +0,0 @@
package core_service.modules.steam_adapter.api.dto.response;
import java.util.List;
public record SteamProfileAndGamesResponse(
SteamPlayerSummaryResponse.Player player,
List<SteamOwnedGamesResponse.Game> games
) {
}
@@ -4,7 +4,6 @@ import core_service.modules.steam_adapter.api.dto.response.SteamGlobalAchievemen
import core_service.modules.steam_adapter.api.dto.response.SteamOwnedGamesResponse;
import core_service.modules.steam_adapter.api.dto.response.SteamPlayerAchievementsResponse;
import core_service.modules.steam_adapter.api.dto.response.SteamPlayerSummaryResponse;
import core_service.modules.steam_adapter.api.dto.response.SteamProfileAndGamesResponse;
import core_service.modules.steam_adapter.api.dto.response.SteamSchemaForGameResponse;
import core_service.modules.steam_adapter.api.dto.response.SteamStoreItemsResponse;
import core_service.modules.steam_adapter.api.dto.response.SteamWishlistResponse;
@@ -54,17 +53,13 @@ public class SteamApiService {
@Cacheable(value = "steamGameCount", key = "#steamId")
@RateLimiter(name = "steamApi")
@Retry(name = "steamApi")
@Retry(name = "steamApiOnce", fallbackMethod = "noGamesCount")
public int getOwnedGamesCount(String steamId) {
if (apiKey == null || apiKey.isEmpty() || apiKey.equals("${STEAM_API_KEY}")) {
log.error("Steam API Key is not configured correctly!");
return 0;
}
try {
log.info("Fetching owned games count for SteamID: {} using API Key starting with: {}",
steamId, apiKey.substring(0, Math.min(apiKey.length(), 4)));
SteamOwnedGamesResponse response = steamRestClient.get()
.uri(uriBuilder -> uriBuilder
.path("/IPlayerService/GetOwnedGames/v1/")
@@ -78,8 +73,6 @@ public class SteamApiService {
.retrieve()
.body(SteamOwnedGamesResponse.class);
log.info("Steam API Response object for {}: {}", steamId, response);
if (response != null && response.response() != null) {
Integer count = response.response().gameCount();
if (count == null) {
@@ -94,30 +87,20 @@ public class SteamApiService {
log.info("Successfully fetched {} games for SteamID: {}", count, steamId);
return count;
}
} catch (Exception e) {
log.error("Error communicating with Steam API for game count of SteamID: {}", steamId, e);
}
return 0;
}
public Optional<SteamProfileAndGamesResponse> getFullPlayerProfile(String steamId) {
var playerOpt = getPlayerSummary(steamId);
if (playerOpt.isEmpty()) {
return Optional.empty();
}
var games = getOwnedGames(steamId);
return Optional.of(new SteamProfileAndGamesResponse(playerOpt.get(), games));
// The fallbacks run after the last try; the callers keep their "Steam failed" rule.
private int noGamesCount(String steamId, Exception e) {
log.error("Error communicating with Steam API for game count of SteamID: {}", steamId, e);
return 0;
}
// Asked on every page load for the avatar in the header.
@Cacheable(value = "steamPlayerSummary", key = "#steamId", unless = "#result == null")
@RateLimiter(name = "steamApi")
@Retry(name = "steamApi")
@Retry(name = "steamApiOnce", fallbackMethod = "noPlayerSummary")
public Optional<SteamPlayerSummaryResponse.Player> getPlayerSummary(String steamId) {
try {
log.info("Fetching Steam profile for SteamID: {}", steamId);
SteamPlayerSummaryResponse response = steamRestClient.get()
@@ -132,18 +115,18 @@ public class SteamApiService {
if (response != null && response.response() != null && !response.response().players().isEmpty()) {
return Optional.of(response.response().players().getFirst());
}
} catch (Exception e) {
log.error("Error communicating with Steam API for SteamID: {}", steamId, e);
return Optional.empty();
}
private Optional<SteamPlayerSummaryResponse.Player> noPlayerSummary(String steamId, Exception e) {
log.error("Error communicating with Steam API for SteamID: {}", steamId, e);
return Optional.empty();
}
@RateLimiter(name = "steamApi")
// No fallback: after the last try the sync stores FAILED.
@Retry(name = "steamApi")
public List<SteamOwnedGamesResponse.Game> getOwnedGames(String steamId) {
try {
log.info("Fetching owned games for SteamID: {}", steamId);
SteamOwnedGamesResponse response = steamRestClient.get()
@@ -162,11 +145,6 @@ public class SteamApiService {
if (response != null && response.response() != null && response.response().games() != null) {
return response.response().games();
}
} catch (Exception e) {
log.error("Error communicating with Steam API for games of SteamID: {}", steamId, e);
}
return Collections.emptyList();
}
@@ -183,9 +161,8 @@ public class SteamApiService {
* @return empty on a Steam error, so the caller can keep the list it has
*/
@RateLimiter(name = "steamApi")
@Retry(name = "steamApi")
@Retry(name = "steamApiOnce", fallbackMethod = "noWishlist")
public Optional<List<WishlistItem>> getWishlist(String steamId) {
try {
SteamWishlistResponse response = steamRestClient.get()
.uri(uriBuilder -> uriBuilder
.path("/IWishlistService/GetWishlist/v1/")
@@ -202,11 +179,12 @@ public class SteamApiService {
.map(item -> new WishlistItem(item.appid(),
item.dateAdded() != null ? Instant.ofEpochSecond(item.dateAdded()) : null))
.toList());
} catch (Exception e) {
}
private Optional<List<WishlistItem>> noWishlist(String steamId, Exception e) {
log.error("Error communicating with Steam API for the wishlist of SteamID: {}", steamId, e);
return Optional.empty();
}
}
/**
* Store name and current price per app, asked in batches of {@value #STORE_ITEMS_BATCH}. Apps Steam does not
@@ -0,0 +1,21 @@
package core_service.shared.api;
import com.fasterxml.jackson.annotation.JsonInclude;
import java.util.List;
/**
* The body of every error answer.
*
* @param fields the refused request fields of a {@code VALIDATION_FAILED}, else left out
*/
@JsonInclude(JsonInclude.Include.NON_NULL)
public record ErrorResponse(String code, String message, List<FieldError> fields) {
public ErrorResponse(String code, String message) {
this(code, message, null);
}
public record FieldError(String field, String message) {
}
}
@@ -0,0 +1,145 @@
package core_service.shared.api;
import lombok.extern.slf4j.Slf4j;
import org.springframework.core.MethodParameter;
import org.springframework.core.Ordered;
import org.springframework.core.annotation.Order;
import org.springframework.dao.DataIntegrityViolationException;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.http.HttpStatusCode;
import org.springframework.http.ResponseEntity;
import org.springframework.security.access.AccessDeniedException;
import org.springframework.security.core.AuthenticationException;
import org.springframework.validation.method.ParameterValidationResult;
import org.springframework.validation.FieldError;
import org.springframework.web.bind.MethodArgumentNotValidException;
import org.springframework.web.bind.MissingServletRequestParameterException;
import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestControllerAdvice;
import org.springframework.web.context.request.WebRequest;
import org.springframework.web.method.annotation.HandlerMethodValidationException;
import org.springframework.web.method.annotation.MethodArgumentTypeMismatchException;
import org.springframework.web.servlet.mvc.method.annotation.ResponseEntityExceptionHandler;
import java.sql.SQLException;
import java.util.List;
/**
* Gives every error the {@link ErrorResponse} body. Spring's own exceptions keep their status; a 400 is
* {@code VALIDATION_FAILED}. A module's own exceptions belong in its own handler.
*/
@RestControllerAdvice
@Order(Ordered.LOWEST_PRECEDENCE)
@Slf4j
public class GlobalExceptionController extends ResponseEntityExceptionHandler {
static final String VALIDATION_FAILED = "VALIDATION_FAILED";
private static final String INTERNAL_ERROR = "INTERNAL_ERROR";
// Postgres unique_violation. The state decides: the message text follows the server's language.
private static final String UNIQUE_VIOLATION = "23505";
@Override
protected ResponseEntity<Object> handleMethodArgumentNotValid(MethodArgumentNotValidException ex,
HttpHeaders headers, HttpStatusCode status,
WebRequest request) {
return validationFailed(ex.getBindingResult().getFieldErrors().stream()
.map(error -> new ErrorResponse.FieldError(error.getField(), error.getDefaultMessage()))
.toList());
}
@Override
protected ResponseEntity<Object> handleHandlerMethodValidationException(HandlerMethodValidationException ex,
HttpHeaders headers,
HttpStatusCode status,
WebRequest request) {
return validationFailed(ex.getParameterValidationResults().stream()
.flatMap(result -> result.getResolvableErrors().stream()
.map(error -> new ErrorResponse.FieldError(
error instanceof FieldError field ? field.getField() : parameterName(result),
error.getDefaultMessage())))
.toList());
}
/**
* The base keeps its checks (a committed response, the error attribute of a 500); only the body is ours.
*/
@Override
protected ResponseEntity<Object> handleExceptionInternal(Exception ex, Object body, HttpHeaders headers,
HttpStatusCode statusCode, WebRequest request) {
ResponseEntity<Object> answer = super.handleExceptionInternal(ex, body, headers, statusCode, request);
if (answer == null) {
return null;
}
ErrorResponse error;
if (statusCode.is5xxServerError()) {
log.error("Unexpected error", ex);
error = new ErrorResponse(INTERNAL_ERROR, "Unexpected error");
} else if (statusCode.value() == HttpStatus.BAD_REQUEST.value()) {
String field = ex instanceof MethodArgumentTypeMismatchException mismatch ? mismatch.getName()
: ex instanceof MissingServletRequestParameterException missing ? missing.getParameterName() : null;
error = new ErrorResponse(VALIDATION_FAILED, "The request is not valid",
field == null ? null : List.of(new ErrorResponse.FieldError(field, "Missing or not readable")));
} else {
HttpStatus status = HttpStatus.resolve(statusCode.value());
error = new ErrorResponse(status != null ? status.name() : "ERROR",
status != null ? status.getReasonPhrase() : "Error");
}
return ResponseEntity.status(answer.getStatusCode()).headers(answer.getHeaders()).body(error);
}
/**
* A duplicate key is a parallel request that won. Any other refused write (a value too long, a missing row) is a
* bug and stays a 500.
*/
@ExceptionHandler(DataIntegrityViolationException.class)
public ResponseEntity<ErrorResponse> dataIntegrity(DataIntegrityViolationException ex) throws Exception {
if (!isUniqueViolation(ex)) {
return unexpected(ex);
}
log.warn("Database refused a write: {}", ex.getMostSpecificCause().getMessage());
return ResponseEntity.status(HttpStatus.CONFLICT)
.body(new ErrorResponse("CONFLICT", "The data changed at the same time; try again"));
}
/**
* Rethrows Spring Security's exceptions: its filters answer them with 401 and 403.
*/
@ExceptionHandler(Exception.class)
public ResponseEntity<ErrorResponse> unexpected(Exception ex) throws Exception {
if (ex instanceof AccessDeniedException || ex instanceof AuthenticationException) {
throw ex;
}
log.error("Unexpected error", ex);
return ResponseEntity.internalServerError().body(new ErrorResponse(INTERNAL_ERROR, "Unexpected error"));
}
private static boolean isUniqueViolation(Throwable ex) {
for (Throwable cause = ex; cause != null; cause = cause.getCause()) {
if (cause instanceof SQLException sql && UNIQUE_VIOLATION.equals(sql.getSQLState())) {
return true;
}
}
return false;
}
private static ResponseEntity<Object> validationFailed(List<ErrorResponse.FieldError> fields) {
return ResponseEntity.badRequest().body(new ErrorResponse(VALIDATION_FAILED, "The request is not valid", fields));
}
// The name in the request, e.g. "q" of @RequestParam("q") String query.
private static String parameterName(ParameterValidationResult result) {
MethodParameter parameter = result.getMethodParameter();
RequestParam requestParam = parameter.getParameterAnnotation(RequestParam.class);
if (requestParam != null && !requestParam.name().isEmpty()) {
return requestParam.name();
}
PathVariable pathVariable = parameter.getParameterAnnotation(PathVariable.class);
if (pathVariable != null && !pathVariable.name().isEmpty()) {
return pathVariable.name();
}
return parameter.getParameterName();
}
}
@@ -7,13 +7,12 @@ import java.security.Principal;
*/
public record UserPrincipal(
String id,
String email,
Long steamId,
Role role
) implements Principal {
public UserPrincipal(String id, String email, Long steamId) {
this(id, email, steamId, Role.USER);
public UserPrincipal(String id, Long steamId) {
this(id, steamId, Role.USER);
}
@Override
+7
View File
@@ -87,6 +87,13 @@ resilience4j:
retry-exceptions:
- org.springframework.web.client.RestClientException
- org.springframework.web.client.ResourceAccessException
# A 4xx (a refused key, a private list) is the same on the next try.
ignore-exceptions:
- org.springframework.web.client.HttpClientErrorException
# Calls a page waits for, and the optional wishlist: one try, else a Steam outage holds every page (or the
# sync) for three read timeouts.
steamApiOnce:
max-attempts: 1
ratelimiter:
instances:
steamApi:
@@ -0,0 +1,51 @@
-- Imports of many games at once (item 23): a list or a CSV, matched with IGDB in the background, then saved as
-- games added by hand. A game an import added keeps its import_id, so the import can be undone.
-- The table is also the queue of the two workers (GameImportWorker):
-- QUEUED -> MATCHING -> READY (the user reviews) -> SAVING -> SAVED (-> UNDONE), or FAILED after 3 tries.
-- queued_at orders each queue. progress_at is set while a worker runs the import and after every line; NULL while
-- it waits. attempts counts the tries a watchdog or a restart stopped.
CREATE TABLE game_import (
id BIGSERIAL PRIMARY KEY,
user_id BIGINT NOT NULL,
source VARCHAR(16) NOT NULL,
status VARCHAR(16) NOT NULL,
total INT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
queued_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
progress_at TIMESTAMPTZ,
attempts INT NOT NULL DEFAULT 0,
-- While SAVING: NULL = adding the games, DETAILS = loading their time to beat and moods.
step VARCHAR(16),
saved_at TIMESTAMPTZ
);
CREATE INDEX game_import_user ON game_import (user_id);
CREATE INDEX game_import_queue ON game_import (status, queued_at);
-- One import in the matching queue per user; an import of 1000 lines takes minutes of the shared IGDB budget.
CREATE UNIQUE INDEX game_import_one_matching ON game_import (user_id) WHERE status IN ('QUEUED', 'MATCHING');
-- One row per line of the input. igdb_id: the game the user already picked in the search (source SEARCH).
-- matched_at NULL: not matched yet. The chosen_* columns hold the user's review, save_result what the save did.
CREATE TABLE game_import_row (
import_id BIGINT NOT NULL REFERENCES game_import (id) ON DELETE CASCADE,
line_no INT NOT NULL,
title VARCHAR(200),
platform VARCHAR(32),
playtime_minutes INT,
status VARCHAR(16),
igdb_id BIGINT,
matched_at TIMESTAMPTZ,
search_failed BOOLEAN NOT NULL DEFAULT FALSE,
exact BOOLEAN NOT NULL DEFAULT FALSE,
-- The IGDB hits, best first, kept whole (IgdbApiService.Game), so the save needs no IGDB call.
candidates JSONB NOT NULL DEFAULT '[]',
chosen_igdb_id BIGINT,
chosen_platform VARCHAR(32),
chosen_status VARCHAR(16),
chosen_minutes INT,
-- ADDED or SKIPPED (in the library already, or IGDB does not know the game).
save_result VARCHAR(8),
PRIMARY KEY (import_id, line_no)
);
ALTER TABLE user_game ADD COLUMN import_id BIGINT REFERENCES game_import (id) ON DELETE SET NULL;
CREATE INDEX user_game_import ON user_game (import_id) WHERE import_id IS NOT NULL;
@@ -0,0 +1,3 @@
-- The claim of a sync is committed on its own now; a RUNNING row is stale by the start of its run, not by the end of
-- the last one.
ALTER TABLE user_steam_sync ADD COLUMN sync_started_at TIMESTAMPTZ;
@@ -23,7 +23,7 @@ import static org.mockito.Mockito.lenient;
*/
class AccessTokenFiltersTest extends AbstractUnitTest {
private static final UserPrincipal USER = new UserPrincipal("42", "a@b.de", 76561198000000042L);
private static final UserPrincipal USER = new UserPrincipal("42", 76561198000000042L);
@Mock
private JwtService jwtService;
@@ -1,16 +1,15 @@
package core_service.configuration.security;
import core_service.shared.security.Role;
import core_service.shared.test.AbstractIntegrationTest;
import core_service.shared.test.ContractTokens;
import jakarta.servlet.http.Cookie;
import io.jsonwebtoken.Jwts;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.webmvc.test.autoconfigure.AutoConfigureMockMvc;
import org.springframework.test.context.ActiveProfiles;
import org.springframework.test.web.servlet.MockMvc;
import java.util.Date;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;
@@ -25,13 +24,7 @@ class ExpiredAccessTokenTest extends AbstractIntegrationTest {
private MockMvc mockMvc;
private static String expiredAccessToken() {
return Jwts.builder()
.subject("a@b.de")
.claim("userId", "42")
.issuedAt(new Date(System.currentTimeMillis() - 120_000))
.expiration(new Date(System.currentTimeMillis() - 60_000))
.signWith(TEST_KEYS.getPrivate(), Jwts.SIG.EdDSA)
.compact();
return ContractTokens.sign(ContractTokens.login(42, Role.USER, null), TEST_KEYS.getPrivate(), -60_000);
}
@Test
@@ -4,6 +4,7 @@ import core_service.shared.security.ProfileTicket;
import core_service.shared.security.Role;
import core_service.shared.security.UserPrincipal;
import core_service.shared.test.AbstractUnitTest;
import core_service.shared.test.ContractTokens;
import io.jsonwebtoken.Jwts;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
@@ -16,6 +17,7 @@ import java.security.PrivateKey;
import java.util.Base64;
import java.util.Date;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
@@ -62,7 +64,7 @@ class JwtServiceTest extends AbstractUnitTest {
void parseLoginToken_ReturnsTheUser() {
String token = signToken("0815", "test@example.com", 123456789L);
assertEquals(Optional.of(new UserPrincipal("0815", "test@example.com", 123456789L)),
assertEquals(Optional.of(new UserPrincipal("0815", 123456789L)),
jwtService.parseLoginToken(token));
}
@@ -205,4 +207,66 @@ class JwtServiceTest extends AbstractUnitTest {
assertEquals(Role.USER, jwtService.parseLoginToken(token).orElseThrow().role());
}
private String sign(Map<String, Object> claims) {
return ContractTokens.sign(claims, signingKey, 60_000);
}
// The shapes of item 29: without the claims an Auth from before item 26 needs.
private static Map<String, Object> without(Map<String, Object> claims, String... names) {
for (String name : names) {
claims.remove(name);
}
return claims;
}
@Test
void contractTokens_OfEveryKind_AreAccepted() {
assertEquals("42", jwtService.parseLoginToken(sign(ContractTokens.claims(ContractTokens.LOGIN))).orElseThrow().id());
assertEquals(42L, jwtService.verifyProfileTicket(sign(ContractTokens.claims(ContractTokens.PUBLIC_PROFILE)))
.orElseThrow().userId());
assertTrue(jwtService.isServiceToken(sign(ContractTokens.claims(ContractTokens.SERVICE))));
}
@Test
void legacyTokens_OfAnAuthFromBeforeItem26_AreAccepted() {
assertEquals("42", jwtService.parseLoginToken(sign(ContractTokens.legacyClaims(ContractTokens.LOGIN))).orElseThrow().id());
assertEquals(42L, jwtService.verifyProfileTicket(sign(ContractTokens.legacyClaims(ContractTokens.PUBLIC_PROFILE)))
.orElseThrow().userId());
assertTrue(jwtService.isServiceToken(sign(ContractTokens.legacyClaims(ContractTokens.SERVICE))));
}
@Test
void tokensWithoutTheOldClaims_AreAccepted() {
assertEquals("42", jwtService.parseLoginToken(sign(without(ContractTokens.claims(ContractTokens.LOGIN), "userId")))
.orElseThrow().id());
assertEquals(42L, jwtService.verifyProfileTicket(sign(without(ContractTokens.claims(ContractTokens.PUBLIC_PROFILE), "typ")))
.orElseThrow().userId());
assertTrue(jwtService.isServiceToken(sign(without(ContractTokens.claims(ContractTokens.SERVICE), "typ"))));
}
@Test
void aTokenForAnotherAudience_IsRejected() {
Map<String, Object> ticketAsLogin = ContractTokens.claims(ContractTokens.LOGIN);
ticketAsLogin.put("aud", List.of(JwtService.PROFILE_TICKET_AUDIENCE));
Map<String, Object> loginAsService = ContractTokens.claims(ContractTokens.SERVICE);
loginAsService.put("aud", List.of(JwtService.LOGIN_AUDIENCE));
Map<String, Object> loginAsTicket = ContractTokens.claims(ContractTokens.PUBLIC_PROFILE);
loginAsTicket.put("aud", List.of(JwtService.LOGIN_AUDIENCE));
assertEquals(Optional.empty(), jwtService.parseLoginToken(sign(ticketAsLogin)));
assertFalse(jwtService.isServiceToken(sign(loginAsService)));
assertEquals(Optional.empty(), jwtService.verifyProfileTicket(sign(loginAsTicket)));
}
@Test
void aTokenWithAnAudienceButAnotherIssuer_IsRejected() {
Map<String, Object> login = ContractTokens.claims(ContractTokens.LOGIN);
login.put("iss", "someone-else");
Map<String, Object> service = ContractTokens.claims(ContractTokens.SERVICE);
service.remove("iss");
assertEquals(Optional.empty(), jwtService.parseLoginToken(sign(login)));
assertFalse(jwtService.isServiceToken(sign(service)));
}
}
@@ -1,6 +1,6 @@
package core_service.configuration.security;
import io.jsonwebtoken.Jwts;
import core_service.shared.test.ContractTokens;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
@@ -19,8 +19,8 @@ import org.testcontainers.junit.jupiter.Testcontainers;
import java.security.KeyPair;
import java.security.KeyPairGenerator;
import java.util.Base64;
import java.util.Date;
import java.util.List;
import java.util.Map;
import java.util.Set;
import static org.junit.jupiter.api.Assertions.assertTrue;
@@ -77,9 +77,10 @@ class PublicEndpointsTest {
@Test
void profileTicketIsNoLogin() throws Exception {
String ticket = Jwts.builder().subject("public-profile").claim("typ", "public-profile").claim("profileUserId", "1")
.claim("userId", "1").expiration(new Date(System.currentTimeMillis() + 60_000))
.signWith(KEYS.getPrivate(), Jwts.SIG.EdDSA).compact();
Map<String, Object> claims = ContractTokens.ticket(1, null, null);
// A ticket that also carries a user id is still no login.
claims.put("userId", "1");
String ticket = ContractTokens.sign(claims, KEYS.getPrivate(), 60_000);
int status = mockMvc.perform(get("/api/library/games-count").header("Authorization", "Bearer " + ticket))
.andReturn().getResponse().getStatus();
@@ -95,9 +96,8 @@ class PublicEndpointsTest {
// Without a login: a class role marker on the way would answer 401.
@Test
void publicProfileWithSharedStats_WorksWithoutALogin() throws Exception {
String ticket = Jwts.builder().subject("public-profile").claim("typ", "public-profile").claim("profileUserId", "1")
.claim("profileStats", List.of("TOP_GAMES", "OVERVIEW")).expiration(new Date(System.currentTimeMillis() + 60_000))
.signWith(KEYS.getPrivate(), Jwts.SIG.EdDSA).compact();
String ticket = ContractTokens.sign(ContractTokens.ticket(1, List.of("TOP_GAMES", "OVERVIEW"), null),
KEYS.getPrivate(), 60_000);
mockMvc.perform(get("/api/public/profiles/library").header("X-Profile-Ticket", ticket))
.andExpect(status().isOk())
@@ -4,7 +4,7 @@ import core_service.shared.security.Public;
import core_service.shared.security.RequiresRole;
import core_service.shared.security.Role;
import core_service.shared.test.AbstractIntegrationTest;
import io.jsonwebtoken.Jwts;
import core_service.shared.test.ContractTokens;
import jakarta.servlet.http.Cookie;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
@@ -18,8 +18,6 @@ import org.springframework.test.web.servlet.request.MockHttpServletRequestBuilde
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
import java.util.Date;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;
@@ -93,13 +91,7 @@ class RoleCheckFlowTest extends AbstractIntegrationTest {
private ProbeService probeService;
private static String token(Role role) {
return Jwts.builder()
.subject("probe@example.com")
.claim("userId", "4711")
.claim("role", role.name())
.expiration(new Date(System.currentTimeMillis() + 60_000))
.signWith(TEST_KEYS.getPrivate(), Jwts.SIG.EdDSA)
.compact();
return ContractTokens.loginToken(4711, role);
}
private ResultActions call(String path, Role role) throws Exception {
@@ -1,25 +1,29 @@
package core_service.modules.account.api;
import core_service.shared.test.AbstractIntegrationTest;
import io.jsonwebtoken.JwtBuilder;
import io.jsonwebtoken.Jwts;
import core_service.shared.test.ContractTokens;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.webmvc.test.autoconfigure.AutoConfigureMockMvc;
import org.springframework.http.HttpMethod;
import org.springframework.http.MediaType;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.test.context.ActiveProfiles;
import org.springframework.test.web.servlet.MockMvc;
import org.springframework.test.web.servlet.ResultActions;
import org.springframework.test.web.servlet.request.MockHttpServletRequestBuilder;
import tools.jackson.databind.JsonNode;
import java.util.Date;
import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.post;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.put;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.request;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.jsonPath;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;
@@ -40,20 +44,15 @@ class InternalAccountControllerTest extends AbstractIntegrationTest {
@AfterEach
void cleanUp() {
jdbc.update("delete from deleted_user where user_id = ?", USER_ID);
}
private static String token(JwtBuilder builder) {
return builder.expiration(new Date(System.currentTimeMillis() + 60_000))
.signWith(TEST_KEYS.getPrivate(), Jwts.SIG.EdDSA).compact();
jdbc.update("delete from deleted_user where user_id in (?, 42)", USER_ID);
}
private static String serviceToken() {
return token(Jwts.builder().subject("tbp-auth").claim("typ", "service"));
return ContractTokens.serviceToken();
}
private static String loginToken() {
return token(Jwts.builder().subject("a@b.de").claim("userId", String.valueOf(USER_ID)));
return ContractTokens.loginToken(USER_ID);
}
private ResultActions send(MockHttpServletRequestBuilder request, String body, String token) throws Exception {
@@ -107,6 +106,38 @@ class InternalAccountControllerTest extends AbstractIntegrationTest {
@Test
void prune_WithAnEmptyListIs400() throws Exception {
send(post("/internal/accounts/prune"), "{\"knownUserIds\":[],\"dryRun\":false}", serviceToken()).andExpect(status().isBadRequest());
send(post("/internal/accounts/prune"), "{\"knownUserIds\":[],\"dryRun\":false}", serviceToken())
.andExpect(status().isBadRequest())
.andExpect(jsonPath("$.code").value("VALIDATION_FAILED"))
.andExpect(jsonPath("$.fields[0].field").value("knownUserIds"));
}
private static MockHttpServletRequestBuilder contractRequest(JsonNode call) {
return request(HttpMethod.valueOf(call.path("method").asString()), call.path("path").asString());
}
// The calls as Auth's CoreAccountService sends them, with the service tokens of an old and a new Auth.
@ParameterizedTest
@ValueSource(booleans = {false, true})
void contractCalls_AreAnswered(boolean legacyToken) throws Exception {
String token = legacyToken ? ContractTokens.sign(ContractTokens.legacyClaims(ContractTokens.SERVICE))
: ContractTokens.serviceToken();
// replaceDeleted with [] removes every tombstone, also those of other tests in this database.
List<Long> tombstones = jdbc.queryForList("select user_id from deleted_user", Long.class);
try {
for (String name : List.of("markDeleted", "replaceDeleted")) {
JsonNode call = ContractTokens.internalCall(name);
send(contractRequest(call), call.path("request").toString(), token).andExpect(status().isNoContent());
}
} finally {
tombstones.forEach(userId -> jdbc.update("insert into deleted_user (user_id) values (?) on conflict do nothing", userId));
}
for (String name : List.of("purge", "prune")) {
JsonNode call = ContractTokens.internalCall(name);
ResultActions result = send(contractRequest(call), call.path("request").toString(), token).andExpect(status().isOk());
for (String field : call.path("response").propertyNames()) {
result.andExpect(jsonPath("$." + field).exists());
}
}
}
}
@@ -17,7 +17,7 @@ class AccountCleanupRepositoryTest extends AbstractPersistenceTest {
private static final List<String> USER_TABLES = List.of("user_game_status_change", "user_game_achievement",
"user_game_achievement_meta", "user_game_playtime_day", "wishlist_game", "backlog_snapshot",
"user_steam_sync", "user_game");
"user_steam_sync", "user_game", "game_import");
@Autowired
private AccountCleanupRepository repository;
@@ -42,6 +42,7 @@ class AccountCleanupRepositoryTest extends AbstractPersistenceTest {
jdbc.update("""
insert into user_steam_sync (user_id, steam_id, user_seen_at) values (?, '76561198000000001', now())""",
userId);
jdbc.update("insert into game_import (user_id, source, status, total) values (?, 'TEXT', 'SAVED', 0)", userId);
}
}
@@ -113,4 +114,13 @@ class AccountCleanupRepositoryTest extends AbstractPersistenceTest {
assertThat(repository.findOrphans(List.of(2L))).containsExactly(1L, 7L);
assertThat(repository.findOrphans(List.of(1L, 2L, 7L))).isEmpty();
}
// A new table with user rows must be in the purge list, or "Delete my account" leaves its rows behind.
@Test
void tables_CoverEveryTableWithAUserId() {
List<String> withUserId = jdbc.queryForList("select table_name from information_schema.columns "
+ "where table_schema = 'public' and column_name = 'user_id' order by table_name", String.class);
assertThat(AccountCleanupRepository.TABLES).containsExactlyInAnyOrderElementsOf(withUserId);
}
}
@@ -11,7 +11,7 @@ import core_service.modules.library.repo.SteamGameRepository;
import core_service.modules.library.repo.UserGameRepository;
import core_service.shared.security.Role;
import core_service.shared.test.AbstractIntegrationTest;
import io.jsonwebtoken.Jwts;
import core_service.shared.test.ContractTokens;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
@@ -22,7 +22,6 @@ import org.springframework.test.web.servlet.MockMvc;
import org.springframework.test.web.servlet.ResultActions;
import java.time.Instant;
import java.util.Date;
import static org.assertj.core.api.Assertions.assertThat;
import static org.hamcrest.Matchers.contains;
@@ -77,13 +76,7 @@ class GameCorrectionFlowTest extends AbstractIntegrationTest {
}
private static String token(Role role) {
return Jwts.builder()
.subject("auditor@example.com")
.claim("userId", String.valueOf(AUDITOR_ID))
.claim("role", role.name())
.expiration(new Date(System.currentTimeMillis() + 60_000))
.signWith(TEST_KEYS.getPrivate(), Jwts.SIG.EdDSA)
.compact();
return ContractTokens.loginToken(AUDITOR_ID, role);
}
private ResultActions correct(Role role, long appid, String field, String valueJson) throws Exception {
@@ -0,0 +1,410 @@
package core_service.modules.library.api;
import core_service.configuration.security.ClientHeaderFilter;
import core_service.modules.igdb_adapter.service.IgdbApiService;
import core_service.modules.library.domain.GameSource;
import core_service.modules.library.domain.GameStatus;
import core_service.modules.library.domain.UserGame;
import core_service.modules.library.domain.UserGameId;
import core_service.modules.library.repo.UserGameRepository;
import core_service.modules.library.service.GameImportWorker;
import core_service.shared.test.AbstractIntegrationTest;
import com.jayway.jsonpath.JsonPath;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.webmvc.test.autoconfigure.AutoConfigureMockMvc;
import org.springframework.http.MediaType;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.test.context.bean.override.mockito.MockitoBean;
import org.springframework.test.web.servlet.MockMvc;
import org.springframework.test.web.servlet.ResultActions;
import java.time.Instant;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import static org.hamcrest.Matchers.contains;
import static org.hamcrest.Matchers.nullValue;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyLong;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.delete;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.post;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.jsonPath;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;
/**
* The dev profile authenticates every request as user 1.
*/
@AutoConfigureMockMvc
class GameImportControllerTest extends AbstractIntegrationTest {
private static final long USER_ID = 1L;
private static final IgdbApiService.Game ZELDA = new IgdbApiService.Game(119388, "Zelda: Tears of the Kingdom", 2023,
null, List.of("Switch"), null);
private static final IgdbApiService.Game ELDEN_RING = new IgdbApiService.Game(119133, "Elden Ring", 2022,
null, List.of("PS5", "PC"), 1245620L);
private static final IgdbApiService.Game NIGHTREIGN = new IgdbApiService.Game(325591, "Elden Ring Nightreign", 2025,
null, List.of("PS5", "PC"), 2622380L);
@Autowired
private MockMvc mockMvc;
@Autowired
private UserGameRepository userGameRepository;
@Autowired
private GameImportWorker worker;
@Autowired
private JdbcTemplate jdbc;
@MockitoBean
private IgdbApiService igdbApiService;
@BeforeEach
void seed() {
jdbc.update("delete from game_alias");
userGameRepository.deleteAll();
jdbc.update("delete from game_import");
when(igdbApiService.isEnabled()).thenReturn(true);
when(igdbApiService.searchGames("Elden Ring")).thenReturn(Optional.of(List.of(NIGHTREIGN, ELDEN_RING)));
when(igdbApiService.searchGames("Zelda Tears")).thenReturn(Optional.of(List.of(ZELDA)));
when(igdbApiService.getGame(119388)).thenReturn(Optional.of(ZELDA));
when(igdbApiService.getTimeToBeat(any())).thenReturn(Optional.of(Map.of()));
when(igdbApiService.getTimeToBeatByGameIds(any())).thenReturn(Optional.of(Map.of()));
}
private ResultActions create(String json) throws Exception {
return mockMvc.perform(post("/api/library/imports")
.header(ClientHeaderFilter.CLIENT_HEADER, "test")
.contentType(MediaType.APPLICATION_JSON)
.content(json));
}
private long createReady(String json) throws Exception {
long id = ((Number) JsonPath.read(create(json).andExpect(status().isAccepted())
.andReturn().getResponse().getContentAsString(), "$.id")).longValue();
awaitReady(id);
return id;
}
private void awaitReady(long id) throws InterruptedException {
awaitStatus(id, "READY");
}
// The matching and the save run in the background.
private void awaitStatus(long id, String expected) throws InterruptedException {
for (int attempt = 0; attempt < 50; attempt++) {
if (expected.equals(jdbc.queryForObject("select status from game_import where id = ?", String.class, id))) {
return;
}
Thread.sleep(100);
}
throw new AssertionError("Import " + id + " not " + expected);
}
private ResultActions save(long id, String rows) throws Exception {
return mockMvc.perform(post("/api/library/imports/" + id + "/save")
.header(ClientHeaderFilter.CLIENT_HEADER, "test")
.contentType(MediaType.APPLICATION_JSON)
.content("{\"rows\": [" + rows + "]}"));
}
private void saveAndWait(long id, String rows) throws Exception {
save(id, rows).andExpect(status().isAccepted());
awaitStatus(id, "SAVED");
}
// Not woken: no worker takes it unless the test wakes one.
private long insertImport(long userId, String status, String set) {
long id = jdbc.queryForObject("insert into game_import (user_id, source, status, total) "
+ "values (?, 'TEXT', ?, 0) returning id", Long.class, userId, status);
if (!set.isEmpty()) {
jdbc.update("update game_import set " + set + " where id = ?", id);
}
return id;
}
private UserGame stored(long appid) {
return userGameRepository.findById(new UserGameId(USER_ID, appid)).orElseThrow();
}
private Long importIdOf(long appid) {
return jdbc.queryForObject("select import_id from user_game where user_id = ? and appid = ?", Long.class,
USER_ID, appid);
}
@Test
void create_MatchesInTheBackground_SameNameFirstAndMarksGamesInTheLibrary() throws Exception {
jdbc.update("insert into steam_game (appid, name) values (1245620, 'ELDEN RING') on conflict do nothing");
userGameRepository.save(UserGame.manual(USER_ID, 1245620L, "PC", GameStatus.BACKLOG, 0, Instant.now()));
long id = createReady("{\"source\": \"TEXT\", \"rows\": [{\"title\": \" Elden Ring \", \"platform\": \"PS5\"}, "
+ "{\"title\": \"Unknown Game\"}]}");
mockMvc.perform(get("/api/library/imports/" + id))
.andExpect(status().isOk())
.andExpect(jsonPath("$.total").value(2))
.andExpect(jsonPath("$.matched").value(2))
.andExpect(jsonPath("$.rows[0].title").value("Elden Ring"))
.andExpect(jsonPath("$.rows[0].platform").value("PS5"))
.andExpect(jsonPath("$.rows[0].exact").value(true))
.andExpect(jsonPath("$.rows[0].candidates[*].name").value(contains("Elden Ring", "Elden Ring Nightreign")))
.andExpect(jsonPath("$.rows[0].candidates[*].inLibrary").value(contains(true, false)))
// Mockito answers every other search with empty: IGDB failed.
.andExpect(jsonPath("$.rows[1].searchFailed").value(true))
.andExpect(jsonPath("$.rows[1].candidates").isEmpty());
mockMvc.perform(get("/api/library/imports"))
.andExpect(jsonPath("$.imports[0].id").value(id))
.andExpect(jsonPath("$.imports[0].status").value("READY"))
.andExpect(jsonPath("$.imports[0].rows").value(nullValue()));
}
@Test
void create_SearchRow_TakesThePickedGameWithoutASearch() throws Exception {
long id = createReady("{\"source\": \"SEARCH\", \"rows\": [{\"igdbId\": 119388}]}");
mockMvc.perform(get("/api/library/imports/" + id))
.andExpect(jsonPath("$.rows[0].exact").value(true))
.andExpect(jsonPath("$.rows[0].candidates[0].igdbId").value(119388));
verify(igdbApiService, never()).searchGames(any());
}
@Test
void create_TooManyRowsOrNoTitle_Is400() throws Exception {
String row = "{\"title\": \"Game\"}";
create("{\"source\": \"TEXT\", \"rows\": [" + String.join(",", Collections.nCopies(1001, row)) + "]}")
.andExpect(status().isBadRequest());
create("{\"source\": \"TEXT\", \"rows\": [{\"platform\": \"PC\"}]}").andExpect(status().isBadRequest());
}
@Test
void create_WhileAnotherImportIsMatching_Is409() throws Exception {
jdbc.update("insert into game_import (user_id, source, status, total) values (?, 'TEXT', 'MATCHING', 0)", USER_ID);
create("{\"source\": \"TEXT\", \"rows\": [{\"title\": \"Elden Ring\"}]}")
.andExpect(status().isConflict())
.andExpect(jsonPath("$.code").value("IMPORT_RUNNING"));
}
@Test
void get_ImportOfAnotherUser_Is404() throws Exception {
Long id = jdbc.queryForObject("insert into game_import (user_id, source, status, total) "
+ "values (2, 'TEXT', 'READY', 0) returning id", Long.class);
mockMvc.perform(get("/api/library/imports/" + id)).andExpect(status().isNotFound());
save(id, "").andExpect(status().isNotFound());
mockMvc.perform(delete("/api/library/imports/" + id + "/games").header(ClientHeaderFilter.CLIENT_HEADER, "test"))
.andExpect(status().isNotFound());
}
@Test
void save_AddsTheCheckedLinesWithTheImportId_SkipsDuplicatesAndOnlyOnce() throws Exception {
long id = createReady("{\"source\": \"CSV\", \"rows\": [{\"title\": \"Elden Ring\"}, {\"title\": \"Elden Ring\"}]}");
when(igdbApiService.getGame(anyLong())).thenReturn(Optional.empty());
saveAndWait(id, "{\"lineNo\": 1, \"igdbId\": 119133, \"platform\": \" PS5 \", \"status\": \"PLAYING\", \"playtimeMinutes\": 90},"
+ "{\"lineNo\": 2, \"igdbId\": 119133, \"platform\": \"PC\", \"status\": \"BACKLOG\"},"
+ "{\"lineNo\": 9, \"igdbId\": 119133, \"platform\": \"PC\", \"status\": \"BACKLOG\"}");
mockMvc.perform(get("/api/library/imports/" + id))
.andExpect(jsonPath("$.chosen").value(2))
.andExpect(jsonPath("$.saved").value(2))
.andExpect(jsonPath("$.added").value(1))
.andExpect(jsonPath("$.step").value(nullValue()))
.andExpect(jsonPath("$.savedAt").isNotEmpty());
UserGame game = stored(1245620L);
assertEquals(GameSource.MANUAL, game.getSource());
assertEquals("PS5", game.getPlatform());
assertEquals(GameStatus.PLAYING, game.getStatus());
assertEquals(90, game.getPlaytimeMinutes());
assertEquals(id, importIdOf(1245620L));
// The candidate was stored whole: no IGDB call for the save.
verify(igdbApiService, never()).getGame(119133);
save(id, "").andExpect(status().isConflict());
}
@Test
void save_GameFromANewSearch_IsReadFromIgdb() throws Exception {
long id = createReady("{\"source\": \"TEXT\", \"rows\": [{\"title\": \"Elden Ring\"}]}");
saveAndWait(id, "{\"lineNo\": 1, \"igdbId\": 119388, \"platform\": \"Switch\", \"status\": \"BACKLOG\"}");
assertEquals(id, importIdOf(-119388L));
}
@Test
void undo_RemovesTheGamesStillAddedByHand_AndCountsTheChangedOnes() throws Exception {
long id = createReady("{\"source\": \"TEXT\", \"rows\": [{\"title\": \"Elden Ring\"}, {\"title\": \"Zelda Tears\"}]}");
saveAndWait(id, "{\"lineNo\": 1, \"igdbId\": 119133, \"platform\": \"PC\", \"status\": \"BACKLOG\"},"
+ "{\"lineNo\": 2, \"igdbId\": 119388, \"platform\": \"Switch\", \"status\": \"BACKLOG\"}");
mockMvc.perform(get("/api/library/imports/" + id + "/undo-check"))
.andExpect(jsonPath("$.games").value(2))
.andExpect(jsonPath("$.changed").value(0));
jdbc.update("update user_game set status = 'PLAYING', status_set_by_user = true where appid = -119388");
mockMvc.perform(get("/api/library/imports/" + id + "/undo-check"))
.andExpect(jsonPath("$.changed").value(1));
// The Steam sync took Elden Ring over: the user owns it on Steam now.
jdbc.update("update user_game set source = 'STEAM' where appid = 1245620");
mockMvc.perform(get("/api/library/imports/" + id + "/undo-check"))
.andExpect(jsonPath("$.games").value(1));
mockMvc.perform(delete("/api/library/imports/" + id + "/games").header(ClientHeaderFilter.CLIENT_HEADER, "test"))
.andExpect(status().isNoContent());
assertTrue(userGameRepository.findById(new UserGameId(USER_ID, -119388L)).isEmpty());
assertEquals(GameSource.STEAM, stored(1245620L).getSource());
assertEquals("UNDONE", jdbc.queryForObject("select status from game_import where id = ?", String.class, id));
mockMvc.perform(delete("/api/library/imports/" + id + "/games").header(ClientHeaderFilter.CLIENT_HEADER, "test"))
.andExpect(status().isConflict());
}
@Test
void resume_CountsTheStoppedTryAndGoesOnWhereItStopped() throws Exception {
// Before this app started: a try the restart stopped.
long id = insertImport(USER_ID, "MATCHING", "progress_at = now() - interval '1 day'");
jdbc.update("insert into game_import_row (import_id, line_no, title, matched_at, exact) "
+ "values (?, 1, 'Done before', now(), true)", id);
jdbc.update("insert into game_import_row (import_id, line_no, title) values (?, 2, 'Elden Ring')", id);
worker.resume();
awaitReady(id);
verify(igdbApiService, never()).searchGames("Done before");
assertEquals(2, jdbc.queryForObject("select jsonb_array_length(candidates) from game_import_row "
+ "where import_id = ? and line_no = 2", Integer.class, id));
assertEquals(1, jdbc.queryForObject("select attempts from game_import where id = ?", Integer.class, id));
}
@Test
void watchdog_StopsATryWithoutProgress_AndFailsAfterThreeTries() throws Exception {
long stalled = insertImport(USER_ID, "MATCHING", "progress_at = now() - interval '10 minutes'");
long lastTry = insertImport(2L, "SAVING", "progress_at = now() - interval '10 minutes', attempts = 2");
long fine = insertImport(3L, "MATCHING", "progress_at = now()");
worker.watchdog();
awaitReady(stalled);
assertEquals(1, jdbc.queryForObject("select attempts from game_import where id = ?", Integer.class, stalled));
assertEquals("FAILED", jdbc.queryForObject("select status from game_import where id = ?", String.class, lastTry));
assertEquals(0, jdbc.queryForObject("select attempts from game_import where id = ?", Integer.class, fine));
}
@Test
void retry_PutsAFailedImportBackInTheQueueOfItsPhase() throws Exception {
long id = insertImport(USER_ID, "FAILED", "attempts = 3");
jdbc.update("insert into game_import_row (import_id, line_no, title) values (?, 1, 'Elden Ring')", id);
mockMvc.perform(post("/api/library/imports/" + id + "/retry").header(ClientHeaderFilter.CLIENT_HEADER, "test"))
.andExpect(status().isAccepted());
awaitReady(id);
assertEquals(0, jdbc.queryForObject("select attempts from game_import where id = ?", Integer.class, id));
mockMvc.perform(post("/api/library/imports/" + id + "/retry").header(ClientHeaderFilter.CLIENT_HEADER, "test"))
.andExpect(status().isConflict())
.andExpect(jsonPath("$.code").value("IMPORT_NOT_FAILED"));
}
@Test
void get_WaitingImport_ShowsHowManyImportsAreBeforeIt() throws Exception {
insertImport(2L, "QUEUED", "queued_at = now() - interval '2 minutes'");
insertImport(3L, "MATCHING", "queued_at = now() - interval '1 minutes', progress_at = now()");
long mine = insertImport(USER_ID, "QUEUED", "");
long saving = insertImport(USER_ID, "SAVING", "");
mockMvc.perform(get("/api/library/imports/" + mine))
.andExpect(jsonPath("$.status").value("QUEUED"))
.andExpect(jsonPath("$.ahead").value(2));
mockMvc.perform(get("/api/library/imports/" + saving)).andExpect(jsonPath("$.ahead").value(0));
jdbc.update("delete from game_import");
}
@Test
void list_AnswersAnObjectWithTheImports() throws Exception {
long id = insertImport(USER_ID, "READY", "");
mockMvc.perform(get("/api/library/imports"))
.andExpect(status().isOk())
.andExpect(jsonPath("$.imports[0].id").value(id));
}
// A lane can start before ApplicationReadyEvent (an early create, the first watchdog run).
@Test
void resume_DoesNotCountAnImportALaneRunsNow() throws Exception {
CountDownLatch release = new CountDownLatch(1);
when(igdbApiService.searchGames("Slow Game")).thenAnswer(invocation -> {
release.await(5, TimeUnit.SECONDS);
return Optional.of(List.of());
});
long id = insertImport(USER_ID, "QUEUED", "");
jdbc.update("insert into game_import_row (import_id, line_no, title) values (?, 1, 'Slow Game')", id);
worker.wakeMatching();
awaitStatus(id, "MATCHING");
try {
worker.resume();
} finally {
release.countDown();
}
awaitReady(id);
assertEquals(0, jdbc.queryForObject("select attempts from game_import where id = ?", Integer.class, id));
}
@Test
void save_RevivedRowOfTheLibrary_IsNotRemovedByTheUndo() throws Exception {
jdbc.update("insert into steam_game (appid, name) values (1245620, 'ELDEN RING') on conflict do nothing");
jdbc.update("insert into user_game (user_id, appid, status, rating, notes, source, removed_at) "
+ "values (?, 1245620, 'FINISHED', 5, 'Loved it', 'STEAM', now())", USER_ID);
long id = createReady("{\"source\": \"TEXT\", \"rows\": [{\"title\": \"Elden Ring\"}]}");
saveAndWait(id, "{\"lineNo\": 1, \"igdbId\": 119133, \"platform\": \"PS5\", \"status\": \"PLAYING\"}");
mockMvc.perform(delete("/api/library/imports/" + id + "/games").header(ClientHeaderFilter.CLIENT_HEADER, "test"))
.andExpect(status().isNoContent());
UserGame kept = stored(1245620L);
assertEquals(5, kept.getRating());
assertEquals("Loved it", kept.getNotes());
assertNull(importIdOf(1245620L));
}
@Test
void save_IgdbFailsForAPickFromANewSearch_KeepsTheLineForTheNextTry() throws Exception {
long id = createReady("{\"source\": \"TEXT\", \"rows\": [{\"title\": \"Elden Ring\"}]}");
when(igdbApiService.getGame(119388)).thenReturn(Optional.empty());
save(id, "{\"lineNo\": 1, \"igdbId\": 119388, \"platform\": \"Switch\", \"status\": \"BACKLOG\"}")
.andExpect(status().isAccepted());
awaitStatus(id, "FAILED");
assertNull(jdbc.queryForObject("select save_result from game_import_row where import_id = ?", String.class, id));
assertEquals(3, jdbc.queryForObject("select attempts from game_import where id = ?", Integer.class, id));
}
@Test
void undo_ImportThatFailedWhileSaving_RemovesTheGamesItAdded() throws Exception {
long id = insertImport(USER_ID, "FAILED", "attempts = 3");
jdbc.update("insert into steam_game (appid, name) values (-119388, 'Zelda') on conflict do nothing");
jdbc.update("insert into user_game (user_id, appid, status, source, platform, import_id) "
+ "values (?, -119388, 'BACKLOG', 'MANUAL', 'Switch', ?)", USER_ID, id);
mockMvc.perform(delete("/api/library/imports/" + id + "/games").header(ClientHeaderFilter.CLIENT_HEADER, "test"))
.andExpect(status().isNoContent());
assertTrue(userGameRepository.findById(new UserGameId(USER_ID, -119388L)).isEmpty());
}
}
@@ -7,16 +7,16 @@ import core_service.modules.library.domain.GameStatus;
import core_service.modules.library.domain.SteamGame;
import core_service.modules.library.domain.UserGame;
import core_service.modules.library.domain.UserGameId;
import core_service.modules.library.domain.UserStats;
import core_service.modules.library.repo.GameCorrectionRepository;
import core_service.modules.library.repo.GameMoodsRepository;
import core_service.modules.library.repo.GameTimeToBeatRepository;
import core_service.modules.library.repo.SteamGameRepository;
import core_service.modules.library.repo.UserGameRepository;
import core_service.modules.library.repo.UserStatsRepository;
import core_service.modules.library.domain.UserStats;
import core_service.shared.security.Role;
import core_service.shared.test.AbstractIntegrationTest;
import io.jsonwebtoken.Jwts;
import core_service.shared.test.ContractTokens;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
@@ -30,7 +30,6 @@ import org.springframework.test.web.servlet.ResultActions;
import java.time.Instant;
import java.time.ZoneId;
import java.time.temporal.ChronoUnit;
import java.util.Date;
import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
@@ -94,13 +93,7 @@ class GameMergeFlowTest extends AbstractIntegrationTest {
}
private static String token(Role role) {
return Jwts.builder()
.subject("auditor@example.com")
.claim("userId", String.valueOf(AUDITOR_ID))
.claim("role", role.name())
.expiration(new Date(System.currentTimeMillis() + 60_000))
.signWith(TEST_KEYS.getPrivate(), Jwts.SIG.EdDSA)
.compact();
return ContractTokens.loginToken(AUDITOR_ID, role);
}
private ResultActions merge(Role role, long appid, long otherAppid) throws Exception {
@@ -14,17 +14,28 @@ import core_service.shared.test.AbstractIntegrationTest;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.dao.DataIntegrityViolationException;
import org.junit.jupiter.api.extension.ExtendWith;
import org.springframework.boot.test.system.CapturedOutput;
import org.springframework.boot.test.system.OutputCaptureExtension;
import org.springframework.http.converter.HttpMessageNotWritableException;
import org.springframework.boot.webmvc.test.autoconfigure.AutoConfigureMockMvc;
import org.springframework.test.context.bean.override.mockito.MockitoBean;
import org.springframework.test.web.servlet.MockMvc;
import java.sql.SQLException;
import java.time.Instant;
import java.util.List;
import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.not;
import static org.hamcrest.Matchers.nullValue;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.verify;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.content;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.jsonPath;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;
@@ -102,6 +113,50 @@ class LibraryControllerStatsTest extends AbstractIntegrationTest {
@Test
void stats_UnknownGroup_Is400() throws Exception {
mockMvc.perform(get("/api/library/stats").param("groups", "MONEY"))
.andExpect(status().isBadRequest());
.andExpect(status().isBadRequest())
.andExpect(jsonPath("$.code").value("VALIDATION_FAILED"))
.andExpect(jsonPath("$.fields[0].field").value("groups"));
}
@Test
void anUnexpectedError_Is500WithoutDetails() throws Exception {
doThrow(new IllegalStateException("secret detail")).when(moodService).refreshListedInBackground(USER_ID);
mockMvc.perform(get("/api/library/stats"))
.andExpect(status().isInternalServerError())
.andExpect(jsonPath("$.code").value("INTERNAL_ERROR"))
.andExpect(content().string(not(containsString("secret detail"))));
}
@Test
void aDatabaseConflict_Is409() throws Exception {
doThrow(new DataIntegrityViolationException("doppelter Schlüsselwert", new SQLException("doppelter Schlüsselwert", "23505"))).when(moodService).refreshListedInBackground(USER_ID);
mockMvc.perform(get("/api/library/stats"))
.andExpect(status().isConflict())
.andExpect(jsonPath("$.code").value("CONFLICT"));
}
@Test
void anotherRefusedWrite_Is500() throws Exception {
doThrow(new DataIntegrityViolationException("Wert zu lang", new SQLException("Wert zu lang", "22001")))
.when(moodService).refreshListedInBackground(USER_ID);
mockMvc.perform(get("/api/library/stats"))
.andExpect(status().isInternalServerError())
.andExpect(jsonPath("$.code").value("INTERNAL_ERROR"));
}
@Test
@ExtendWith(OutputCaptureExtension.class)
void aSpringServerError_IsLoggedAsInternalError(CapturedOutput output) throws Exception {
doThrow(new HttpMessageNotWritableException("body not writable"))
.when(moodService).refreshListedInBackground(USER_ID);
mockMvc.perform(get("/api/library/stats"))
.andExpect(status().isInternalServerError())
.andExpect(jsonPath("$.code").value("INTERNAL_ERROR"));
assertTrue(output.getAll().contains("body not writable"), "the 500 left no trace in the log");
}
}
@@ -2,7 +2,7 @@ package core_service.modules.library.api;
import core_service.modules.library.repo.UserSteamSyncRepository;
import core_service.shared.test.AbstractIntegrationTest;
import io.jsonwebtoken.Jwts;
import core_service.shared.test.ContractTokens;
import jakarta.servlet.http.Cookie;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
@@ -13,7 +13,6 @@ import org.springframework.test.web.servlet.MockMvc;
import java.time.Duration;
import java.time.Instant;
import java.util.Date;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
@@ -42,12 +41,7 @@ class LibraryControllerSteamIdGoneTest extends AbstractIntegrationTest {
}
private static Cookie loginWithoutSteamId() {
return new Cookie("tbp_access", Jwts.builder()
.subject("unlinked@example.com")
.claim("userId", String.valueOf(USER_ID))
.expiration(new Date(System.currentTimeMillis() + 60_000))
.signWith(TEST_KEYS.getPrivate(), Jwts.SIG.EdDSA)
.compact());
return new Cookie("tbp_access", ContractTokens.loginToken(USER_ID));
}
@Test
@@ -3,8 +3,8 @@ package core_service.modules.library.api;
import core_service.configuration.security.ClientHeaderFilter;
import core_service.modules.igdb_adapter.service.IgdbApiService;
import core_service.modules.library.domain.GameSource;
import core_service.modules.library.domain.GameTimeToBeat;
import core_service.modules.library.domain.GameStatus;
import core_service.modules.library.domain.GameTimeToBeat;
import core_service.modules.library.domain.SteamGame;
import core_service.modules.library.domain.UserGame;
import core_service.modules.library.domain.UserGameId;
@@ -12,6 +12,7 @@ import core_service.modules.library.repo.GameTimeToBeatRepository;
import core_service.modules.library.repo.SteamGameRepository;
import core_service.modules.library.repo.UserGameRepository;
import core_service.shared.test.AbstractIntegrationTest;
import core_service.shared.test.ContractTokens;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
@@ -215,6 +216,14 @@ class ManualGameControllerTest extends AbstractIntegrationTest {
add(119388, " ").andExpect(status().isBadRequest());
}
@Test
void search_OneLetter_Is400WithTheField() throws Exception {
mockMvc.perform(get("/api/library/game-search").param("q", "a"))
.andExpect(status().isBadRequest())
.andExpect(jsonPath("$.code").value("VALIDATION_FAILED"))
.andExpect(jsonPath("$.fields[0].field").value("q"));
}
@Test
void search_MarksGamesAlreadyInTheLibrary() throws Exception {
steamGameRepository.save(new SteamGame(1245620L, "ELDEN RING", null, Instant.now()));
@@ -342,12 +351,7 @@ class ManualGameControllerTest extends AbstractIntegrationTest {
// Signed like an Auth login token, for a user who saved no Steam ID.
private static String tokenWithoutSteamId() {
return io.jsonwebtoken.Jwts.builder()
.subject("no-steam@example.com")
.claim("userId", String.valueOf(USER_ID))
.expiration(new java.util.Date(System.currentTimeMillis() + 60_000))
.signWith(TEST_KEYS.getPrivate(), io.jsonwebtoken.Jwts.SIG.EdDSA)
.compact();
return ContractTokens.loginToken(USER_ID);
}
@Test
@@ -11,7 +11,7 @@ import core_service.modules.library.repo.SteamGameRepository;
import core_service.modules.library.repo.UserGameAchievementMetaRepository;
import core_service.modules.library.repo.UserGameRepository;
import core_service.shared.test.AbstractIntegrationTest;
import io.jsonwebtoken.Jwts;
import core_service.shared.test.ContractTokens;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
@@ -19,7 +19,6 @@ import org.springframework.boot.webmvc.test.autoconfigure.AutoConfigureMockMvc;
import org.springframework.test.web.servlet.MockMvc;
import java.time.Instant;
import java.util.Date;
import java.util.List;
import static org.hamcrest.Matchers.hasSize;
@@ -49,20 +48,15 @@ class PublicProfileControllerTest extends AbstractIntegrationTest {
private UserGameAchievementMetaRepository userAchievementMetaRepository;
private static String ticket(long userId) {
return Jwts.builder().subject("public-profile").claim("typ", "public-profile").claim("profileUserId", String.valueOf(userId))
.expiration(new Date(System.currentTimeMillis() + 60_000)).signWith(TEST_KEYS.getPrivate(), Jwts.SIG.EdDSA).compact();
return ContractTokens.ticketToken(userId, null, null);
}
private static String ticket(long userId, List<String> profileStats) {
return Jwts.builder().subject("public-profile").claim("typ", "public-profile").claim("profileUserId", String.valueOf(userId))
.claim("profileStats", profileStats)
.expiration(new Date(System.currentTimeMillis() + 60_000)).signWith(TEST_KEYS.getPrivate(), Jwts.SIG.EdDSA).compact();
return ContractTokens.ticketToken(userId, profileStats, null);
}
private static String ticket(long userId, List<String> profileStats, List<String> sections) {
return Jwts.builder().subject("public-profile").claim("typ", "public-profile").claim("profileUserId", String.valueOf(userId))
.claim("profileStats", profileStats).claim("sections", sections)
.expiration(new Date(System.currentTimeMillis() + 60_000)).signWith(TEST_KEYS.getPrivate(), Jwts.SIG.EdDSA).compact();
return ContractTokens.ticketToken(userId, profileStats, sections);
}
private UserGame game(long appid, int playtime, GameStatus status, Integer rating) {
@@ -21,7 +21,7 @@ class UserSteamSyncRepositoryTest extends AbstractPersistenceTest {
private JdbcTemplate jdbc;
private int claim(long userId) {
return repository.tryClaimRun(userId, STEAM_ID, Instant.now().minusSeconds(600));
return repository.tryClaimRun(userId, STEAM_ID, Instant.now(), Instant.now().minusSeconds(600));
}
@Test
@@ -44,4 +44,20 @@ class UserSteamSyncRepositoryTest extends AbstractPersistenceTest {
assertThat(claim(3L)).isZero();
}
@Test
void tryClaimRun_RunningRowStartedLongAgo_IsClaimedEvenWhenTheLastSyncEndedRecently() {
jdbc.update("insert into user_steam_sync (user_id, steam_id, last_synced_at, last_sync_status, sync_started_at) " +
"values (4, ?, now(), 'RUNNING', now() - interval '20 minutes')", STEAM_ID);
assertThat(claim(4L)).isEqualTo(1);
}
@Test
void tryClaimRun_RunningRowStartedJustNow_IsRefusedEvenWhenTheLastSyncEndedLongAgo() {
jdbc.update("insert into user_steam_sync (user_id, steam_id, last_synced_at, last_sync_status, sync_started_at) " +
"values (5, ?, now() - interval '1 day', 'RUNNING', now())", STEAM_ID);
assertThat(claim(5L)).isZero();
}
}
@@ -0,0 +1,72 @@
package core_service.modules.library.service;
import core_service.modules.igdb_adapter.service.IgdbApiService;
import core_service.modules.library.domain.GameImport;
import core_service.modules.library.repo.GameImportRepository;
import org.junit.jupiter.api.Test;
import org.springframework.dao.DataAccessResourceFailureException;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.stream.IntStream;
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.timeout;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
class GameImportWorkerTest {
private static IgdbApiService.Game game(long igdbId, String name, String... platforms) {
return new IgdbApiService.Game(igdbId, name, null, null, List.of(platforms), null);
}
private static GameImport.Line line(String title, String platform) {
return new GameImport.Line(1, title, platform, null, null, null);
}
@Test
void sameName_IgnoresCaseAccentsPunctuationAndMarks() {
assertThat(GameImportWorker.sameName("The Witcher® 3: Wild Hunt", "the witcher 3 wild hunt")).isTrue();
assertThat(GameImportWorker.sameName("Pokémon Legends: Arceus", "Pokemon Legends Arceus")).isTrue();
assertThat(GameImportWorker.sameName("Doom", "Doom II")).isFalse();
}
@Test
void rank_SameNameFirst_ThenTheLinesPlatform_ThenIgdbsOrder_AtMostFive() {
List<IgdbApiService.Game> found = List.of(game(1, "Doom Eternal", "PC"), game(2, "DOOM", "PC"),
game(3, "Doom", "Switch"), game(4, "Doom 3"), game(5, "Doom 64"), game(6, "Doom II"));
assertThat(GameImportWorker.rank(line("Doom", "switch"), found)).extracting(IgdbApiService.Game::igdbId)
.containsExactly(3L, 2L, 1L, 4L, 5L);
assertThat(GameImportWorker.rank(line("Doom", null), found)).extracting(IgdbApiService.Game::igdbId)
.startsWith(2L, 3L);
assertThat(GameImportWorker.rank(line("x", null), IntStream.range(0, 9)
.mapToObj(i -> game(i, "Game " + i)).toList())).hasSize(GameImportWorker.MAX_CANDIDATES);
}
@Test
void watchdog_WakesALaneWhoseRunDiedOnADatabaseError() {
GameImportRepository repository = mock(GameImportRepository.class);
when(repository.findNext(GameImportWorker.MATCHING_QUEUE))
.thenThrow(new DataAccessResourceFailureException("connection lost"))
.thenReturn(Optional.of(7L), Optional.empty());
when(repository.findStalled(any())).thenReturn(Map.of());
GameImportWorker worker = new GameImportWorker(repository, mock(ManualGameService.class),
mock(IgdbApiService.class), mock(TimeToBeatService.class), mock(MoodService.class), 5);
try {
worker.wakeMatching();
verify(repository, timeout(2000)).findNext(GameImportWorker.MATCHING_QUEUE);
worker.watchdog();
// Import 7 waits in QUEUED: no try of it is running, so no watchdog rule ever picks it up.
verify(repository, timeout(2000)).markReady(7L);
} finally {
worker.stop();
}
}
}
@@ -63,6 +63,7 @@ class LibrarySyncServiceTest extends AbstractUnitTest {
UserSteamSync sync = new UserSteamSync(USER_ID, steamId);
sync.setLastSyncStatus(status);
sync.setLastSyncedAt(Instant.now().minus(age));
sync.setSyncStartedAt(Instant.now().minus(age));
when(userSteamSyncRepository.findById(USER_ID)).thenReturn(Optional.of(sync));
}
@@ -0,0 +1,140 @@
package core_service.modules.library.service;
import core_service.modules.library.domain.UserSteamSync;
import core_service.modules.library.repo.UserSteamSyncRepository;
import core_service.modules.steam_adapter.api.dto.response.SteamOwnedGamesResponse;
import core_service.modules.steam_adapter.service.SteamApiService;
import core_service.shared.test.AbstractIntegrationTest;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.test.context.bean.override.mockito.MockitoBean;
import java.util.List;
import java.util.Optional;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
* The claim and the finish of a sync are committed on their own, so other requests see RUNNING and FAILED.
*/
class LibrarySyncWorkerConcurrencyTest extends AbstractIntegrationTest {
private static final long USER_ID = 4711L;
private static final long APPID = 991220L;
private static final String STEAM_ID = "76561198000004711";
@Autowired
private LibrarySyncWorker worker;
@Autowired
private UserSteamSyncRepository userSteamSyncRepository;
@Autowired
private JdbcTemplate jdbcTemplate;
@MockitoBean
private SteamApiService steamApiService;
@BeforeEach
@AfterEach
void clean() {
jdbcTemplate.update("DELETE FROM user_game_playtime_day WHERE user_id = ?", USER_ID);
jdbcTemplate.update("DELETE FROM user_game WHERE user_id = ?", USER_ID);
jdbcTemplate.update("DELETE FROM wishlist_game WHERE user_id = ?", USER_ID);
jdbcTemplate.update("DELETE FROM user_steam_sync WHERE user_id = ?", USER_ID);
jdbcTemplate.update("DELETE FROM steam_game WHERE appid = ?", APPID);
}
@Test
void runSyncNow_SecondSyncWhileTheFirstWaitsForSteam_IsSkipped() throws Exception {
// Last synced yesterday: the running check must not take that as a stale run.
jdbcTemplate.update("INSERT INTO user_steam_sync (user_id, steam_id, last_synced_at, last_sync_status) " +
"VALUES (?, ?, NOW() - INTERVAL '1 day', 'OK')", USER_ID, STEAM_ID);
CountDownLatch steamAsked = new CountDownLatch(1);
CountDownLatch secondDone = new CountDownLatch(1);
when(steamApiService.getWishlist(STEAM_ID)).thenReturn(Optional.empty());
when(steamApiService.getOwnedGames(STEAM_ID)).thenAnswer(invocation -> {
steamAsked.countDown();
secondDone.await(5, TimeUnit.SECONDS);
return List.of(game("icon"));
});
CompletableFuture<Void> first = CompletableFuture.runAsync(() -> worker.runSyncNow(USER_ID, STEAM_ID));
assertThat(steamAsked.await(10, TimeUnit.SECONDS)).isTrue();
assertThat(userSteamSyncRepository.findById(USER_ID).orElseThrow().getLastSyncStatus())
.isEqualTo(UserSteamSync.Status.RUNNING);
CompletableFuture<Void> second = CompletableFuture.runAsync(() -> {
worker.runSyncNow(USER_ID, STEAM_ID);
secondDone.countDown();
});
CompletableFuture.allOf(first, second).get(30, TimeUnit.SECONDS);
verify(steamApiService, times(1)).getOwnedGames(STEAM_ID);
assertThat(userSteamSyncRepository.findById(USER_ID).orElseThrow().getLastSyncStatus())
.isEqualTo(UserSteamSync.Status.OK);
}
@Test
void runSyncNow_DatabaseErrorWhileSaving_StoresFailed() {
when(steamApiService.getWishlist(STEAM_ID)).thenReturn(Optional.empty());
// img_icon_url is VARCHAR(512).
when(steamApiService.getOwnedGames(STEAM_ID)).thenReturn(List.of(game("x".repeat(600))));
worker.runSyncNow(USER_ID, STEAM_ID);
UserSteamSync row = userSteamSyncRepository.findById(USER_ID).orElseThrow();
assertThat(row.getLastSyncStatus()).isEqualTo(UserSteamSync.Status.FAILED);
assertThat(jdbcTemplate.queryForObject("SELECT count(*) FROM user_game WHERE user_id = ?", Integer.class, USER_ID))
.isZero();
}
@Test
void runSyncNow_StaleRunFinishesWhileTheRunThatTookOverStillRuns_RowStaysRunning() throws Exception {
CountDownLatch firstAsked = new CountDownLatch(1);
CountDownLatch secondAsked = new CountDownLatch(1);
CountDownLatch releaseFirst = new CountDownLatch(1);
CountDownLatch releaseSecond = new CountDownLatch(1);
AtomicInteger calls = new AtomicInteger();
when(steamApiService.getWishlist(STEAM_ID)).thenReturn(Optional.empty());
when(steamApiService.getOwnedGames(STEAM_ID)).thenAnswer(invocation -> {
if (calls.incrementAndGet() == 1) {
firstAsked.countDown();
releaseFirst.await(10, TimeUnit.SECONDS);
} else {
secondAsked.countDown();
releaseSecond.await(10, TimeUnit.SECONDS);
}
return List.of(game("icon"));
});
CompletableFuture<Void> first = CompletableFuture.runAsync(() -> worker.runSyncNow(USER_ID, STEAM_ID));
assertThat(firstAsked.await(10, TimeUnit.SECONDS)).isTrue();
// The first run hangs past running-stale-minutes, so the next sync takes over.
jdbcTemplate.update("UPDATE user_steam_sync SET sync_started_at = NOW() - INTERVAL '10 minutes' WHERE user_id = ?",
USER_ID);
CompletableFuture<Void> second = CompletableFuture.runAsync(() -> worker.runSyncNow(USER_ID, STEAM_ID));
assertThat(secondAsked.await(10, TimeUnit.SECONDS)).isTrue();
releaseFirst.countDown();
first.get(30, TimeUnit.SECONDS);
UserSteamSync.Status whileSecondRuns = userSteamSyncRepository.findById(USER_ID).orElseThrow().getLastSyncStatus();
releaseSecond.countDown();
second.get(30, TimeUnit.SECONDS);
// OK here lets a third sync claim and run next to the second one.
assertThat(whileSecondRuns).isEqualTo(UserSteamSync.Status.RUNNING);
}
private static SteamOwnedGamesResponse.Game game(String iconUrl) {
return new SteamOwnedGamesResponse.Game((int) APPID, "Half-Life 2", 60, iconUrl, true, 0);
}
}
@@ -5,7 +5,6 @@ import core_service.modules.library.domain.GameSource;
import core_service.modules.library.domain.GameStatus;
import core_service.modules.library.domain.SteamGame;
import core_service.modules.library.domain.UserGame;
import core_service.modules.library.domain.UserGameId;
import core_service.modules.library.domain.UserSteamSync;
import core_service.modules.library.domain.WishlistGame;
import core_service.modules.library.repo.GameCorrectionRepository;
@@ -25,8 +24,11 @@ import org.mockito.ArgumentCaptor;
import org.mockito.InOrder;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.Spy;
import org.springframework.context.ApplicationEventPublisher;
import org.springframework.test.util.ReflectionTestUtils;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.support.TransactionTemplate;
import java.time.Instant;
import java.time.LocalDate;
@@ -47,6 +49,8 @@ import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.argThat;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.inOrder;
import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.verifyNoInteractions;
@@ -71,9 +75,9 @@ class LibrarySyncWorkerTest extends AbstractUnitTest {
*/
private void steamReturns(SteamOwnedGamesResponse.Game game, String previousSteamId) {
if (previousSteamId != null) {
when(userSteamSyncRepository.findById(USER_ID)).thenReturn(Optional.of(syncRow(previousSteamId, previousSteamId)));
when(userSteamSyncRepository.findForClaim(USER_ID)).thenReturn(Optional.of(syncRow(previousSteamId, previousSteamId)));
}
when(userSteamSyncRepository.tryClaimRun(eq(USER_ID), eq(STEAM_ID), any())).thenReturn(1);
when(userSteamSyncRepository.tryClaimRun(eq(USER_ID), eq(STEAM_ID), any(), any())).thenReturn(1);
when(steamApiService.getOwnedGames(STEAM_ID)).thenReturn(List.of(game));
}
@@ -85,7 +89,7 @@ class LibrarySyncWorkerTest extends AbstractUnitTest {
private UserGame owned(int playtime) {
UserGame existing = new UserGame(USER_ID, 220L, playtime, Instant.EPOCH, Instant.EPOCH);
when(userGameRepository.findById(new UserGameId(USER_ID, 220L))).thenReturn(Optional.of(existing));
when(userGameRepository.findByUserId(USER_ID)).thenReturn(List.of(existing));
return existing;
}
@@ -119,69 +123,83 @@ class LibrarySyncWorkerTest extends AbstractUnitTest {
@Mock
private ApplicationEventPublisher eventPublisher;
@Spy
private TransactionTemplate transaction = new TransactionTemplate(mock(PlatformTransactionManager.class));
@InjectMocks
private LibrarySyncWorker worker;
@BeforeEach
void setUp() {
ReflectionTestUtils.setField(worker, "zone", ZoneId.of("Europe/Berlin"));
lenient().when(userSteamSyncRepository.markFinished(eq(USER_ID), any(), anyString(), any())).thenReturn(1);
}
@Test
void runSync_AnotherRunTookOverTheStaleClaim_PublishesNothing() {
steamReturns(game(100, 0), null);
when(userSteamSyncRepository.markFinished(eq(USER_ID), any(), eq("OK"), any())).thenReturn(0);
worker.runSync(USER_ID, STEAM_ID);
verifyNoInteractions(eventPublisher);
}
@Test
void runSync_EmptySteamResult_MarksFailedAndKeepsLibrary() {
when(userSteamSyncRepository.tryClaimRun(eq(USER_ID), eq(STEAM_ID), any())).thenReturn(1);
when(userSteamSyncRepository.tryClaimRun(eq(USER_ID), eq(STEAM_ID), any(), any())).thenReturn(1);
when(steamApiService.getOwnedGames(STEAM_ID)).thenReturn(List.of());
worker.runSync(USER_ID, STEAM_ID);
verify(userGameRepository, never()).softDeleteMissingSince(anyLong(), any(), any());
verify(userSteamSyncRepository).markFinished(eq(USER_ID), eq("FAILED"), any());
verify(userSteamSyncRepository).markFinished(eq(USER_ID), any(), eq("FAILED"), any());
verifyNoInteractions(eventPublisher);
}
@Test
void runSync_EmptyResultForNewSteamId_RemovesOldLibrary() {
when(userSteamSyncRepository.findById(USER_ID))
when(userSteamSyncRepository.findForClaim(USER_ID))
.thenReturn(Optional.of(new UserSteamSync(USER_ID, "76561198005679790")));
when(userSteamSyncRepository.tryClaimRun(eq(USER_ID), eq(STEAM_ID), any())).thenReturn(1);
when(userSteamSyncRepository.tryClaimRun(eq(USER_ID), eq(STEAM_ID), any(), any())).thenReturn(1);
when(steamApiService.getOwnedGames(STEAM_ID)).thenReturn(List.of());
worker.runSync(USER_ID, STEAM_ID);
verify(userGameRepository).softDeleteMissingSince(eq(USER_ID), any(), any());
verify(userSteamSyncRepository).markFinished(eq(USER_ID), eq("FAILED"), any());
verify(userSteamSyncRepository).markFinished(eq(USER_ID), any(), eq("FAILED"), any());
}
@Test
void runSync_SteamErrorForNewSteamId_RemovesOldLibrary() {
when(userSteamSyncRepository.findById(USER_ID))
when(userSteamSyncRepository.findForClaim(USER_ID))
.thenReturn(Optional.of(new UserSteamSync(USER_ID, "76561198005679790")));
when(userSteamSyncRepository.tryClaimRun(eq(USER_ID), eq(STEAM_ID), any())).thenReturn(1);
when(userSteamSyncRepository.tryClaimRun(eq(USER_ID), eq(STEAM_ID), any(), any())).thenReturn(1);
when(steamApiService.getOwnedGames(STEAM_ID)).thenThrow(new IllegalStateException("Steam down"));
worker.runSync(USER_ID, STEAM_ID);
verify(userGameRepository).softDeleteMissingSince(eq(USER_ID), any(), any());
verify(userSteamSyncRepository).markFinished(eq(USER_ID), eq("FAILED"), any());
verify(userSteamSyncRepository).markFinished(eq(USER_ID), any(), eq("FAILED"), any());
}
@Test
void runSync_WithGames_MarksOk() {
when(userSteamSyncRepository.tryClaimRun(eq(USER_ID), eq(STEAM_ID), any())).thenReturn(1);
when(userSteamSyncRepository.tryClaimRun(eq(USER_ID), eq(STEAM_ID), any(), any())).thenReturn(1);
when(steamApiService.getOwnedGames(STEAM_ID)).thenReturn(List.of(
game(0, 0)));
worker.runSync(USER_ID, STEAM_ID);
verify(userGameRepository).softDeleteMissingSince(eq(USER_ID), any(), any());
verify(userSteamSyncRepository).markFinished(eq(USER_ID), eq("OK"), any());
verify(userSteamSyncRepository).markFinished(eq(USER_ID), any(), eq("OK"), any());
verify(eventPublisher).publishEvent(new LibrarySyncedEvent(USER_ID, STEAM_ID));
}
@ParameterizedTest
@CsvSource({"0, BACKLOG", "1, PLAYING"})
void runSync_NewGame_GetsStatusFromPlaytime(int playtime, GameStatus expected) {
when(userSteamSyncRepository.tryClaimRun(eq(USER_ID), eq(STEAM_ID), any())).thenReturn(1);
when(userSteamSyncRepository.tryClaimRun(eq(USER_ID), eq(STEAM_ID), any(), any())).thenReturn(1);
when(steamApiService.getOwnedGames(STEAM_ID)).thenReturn(List.of(
game(playtime, 0)));
@@ -197,7 +215,7 @@ class LibrarySyncWorkerTest extends AbstractUnitTest {
UserGame manual = UserGame.manual(USER_ID, 220L, "PS5", GameStatus.FINISHED, 60, Instant.EPOCH);
manual.setRating(5);
manual.setNotes("Played on the couch");
when(userGameRepository.findById(new UserGameId(USER_ID, 220L))).thenReturn(Optional.of(manual));
when(userGameRepository.findByUserId(USER_ID)).thenReturn(List.of(manual));
steamReturns(game(90, 30), STEAM_ID);
worker.runSync(USER_ID, STEAM_ID);
@@ -213,7 +231,7 @@ class LibrarySyncWorkerTest extends AbstractUnitTest {
@Test
void runSync_GameAddedByHandIsNowOnSteam_TheHandTypedHoursAreNoPlaytimeGrowth() {
UserGame manual = UserGame.manual(USER_ID, 220L, "PS5", GameStatus.PLAYING, 60, Instant.EPOCH);
when(userGameRepository.findById(new UserGameId(USER_ID, 220L))).thenReturn(Optional.of(manual));
when(userGameRepository.findByUserId(USER_ID)).thenReturn(List.of(manual));
steamReturns(game(90, 30), STEAM_ID);
worker.runSync(USER_ID, STEAM_ID);
@@ -225,8 +243,8 @@ class LibrarySyncWorkerTest extends AbstractUnitTest {
void runSync_ExistingGame_KeepsStatus() {
UserGame existing = new UserGame(USER_ID, 220L, 0, Instant.EPOCH, Instant.EPOCH);
existing.setStatus(GameStatus.DROPPED);
when(userGameRepository.findById(new UserGameId(USER_ID, 220L))).thenReturn(Optional.of(existing));
when(userSteamSyncRepository.tryClaimRun(eq(USER_ID), eq(STEAM_ID), any())).thenReturn(1);
when(userGameRepository.findByUserId(USER_ID)).thenReturn(List.of(existing));
when(userSteamSyncRepository.tryClaimRun(eq(USER_ID), eq(STEAM_ID), any(), any())).thenReturn(1);
when(steamApiService.getOwnedGames(STEAM_ID)).thenReturn(List.of(
game(90, 0)));
@@ -236,6 +254,18 @@ class LibrarySyncWorkerTest extends AbstractUnitTest {
assertEquals(90, existing.getPlaytimeMinutes());
}
@Test
void runSync_SteamListsAGameTwice_CountsItOnce() {
when(userSteamSyncRepository.findForClaim(USER_ID)).thenReturn(Optional.of(syncRow(STEAM_ID, STEAM_ID)));
when(userSteamSyncRepository.tryClaimRun(eq(USER_ID), eq(STEAM_ID), any(), any())).thenReturn(1);
when(steamApiService.getOwnedGames(STEAM_ID)).thenReturn(List.of(game(600, 120), game(600, 120)));
worker.runSync(USER_ID, STEAM_ID);
verify(userGameRepository).save(any(UserGame.class));
verify(playtimeDayRepository).addMinutes(USER_ID, 220L, today(), 120);
}
@Test
void runSync_PlaytimeGrew_RecordsTheDifferenceForToday() {
owned(100);
@@ -358,10 +388,10 @@ class LibrarySyncWorkerTest extends AbstractUnitTest {
@Test
void runSync_FailedFirstSyncOfNewSteamId_NextSyncIsStillOnlyTheBaseline() {
when(userSteamSyncRepository.findById(USER_ID)).thenReturn(
when(userSteamSyncRepository.findForClaim(USER_ID)).thenReturn(
Optional.of(syncRow(OTHER_STEAM_ID, OTHER_STEAM_ID)),
Optional.of(syncRow(STEAM_ID, OTHER_STEAM_ID)));
when(userSteamSyncRepository.tryClaimRun(eq(USER_ID), eq(STEAM_ID), any())).thenReturn(1);
when(userSteamSyncRepository.tryClaimRun(eq(USER_ID), eq(STEAM_ID), any(), any())).thenReturn(1);
when(steamApiService.getOwnedGames(STEAM_ID)).thenReturn(List.of(), List.of(game(5000, 60)));
worker.runSync(USER_ID, STEAM_ID);
@@ -425,19 +455,19 @@ class LibrarySyncWorkerTest extends AbstractUnitTest {
worker.runSync(USER_ID, STEAM_ID);
verify(wishlistGameRepository, never()).deleteByUserId(anyLong());
verify(userSteamSyncRepository).markFinished(eq(USER_ID), eq("OK"), any());
verify(userSteamSyncRepository).markFinished(eq(USER_ID), any(), eq("OK"), any());
}
@Test
void runSync_NoOwnedGames_StillSyncsTheWishlist() {
when(userSteamSyncRepository.tryClaimRun(eq(USER_ID), eq(STEAM_ID), any())).thenReturn(1);
when(userSteamSyncRepository.tryClaimRun(eq(USER_ID), eq(STEAM_ID), any(), any())).thenReturn(1);
when(steamApiService.getOwnedGames(STEAM_ID)).thenReturn(List.of());
when(steamApiService.getWishlist(STEAM_ID)).thenReturn(Optional.of(List.of(new SteamApiService.WishlistItem(1257610L, null))));
worker.runSync(USER_ID, STEAM_ID);
verify(wishlistGameRepository).saveAll(anyList());
verify(userSteamSyncRepository).markFinished(eq(USER_ID), eq("FAILED"), any());
verify(userSteamSyncRepository).markFinished(eq(USER_ID), any(), eq("FAILED"), any());
}
@Test
@@ -469,8 +499,8 @@ class LibrarySyncWorkerTest extends AbstractUnitTest {
@Test
void runSync_SteamIdChanged_LibraryFailed_RemovesTheOldAccountsWishlist() {
when(userSteamSyncRepository.findById(USER_ID)).thenReturn(Optional.of(syncRow(OTHER_STEAM_ID, OTHER_STEAM_ID)));
when(userSteamSyncRepository.tryClaimRun(eq(USER_ID), eq(STEAM_ID), any())).thenReturn(1);
when(userSteamSyncRepository.findForClaim(USER_ID)).thenReturn(Optional.of(syncRow(OTHER_STEAM_ID, OTHER_STEAM_ID)));
when(userSteamSyncRepository.tryClaimRun(eq(USER_ID), eq(STEAM_ID), any(), any())).thenReturn(1);
when(steamApiService.getOwnedGames(STEAM_ID)).thenReturn(List.of());
worker.runSync(USER_ID, STEAM_ID);
@@ -525,7 +555,7 @@ class LibrarySyncWorkerTest extends AbstractUnitTest {
@Test
void runSync_GameWithoutName_GetsTheAppidAsName() {
when(userSteamSyncRepository.tryClaimRun(eq(USER_ID), eq(STEAM_ID), any())).thenReturn(1);
when(userSteamSyncRepository.tryClaimRun(eq(USER_ID), eq(STEAM_ID), any(), any())).thenReturn(1);
when(steamApiService.getOwnedGames(STEAM_ID)).thenReturn(List.of(
new SteamOwnedGamesResponse.Game(2000001, null, 12, null, null, null),
game(90, 0)));
@@ -539,8 +569,8 @@ class LibrarySyncWorkerTest extends AbstractUnitTest {
@Test
void runSync_GameWithoutName_KeepsTheStoredName() {
SteamGame stored = new SteamGame(2000001L, "El Mucho", null, Instant.EPOCH);
when(userSteamSyncRepository.tryClaimRun(eq(USER_ID), eq(STEAM_ID), any())).thenReturn(1);
when(steamGameRepository.findById(2000001L)).thenReturn(Optional.of(stored));
when(userSteamSyncRepository.tryClaimRun(eq(USER_ID), eq(STEAM_ID), any(), any())).thenReturn(1);
when(steamGameRepository.findAllById(List.of(2000001L))).thenReturn(List.of(stored));
when(steamApiService.getOwnedGames(STEAM_ID)).thenReturn(List.of(
new SteamOwnedGamesResponse.Game(2000001, " ", 12, null, null, null)));
@@ -552,7 +582,7 @@ class LibrarySyncWorkerTest extends AbstractUnitTest {
@Test
void runSync_NewName_GoesThroughTheGuardedUpdate() {
SteamGame stored = new SteamGame(220L, "Half-Life 2: Corrected", null, Instant.EPOCH);
when(steamGameRepository.findById(220L)).thenReturn(Optional.of(stored));
when(steamGameRepository.findAllById(List.of(220L))).thenReturn(List.of(stored));
steamReturns(game(10, 0), null);
worker.runSync(USER_ID, STEAM_ID);
@@ -565,7 +595,7 @@ class LibrarySyncWorkerTest extends AbstractUnitTest {
@Test
void runSync_CorrectedName_SendsNoUpdate() {
when(steamGameRepository.findById(220L)).thenReturn(Optional.of(new SteamGame(220L, "HL2 (corrected)", null, Instant.EPOCH)));
when(steamGameRepository.findAllById(List.of(220L))).thenReturn(List.of(new SteamGame(220L, "HL2 (corrected)", null, Instant.EPOCH)));
when(gameCorrectionRepository.findCorrectedAppids(eq(CorrectionField.NAME), any())).thenReturn(Set.of(220L));
steamReturns(game(10, 0), null);
@@ -1,6 +1,8 @@
package core_service.modules.steam_adapter.service;
import core_service.modules.steam_adapter.api.dto.response.SteamOwnedGamesResponse;
import core_service.modules.steam_adapter.api.dto.response.SteamPlayerSummaryResponse;
import core_service.modules.steam_adapter.api.dto.response.SteamWishlistResponse;
import core_service.shared.test.AbstractIntegrationTest;
import io.github.resilience4j.ratelimiter.RateLimiter;
import io.github.resilience4j.ratelimiter.RateLimiterRegistry;
@@ -10,6 +12,10 @@ import org.mockito.Answers;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.cache.CacheManager;
import org.springframework.test.context.bean.override.mockito.MockitoBean;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.web.client.HttpClientErrorException;
import org.springframework.web.client.ResourceAccessException;
import org.springframework.web.client.RestClient;
import java.time.Duration;
@@ -18,6 +24,7 @@ import java.util.concurrent.atomic.AtomicInteger;
import java.util.function.Function;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.when;
@@ -98,4 +105,57 @@ class SteamApiServiceCacheTest extends AbstractIntegrationTest {
assertThat(steamApiService.getPlayerSummary(STEAM_ID)).isPresent();
assertThat(steamCalls).hasValue(1);
}
@Test
void getOwnedGames_TransientSteamError_IsTriedAgain() {
SteamOwnedGamesResponse games = new SteamOwnedGamesResponse(new SteamOwnedGamesResponse.OwnedGamesResponse(1,
List.of(new SteamOwnedGamesResponse.Game(220, "Half-Life 2", 60, null, true, 0))));
when(steamRestClient.get().uri(any(Function.class)).retrieve().body(SteamOwnedGamesResponse.class))
.thenAnswer(invocation -> {
if (steamCalls.incrementAndGet() == 1) {
throw new ResourceAccessException("Read timed out");
}
return games;
});
assertThat(steamApiService.getOwnedGames(STEAM_ID)).hasSize(1);
assertThat(steamCalls).hasValue(2);
}
@Test
void getOwnedGames_SteamDownForEveryTry_Throws() {
when(steamRestClient.get().uri(any(Function.class)).retrieve().body(SteamOwnedGamesResponse.class))
.thenAnswer(invocation -> {
steamCalls.incrementAndGet();
throw new ResourceAccessException("Read timed out");
});
assertThatThrownBy(() -> steamApiService.getOwnedGames(STEAM_ID)).isInstanceOf(ResourceAccessException.class);
assertThat(steamCalls).hasValue(3);
}
@Test
void getOwnedGames_SteamRefusesTheKey_IsNotTriedAgain() {
when(steamRestClient.get().uri(any(Function.class)).retrieve().body(SteamOwnedGamesResponse.class))
.thenAnswer(invocation -> {
steamCalls.incrementAndGet();
throw HttpClientErrorException.create(HttpStatus.FORBIDDEN, "Forbidden", HttpHeaders.EMPTY, null, null);
});
assertThatThrownBy(() -> steamApiService.getOwnedGames(STEAM_ID)).isInstanceOf(HttpClientErrorException.class);
assertThat(steamCalls).hasValue(1);
}
@Test
void getWishlist_SteamDownForEveryTry_IsEmptySoTheCallerKeepsTheOldList() {
when(steamRestClient.get().uri(any(Function.class)).retrieve().body(SteamWishlistResponse.class))
.thenAnswer(invocation -> {
steamCalls.incrementAndGet();
throw new ResourceAccessException("Read timed out");
});
assertThat(steamApiService.getWishlist(STEAM_ID)).isEmpty();
// One try: the wishlist is optional and must not hold up the sync of the library.
assertThat(steamCalls).hasValue(1);
}
}
@@ -3,7 +3,6 @@ package core_service.modules.steam_adapter.service;
import core_service.modules.steam_adapter.api.dto.response.SteamOwnedGamesResponse;
import core_service.modules.steam_adapter.api.dto.response.SteamPlayerAchievementsResponse;
import core_service.modules.steam_adapter.api.dto.response.SteamPlayerSummaryResponse;
import core_service.modules.steam_adapter.api.dto.response.SteamProfileAndGamesResponse;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
@@ -21,6 +20,7 @@ import java.util.Map;
import java.util.Optional;
import java.util.stream.LongStream;
import static org.springframework.test.web.client.response.MockRestResponseCreators.withException;
import static org.hamcrest.CoreMatchers.allOf;
import static org.hamcrest.CoreMatchers.containsString;
import static org.hamcrest.CoreMatchers.not;
@@ -172,54 +172,6 @@ class SteamApiServiceTest {
assertThat(result.getFirst().hasCommunityVisibleStats()).isNull();
}
@Test
void getFullPlayerProfile_shouldReturnCombinedResult() {
String steamId = "76561197960435530";
String summaryJson = """
{
"response": {
"players": [
{
"steamid": "76561197960435530",
"personaname": "TestUser",
"profileurl": "url",
"avatarfull": "avatar"
}
]
}
}
""";
String gamesJson = """
{
"response": {
"game_count": 1,
"games": [
{
"appid": 10,
"name": "Counter-Strike",
"playtime_forever": 32,
"img_icon_url": "icon_url",
"has_community_visible_stats": true
}
]
}
}
""";
server.expect(requestTo(containsString("/ISteamUser/GetPlayerSummaries/v0002/")))
.andRespond(withSuccess(summaryJson, MediaType.APPLICATION_JSON));
server.expect(requestTo(containsString("/IPlayerService/GetOwnedGames/v1/")))
.andRespond(withSuccess(gamesJson, MediaType.APPLICATION_JSON));
Optional<SteamProfileAndGamesResponse> result = steamApiService.getFullPlayerProfile(steamId);
assertThat(result).isPresent();
assertThat(result.get().player().personaname()).isEqualTo("TestUser");
assertThat(result.get().games()).hasSize(1);
assertThat(result.get().games().getFirst().name()).isEqualTo("Counter-Strike");
}
@Test
void getWishlist_ReturnsItemsWithTheDateAdded() {
server.expect(requestTo(containsString("/IWishlistService/GetWishlist/v1/")))
@@ -241,12 +193,14 @@ class SteamApiServiceTest {
assertThat(steamApiService.getWishlist("76561197960435530")).contains(List.of());
}
// Without the resilience4j proxy the error reaches the caller; with it the fallback makes it empty
// (SteamApiServiceCacheTest).
@Test
void getWishlist_SteamError_IsEmptySoTheCallerKeepsTheOldList() {
void getWishlist_SteamError_IsThrownForTheFallback() {
server.expect(requestTo(containsString("/IWishlistService/GetWishlist/v1/")))
.andRespond(withServerError());
assertThat(steamApiService.getWishlist("76561197960435530")).isEmpty();
assertThatThrownBy(() -> steamApiService.getWishlist("76561197960435530")).isInstanceOf(RestClientException.class);
}
@Test
@@ -408,4 +362,14 @@ class SteamApiServiceTest {
assertThat(steamApiService.getGlobalAchievementPercentages(620)).isEmpty();
assertThat(steamApiService.getGlobalAchievementPercentages(620)).isEmpty();
}
// The key is a query parameter; Spring 7 leaves the query out of an I/O error, and it must stay so in our logs.
@Test
void getOwnedGames_Timeout_HasNoApiKeyInItsMessage() {
server.expect(requestTo(containsString("/IPlayerService/GetOwnedGames/v1/")))
.andRespond(withException(new java.net.SocketTimeoutException("Read timed out")));
assertThatThrownBy(() -> steamApiService.getOwnedGames("76561198000000001"))
.hasMessageContaining("Read timed out").hasMessageNotContaining("test-api-key");
}
}
@@ -18,10 +18,12 @@ import java.util.Base64;
* Full Spring context with Testcontainers.
*/
@Tag("IntegrationTest")
// The nightly sync would call Steam if a test ran at 23:55; speedrun.com needs no key, so it is switched off.
// The nightly sync would call Steam if a test ran at 23:55; speedrun.com needs no key, so it is switched off. The
// import watchdog runs only when a test calls it.
// Every cached context keeps its own pool: small pools keep them all below Postgres' 100 clients.
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT,
properties = {"app.library.nightly-sync-cron=-", "app.library.image-repair-cron=-", "app.speedrun.enabled=false",
"app.library.import-watchdog-ms=86400000",
"spring.datasource.hikari.maximum-pool-size=4"})
@ActiveProfiles("dev")
public abstract class AbstractIntegrationTest {
@@ -0,0 +1,120 @@
package core_service.shared.test;
import core_service.shared.security.Role;
import io.jsonwebtoken.Jwts;
import tools.jackson.databind.JsonNode;
import tools.jackson.databind.json.JsonMapper;
import java.io.IOException;
import java.io.InputStream;
import java.io.UncheckedIOException;
import java.security.PrivateKey;
import java.util.Date;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
/**
* Tokens only in the shapes of {@code contract/auth-core-contract.json}, the file Auth checks its tokens against. A
* test changes the values, never the claim names.
*/
public final class ContractTokens {
public static final String LOGIN = "login";
public static final String PUBLIC_PROFILE = "publicProfile";
public static final String SERVICE = "service";
private static final JsonMapper MAPPER = JsonMapper.builder().build();
private static final JsonNode CONTRACT = load();
private ContractTokens() {
}
private static JsonNode load() {
try (InputStream in = ContractTokens.class.getResourceAsStream("/contract/auth-core-contract.json")) {
return MAPPER.readTree(in);
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
/**
* The claims Auth signs for this kind today.
*/
@SuppressWarnings("unchecked")
public static Map<String, Object> claims(String kind) {
return new LinkedHashMap<>(MAPPER.treeToValue(CONTRACT.path("tokens").required(kind), Map.class));
}
/**
* The claims of an Auth from before item 26.
*/
@SuppressWarnings("unchecked")
public static Map<String, Object> legacyClaims(String kind) {
return new LinkedHashMap<>(MAPPER.treeToValue(CONTRACT.path("legacyTokens").required(kind), Map.class));
}
public static JsonNode internalCall(String name) {
return CONTRACT.path("internal").required(name);
}
public static String sign(Map<String, Object> claims, PrivateKey key, long expiresInMs) {
return Jwts.builder()
.claims(claims)
.expiration(new Date(System.currentTimeMillis() + expiresInMs))
.signWith(key, Jwts.SIG.EdDSA)
.compact();
}
public static String sign(Map<String, Object> claims) {
return sign(claims, AbstractIntegrationTest.TEST_KEYS.getPrivate(), 60_000);
}
/**
* @param steamId null for a user without a linked Steam account
*/
public static Map<String, Object> login(long userId, Role role, Long steamId) {
Map<String, Object> claims = claims(LOGIN);
claims.put("sub", String.valueOf(userId));
claims.put("userId", String.valueOf(userId));
claims.put("role", role.name());
putOrRemove(claims, "steamId", steamId);
return claims;
}
public static String loginToken(long userId, Role role) {
return sign(login(userId, role, null));
}
public static String loginToken(long userId) {
return loginToken(userId, Role.USER);
}
/**
* @param profileStats null for a ticket without the claim
* @param sections null for a ticket without the claim
*/
public static Map<String, Object> ticket(long userId, List<String> profileStats, List<String> sections) {
Map<String, Object> claims = claims(PUBLIC_PROFILE);
claims.put("profileUserId", String.valueOf(userId));
putOrRemove(claims, "profileStats", profileStats);
putOrRemove(claims, "sections", sections);
return claims;
}
public static String ticketToken(long userId, List<String> profileStats, List<String> sections) {
return sign(ticket(userId, profileStats, sections));
}
public static String serviceToken() {
return sign(claims(SERVICE));
}
private static void putOrRemove(Map<String, Object> claims, String name, Object value) {
if (value == null) {
claims.remove(name);
} else {
claims.put(name, value);
}
}
}
@@ -0,0 +1,71 @@
{
"_about": "The token and internal-call contract between Auth and Core. The same file is in both repos; ToBePlayed-Deployment/scripts/check-contract.sh compares them. iat and exp are left out. legacyTokens are the shapes of an Auth from before item 26: Core and Auth accept them until item 29.",
"tokens": {
"login": {
"iss": "tbp-auth",
"aud": ["tbp-login"],
"sub": "42",
"userId": "42",
"role": "USER",
"steamId": 76561198000000001
},
"publicProfile": {
"iss": "tbp-auth",
"aud": ["tbp-public-profile"],
"sub": "public-profile",
"typ": "public-profile",
"profileUserId": "42",
"profileStats": ["OVERVIEW"],
"sections": ["SUMMARY"]
},
"service": {
"iss": "tbp-auth",
"aud": ["tbp-internal"],
"sub": "tbp-auth",
"typ": "service"
}
},
"legacyTokens": {
"login": {
"sub": "user@example.com",
"userId": "42",
"role": "USER",
"steamId": 76561198000000001
},
"publicProfile": {
"sub": "public-profile",
"typ": "public-profile",
"profileUserId": "42",
"profileStats": ["OVERVIEW"],
"sections": ["SUMMARY"]
},
"service": {
"sub": "tbp-auth",
"typ": "service"
}
},
"internal": {
"markDeleted": {
"method": "POST",
"path": "/internal/accounts/deleted",
"request": {"userIds": [42]}
},
"replaceDeleted": {
"method": "PUT",
"path": "/internal/accounts/deleted",
"request": {"userIds": []}
},
"purge": {
"method": "POST",
"path": "/internal/accounts/purge",
"request": {"userIds": [42], "dryRun": false},
"response": {"dryRun": true, "users": [42], "rows": {"user_game": 3}, "refused": false}
},
"prune": {
"method": "POST",
"path": "/internal/accounts/prune",
"request": {"knownUserIds": [1, 42], "dryRun": true},
"response": {"dryRun": true, "users": [42], "rows": {"user_game": 3}, "refused": false}
}
}
}