feat(import): import many games with background matching, review and undo

This commit is contained in:
lobo committed 2026-09-29 10:20:54 +02:00
1 parent 3bd1eda0ec
commit 2a5fcc1c00
20 files changed
+1639 -24

No files matched your search

+46 -1
View File
@@ -103,6 +103,11 @@ method calling another service's method that needs more.
| `PUT` | `/api/library/games/{appid}/status` (status only, keeps rating, notes, priority) | JWT | | `PUT` | `/api/library/games/{appid}/status` (status only, keeps rating, notes, priority) | JWT |
| `PUT` | `/api/library/games/{appid}/finished-hint` (`{dismissed: ENDING\|PLAYTIME}`, "Not yet"; `204`) | JWT | | `PUT` | `/api/library/games/{appid}/finished-hint` (`{dismissed: ENDING\|PLAYTIME}`, "Not yet"; `204`) | JWT |
| `PUT` | `/api/library/games/{appid}/hidden` (`{hidden}`; `409 MANUAL_GAME` for a game added by hand) | JWT | | `PUT` | `/api/library/games/{appid}/hidden` (`{hidden}`; `409 MANUAL_GAME` for a game added by hand) | JWT |
| `POST` | `/api/library/imports` (`{source: TEXT\|CSV\|SEARCH, rows: [{title, platform, playtimeMinutes, status, igdbId}]}`, 1–1000 rows; `202 {id}` in `QUEUED`, `409 IMPORT_RUNNING`, `503 IGDB_OFF`) | JWT |
| `GET` | `/api/library/imports` (`{imports}`, the last 20), `/api/library/imports/{id}` (status, progress counts, `ahead` while it waits in a queue, `attempts`; the rows with their candidates while `READY`) | JWT |
| `POST` | `/api/library/imports/{id}/save` (`{rows: [{lineNo, igdbId, platform, status, playtimeMinutes}]}`; `202`, then poll until `SAVED`; `409 IMPORT_NOT_READY`) | JWT |
| `POST` | `/api/library/imports/{id}/retry` (a `FAILED` import; `202`, `409 IMPORT_NOT_FAILED` / `IMPORT_RUNNING`) | JWT |
| `GET`, `DELETE` | `/api/library/imports/{id}/undo-check` (`{games, changed}`), `/api/library/imports/{id}/games` (undo; `204`, `409 IMPORT_NOT_SAVED`) | JWT |
| `POST` | `/api/library/sync` | JWT | | `POST` | `/api/library/sync` | JWT |
| `POST` | `/api/library/broken-images` (`{appid, kind}`, kind `COVER` or `ACHIEVEMENT_ICONS`; `204`) | JWT | | `POST` | `/api/library/broken-images` (`{appid, kind}`, kind `COVER` or `ACHIEVEMENT_ICONS`; `204`) | JWT |
| `GET` | `/api/library/picker?maxMinutes=&mood=&count=&exclude=` | JWT | | `GET` | `/api/library/picker?maxMinutes=&mood=&count=&exclude=` | JWT |
@@ -163,7 +168,7 @@ flowchart LR
| Module | Content | | Module | Content |
|---|---| |---|---|
| `library` | Sync, backlog, manual games, playtime history, wishlist, achievements, public profile data | | `library` | Sync, backlog, manual games, imports, playtime history, wishlist, achievements, public profile data |
| `stats` | Public numbers for the landing page | | `stats` | Public numbers for the landing page |
| `version` | `GET /api/version`: commit and build time of the running image | | `version` | `GET /api/version`: commit and build time of the running image |
| `account` | Deleted accounts, driven by Auth: tombstones in `deleted_user`, purge and orphan prune of the per-user rows | | `account` | Deleted accounts, driven by Auth: tombstones in `deleted_user`, purge and orphan prune of the per-user rows |
@@ -265,6 +270,46 @@ id for negative keys. Search and add check both keys of a game. A game added und
a second row when IGDB learns its Steam appid later and the user owns it on Steam; an auditor then merges the two a second row when IGDB learns its Steam appid later and the user owns it on Steam; an auditor then merges the two
(see "Merging duplicates"). (see "Merging duplicates").
### Imports
Many games at once (V22, `GameImportService`): a list, a CSV or several picks of the game search become one
`game_import` with a `game_import_row` per line. `game_import` is also the queue of `GameImportWorker`, which has
two queues, each on one thread for all users:
- **matching**: IGDB searches with typo help. One thread, so an import never takes the whole IGDB budget from the
interactive search; other users' searches may wait for the limiter meanwhile. Up to 5 candidates per line are
stored whole (JSONB), so the save needs no IGDB call, and `exact` marks a best one with the line's name.
- **saving**: adds the chosen lines the way the manual add does (same keys and duplicate check) with
`user_game.import_id` in the same transaction, then loads their time to beat and moods. A save never waits
behind another user's big matching.
Each line is stored when done, so a stopped try goes on where it stopped. A worker sets `progress_at` when it takes
an import and after every line. A watchdog (every minute) stops a try without progress for 5 minutes
(`app.library.import-stall-minutes`): the import goes to the end of its queue and the queue gets a new thread. A
restart counts as a stopped try too. After 3 tries the import is `FAILED`; the user can try it again. One import
per user can be in the matching queue (a unique partial index).
```mermaid
stateDiagram-v2
[*] --> QUEUED: POST /imports
QUEUED --> MATCHING: matching worker takes it
MATCHING --> READY: every line matched
READY --> SAVING: POST /save (the review)
SAVING --> SAVED: games added, time to beat and moods loaded
SAVED --> UNDONE: DELETE /games (undo)
MATCHING --> QUEUED: no progress for 5 min or restart
SAVING --> SAVING: no progress for 5 min or restart (end of the queue)
MATCHING --> FAILED: 3rd stopped try
SAVING --> FAILED: 3rd stopped try
FAILED --> QUEUED: POST /retry (stopped while matching)
FAILED --> SAVING: POST /retry (stopped while saving)
FAILED --> UNDONE: DELETE /games (the games added before it failed)
```
The undo deletes the games of the import that are still `MANUAL` (a game the Steam sync took over stays). A removed
row the import brings back gets no `import_id`: it holds rating, notes and history from before, so the undo keeps it;
`undo-check` counts those with a rating, notes, priority or a status change after the save.
## Run locally ## Run locally
### Local development without keys (profile `local`) ### Local development without keys (profile `local`)
@@ -21,11 +21,11 @@ public class AccountCleanupRepository {
// Children before user_game (user_game_status_change would also go by its cascade), the tombstones last. // Children before user_game (user_game_status_change would also go by its cascade), the tombstones last.
private static final List<String> TABLES = List.of("user_game_status_change", "user_game_achievement", private static final List<String> TABLES = List.of("user_game_status_change", "user_game_achievement",
"user_game_achievement_meta", "user_game_playtime_day", "wishlist_game", "backlog_snapshot", "user_game_achievement_meta", "user_game_playtime_day", "wishlist_game", "backlog_snapshot",
"user_steam_sync", "user_game", "deleted_user"); "user_steam_sync", "user_game", "game_import", "deleted_user");
// Every user with data has a row in one of these; deleteRows then removes the child rows too. // Every user with data has a row in one of these; deleteRows then removes the child rows too.
private static final List<String> ORPHAN_SOURCES = List.of("user_game", "user_steam_sync", "wishlist_game", private static final List<String> ORPHAN_SOURCES = List.of("user_game", "user_steam_sync", "wishlist_game",
"deleted_user"); "game_import", "deleted_user");
private final NamedParameterJdbcTemplate jdbc; private final NamedParameterJdbcTemplate jdbc;
@@ -0,0 +1,99 @@
package core_service.modules.library.api;
import core_service.modules.library.api.dto.ImportHistoryDto;
import core_service.modules.library.api.dto.ImportRequest;
import core_service.modules.library.api.dto.ImportSaveRequest;
import core_service.modules.library.service.GameImportService;
import core_service.shared.security.UserPrincipal;
import jakarta.validation.Valid;
import lombok.RequiredArgsConstructor;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.security.core.annotation.AuthenticationPrincipal;
import org.springframework.web.bind.annotation.DeleteMapping;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import java.util.Map;
/**
* Imports of many games at once, see GameImport.
*/
@RestController
@RequestMapping("/api/library/imports")
@RequiredArgsConstructor
public class GameImportController {
private static final Map<String, String> NOT_FOUND = Map.of("code", "IMPORT_NOT_FOUND", "message", "No such import");
private static final ResponseEntity<?> IMPORT_RUNNING = ResponseEntity.status(HttpStatus.CONFLICT)
.body(Map.of("code", "IMPORT_RUNNING", "message", "Another import is still being matched"));
private final GameImportService importService;
@PostMapping
public ResponseEntity<?> create(@AuthenticationPrincipal UserPrincipal principal,
@Valid @RequestBody ImportRequest request) {
return switch (importService.create(principal.getIdAsLong(), request.source(), request.rows())) {
case GameImportService.CreateResult.Created created -> ResponseEntity.status(HttpStatus.ACCEPTED)
.body(Map.of("id", created.id()));
case GameImportService.CreateResult.AlreadyMatching ignored -> IMPORT_RUNNING;
case GameImportService.CreateResult.SearchUnavailable ignored -> ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE)
.body(Map.of("code", "IGDB_OFF", "message", "Importing games needs the IGDB app"));
};
}
@GetMapping
public ImportHistoryDto list(@AuthenticationPrincipal UserPrincipal principal) {
return new ImportHistoryDto(importService.list(principal.getIdAsLong()));
}
@GetMapping("/{id}")
public ResponseEntity<?> get(@AuthenticationPrincipal UserPrincipal principal, @PathVariable("id") long id) {
return importService.get(principal.getIdAsLong(), id)
.<ResponseEntity<?>>map(ResponseEntity::ok)
.orElseGet(() -> ResponseEntity.status(HttpStatus.NOT_FOUND).body(NOT_FOUND));
}
@PostMapping("/{id}/save")
public ResponseEntity<?> save(@AuthenticationPrincipal UserPrincipal principal, @PathVariable("id") long id,
@Valid @RequestBody ImportSaveRequest request) {
return switch (importService.save(principal.getIdAsLong(), id, request.rows())) {
case DONE -> ResponseEntity.accepted().build();
case NOT_FOUND -> ResponseEntity.status(HttpStatus.NOT_FOUND).body(NOT_FOUND);
case WRONG_STATUS, ALREADY_MATCHING -> ResponseEntity.status(HttpStatus.CONFLICT)
.body(Map.of("code", "IMPORT_NOT_READY", "message", "The import is still matching or saved already"));
};
}
@PostMapping("/{id}/retry")
public ResponseEntity<?> retry(@AuthenticationPrincipal UserPrincipal principal, @PathVariable("id") long id) {
return switch (importService.retry(principal.getIdAsLong(), id)) {
case DONE -> ResponseEntity.accepted().build();
case NOT_FOUND -> ResponseEntity.status(HttpStatus.NOT_FOUND).body(NOT_FOUND);
case WRONG_STATUS -> ResponseEntity.status(HttpStatus.CONFLICT)
.body(Map.of("code", "IMPORT_NOT_FAILED", "message", "Only a failed import can be tried again"));
case ALREADY_MATCHING -> IMPORT_RUNNING;
};
}
@GetMapping("/{id}/undo-check")
public ResponseEntity<?> undoCheck(@AuthenticationPrincipal UserPrincipal principal, @PathVariable("id") long id) {
return importService.undoCheck(principal.getIdAsLong(), id)
.<ResponseEntity<?>>map(ResponseEntity::ok)
.orElseGet(() -> ResponseEntity.status(HttpStatus.NOT_FOUND).body(NOT_FOUND));
}
@DeleteMapping("/{id}/games")
public ResponseEntity<?> undo(@AuthenticationPrincipal UserPrincipal principal, @PathVariable("id") long id) {
return switch (importService.undo(principal.getIdAsLong(), id)) {
case DONE -> ResponseEntity.noContent().build();
case NOT_FOUND -> ResponseEntity.status(HttpStatus.NOT_FOUND).body(NOT_FOUND);
case WRONG_STATUS, ALREADY_MATCHING -> ResponseEntity.status(HttpStatus.CONFLICT)
.body(Map.of("code", "IMPORT_NOT_SAVED", "message", "Only a saved import can be undone"));
};
}
}
@@ -52,7 +52,7 @@ public class ManualGameController {
public ResponseEntity<?> add(@AuthenticationPrincipal UserPrincipal principal, public ResponseEntity<?> add(@AuthenticationPrincipal UserPrincipal principal,
@Valid @RequestBody AddGameRequest request) { @Valid @RequestBody AddGameRequest request) {
int minutes = request.playtimeMinutes() != null ? request.playtimeMinutes() : 0; int minutes = request.playtimeMinutes() != null ? request.playtimeMinutes() : 0;
return switch (manualGameService.add(principal.getIdAsLong(), request.igdbId(), request.platform().trim(), return switch (manualGameService.add(principal.getIdAsLong(), request.igdbId(), request.platform(),
request.status(), minutes)) { request.status(), minutes)) {
case ManualGameService.AddResult.Added added -> ResponseEntity case ManualGameService.AddResult.Added added -> ResponseEntity
.created(URI.create("/api/library/games/" + added.appid())) .created(URI.create("/api/library/games/" + added.appid()))
@@ -0,0 +1,32 @@
package core_service.modules.library.api.dto;
import core_service.modules.library.domain.GameImport;
import core_service.modules.library.domain.GameStatus;
import java.time.Instant;
import java.util.List;
/**
* An import and its progress, see {@link GameImport}.
*
* @param ahead while it waits in a queue (QUEUED, or SAVING before its save started): the imports of any user before
* it; else null
* @param attempts tries a watchdog or a restart stopped; FAILED after 3
* @param rows only for an import in READY that was asked for by its id; else null
*/
public record GameImportDto(long id, GameImport.Source source, GameImport.Status status, int total, int matched,
int chosen, int saved, int added, Integer ahead, GameImport.Step step, int attempts,
Instant createdAt, Instant savedAt, List<Row> rows) {
/**
* @param exact the first candidate has the line's name; see the review's preselection
*/
public record Row(int lineNo, String title, String platform, Integer playtimeMinutes, GameStatus status,
boolean searchFailed, boolean exact, List<Candidate> candidates) {}
/**
* @param inLibrary the user has this game already, from Steam or by hand
*/
public record Candidate(long igdbId, String name, Integer year, String imageUrl, List<String> platforms,
boolean inLibrary) {}
}
@@ -0,0 +1,8 @@
package core_service.modules.library.api.dto;
import java.util.List;
/**
* @param imports the user's last imports, newest first
*/
public record ImportHistoryDto(List<GameImportDto> imports) {}
@@ -0,0 +1,38 @@
package core_service.modules.library.api.dto;
import core_service.modules.library.domain.GameImport;
import core_service.modules.library.domain.GameStatus;
import jakarta.validation.Valid;
import jakarta.validation.constraints.AssertTrue;
import jakarta.validation.constraints.Max;
import jakarta.validation.constraints.Min;
import jakarta.validation.constraints.NotEmpty;
import jakarta.validation.constraints.NotNull;
import jakarta.validation.constraints.Size;
import java.util.List;
public record ImportRequest(
@NotNull GameImport.Source source,
@NotEmpty @Size(max = MAX_ROWS) List<@NotNull @Valid Row> rows
) {
public static final int MAX_ROWS = 1000;
/**
* @param igdbId the game already picked in the search; then the title is not searched
*/
public record Row(
@Size(max = 100) String title,
@Size(max = ManualGameUpdate.MAX_PLATFORM_LENGTH) String platform,
@Min(0) @Max(ManualGameUpdate.MAX_MINUTES) Integer playtimeMinutes,
GameStatus status,
@Min(1) Long igdbId
) {
@AssertTrue(message = "title or igdbId is needed")
public boolean isTitleOrIgdbId() {
return igdbId != null || (title != null && !title.isBlank());
}
}
}
@@ -0,0 +1,28 @@
package core_service.modules.library.api.dto;
import core_service.modules.library.domain.GameStatus;
import jakarta.validation.Valid;
import jakarta.validation.constraints.Max;
import jakarta.validation.constraints.Min;
import jakarta.validation.constraints.NotBlank;
import jakarta.validation.constraints.NotNull;
import jakarta.validation.constraints.Size;
import java.util.List;
/**
* The lines the user checked in the review, each with the game picked.
*/
public record ImportSaveRequest(@NotNull @Size(max = ImportRequest.MAX_ROWS) List<@NotNull @Valid Row> rows) {
/**
* @param igdbId a candidate of the line, or a game from a new search
*/
public record Row(
@Min(1) int lineNo,
@NotNull @Min(1) Long igdbId,
@NotBlank @Size(max = ManualGameUpdate.MAX_PLATFORM_LENGTH) String platform,
@NotNull GameStatus status,
@Min(0) @Max(ManualGameUpdate.MAX_MINUTES) Integer playtimeMinutes
) {}
}
@@ -0,0 +1,7 @@
package core_service.modules.library.api.dto;
/**
* @param games the import's games an undo would remove
* @param changed of these, the games with a rating, notes, priority or a new status since the save
*/
public record UndoCheckDto(int games, int changed) {}
@@ -0,0 +1,55 @@
package core_service.modules.library.domain;
import core_service.modules.igdb_adapter.service.IgdbApiService;
import java.time.Instant;
import java.util.List;
/**
* Many games at once (V22): the lines of a list or a CSV, matched with IGDB in the background, reviewed by the user
* and then saved in the background as games added by hand. The games keep the import's id, so the import can be
* undone.
*
* @param matched lines matched so far, of total
* @param chosen lines the user checked in the review
* @param saved of chosen, the lines the save has done
* @param added of saved, the lines that became a game in the library
* @param running a worker runs the import now; false while it waits in its queue
*/
public record GameImport(long id, long userId, Source source, Status status, int total, int matched, int chosen,
int saved, int added, boolean running, int attempts, Step step, Instant createdAt,
Instant queuedAt, Instant savedAt) {
public enum Source { TEXT, CSV, SEARCH }
public enum Status { QUEUED, MATCHING, READY, SAVING, SAVED, UNDONE, FAILED }
/**
* Of a save: null while the games are added, then DETAILS while their time to beat and moods load.
*/
public enum Step { DETAILS }
public enum SaveResult { ADDED, SKIPPED }
/**
* One input line; every field but lineNo may be missing.
*
* @param igdbId the game the user already picked in the search (source SEARCH); then title is not searched
*/
public record Line(int lineNo, String title, String platform, Integer playtimeMinutes, GameStatus status,
Long igdbId) {}
/**
* A line after the matching.
*
* @param searchFailed IGDB did not answer for this line
* @param exact the first candidate has the line's name (case and punctuation ignored)
* @param candidates best first, at most five
*/
public record Row(Line line, boolean searchFailed, boolean exact, List<IgdbApiService.Game> candidates) {}
/**
* The user's pick for a line in the review.
*/
public record Choice(int lineNo, long igdbId, String platform, GameStatus status, int playtimeMinutes) {}
}
@@ -87,6 +87,10 @@ public class UserGame {
@Column(name = "platform", length = 32) @Column(name = "platform", length = 32)
private String platform; private String platform;
/** The import that added this game (V22), so the import can be undone. Null for any other add. */
@Column(name = "import_id")
private Long importId;
public UserGame(Long userId, Long appid, int playtimeMinutes, Instant firstSeenAt, Instant lastSeenAt) { public UserGame(Long userId, Long appid, int playtimeMinutes, Instant firstSeenAt, Instant lastSeenAt) {
this.userId = userId; this.userId = userId;
this.appid = appid; this.appid = appid;
@@ -0,0 +1,283 @@
package core_service.modules.library.repo;
import core_service.modules.igdb_adapter.service.IgdbApiService;
import core_service.modules.library.domain.GameImport;
import core_service.modules.library.domain.GameStatus;
import lombok.RequiredArgsConstructor;
import org.springframework.jdbc.core.RowMapper;
import org.springframework.jdbc.core.namedparam.MapSqlParameterSource;
import org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate;
import org.springframework.jdbc.support.GeneratedKeyHolder;
import org.springframework.stereotype.Repository;
import tools.jackson.databind.json.JsonMapper;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Timestamp;
import java.time.Instant;
import java.util.Collection;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
/**
* Imports and their lines (V22), the queues of the import workers, and the games an import added.
*/
@Repository
@RequiredArgsConstructor
public class GameImportRepository {
private static final String SELECT_IMPORT = """
select i.*, r.* from game_import i cross join lateral (
select count(*) filter (where matched_at is not null) as matched,
count(*) filter (where chosen_igdb_id is not null) as chosen,
count(*) filter (where save_result is not null) as saved,
count(*) filter (where save_result = 'ADDED') as added
from game_import_row where import_id = i.id) r
""";
private static final RowMapper<GameImport> IMPORT = (rs, row) -> new GameImport(rs.getLong("id"),
rs.getLong("user_id"), GameImport.Source.valueOf(rs.getString("source")),
GameImport.Status.valueOf(rs.getString("status")), rs.getInt("total"), rs.getInt("matched"),
rs.getInt("chosen"), rs.getInt("saved"), rs.getInt("added"), rs.getTimestamp("progress_at") != null,
rs.getInt("attempts"), rs.getString("step") != null ? GameImport.Step.valueOf(rs.getString("step")) : null,
rs.getTimestamp("created_at").toInstant(), rs.getTimestamp("queued_at").toInstant(),
instant(rs.getTimestamp("saved_at")));
private final NamedParameterJdbcTemplate jdbc;
private final JsonMapper jsonMapper;
/**
* Puts the import in the matching queue. The V22 index refuses a second import there per user with a
* DuplicateKeyException.
*/
public long create(long userId, GameImport.Source source, List<GameImport.Line> lines) {
GeneratedKeyHolder key = new GeneratedKeyHolder();
jdbc.update("insert into game_import (user_id, source, status, total) values (:user, :source, 'QUEUED', :total)",
new MapSqlParameterSource("user", userId).addValue("source", source.name()).addValue("total", lines.size()),
key, new String[]{"id"});
long id = key.getKeyAs(Long.class);
jdbc.batchUpdate("insert into game_import_row (import_id, line_no, title, platform, playtime_minutes, status, "
+ "igdb_id) values (:id, :line, :title, :platform, :minutes, :status, :igdb)",
lines.stream().map(line -> new MapSqlParameterSource("id", id).addValue("line", line.lineNo())
.addValue("title", line.title()).addValue("platform", line.platform())
.addValue("minutes", line.playtimeMinutes())
.addValue("status", line.status() != null ? line.status().name() : null)
.addValue("igdb", line.igdbId())).toArray(MapSqlParameterSource[]::new));
return id;
}
public Optional<GameImport> find(long userId, long id) {
return jdbc.query(SELECT_IMPORT + "where i.id = :id and i.user_id = :user",
new MapSqlParameterSource("id", id).addValue("user", userId), IMPORT).stream().findFirst();
}
public List<GameImport> findByUser(long userId, int limit) {
return jdbc.query(SELECT_IMPORT + "where i.user_id = :user order by i.id desc limit :limit",
new MapSqlParameterSource("user", userId).addValue("limit", limit), IMPORT);
}
/**
* The imports before this one in its queue (of any user).
*/
public int countAhead(GameImport waiting, Collection<GameImport.Status> queue) {
return jdbc.queryForObject("select count(*) from game_import where status in (:queue) "
+ "and (queued_at, id) < (:queuedAt, :id)",
new MapSqlParameterSource("queue", queue.stream().map(Enum::name).toList())
.addValue("queuedAt", Timestamp.from(waiting.queuedAt())).addValue("id", waiting.id()),
Integer.class);
}
/**
* The oldest import of a queue.
*/
public Optional<Long> findNext(Collection<GameImport.Status> queue) {
return jdbc.queryForList("select id from game_import where status in (:queue) order by queued_at, id limit 1",
new MapSqlParameterSource("queue", queue.stream().map(Enum::name).toList()), Long.class)
.stream().findFirst();
}
/**
* Imports a worker runs whose last progress is before {@code before}, with their status.
*/
public Map<Long, GameImport.Status> findStalled(Instant before) {
Map<Long, GameImport.Status> stalled = new LinkedHashMap<>();
jdbc.query("select id, status from game_import where status in ('MATCHING', 'SAVING') "
+ "and progress_at < :before order by id",
new MapSqlParameterSource("before", Timestamp.from(before)),
rs -> { stalled.put(rs.getLong("id"), GameImport.Status.valueOf(rs.getString("status"))); });
return stalled;
}
public long findUserId(long id) {
return jdbc.queryForObject("select user_id from game_import where id = :id", new MapSqlParameterSource("id", id),
Long.class);
}
/**
* Marks the import as run by a worker, from now on watched by the watchdog.
*/
public void start(long id, GameImport.Status status) {
jdbc.update("update game_import set status = :status, progress_at = now() where id = :id",
new MapSqlParameterSource("id", id).addValue("status", status.name()));
}
public void setStep(long id, GameImport.Step step) {
jdbc.update("update game_import set step = :step, progress_at = now() where id = :id",
new MapSqlParameterSource("id", id).addValue("step", step.name()));
}
public void markReady(long id) {
jdbc.update("update game_import set status = 'READY', progress_at = null where id = :id",
new MapSqlParameterSource("id", id));
}
/**
* After the games are added: a status change after saved_at is the user's.
*/
public void markSaved(long id) {
jdbc.update("update game_import set status = 'SAVED', saved_at = now(), step = null, progress_at = null "
+ "where id = :id", new MapSqlParameterSource("id", id));
}
/**
* Counts a stopped try: back to the end of the queue of its phase, or FAILED after {@code maxAttempts} tries.
*/
public void retryOrFail(long id, int maxAttempts) {
jdbc.update("""
update game_import set attempts = attempts + 1, queued_at = now(), progress_at = null, step = null,
status = case when attempts + 1 >= :max then 'FAILED'
when status in ('QUEUED', 'MATCHING') then 'QUEUED' else 'SAVING' end
where id = :id and status in ('QUEUED', 'MATCHING', 'SAVING')
""", new MapSqlParameterSource("id", id).addValue("max", maxAttempts));
}
/**
* A FAILED import goes back to the queue of the phase it stopped in, with new tries. In the matching queue the
* V22 index refuses a second import per user with a DuplicateKeyException.
*
* @return false when the import did not fail
*/
public boolean retry(long userId, long id) {
return jdbc.update("""
update game_import i set attempts = 0, queued_at = now(), progress_at = null,
status = case when exists (select 1 from game_import_row r
where r.import_id = i.id and r.chosen_igdb_id is not null)
then 'SAVING' else 'QUEUED' end
where id = :id and user_id = :user and status = 'FAILED'
""", new MapSqlParameterSource("id", id).addValue("user", userId)) == 1;
}
/**
* @return false when the import was in none of {@code from} (a second save, or an undo of an import not saved)
*/
public boolean changeStatus(long userId, long id, Collection<GameImport.Status> from, GameImport.Status to) {
return jdbc.update("update game_import set status = :to, queued_at = now() "
+ "where id = :id and user_id = :user and status in (:from)",
new MapSqlParameterSource("id", id).addValue("user", userId)
.addValue("from", from.stream().map(Enum::name).toList()).addValue("to", to.name())) == 1;
}
public List<GameImport.Line> findUnmatchedLines(long importId) {
return jdbc.query("select * from game_import_row where import_id = :id and matched_at is null order by line_no",
new MapSqlParameterSource("id", importId), (rs, row) -> line(rs));
}
public List<GameImport.Row> findRows(long importId) {
return jdbc.query("select * from game_import_row where import_id = :id order by line_no",
new MapSqlParameterSource("id", importId), (rs, row) -> new GameImport.Row(line(rs),
rs.getBoolean("search_failed"), rs.getBoolean("exact"),
jsonMapper.readValue(rs.getString("candidates"),
jsonMapper.getTypeFactory().constructCollectionType(List.class, IgdbApiService.Game.class))));
}
public void saveMatch(long importId, int lineNo, boolean searchFailed, boolean exact,
List<IgdbApiService.Game> candidates) {
jdbc.update("update game_import_row set matched_at = now(), search_failed = :failed, exact = :exact, "
+ "candidates = cast(:candidates as jsonb) where import_id = :id and line_no = :line",
new MapSqlParameterSource("id", importId).addValue("line", lineNo).addValue("failed", searchFailed)
.addValue("exact", exact).addValue("candidates", jsonMapper.writeValueAsString(candidates)));
touch(importId);
}
/**
* Stores the review; a line number the import does not have is left out.
*/
public void saveChoices(long importId, List<GameImport.Choice> choices) {
jdbc.batchUpdate("update game_import_row set chosen_igdb_id = :igdb, chosen_platform = :platform, "
+ "chosen_status = :status, chosen_minutes = :minutes where import_id = :id and line_no = :line",
choices.stream().map(choice -> new MapSqlParameterSource("id", importId)
.addValue("line", choice.lineNo()).addValue("igdb", choice.igdbId())
.addValue("platform", choice.platform()).addValue("status", choice.status().name())
.addValue("minutes", choice.playtimeMinutes())).toArray(MapSqlParameterSource[]::new));
}
/**
* The chosen lines the save has not done yet, in line order.
*/
public List<GameImport.Choice> findUnsavedChoices(long importId) {
return jdbc.query("select * from game_import_row where import_id = :id and chosen_igdb_id is not null "
+ "and save_result is null order by line_no",
new MapSqlParameterSource("id", importId), (rs, row) -> new GameImport.Choice(rs.getInt("line_no"),
rs.getLong("chosen_igdb_id"), rs.getString("chosen_platform"),
GameStatus.valueOf(rs.getString("chosen_status")), rs.getInt("chosen_minutes")));
}
public void saveResult(long importId, int lineNo, GameImport.SaveResult result) {
jdbc.update("update game_import_row set save_result = :result where import_id = :id and line_no = :line",
new MapSqlParameterSource("id", importId).addValue("line", lineNo).addValue("result", result.name()));
touch(importId);
}
/**
* The import's games still added by hand; a game the Steam sync took over stays in the library.
*/
public int countGames(long userId, long importId) {
return jdbc.queryForObject("select count(*) from user_game where user_id = :user and import_id = :import "
+ "and source = 'MANUAL'", new MapSqlParameterSource("user", userId).addValue("import", importId),
Integer.class);
}
public List<Long> findAddedAppids(long importId) {
return jdbc.queryForList("select appid from user_game where import_id = :import",
new MapSqlParameterSource("import", importId), Long.class);
}
/**
* Of {@link #countGames}: those with a rating, notes, priority or a status change after the save.
*/
public int countChangedGames(long userId, long importId) {
return jdbc.queryForObject("""
select count(*) from user_game g join game_import i on i.id = g.import_id
where g.user_id = :user and g.import_id = :import and g.source = 'MANUAL'
and (g.rating is not null or g.notes is not null or g.priority is not null
or exists (select 1 from user_game_status_change c where c.user_id = g.user_id
and c.appid = g.appid and c.changed_at > i.saved_at))
""", new MapSqlParameterSource("user", userId).addValue("import", importId), Integer.class);
}
/**
* Deletes {@link #countGames} the way the manual remove does: with status, rating, notes and history.
*/
public int deleteGames(long userId, long importId) {
return jdbc.update("delete from user_game where user_id = :user and import_id = :import and source = 'MANUAL'",
new MapSqlParameterSource("user", userId).addValue("import", importId));
}
// The watchdog's sign of life.
private void touch(long importId) {
jdbc.update("update game_import set progress_at = now() where id = :id", new MapSqlParameterSource("id", importId));
}
private static GameImport.Line line(ResultSet rs) throws SQLException {
String status = rs.getString("status");
return new GameImport.Line(rs.getInt("line_no"), rs.getString("title"), rs.getString("platform"),
rs.getObject("playtime_minutes", Integer.class), status != null ? GameStatus.valueOf(status) : null,
rs.getObject("igdb_id", Long.class));
}
private static Instant instant(Timestamp timestamp) {
return timestamp != null ? timestamp.toInstant() : null;
}
}
@@ -0,0 +1,173 @@
package core_service.modules.library.service;
import core_service.modules.igdb_adapter.service.IgdbApiService;
import core_service.modules.library.api.dto.GameImportDto;
import core_service.modules.library.api.dto.ImportRequest;
import core_service.modules.library.api.dto.ImportSaveRequest;
import core_service.modules.library.api.dto.UndoCheckDto;
import core_service.modules.library.domain.GameImport;
import core_service.modules.library.repo.GameImportRepository;
import core_service.shared.security.RequiresRole;
import core_service.shared.security.Role;
import org.springframework.dao.DuplicateKeyException;
import org.springframework.stereotype.Service;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.support.TransactionTemplate;
import java.util.List;
import java.util.Optional;
import java.util.Set;
import java.util.stream.IntStream;
/**
* Imports of many games at once, see {@link GameImport}. The matching and the save run in {@link GameImportWorker};
* the save adds each chosen line the way {@link ManualGameService#add} does.
*/
@Service
@RequiresRole(Role.USER)
public class GameImportService {
public sealed interface CreateResult {
record Created(long id) implements CreateResult {}
record AlreadyMatching() implements CreateResult {}
record SearchUnavailable() implements CreateResult {}
}
public enum ChangeResult { DONE, NOT_FOUND, WRONG_STATUS, ALREADY_MATCHING }
private static final int HISTORY_SIZE = 20;
private final GameImportRepository importRepository;
private final GameImportWorker worker;
private final ManualGameService manualGameService;
private final IgdbApiService igdbApiService;
private final TransactionTemplate transaction;
public GameImportService(GameImportRepository importRepository, GameImportWorker worker,
ManualGameService manualGameService, IgdbApiService igdbApiService,
PlatformTransactionManager transactionManager) {
this.importRepository = importRepository;
this.worker = worker;
this.manualGameService = manualGameService;
this.igdbApiService = igdbApiService;
this.transaction = new TransactionTemplate(transactionManager);
}
public CreateResult create(Long userId, GameImport.Source source, List<ImportRequest.Row> rows) {
if (!igdbApiService.isEnabled()) {
return new CreateResult.SearchUnavailable();
}
List<GameImport.Line> lines = IntStream.range(0, rows.size()).mapToObj(index -> {
ImportRequest.Row row = rows.get(index);
return new GameImport.Line(index + 1, strippedOrNull(row.title()), strippedOrNull(row.platform()),
row.playtimeMinutes(), row.status(), row.igdbId());
}).toList();
long id;
try {
id = transaction.execute(tx -> importRepository.create(userId, source, lines));
} catch (DuplicateKeyException e) {
return new CreateResult.AlreadyMatching();
}
worker.wakeMatching();
return new CreateResult.Created(id);
}
public List<GameImportDto> list(Long userId) {
return importRepository.findByUser(userId, HISTORY_SIZE).stream().map(found -> dto(found, null)).toList();
}
public Optional<GameImportDto> get(Long userId, long id) {
return importRepository.find(userId, id).map(found -> dto(found,
found.status() == GameImport.Status.READY ? rows(userId, importRepository.findRows(id)) : null));
}
/**
* Once per import: stores the review and puts the import in the saving queue.
*/
public ChangeResult save(Long userId, long id, List<ImportSaveRequest.Row> chosen) {
if (importRepository.find(userId, id).isEmpty()) {
return ChangeResult.NOT_FOUND;
}
boolean queued = transaction.execute(tx -> {
if (!importRepository.changeStatus(userId, id, Set.of(GameImport.Status.READY), GameImport.Status.SAVING)) {
return false;
}
importRepository.saveChoices(id, chosen.stream().map(row -> new GameImport.Choice(row.lineNo(),
row.igdbId(), row.platform(), row.status(),
row.playtimeMinutes() != null ? row.playtimeMinutes() : 0)).toList());
return true;
});
if (!queued) {
return ChangeResult.WRONG_STATUS;
}
worker.wakeSaving();
return ChangeResult.DONE;
}
public ChangeResult retry(Long userId, long id) {
if (importRepository.find(userId, id).isEmpty()) {
return ChangeResult.NOT_FOUND;
}
try {
if (!importRepository.retry(userId, id)) {
return ChangeResult.WRONG_STATUS;
}
} catch (DuplicateKeyException e) {
return ChangeResult.ALREADY_MATCHING;
}
worker.wakeMatching();
worker.wakeSaving();
return ChangeResult.DONE;
}
public Optional<UndoCheckDto> undoCheck(Long userId, long id) {
return importRepository.find(userId, id).map(found -> new UndoCheckDto(
importRepository.countGames(userId, id), importRepository.countChangedGames(userId, id)));
}
/**
* Removes the import's games that are still added by hand, like {@link ManualGameService#remove}. Also for an
* import that FAILED while saving.
*/
public ChangeResult undo(Long userId, long id) {
if (importRepository.find(userId, id).isEmpty()) {
return ChangeResult.NOT_FOUND;
}
return transaction.execute(tx -> {
if (!importRepository.changeStatus(userId, id, Set.of(GameImport.Status.SAVED, GameImport.Status.FAILED),
GameImport.Status.UNDONE)) {
return ChangeResult.WRONG_STATUS;
}
importRepository.deleteGames(userId, id);
return ChangeResult.DONE;
});
}
private List<GameImportDto.Row> rows(Long userId, List<GameImport.Row> rows) {
Set<Long> owned = manualGameService.ownedIgdbIds(userId,
rows.stream().flatMap(row -> row.candidates().stream()).toList());
return rows.stream().map(row -> new GameImportDto.Row(row.line().lineNo(), row.line().title(),
row.line().platform(), row.line().playtimeMinutes(), row.line().status(), row.searchFailed(),
row.exact(), row.candidates().stream().map(game -> new GameImportDto.Candidate(game.igdbId(),
game.name(), game.year(), game.imageUrl(), ManualGameService.fittingPlatforms(game),
owned.contains(game.igdbId()))).toList())).toList();
}
private GameImportDto dto(GameImport gameImport, List<GameImportDto.Row> rows) {
Integer ahead = switch (gameImport.status()) {
case QUEUED -> importRepository.countAhead(gameImport, GameImportWorker.MATCHING_QUEUE);
case SAVING -> gameImport.running() ? null
: importRepository.countAhead(gameImport, GameImportWorker.SAVING_QUEUE);
default -> null;
};
return new GameImportDto(gameImport.id(), gameImport.source(), gameImport.status(), gameImport.total(),
gameImport.matched(), gameImport.chosen(), gameImport.saved(), gameImport.added(), ahead,
gameImport.step(), gameImport.attempts(), gameImport.createdAt(), gameImport.savedAt(), rows);
}
private static String strippedOrNull(String text) {
return text == null || text.isBlank() ? null : text.strip();
}
}
@@ -0,0 +1,267 @@
package core_service.modules.library.service;
import core_service.modules.igdb_adapter.service.IgdbApiService;
import core_service.modules.library.domain.GameImport;
import core_service.modules.library.repo.GameImportRepository;
import jakarta.annotation.PreDestroy;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.context.event.ApplicationReadyEvent;
import org.springframework.context.event.EventListener;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;
import java.time.Duration;
import java.time.Instant;
import java.util.Comparator;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.function.LongConsumer;
import java.util.stream.Collectors;
/**
* Runs imports in two queues, each on one thread for all users, with game_import as the queue:
* <ul>
* <li>matching: IGDB searches, so an import of 1000 lines never takes the whole IGDB budget from the interactive
* search;</li>
* <li>saving: adds the chosen games, then loads their time to beat and moods. A save never waits behind another
* user's big matching.</li>
* </ul>
* Each line is stored when done, so a stopped try goes on where it stopped. A watchdog stops a try without progress
* for {@code import-stall-minutes}; a restart counts as a stopped try. After {@link #MAX_ATTEMPTS} tries the import
* is FAILED.
*/
@Service
@Slf4j
public class GameImportWorker {
static final int MAX_CANDIDATES = 5;
static final int MAX_ATTEMPTS = 3;
private static final int RESTART_WAIT_SECONDS = 30;
static final Set<GameImport.Status> MATCHING_QUEUE = Set.of(GameImport.Status.QUEUED, GameImport.Status.MATCHING);
static final Set<GameImport.Status> SAVING_QUEUE = Set.of(GameImport.Status.SAVING);
private final GameImportRepository importRepository;
private final ManualGameService manualGameService;
private final IgdbApiService igdbApiService;
private final TimeToBeatService timeToBeatService;
private final MoodService moodService;
private final Duration stallTime;
// Only a try that started before this app is one a restart stopped.
private final Instant startedAt = Instant.now();
private final Lane matching;
private final Lane saving;
public GameImportWorker(GameImportRepository importRepository, ManualGameService manualGameService,
IgdbApiService igdbApiService, TimeToBeatService timeToBeatService, MoodService moodService,
@Value("${app.library.import-stall-minutes:5}") long stallMinutes) {
this.importRepository = importRepository;
this.manualGameService = manualGameService;
this.igdbApiService = igdbApiService;
this.timeToBeatService = timeToBeatService;
this.moodService = moodService;
this.stallTime = Duration.ofMinutes(stallMinutes);
this.matching = new Lane("game-import-match", MATCHING_QUEUE, this::match);
this.saving = new Lane("game-import-save", SAVING_QUEUE, this::save);
}
public void wakeMatching() {
matching.wake();
}
public void wakeSaving() {
saving.wake();
}
/**
* Imports a worker ran when the app stopped count as a stopped try; then both queues start.
*/
@EventListener(ApplicationReadyEvent.class)
public void resume() {
importRepository.findStalled(startedAt).keySet().forEach(id -> importRepository.retryOrFail(id, MAX_ATTEMPTS));
matching.wake();
saving.wake();
}
/**
* Stops a try without progress: the import goes back to the end of its queue (or FAILED), and its queue gets a
* new thread. The old thread is interrupted and writes nothing more once it sees that. Also wakes both queues:
* a run that died on a database error left its queue idle.
*/
@Scheduled(fixedDelayString = "${app.library.import-watchdog-ms:60000}")
public void watchdog() {
Map<Long, GameImport.Status> stalled = importRepository.findStalled(Instant.now().minus(stallTime));
if (!stalled.isEmpty()) {
log.warn("Imports {} made no progress for {}, trying again", stalled, stallTime);
stalled.keySet().forEach(id -> importRepository.retryOrFail(id, MAX_ATTEMPTS));
}
if (stalled.containsValue(GameImport.Status.MATCHING)) {
matching.restart();
} else {
matching.wake();
}
if (stalled.containsValue(GameImport.Status.SAVING)) {
saving.restart();
} else {
saving.wake();
}
}
@PreDestroy
void stop() {
matching.stop();
saving.stop();
}
private void match(long importId) {
importRepository.start(importId, GameImport.Status.MATCHING);
for (GameImport.Line line : importRepository.findUnmatchedLines(importId)) {
Match found = match(line);
// An IGDB call cut by the watchdog or the shutdown is no failed search.
if (Thread.currentThread().isInterrupted()) {
return;
}
importRepository.saveMatch(importId, line.lineNo(), found.searchFailed(), found.exact(), found.candidates());
}
importRepository.markReady(importId);
}
private void save(long importId) {
importRepository.start(importId, GameImport.Status.SAVING);
List<GameImport.Choice> choices = importRepository.findUnsavedChoices(importId);
if (!choices.isEmpty()) {
long userId = importRepository.findUserId(importId);
Map<Integer, GameImport.Row> rows = importRepository.findRows(importId).stream()
.collect(Collectors.toMap(row -> row.line().lineNo(), row -> row));
for (GameImport.Choice choice : choices) {
IgdbApiService.Game game = rows.get(choice.lineNo()).candidates().stream()
.filter(candidate -> candidate.igdbId() == choice.igdbId()).findFirst()
// A game from a new search in the review. IGDB knew it minutes ago, so no answer is an IGDB
// failure: the try stops and the line waits for the next one.
.or(() -> igdbApiService.getGame(choice.igdbId()))
.orElseThrow(() -> new IllegalStateException("IGDB did not answer for game " + choice.igdbId()));
if (Thread.currentThread().isInterrupted()) {
return;
}
boolean added = manualGameService.addGame(userId, game, choice.platform(), choice.status(),
choice.playtimeMinutes(), importId) instanceof ManualGameService.AddResult.Added;
importRepository.saveResult(importId, choice.lineNo(),
added ? GameImport.SaveResult.ADDED : GameImport.SaveResult.SKIPPED);
}
}
importRepository.setStep(importId, GameImport.Step.DETAILS);
List<Long> appids = importRepository.findAddedAppids(importId);
try {
timeToBeatService.refresh(appids);
moodService.refresh(appids);
} catch (RuntimeException e) {
// The next sync or page view asks again.
log.warn("Time to beat or moods of import {} not loaded: {}", importId, e.getMessage());
}
if (Thread.currentThread().isInterrupted()) {
return;
}
importRepository.markSaved(importId);
}
private record Match(boolean searchFailed, boolean exact, List<IgdbApiService.Game> candidates) {}
private Match match(GameImport.Line line) {
if (line.igdbId() != null) {
Optional<IgdbApiService.Game> game = igdbApiService.getGame(line.igdbId());
return new Match(game.isEmpty(), game.isPresent(), game.stream().toList());
}
return manualGameService.findGames(line.title())
.map(found -> rank(line, found))
.map(candidates -> new Match(false,
!candidates.isEmpty() && sameName(candidates.getFirst().name(), line.title()), candidates))
.orElseGet(() -> new Match(true, false, List.of()));
}
// Same name first, among them one on the line's platform; else IGDB's order.
static List<IgdbApiService.Game> rank(GameImport.Line line, List<IgdbApiService.Game> found) {
Comparator<IgdbApiService.Game> order = Comparator
.comparing((IgdbApiService.Game game) -> !sameName(game.name(), line.title()))
.thenComparing(game -> line.platform() == null
|| game.platforms().stream().noneMatch(line.platform()::equalsIgnoreCase));
return found.stream().sorted(order).limit(MAX_CANDIDATES).toList();
}
/**
* Case, accents, punctuation and marks like ™ ignored, as in the speedrun.com match.
*/
static boolean sameName(String a, String b) {
return SpeedrunMatch.normalize(a).equals(SpeedrunMatch.normalize(b));
}
/**
* One thread that runs the imports of a queue, oldest first, until the queue is empty. A wake while it runs
* leaves one more run waiting, so an import queued meanwhile is not missed.
*/
private final class Lane {
private final String name;
private final Set<GameImport.Status> queue;
private final LongConsumer work;
private volatile ExecutorService executor;
Lane(String name, Set<GameImport.Status> queue, LongConsumer work) {
this.name = name;
this.queue = queue;
this.work = work;
this.executor = newExecutor();
}
void wake() {
executor.execute(this::drain);
}
// Waits a while for the old thread, which stops at its next check, so two threads do not run one import.
synchronized void restart() {
ExecutorService old = executor;
old.shutdownNow();
try {
if (!old.awaitTermination(RESTART_WAIT_SECONDS, TimeUnit.SECONDS)) {
log.warn("Thread {} did not stop; a new one starts anyway", name);
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
executor = newExecutor();
wake();
}
void stop() {
executor.shutdownNow();
}
private void drain() {
Optional<Long> next;
while (!Thread.currentThread().isInterrupted() && (next = importRepository.findNext(queue)).isPresent()) {
long id = next.get();
try {
work.accept(id);
} catch (RuntimeException e) {
if (Thread.currentThread().isInterrupted()) {
return;
}
// To the end of the queue, so the next import is not blocked by this one.
log.error("Import {} stopped", id, e);
importRepository.retryOrFail(id, MAX_ATTEMPTS);
}
}
}
private ExecutorService newExecutor() {
// One run and at most one waiting run; more wakes add nothing.
return new ThreadPoolExecutor(1, 1, 0, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<>(1),
Thread.ofPlatform().name(name).daemon().factory(), new ThreadPoolExecutor.DiscardPolicy());
}
}
}
@@ -23,6 +23,7 @@ import org.springframework.transaction.support.TransactionTemplate;
import java.time.Instant; import java.time.Instant;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.Collection;
import java.util.Comparator; import java.util.Comparator;
import java.util.HashSet; import java.util.HashSet;
import java.util.List; import java.util.List;
@@ -86,8 +87,7 @@ public class ManualGameService {
if (!igdbApiService.isEnabled()) { if (!igdbApiService.isEnabled()) {
return Optional.of(new GameSearchDto(false, List.of())); return Optional.of(new GameSearchDto(false, List.of()));
} }
return igdbApiService.searchGames(query).map(found -> withTypoHelp(query, found)) return findGames(query).map(games -> new GameSearchDto(true, hits(userId, query, games)));
.map(games -> new GameSearchDto(true, hits(userId, query, games)));
} }
// Games already in ToBePlayed first, with their catalog name and image (an auditor's correction included); then // Games already in ToBePlayed first, with their catalog name and image (an auditor's correction included); then
@@ -166,19 +166,7 @@ public class ManualGameService {
if (found.isEmpty()) { if (found.isEmpty()) {
return new AddResult.UnknownGame(); return new AddResult.UnknownGame();
} }
IgdbApiService.Game game = found.get(); AddResult result = addGame(userId, found.get(), platform, status, playtimeMinutes, null);
AddResult result;
try {
result = addInTransaction(userId, game, platform, status, playtimeMinutes);
} catch (DataIntegrityViolationException e) {
// A parallel add, sync or merge changed the rows after the check. The second try sees it.
log.info("Add of igdbId={} for user={} raced another write, trying again", igdbId, userId);
try {
result = addInTransaction(userId, game, platform, status, playtimeMinutes);
} catch (DataIntegrityViolationException again) {
return new AddResult.AlreadyInLibrary();
}
}
if (result instanceof AddResult.Added added) { if (result instanceof AddResult.Added added) {
timeToBeatService.refreshInBackground(List.of(added.appid())); timeToBeatService.refreshInBackground(List.of(added.appid()));
moodService.refreshInBackground(List.of(added.appid())); moodService.refreshInBackground(List.of(added.appid()));
@@ -186,8 +174,56 @@ public class ManualGameService {
return result; return result;
} }
/**
* {@link #add} of a game already loaded from IGDB, without the background refresh; an import adds many games
* and loads their time to beat and moods once.
*
* @param importId the import that adds the game, so its undo can remove it; null for a single add. A removed
* row that comes back keeps no import id.
*/
public AddResult addGame(Long userId, IgdbApiService.Game game, String platform, GameStatus status,
int playtimeMinutes, Long importId) {
AddResult result;
try {
result = addInTransaction(userId, game, platform.strip(), status, playtimeMinutes, importId);
} catch (DataIntegrityViolationException e) {
// A parallel add, sync or merge changed the rows after the check. The second try sees it.
log.info("Add of igdbId={} for user={} raced another write, trying again", game.igdbId(), userId);
try {
result = addInTransaction(userId, game, platform.strip(), status, playtimeMinutes, importId);
} catch (DataIntegrityViolationException again) {
return new AddResult.AlreadyInLibrary();
}
}
return result;
}
/**
* IGDB's hits for a title, with the same typo help as {@link #search}; empty when IGDB failed or is off.
*/
public Optional<List<IgdbApiService.Game>> findGames(String query) {
if (!igdbApiService.isEnabled()) {
return Optional.empty();
}
return igdbApiService.searchGames(query).map(found -> withTypoHelp(query, found));
}
/**
* The IGDB ids of the games the user already has, under any of their keys (a merge included).
*/
public Set<Long> ownedIgdbIds(Long userId, Collection<IgdbApiService.Game> games) {
if (games.isEmpty()) {
return Set.of();
}
Map<Long, Long> targets = mergeRepository.findTargets(games.stream().flatMap(game -> ownKeys(game).stream()).toList());
Set<Long> owned = new HashSet<>(userGameRepository.findActiveAppids(userId,
games.stream().flatMap(game -> possibleKeys(game, targets).stream()).distinct().toList()));
return games.stream().filter(game -> possibleKeys(game, targets).stream().anyMatch(owned::contains))
.map(IgdbApiService.Game::igdbId).collect(Collectors.toSet());
}
private AddResult addInTransaction(Long userId, IgdbApiService.Game game, String platform, GameStatus status, private AddResult addInTransaction(Long userId, IgdbApiService.Game game, String platform, GameStatus status,
int playtimeMinutes) { int playtimeMinutes, Long importId) {
return transaction.execute(tx -> { return transaction.execute(tx -> {
Instant now = Instant.now(); Instant now = Instant.now();
Map<Long, Long> targets = mergeRepository.findTargets(ownKeys(game)); Map<Long, Long> targets = mergeRepository.findTargets(ownKeys(game));
@@ -210,8 +246,12 @@ public class ManualGameService {
revived.setRemovedAt(null); revived.setRemovedAt(null);
// Only a Steam game can be hidden. // Only a Steam game can be hidden.
revived.setHiddenAt(null); revived.setHiddenAt(null);
// The row holds rating, notes and history from before, so an undo of the import must not delete it.
revived.setImportId(null);
} else { } else {
userGameRepository.save(UserGame.manual(userId, appid, platform, status, playtimeMinutes, now)); UserGame added = UserGame.manual(userId, appid, platform, status, playtimeMinutes, now);
added.setImportId(importId);
userGameRepository.save(added);
} }
return new AddResult.Added(appid); return new AddResult.Added(appid);
}); });
@@ -263,7 +303,7 @@ public class ManualGameService {
} }
// A longer name would not fit the column; with no platform left, the user types one. // A longer name would not fit the column; with no platform left, the user types one.
private static List<String> fittingPlatforms(IgdbApiService.Game game) { static List<String> fittingPlatforms(IgdbApiService.Game game) {
return game.platforms().stream().filter(platform -> platform.length() <= ManualGameUpdate.MAX_PLATFORM_LENGTH).toList(); return game.platforms().stream().filter(platform -> platform.length() <= ManualGameUpdate.MAX_PLATFORM_LENGTH).toList();
} }
@@ -0,0 +1,51 @@
-- Imports of many games at once (item 23): a list or a CSV, matched with IGDB in the background, then saved as
-- games added by hand. A game an import added keeps its import_id, so the import can be undone.
-- The table is also the queue of the two workers (GameImportWorker):
-- QUEUED -> MATCHING -> READY (the user reviews) -> SAVING -> SAVED (-> UNDONE), or FAILED after 3 tries.
-- queued_at orders each queue. progress_at is set while a worker runs the import and after every line; NULL while
-- it waits. attempts counts the tries a watchdog or a restart stopped.
CREATE TABLE game_import (
id BIGSERIAL PRIMARY KEY,
user_id BIGINT NOT NULL,
source VARCHAR(16) NOT NULL,
status VARCHAR(16) NOT NULL,
total INT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
queued_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
progress_at TIMESTAMPTZ,
attempts INT NOT NULL DEFAULT 0,
-- While SAVING: NULL = adding the games, DETAILS = loading their time to beat and moods.
step VARCHAR(16),
saved_at TIMESTAMPTZ
);
CREATE INDEX game_import_user ON game_import (user_id);
CREATE INDEX game_import_queue ON game_import (status, queued_at);
-- One import in the matching queue per user; an import of 1000 lines takes minutes of the shared IGDB budget.
CREATE UNIQUE INDEX game_import_one_matching ON game_import (user_id) WHERE status IN ('QUEUED', 'MATCHING');
-- One row per line of the input. igdb_id: the game the user already picked in the search (source SEARCH).
-- matched_at NULL: not matched yet. The chosen_* columns hold the user's review, save_result what the save did.
CREATE TABLE game_import_row (
import_id BIGINT NOT NULL REFERENCES game_import (id) ON DELETE CASCADE,
line_no INT NOT NULL,
title VARCHAR(200),
platform VARCHAR(32),
playtime_minutes INT,
status VARCHAR(16),
igdb_id BIGINT,
matched_at TIMESTAMPTZ,
search_failed BOOLEAN NOT NULL DEFAULT FALSE,
exact BOOLEAN NOT NULL DEFAULT FALSE,
-- The IGDB hits, best first, kept whole (IgdbApiService.Game), so the save needs no IGDB call.
candidates JSONB NOT NULL DEFAULT '[]',
chosen_igdb_id BIGINT,
chosen_platform VARCHAR(32),
chosen_status VARCHAR(16),
chosen_minutes INT,
-- ADDED or SKIPPED (in the library already, or IGDB does not know the game).
save_result VARCHAR(8),
PRIMARY KEY (import_id, line_no)
);
ALTER TABLE user_game ADD COLUMN import_id BIGINT REFERENCES game_import (id) ON DELETE SET NULL;
CREATE INDEX user_game_import ON user_game (import_id) WHERE import_id IS NOT NULL;
@@ -17,7 +17,7 @@ class AccountCleanupRepositoryTest extends AbstractPersistenceTest {
private static final List<String> USER_TABLES = List.of("user_game_status_change", "user_game_achievement", private static final List<String> USER_TABLES = List.of("user_game_status_change", "user_game_achievement",
"user_game_achievement_meta", "user_game_playtime_day", "wishlist_game", "backlog_snapshot", "user_game_achievement_meta", "user_game_playtime_day", "wishlist_game", "backlog_snapshot",
"user_steam_sync", "user_game"); "user_steam_sync", "user_game", "game_import");
@Autowired @Autowired
private AccountCleanupRepository repository; private AccountCleanupRepository repository;
@@ -42,6 +42,7 @@ class AccountCleanupRepositoryTest extends AbstractPersistenceTest {
jdbc.update(""" jdbc.update("""
insert into user_steam_sync (user_id, steam_id, user_seen_at) values (?, '76561198000000001', now())""", insert into user_steam_sync (user_id, steam_id, user_seen_at) values (?, '76561198000000001', now())""",
userId); userId);
jdbc.update("insert into game_import (user_id, source, status, total) values (?, 'TEXT', 'SAVED', 0)", userId);
} }
} }
@@ -0,0 +1,410 @@
package core_service.modules.library.api;
import core_service.configuration.security.ClientHeaderFilter;
import core_service.modules.igdb_adapter.service.IgdbApiService;
import core_service.modules.library.domain.GameSource;
import core_service.modules.library.domain.GameStatus;
import core_service.modules.library.domain.UserGame;
import core_service.modules.library.domain.UserGameId;
import core_service.modules.library.repo.UserGameRepository;
import core_service.modules.library.service.GameImportWorker;
import core_service.shared.test.AbstractIntegrationTest;
import com.jayway.jsonpath.JsonPath;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.webmvc.test.autoconfigure.AutoConfigureMockMvc;
import org.springframework.http.MediaType;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.test.context.bean.override.mockito.MockitoBean;
import org.springframework.test.web.servlet.MockMvc;
import org.springframework.test.web.servlet.ResultActions;
import java.time.Instant;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import static org.hamcrest.Matchers.contains;
import static org.hamcrest.Matchers.nullValue;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyLong;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.delete;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.post;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.jsonPath;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;
/**
* The dev profile authenticates every request as user 1.
*/
@AutoConfigureMockMvc
class GameImportControllerTest extends AbstractIntegrationTest {
private static final long USER_ID = 1L;
private static final IgdbApiService.Game ZELDA = new IgdbApiService.Game(119388, "Zelda: Tears of the Kingdom", 2023,
null, List.of("Switch"), null);
private static final IgdbApiService.Game ELDEN_RING = new IgdbApiService.Game(119133, "Elden Ring", 2022,
null, List.of("PS5", "PC"), 1245620L);
private static final IgdbApiService.Game NIGHTREIGN = new IgdbApiService.Game(325591, "Elden Ring Nightreign", 2025,
null, List.of("PS5", "PC"), 2622380L);
@Autowired
private MockMvc mockMvc;
@Autowired
private UserGameRepository userGameRepository;
@Autowired
private GameImportWorker worker;
@Autowired
private JdbcTemplate jdbc;
@MockitoBean
private IgdbApiService igdbApiService;
@BeforeEach
void seed() {
jdbc.update("delete from game_alias");
userGameRepository.deleteAll();
jdbc.update("delete from game_import");
when(igdbApiService.isEnabled()).thenReturn(true);
when(igdbApiService.searchGames("Elden Ring")).thenReturn(Optional.of(List.of(NIGHTREIGN, ELDEN_RING)));
when(igdbApiService.searchGames("Zelda Tears")).thenReturn(Optional.of(List.of(ZELDA)));
when(igdbApiService.getGame(119388)).thenReturn(Optional.of(ZELDA));
when(igdbApiService.getTimeToBeat(any())).thenReturn(Optional.of(Map.of()));
when(igdbApiService.getTimeToBeatByGameIds(any())).thenReturn(Optional.of(Map.of()));
}
private ResultActions create(String json) throws Exception {
return mockMvc.perform(post("/api/library/imports")
.header(ClientHeaderFilter.CLIENT_HEADER, "test")
.contentType(MediaType.APPLICATION_JSON)
.content(json));
}
private long createReady(String json) throws Exception {
long id = ((Number) JsonPath.read(create(json).andExpect(status().isAccepted())
.andReturn().getResponse().getContentAsString(), "$.id")).longValue();
awaitReady(id);
return id;
}
private void awaitReady(long id) throws InterruptedException {
awaitStatus(id, "READY");
}
// The matching and the save run in the background.
private void awaitStatus(long id, String expected) throws InterruptedException {
for (int attempt = 0; attempt < 50; attempt++) {
if (expected.equals(jdbc.queryForObject("select status from game_import where id = ?", String.class, id))) {
return;
}
Thread.sleep(100);
}
throw new AssertionError("Import " + id + " not " + expected);
}
private ResultActions save(long id, String rows) throws Exception {
return mockMvc.perform(post("/api/library/imports/" + id + "/save")
.header(ClientHeaderFilter.CLIENT_HEADER, "test")
.contentType(MediaType.APPLICATION_JSON)
.content("{\"rows\": [" + rows + "]}"));
}
private void saveAndWait(long id, String rows) throws Exception {
save(id, rows).andExpect(status().isAccepted());
awaitStatus(id, "SAVED");
}
// Not woken: no worker takes it unless the test wakes one.
private long insertImport(long userId, String status, String set) {
long id = jdbc.queryForObject("insert into game_import (user_id, source, status, total) "
+ "values (?, 'TEXT', ?, 0) returning id", Long.class, userId, status);
if (!set.isEmpty()) {
jdbc.update("update game_import set " + set + " where id = ?", id);
}
return id;
}
private UserGame stored(long appid) {
return userGameRepository.findById(new UserGameId(USER_ID, appid)).orElseThrow();
}
private Long importIdOf(long appid) {
return jdbc.queryForObject("select import_id from user_game where user_id = ? and appid = ?", Long.class,
USER_ID, appid);
}
@Test
void create_MatchesInTheBackground_SameNameFirstAndMarksGamesInTheLibrary() throws Exception {
jdbc.update("insert into steam_game (appid, name) values (1245620, 'ELDEN RING') on conflict do nothing");
userGameRepository.save(UserGame.manual(USER_ID, 1245620L, "PC", GameStatus.BACKLOG, 0, Instant.now()));
long id = createReady("{\"source\": \"TEXT\", \"rows\": [{\"title\": \" Elden Ring \", \"platform\": \"PS5\"}, "
+ "{\"title\": \"Unknown Game\"}]}");
mockMvc.perform(get("/api/library/imports/" + id))
.andExpect(status().isOk())
.andExpect(jsonPath("$.total").value(2))
.andExpect(jsonPath("$.matched").value(2))
.andExpect(jsonPath("$.rows[0].title").value("Elden Ring"))
.andExpect(jsonPath("$.rows[0].platform").value("PS5"))
.andExpect(jsonPath("$.rows[0].exact").value(true))
.andExpect(jsonPath("$.rows[0].candidates[*].name").value(contains("Elden Ring", "Elden Ring Nightreign")))
.andExpect(jsonPath("$.rows[0].candidates[*].inLibrary").value(contains(true, false)))
// Mockito answers every other search with empty: IGDB failed.
.andExpect(jsonPath("$.rows[1].searchFailed").value(true))
.andExpect(jsonPath("$.rows[1].candidates").isEmpty());
mockMvc.perform(get("/api/library/imports"))
.andExpect(jsonPath("$.imports[0].id").value(id))
.andExpect(jsonPath("$.imports[0].status").value("READY"))
.andExpect(jsonPath("$.imports[0].rows").value(nullValue()));
}
@Test
void create_SearchRow_TakesThePickedGameWithoutASearch() throws Exception {
long id = createReady("{\"source\": \"SEARCH\", \"rows\": [{\"igdbId\": 119388}]}");
mockMvc.perform(get("/api/library/imports/" + id))
.andExpect(jsonPath("$.rows[0].exact").value(true))
.andExpect(jsonPath("$.rows[0].candidates[0].igdbId").value(119388));
verify(igdbApiService, never()).searchGames(any());
}
@Test
void create_TooManyRowsOrNoTitle_Is400() throws Exception {
String row = "{\"title\": \"Game\"}";
create("{\"source\": \"TEXT\", \"rows\": [" + String.join(",", Collections.nCopies(1001, row)) + "]}")
.andExpect(status().isBadRequest());
create("{\"source\": \"TEXT\", \"rows\": [{\"platform\": \"PC\"}]}").andExpect(status().isBadRequest());
}
@Test
void create_WhileAnotherImportIsMatching_Is409() throws Exception {
jdbc.update("insert into game_import (user_id, source, status, total) values (?, 'TEXT', 'MATCHING', 0)", USER_ID);
create("{\"source\": \"TEXT\", \"rows\": [{\"title\": \"Elden Ring\"}]}")
.andExpect(status().isConflict())
.andExpect(jsonPath("$.code").value("IMPORT_RUNNING"));
}
@Test
void get_ImportOfAnotherUser_Is404() throws Exception {
Long id = jdbc.queryForObject("insert into game_import (user_id, source, status, total) "
+ "values (2, 'TEXT', 'READY', 0) returning id", Long.class);
mockMvc.perform(get("/api/library/imports/" + id)).andExpect(status().isNotFound());
save(id, "").andExpect(status().isNotFound());
mockMvc.perform(delete("/api/library/imports/" + id + "/games").header(ClientHeaderFilter.CLIENT_HEADER, "test"))
.andExpect(status().isNotFound());
}
@Test
void save_AddsTheCheckedLinesWithTheImportId_SkipsDuplicatesAndOnlyOnce() throws Exception {
long id = createReady("{\"source\": \"CSV\", \"rows\": [{\"title\": \"Elden Ring\"}, {\"title\": \"Elden Ring\"}]}");
when(igdbApiService.getGame(anyLong())).thenReturn(Optional.empty());
saveAndWait(id, "{\"lineNo\": 1, \"igdbId\": 119133, \"platform\": \" PS5 \", \"status\": \"PLAYING\", \"playtimeMinutes\": 90},"
+ "{\"lineNo\": 2, \"igdbId\": 119133, \"platform\": \"PC\", \"status\": \"BACKLOG\"},"
+ "{\"lineNo\": 9, \"igdbId\": 119133, \"platform\": \"PC\", \"status\": \"BACKLOG\"}");
mockMvc.perform(get("/api/library/imports/" + id))
.andExpect(jsonPath("$.chosen").value(2))
.andExpect(jsonPath("$.saved").value(2))
.andExpect(jsonPath("$.added").value(1))
.andExpect(jsonPath("$.step").value(nullValue()))
.andExpect(jsonPath("$.savedAt").isNotEmpty());
UserGame game = stored(1245620L);
assertEquals(GameSource.MANUAL, game.getSource());
assertEquals("PS5", game.getPlatform());
assertEquals(GameStatus.PLAYING, game.getStatus());
assertEquals(90, game.getPlaytimeMinutes());
assertEquals(id, importIdOf(1245620L));
// The candidate was stored whole: no IGDB call for the save.
verify(igdbApiService, never()).getGame(119133);
save(id, "").andExpect(status().isConflict());
}
@Test
void save_GameFromANewSearch_IsReadFromIgdb() throws Exception {
long id = createReady("{\"source\": \"TEXT\", \"rows\": [{\"title\": \"Elden Ring\"}]}");
saveAndWait(id, "{\"lineNo\": 1, \"igdbId\": 119388, \"platform\": \"Switch\", \"status\": \"BACKLOG\"}");
assertEquals(id, importIdOf(-119388L));
}
@Test
void undo_RemovesTheGamesStillAddedByHand_AndCountsTheChangedOnes() throws Exception {
long id = createReady("{\"source\": \"TEXT\", \"rows\": [{\"title\": \"Elden Ring\"}, {\"title\": \"Zelda Tears\"}]}");
saveAndWait(id, "{\"lineNo\": 1, \"igdbId\": 119133, \"platform\": \"PC\", \"status\": \"BACKLOG\"},"
+ "{\"lineNo\": 2, \"igdbId\": 119388, \"platform\": \"Switch\", \"status\": \"BACKLOG\"}");
mockMvc.perform(get("/api/library/imports/" + id + "/undo-check"))
.andExpect(jsonPath("$.games").value(2))
.andExpect(jsonPath("$.changed").value(0));
jdbc.update("update user_game set status = 'PLAYING', status_set_by_user = true where appid = -119388");
mockMvc.perform(get("/api/library/imports/" + id + "/undo-check"))
.andExpect(jsonPath("$.changed").value(1));
// The Steam sync took Elden Ring over: the user owns it on Steam now.
jdbc.update("update user_game set source = 'STEAM' where appid = 1245620");
mockMvc.perform(get("/api/library/imports/" + id + "/undo-check"))
.andExpect(jsonPath("$.games").value(1));
mockMvc.perform(delete("/api/library/imports/" + id + "/games").header(ClientHeaderFilter.CLIENT_HEADER, "test"))
.andExpect(status().isNoContent());
assertTrue(userGameRepository.findById(new UserGameId(USER_ID, -119388L)).isEmpty());
assertEquals(GameSource.STEAM, stored(1245620L).getSource());
assertEquals("UNDONE", jdbc.queryForObject("select status from game_import where id = ?", String.class, id));
mockMvc.perform(delete("/api/library/imports/" + id + "/games").header(ClientHeaderFilter.CLIENT_HEADER, "test"))
.andExpect(status().isConflict());
}
@Test
void resume_CountsTheStoppedTryAndGoesOnWhereItStopped() throws Exception {
// Before this app started: a try the restart stopped.
long id = insertImport(USER_ID, "MATCHING", "progress_at = now() - interval '1 day'");
jdbc.update("insert into game_import_row (import_id, line_no, title, matched_at, exact) "
+ "values (?, 1, 'Done before', now(), true)", id);
jdbc.update("insert into game_import_row (import_id, line_no, title) values (?, 2, 'Elden Ring')", id);
worker.resume();
awaitReady(id);
verify(igdbApiService, never()).searchGames("Done before");
assertEquals(2, jdbc.queryForObject("select jsonb_array_length(candidates) from game_import_row "
+ "where import_id = ? and line_no = 2", Integer.class, id));
assertEquals(1, jdbc.queryForObject("select attempts from game_import where id = ?", Integer.class, id));
}
@Test
void watchdog_StopsATryWithoutProgress_AndFailsAfterThreeTries() throws Exception {
long stalled = insertImport(USER_ID, "MATCHING", "progress_at = now() - interval '10 minutes'");
long lastTry = insertImport(2L, "SAVING", "progress_at = now() - interval '10 minutes', attempts = 2");
long fine = insertImport(3L, "MATCHING", "progress_at = now()");
worker.watchdog();
awaitReady(stalled);
assertEquals(1, jdbc.queryForObject("select attempts from game_import where id = ?", Integer.class, stalled));
assertEquals("FAILED", jdbc.queryForObject("select status from game_import where id = ?", String.class, lastTry));
assertEquals(0, jdbc.queryForObject("select attempts from game_import where id = ?", Integer.class, fine));
}
@Test
void retry_PutsAFailedImportBackInTheQueueOfItsPhase() throws Exception {
long id = insertImport(USER_ID, "FAILED", "attempts = 3");
jdbc.update("insert into game_import_row (import_id, line_no, title) values (?, 1, 'Elden Ring')", id);
mockMvc.perform(post("/api/library/imports/" + id + "/retry").header(ClientHeaderFilter.CLIENT_HEADER, "test"))
.andExpect(status().isAccepted());
awaitReady(id);
assertEquals(0, jdbc.queryForObject("select attempts from game_import where id = ?", Integer.class, id));
mockMvc.perform(post("/api/library/imports/" + id + "/retry").header(ClientHeaderFilter.CLIENT_HEADER, "test"))
.andExpect(status().isConflict())
.andExpect(jsonPath("$.code").value("IMPORT_NOT_FAILED"));
}
@Test
void get_WaitingImport_ShowsHowManyImportsAreBeforeIt() throws Exception {
insertImport(2L, "QUEUED", "queued_at = now() - interval '2 minutes'");
insertImport(3L, "MATCHING", "queued_at = now() - interval '1 minutes', progress_at = now()");
long mine = insertImport(USER_ID, "QUEUED", "");
long saving = insertImport(USER_ID, "SAVING", "");
mockMvc.perform(get("/api/library/imports/" + mine))
.andExpect(jsonPath("$.status").value("QUEUED"))
.andExpect(jsonPath("$.ahead").value(2));
mockMvc.perform(get("/api/library/imports/" + saving)).andExpect(jsonPath("$.ahead").value(0));
jdbc.update("delete from game_import");
}
@Test
void list_AnswersAnObjectWithTheImports() throws Exception {
long id = insertImport(USER_ID, "READY", "");
mockMvc.perform(get("/api/library/imports"))
.andExpect(status().isOk())
.andExpect(jsonPath("$.imports[0].id").value(id));
}
// A lane can start before ApplicationReadyEvent (an early create, the first watchdog run).
@Test
void resume_DoesNotCountAnImportALaneRunsNow() throws Exception {
CountDownLatch release = new CountDownLatch(1);
when(igdbApiService.searchGames("Slow Game")).thenAnswer(invocation -> {
release.await(5, TimeUnit.SECONDS);
return Optional.of(List.of());
});
long id = insertImport(USER_ID, "QUEUED", "");
jdbc.update("insert into game_import_row (import_id, line_no, title) values (?, 1, 'Slow Game')", id);
worker.wakeMatching();
awaitStatus(id, "MATCHING");
try {
worker.resume();
} finally {
release.countDown();
}
awaitReady(id);
assertEquals(0, jdbc.queryForObject("select attempts from game_import where id = ?", Integer.class, id));
}
@Test
void save_RevivedRowOfTheLibrary_IsNotRemovedByTheUndo() throws Exception {
jdbc.update("insert into steam_game (appid, name) values (1245620, 'ELDEN RING') on conflict do nothing");
jdbc.update("insert into user_game (user_id, appid, status, rating, notes, source, removed_at) "
+ "values (?, 1245620, 'FINISHED', 5, 'Loved it', 'STEAM', now())", USER_ID);
long id = createReady("{\"source\": \"TEXT\", \"rows\": [{\"title\": \"Elden Ring\"}]}");
saveAndWait(id, "{\"lineNo\": 1, \"igdbId\": 119133, \"platform\": \"PS5\", \"status\": \"PLAYING\"}");
mockMvc.perform(delete("/api/library/imports/" + id + "/games").header(ClientHeaderFilter.CLIENT_HEADER, "test"))
.andExpect(status().isNoContent());
UserGame kept = stored(1245620L);
assertEquals(5, kept.getRating());
assertEquals("Loved it", kept.getNotes());
assertNull(importIdOf(1245620L));
}
@Test
void save_IgdbFailsForAPickFromANewSearch_KeepsTheLineForTheNextTry() throws Exception {
long id = createReady("{\"source\": \"TEXT\", \"rows\": [{\"title\": \"Elden Ring\"}]}");
when(igdbApiService.getGame(119388)).thenReturn(Optional.empty());
save(id, "{\"lineNo\": 1, \"igdbId\": 119388, \"platform\": \"Switch\", \"status\": \"BACKLOG\"}")
.andExpect(status().isAccepted());
awaitStatus(id, "FAILED");
assertNull(jdbc.queryForObject("select save_result from game_import_row where import_id = ?", String.class, id));
assertEquals(3, jdbc.queryForObject("select attempts from game_import where id = ?", Integer.class, id));
}
@Test
void undo_ImportThatFailedWhileSaving_RemovesTheGamesItAdded() throws Exception {
long id = insertImport(USER_ID, "FAILED", "attempts = 3");
jdbc.update("insert into steam_game (appid, name) values (-119388, 'Zelda') on conflict do nothing");
jdbc.update("insert into user_game (user_id, appid, status, source, platform, import_id) "
+ "values (?, -119388, 'BACKLOG', 'MANUAL', 'Switch', ?)", USER_ID, id);
mockMvc.perform(delete("/api/library/imports/" + id + "/games").header(ClientHeaderFilter.CLIENT_HEADER, "test"))
.andExpect(status().isNoContent());
assertTrue(userGameRepository.findById(new UserGameId(USER_ID, -119388L)).isEmpty());
}
}
@@ -0,0 +1,72 @@
package core_service.modules.library.service;
import core_service.modules.igdb_adapter.service.IgdbApiService;
import core_service.modules.library.domain.GameImport;
import core_service.modules.library.repo.GameImportRepository;
import org.junit.jupiter.api.Test;
import org.springframework.dao.DataAccessResourceFailureException;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.stream.IntStream;
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.timeout;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
class GameImportWorkerTest {
private static IgdbApiService.Game game(long igdbId, String name, String... platforms) {
return new IgdbApiService.Game(igdbId, name, null, null, List.of(platforms), null);
}
private static GameImport.Line line(String title, String platform) {
return new GameImport.Line(1, title, platform, null, null, null);
}
@Test
void sameName_IgnoresCaseAccentsPunctuationAndMarks() {
assertThat(GameImportWorker.sameName("The Witcher® 3: Wild Hunt", "the witcher 3 wild hunt")).isTrue();
assertThat(GameImportWorker.sameName("Pokémon Legends: Arceus", "Pokemon Legends Arceus")).isTrue();
assertThat(GameImportWorker.sameName("Doom", "Doom II")).isFalse();
}
@Test
void rank_SameNameFirst_ThenTheLinesPlatform_ThenIgdbsOrder_AtMostFive() {
List<IgdbApiService.Game> found = List.of(game(1, "Doom Eternal", "PC"), game(2, "DOOM", "PC"),
game(3, "Doom", "Switch"), game(4, "Doom 3"), game(5, "Doom 64"), game(6, "Doom II"));
assertThat(GameImportWorker.rank(line("Doom", "switch"), found)).extracting(IgdbApiService.Game::igdbId)
.containsExactly(3L, 2L, 1L, 4L, 5L);
assertThat(GameImportWorker.rank(line("Doom", null), found)).extracting(IgdbApiService.Game::igdbId)
.startsWith(2L, 3L);
assertThat(GameImportWorker.rank(line("x", null), IntStream.range(0, 9)
.mapToObj(i -> game(i, "Game " + i)).toList())).hasSize(GameImportWorker.MAX_CANDIDATES);
}
@Test
void watchdog_WakesALaneWhoseRunDiedOnADatabaseError() {
GameImportRepository repository = mock(GameImportRepository.class);
when(repository.findNext(GameImportWorker.MATCHING_QUEUE))
.thenThrow(new DataAccessResourceFailureException("connection lost"))
.thenReturn(Optional.of(7L), Optional.empty());
when(repository.findStalled(any())).thenReturn(Map.of());
GameImportWorker worker = new GameImportWorker(repository, mock(ManualGameService.class),
mock(IgdbApiService.class), mock(TimeToBeatService.class), mock(MoodService.class), 5);
try {
worker.wakeMatching();
verify(repository, timeout(2000)).findNext(GameImportWorker.MATCHING_QUEUE);
worker.watchdog();
// Import 7 waits in QUEUED: no try of it is running, so no watchdog rule ever picks it up.
verify(repository, timeout(2000)).markReady(7L);
} finally {
worker.stop();
}
}
}
@@ -18,10 +18,12 @@ import java.util.Base64;
* Full Spring context with Testcontainers. * Full Spring context with Testcontainers.
*/ */
@Tag("IntegrationTest") @Tag("IntegrationTest")
// The nightly sync would call Steam if a test ran at 23:55; speedrun.com needs no key, so it is switched off. // The nightly sync would call Steam if a test ran at 23:55; speedrun.com needs no key, so it is switched off. The
// import watchdog runs only when a test calls it.
// Every cached context keeps its own pool: small pools keep them all below Postgres' 100 clients. // Every cached context keeps its own pool: small pools keep them all below Postgres' 100 clients.
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT, @SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT,
properties = {"app.library.nightly-sync-cron=-", "app.library.image-repair-cron=-", "app.speedrun.enabled=false", properties = {"app.library.nightly-sync-cron=-", "app.library.image-repair-cron=-", "app.speedrun.enabled=false",
"app.library.import-watchdog-ms=86400000",
"spring.datasource.hikari.maximum-pool-size=4"}) "spring.datasource.hikari.maximum-pool-size=4"})
@ActiveProfiles("dev") @ActiveProfiles("dev")
public abstract class AbstractIntegrationTest { public abstract class AbstractIntegrationTest {