I did some manual modifications in a few places to work around some
truly mysterious control flow, and some unsupported stack array stuff.
Also fixes a bug in decompiling static improper lists.
As far as I can tell, guns work, other than some graphical issues and
the crazy particle spawning issue, but I strongly suspect these are
problems with the sparticle/graphics side.
- Split up DGOs between threads in the multithreaded offline test
- fix some random warnings
- make the sig paths decompile a bit nicer to make some files smaller
Running reference tests/decompiler should now be possible on macos
(arm). Most of the changes were just cleaning up places where we were
sloppy with ifdefs, but there were two interesting ones:
- `Printer.cpp` was updated to not use a recursive function for printing
lists, to avoid stack overflow
- I replaced xxhash with another version of the same library that
supports arm (the one that comes in zstd). The interface is C instead of
C++ but it's not bad to use. I confirmed that the extractor succeeds on
jak 1 iso so it looks like this gives us the same results as the old
library.
A big one...
I figure even if we would like to change the way the particle/scene code
is output -- it'd be easier to find patterns with it all decompiled.
I've updated my script so it can easily be used to mass update these
files:
```bash
task update-gsrc-glob GLOB="**/*-part*.gc"
```
> for example will update gsrc files with `part` in their name -- if
they are in ref tests (so uncompleted ones aren't touched)
I found a few issues along the way that I'll have to make issues for
soon.
This solves two main problems:
- the looming threat of running out of memory since every thread would
consume duplicate (and probably not needed) resources
- though I will point out, jak 2's offline tests seem to hardly use any
memory even with 400+ files, duplicated across many threads. Where as
jak 1 does indeed use tons more memory. So I think there is something
going on besides just the source files
- condense the output so it's much easier to see what is happening / how
close the test is to completing.
- one annoying thing about the multiple thread change was errors were
typically buried far in the middle of the output, this fixes that
- refactors the offline test code in general to be a lot more modular
The pretty printing is not enabled by default, run with `-p` or
`--pretty-print` if you want to use it
https://user-images.githubusercontent.com/13153231/205513212-a65c20d4-ce36-44f6-826a-cd475505dbf9.mp4
Mostly easy / particle def files. I did a bit in `ruins` but CFG
failures and missing `drawable` functions make it untestable for now so
I've put that on pause.
- target-part
- ;; ERROR: Failed to convert to atomic ops: Variable could not be
constructed from register r0 in `process-drawable-shock-wall-effect`
- gun-part
- missing sparticle decompiling case it seems related to `L155`
This is pretty rough but...im excited to see it working :)
The VU programs for the ocean renderer have changed a bit and
`ocean-texture` has a bunch of new stuff, otherwise things are
relatively similar to Jak 1.
This is the first time I used mips2c and I'm not sure I did it 100%
right, so that should be double-checked.
The only interesting one is `collide-hash`, which is untested.
The other two are very likely unused. I skipped the annoying code in
`collide-probe` because it's not used and the same as jak 1.
Offline tests ignore comments in their comparison, but there's no reason
to strip them from the file that goes into the reference test folder
when doing the typical update routine.
This just generates superfluous diffs for all the files already done
prior to this change and is meaningless (the lines are dropped anyway)
Couldn't finish any of the enemy/nav-enemy related files for one reason
or another, but quite a bit of work that will be easier to merge and
iterate on instead of keeping track of the branch.
enemy/idle-control has some very weird focus related code.
nav-mesh/nav-control still has a bunch of CFG resolution problems that
need to be manually resolved.
Co-authored-by: water <awaterford111445@gmail.com>
- fix issue described in
https://github.com/open-goal/jak-project/issues/1939
- fix `text`, which was manually patched with the wrong offset (was
reading the symbol value off by one byte)
- clean up some random useless prints
- make the offline tests keep trying if there's a comparison error,
clean up the output a bit so the diffs are all at the end.
The offline-tests are going to end up taking too long for jak 2, I did
some rough math and by the end of it we'll be spending almost 2 minutes
for a full offline test on my machine.
These changes allow us to throw hardware at the problem
Still some work to do to make the output nicer, but seems to be fairly
reliable. By default it still uses 1 thread, use `num_threads` to change
this.
Almost done:
- `target-handler` (`(none)` event handler casts and CFG error)
- `target2` (`(none)` event handler casts)
- `powerups` (`cloud-track` does some weird stuff with `handle`s)
- `gun-states` (CFG error)
Some progress in:
- `water-flow`
Additionally:
- Clean up the two year old Jak 3 config file and add a config skeleton
(disassembling seems to not have worked, but I was able to dump obj
files and the `all_scripts` file)
- Fix automatic skelgroup detection and `defskelgroup` macro for Jak 2
(closes#1950)
- When a function decompiles without any major errors, a warning is
generated with the op id for each unresolved load and store that will
likely fail to compile (closes#1933)
This PR does a few main things:
- finish decompiling the progress related code
- implemented changes necessary to load the text files end-to-end
- japanese/korean character encodings were not added
- finish more camera code, which is required to spawn the progress menu
/ init the default language settings needed for text
- initialized the camera as well
Still havn't opened the menu as there are a lot of checks around
`*target*` which I havn't yet gone through and attempted to comment out.
A few issues:
- lwidea's fr3 is getting loaded and unloaded all the time
- the debug line drawing clipping is wrong (doesn't seem wrong in pcsx2,
so I think this is on us)
- nothing actually using vis data yet
- at a large distance, our view frustum culling seems slightly too
aggressive (might be that viewport scissoring is wrong)
- in the city, things seem darker as you move away. unclear how this is
happening (fog?)
And everything else needed for them!
A couple functions are bad currently.
- fixes#1929 - untested on linux
- fixes#1924 - now you need to type `,` before a lambda you want to put
in a pair.
- fix debugger symbol table in jak 2
- made the decompiler output `(meters 2)` instead of `(meters 2.0)`
- fixed a bug with the bitfield enum special -1 case
- made bad game text decomp not exit the decompiler
- added `editable-player` and `script`
A small quality of life increase that is more impactful since jak 2 has
double the file count.
I often use the offline-tests to find compiler errors / automatically
resolve them when i am finalizing a file. As more and more files are
completed this becomes increasingly more inefficient. When I know that
only 1 file needs to be decompiled / compared / compiled, I'd prefer to
have a feature like this.
- decompile `subdivide`, `wind-work`, `tie-work`, `bsp`, `focus`
- support `ppacb` in compiler
- don't assert when bitfield stuff fails due to constant propgataion
weirdness
- finish up history
- div/mod unsigned assert fix in decompiler
- empty assert fix in decompiler for failed `add` type prop
- make jak 1 performance counters "work" (just measure time)
- fix cast/typos on pcgtb/vftoi15
I wasn't able to 100% complete `debug` due to a bunch of level boundary
debug stuff I couldn't figure out. But, I added a ref test and
documented/copied over everything else into goal_src.
Most of the functions that existed in Jak 1 are identical, others got an
extra param or 2 and some now make copies of arguments instead of
modifying them. There's a bunch of new functions, including all of the
debug functions that used to be in `level-boundary` are now in `debug`.
At the very end of `debug` there's also some weird asm functions
checking for some EE memory controller bug (not sure what's up with
that)?
* sparticle-launcher
* d/jak2: large amount of `sparticle-launcher` done
* d/jak2: finish the majority of `sparticle`
* decomp: improve format code ignoring
* d/jak2: make bits unique in `sp-cpuinfo-flag`
* d/jak1: revert config change
* d/jak2: finish `cylinder`
* d/jak2: mostly finish `history`
* d/jak2: finish a bit of `ripple` and `sync-info`
* d/jak2: finish `font-data` and chip away at some other files
* d/jak2: most of `text` done, a bit of `script` - symbol arr issue in `level-info`
* d/jak2: finish `mech-h`
* lint: format
* tests: fix typeconsistency
* d/jak2: address feedback
* fix utf-8 handling around env-vars
* fix file opening errors related to unicode
* add uncaught exception handler in `gk` to ensure something is logged
* gracefully fail if window icon cant be loaded and work with unicode
* linux fix and add changes to vendor file
* git: ignore vs build dir
* cmake: ditch `clang-cl` on windows in favor of actual `clang`
* build: suppress a significant number of warnings
* build: adjust workflows and vendor nasm
* docs: update docs to remove `clang-cl` mentions
* tests: move jak1 reference tests into their own folder
* tests: update offline tests to support multiple games
* tests: some additional fixes and multi-game handling
* tests: update reference tests
* extractor: refactor and cleanup for multi-game support
* deps: switch to `ghc::filesystem` as it is utf-8 everywhere by default
* extractor: finally working with unicode
* unicode: fix unicode cli args on windows in all `main` functions
* Remove assets folder, use more std::filesystem
* windows fix
* another one for windows
* another one
* better system for different folders
* rm debugging stuff
* let extractor override everything
* dont revert jak1 change
* PAL dumps files
* alrighty then
* make PAL warning slightly more specific
* PAL patches for `title-obs`
* Update all-types.gc
* PAL patch `beach-obs`
* `process-taskable` PAL patch
* `ambient` PAL patch
* `yakow` PAL patch
* `village-obs` PAL patch
* `sparticle-launcher` patch
* `swamp-obs` PAL patch
* `sequence-a-village1` PAL patch
* typo
* errors
* `powerups` PAL patch
* `ogreboss` PAL patch
* jak 1 v2 encoding
* `load-boundary` PAL patch
* `flying-lurker` PAL patch
* `mayor` PAL patch
* update game encoding to PAL (v2) encoding
* `cam-debug` and `cam-update` PAL patch
* `fisher` PAL patch
* `target` PAL patch
* `target2` PAL patch and fix text compiling
* `target-death` PAL patch
* `target-racer-h` PAL patch
* `logic-target` PAL patch
* `main` PAL patch
* `snow-flutflut-obs` PAL patch
* `rolling-obs` PAL patch
* `gsound` PAL patch
* update refs
* `progress` and `progress-draw` PAL patches
* clang
* wrong.
* complain
* clang
* fix test
* fix blurry jp text
* fix weird interrupt lag from setting window size
* patch more text lines, special handling for credits
* Update FontUtils.cpp
* Add xdelta3 and file patching interface
* add window lock toggle and update settings ver
* better particle hacks
* add PAL support to extractor
* Fix credits
* also NTSC-J support
* make xdelta3 a separate library
* address feedback
Co-authored-by: water <awaterford111445@gmail.com>
* update refs
* [decompiler] read and process art groups
* finish decompiler art group selection & detect in `ja-group?`
* make art stuff work on offline tests!
* [decompiler] detect `ja-group!` (primitive)
* corrections.
* more
* use new feature on skel groups!
* find `loop!` as well
* fully fledged `ja` macro & decomp + `loop` detect
* fancy fixed point printing!
* update source
* `:num! max` (i knew i should've done this)
* Update jak1_ntsc_black_label.jsonc
* hi imports
* make compiling the game work
* fix `defskelgroup`
* clang
* update refs
* fix chan
* fix seek and finalboss
* fix tests
* delete unused function
* track let rewrite stats
* reorder `rewrite_let`
* Update .gitattributes
* fix bug with `:num! max`
* Update robotboss-part.gc
* Update goal-lib.gc
* document `ja`
* get rid of pc fixes thing
* use std::abs
* first attempt
* fix
* zip to tar
* windows
* try again, std::filesystem sucks
* std::filesystem is still garbage
* std::filesystem is terrible
* std::filesystem continues to waste my time
* again
* neadsflaldksal;df
* begin generic merc
* more generic stuff
* generic dma generation not crashing, at least until high-speed-reject
* c
* unpacker for generic
* starting to work
* small fixes
* more fixes
* more progress
* cleanup
* fix tests
* no custom imgui assert on windows
* ci: switch to codacy for coverage
* docs: update badges
* decomp: allow overriding config flags via CLI
* cleanup: top level file cleanup
* docs: big README overhaul
Attempt to close#1128 and #1086
* decomp: attempt to detect if `iso_data` is missing or wrongly extracted
* game: switch to `fpng` for screenshots, allow for compression
closes#1035
* game: switch vsync control to a checkbox
* lint: format cpp files
* lint: format json files
* docs/scripts: organize taskfile
* delete unused shaders
* hide some options in debug menu
* change fullscreen logic a bit
* add "all actors" toggle
* borderless fix and fix alpha in direct renderer untextured (do we need a separate shader for that?)
* fix fuel cell orbit icons in widescreen
* fix `curve` types
* refs
* fix levitator task...
* fix some task stuff
* update font code a bit (temp)
* cmake, third-party and visual studio overhaul
* Update .gitmodules
* update modules
* clone repos
* fix encoding in zydis
* where did these come from
* try again
* add submodule
* Update 11zip
* Update 11zip
* Update 11zip
* delete
* try again
* clang
* update compiler flags
* delete 11zip. go away.
* Create memory-dump-p2s.py
* properly
* fix minimum architecture c++ compiler flags
* fix zydis
* oops
* Update all-types.gc
* fix clang-cl tests
* make "all actors" work better, entity debug qol
* update game-text conversion code to be more modularized
* Create vendor.txt
* fix typos and minor things
* update refs
* clang
* Attempt to add clang-cl support to vs2019 and CI
* vs2022 + clang-cl
* srsly? fix clang build
* Update launch.vs.json
* extend windows CI timer
* decomp: Add texture-upload to ref tests
* maybe 50% done?
* 5 functions to go!
* decomp: stuck in `navigate`
* work-around fp issue
* some cleanup and label casts
* working on supporting asm instructions -- this is currently WRONG
* support ASM operations
* fixes for asm op support
* decomp: finish the vast majority of `navigate`
* format
* update test though i think this suggests a regression!
* decomp: cleanup some more of navigate
* decomp: finish `rolling-lightning-mole`
* revert `r0` handling for `pcpyud` and `pextuw`
* update ref tests
* lint
* fix a failing test
* help
* navigate mostly works now, with some potential bugs
* remove my debugging logs
* update ref tests
* review feedback cleanup
* these are all likely fine
* can't get the crab to chase me anymore
* the crab is back
* cleanup: cleanup all-types and all state definitions
* these are fine
* these are fine
* these are fine
* everything is fine
* gsrc updated
* remove define-externs in gsrc
* decomp: finish `target-racer`
target-racing-turn-anim was the `cond` issue here
* decomp: finish `target-tube`
target-tube-turn-anim was the issue here
* temp
* temp
* wip
* more progress on the instance asm
* first half of tie extraction, up to dma lists
* more tie extraction
* first part figured out maybe
* bp1 loop seems to work, bp2 loop does not
* bp1 and bp2 appear working. sadly ip is needed
* ip1 outline, not working ip2
* just kidding, ip2 seems to work
* extraction seems to work
* basic rendering working
* tie fixes
* performance optimization of tie renderer
* hook up tie to engine
* fix more bugs
* cleanup and perf improvements
* fix tests
* ref tests
* mm256i for gcc
* CLANG
* windows
* more compile fixes
* fix fast time of day
* small fixes
* fix after merge
* clang
* stash
* more
* and more
* like 80% done
* trying to update things
* refreshed work on generic-obs
* blocked: down to only decompiler issues!
* decomp: finish `generic-obs` skip over the failing functions for now
* `pair` -> `object`s and fix 2/4 of the functions i had skipped
* address feedback
* decomp: mostly finish `cam-master`
* decomp/scripts: lots of work in cam-states
* stash
* Merge remote-tracking branch 'water111/master' into decomp/camera-master
Updated submodule third-party/googletest
* decompiler: Add support for non power of 2 offsets for inline arr access
* decomp: mostly finish `cam-states` need to fix a macro issue
* blocked: `cam-master` decompiler crash when adding casts
* decomp: finish `cam-states-dbg`
* decomp: mostly finish `pov-camera` with the exception of joint-related code
* decomp: `cam-debug` finished decompiling, no way does this compile yet though
* decomp: considerable work done in `cam-layout`
* decomp: `cam-layout` almost done!
* decomp: `pov-camera` finished, TC tests will fail for now
* decomp: working on resolving issues
* decomp: cam-layout decompiling
* fixing more issues in cam-master...one event handler remains
* skip problematic function in `cam-master` for now
* gsrc: update res macros
* decomp: finish `cam-states`
* decomp: giving up on `cam-debug`
* tests: allow skipping state handlers in ref tests
* decomp: working through cam-layout bugs
* decomp: allow for shifting non-integers
* decomp: finalize `cam-layout` and `cam-master`
* decomp: finalize `cam-states`
* cleanup: bi-annual formatting of the casting files
* formatting
* decomp: start working on beach-obs
* blocked: `beach-obs` mostly finished, but handle cast issues and unknown `prebind` func signature
* blocked: `villagep-obs` done, `s6` not being referred to as `self`
* decomp: finish `citadel-obs`
* decomp: finish `darkcave-obs`
* blocked: need to allow `hud-h` to decompile properly (#f as static pointer)
* decomp: finish `jungle-obs`
* decomp: finish `village-obs`
* blocked: `misty-obs` handle cast issues
* decomp: finish `village2-obs`
* decomp: 1 function left in `swamp-obs`, particle related -- maybe we know now?
* decomp: finish `swamp-obs`
* blocked: `maincave-obs` handle casts
* decomp: finish `sunken-obs`
* blocked: `rolling-obs` handle casts and hud-parts
* decomp: finish `firecanyon-obs`
* decomp: finish `ogre-obs`
* blocked: `village3-obs` gives up type analysis!
* blocked: `snow-obs` has hud-parts and handle casts code
* decomp: finish `snow-flutflut-obs`
* blocked: `lavatube-obs` has s6-1 issue
* blocked: `title-obs` handle cast issues
* fixed post merge problems
* decomp: finish `jungleb-obs`
* blocked: `training-obs` has `s6-1` issue
* fix type consistency
* scripts: Extend update script to handle the game-text-id enum as well
* git: Update git attributes to effectively halve PR burden
* fixed `sound-play-by-name` signature
* fix particle definitions in firecanyon-obs
* fix func signature in racer-states
* update ref tests
* tests: update current ref tests
* tests: add `joint` to ref-tests
* tests: add `process-drawable` to ref-tests
* updated gsrc
* add back manual fix
* address most feedback, update source files
* get rid of forward declarations in `darkcave-obs`
Co-authored-by: ManDude <7569514+ManDude@users.noreply.github.com>
* blocked: `powerups` finicky handle casts
* stuck: decompiled `ocean-texture` but `ocean-verts` is a mystery to me
* decomp: finish `merc-death`
* fix some issues, add a way to remind us for files we've manually touched
* tests: add `ambient` to ref tests, uncover compiler error
* tests: add `main` and `font` to ref tests
* decomp: mostly finish `cam-master`
* decomp/scripts: lots of work in cam-states
* stash
* Merge remote-tracking branch 'water111/master' into decomp/camera-master
Updated submodule third-party/googletest
* decompiler: Add support for non power of 2 offsets for inline arr access
* decomp: mostly finish `cam-states` need to fix a macro issue
* blocked: `cam-master` decompiler crash when adding casts
* decomp: finish `cam-states-dbg`
* decomp: mostly finish `pov-camera` with the exception of joint-related code
* decomp: `cam-debug` finished decompiling, no way does this compile yet though
* decomp: considerable work done in `cam-layout`
* decomp: `cam-layout` almost done!
* decomp: `pov-camera` finished, TC tests will fail for now
* decomp: working on resolving issues
* decomp: cam-layout decompiling
* fixing more issues in cam-master...one event handler remains
* skip problematic function in `cam-master` for now
* gsrc: update res macros
* decomp: finish `cam-states`
* decomp: giving up on `cam-debug`
* tests: allow skipping state handlers in ref tests
* decomp: working through cam-layout bugs
* decomp: allow for shifting non-integers
* decomp: finalize `cam-layout` and `cam-master`
* decomp: finalize `cam-states`
* cleanup: bi-annual formatting of the casting files
* formatting
* address feedback - leave the float labels alone for now
* address feedback
* linting/formatting
* update gsrc and ref tests
Co-authored-by: ManDude <7569514+ManDude@users.noreply.github.com>
* decomp `fisher`, `robotboss`, `light-eco`, `green-eco-lurker`, `sage-finalboss`, `robotboss-weapon`, `robotboss-misc`
* fixes
* add files
* add `:states` list to `deftype` and fix files
* test state forward decl's on a few more types
* also the refs
* add light-eco
* whatever
* time of day
* goal code seems to work
* stars at wrong spot
* stars and sun work
* debugging clouds
* fix texture correction
* sky works
* cleanup, add profiler
* clean up
* final clean up
* offline tests
* missing include
* blocked: `puffer` has state decomp issues
* blocked: decomp crash when adding process-drawble cast in `helix-water`
* blocked: `green-eco-lurker` no nice way to cast value out of EMB
* decomp: started on `citadel-sages`
* decomp: `seagull` blocked state decomp
* blocked: `sunken-pipegame` runs into handle related issues
* blocked: `snow-ram` and `snow-ram-boss` state decomp
* decomp: `rolling-race-ring` blocked, handle casts
* blocked: state decomp in `part-tester`
* decomp: started `joint-exploder`
* blocked: `anim-tester` has rare cmove-#f-zero usage
* decomp: I lied! `anim-tester` finished
* config: update banned list
* `flying-lurker`
* why'd this get left behind?
* also `ambient` and add a `banned_objects` config and fix offline tests
* clang
* fix `ambient-type-sound`
* mips 2 c basic version, not yet tested
* calling works without crashing, but the function doesn't
* it works
* add test
* cleanup and actually add the test
* dont use mips2c by default for font
* clean up formatting
* add script to make adding lambda label casts a bit easier
* decomp: almost finish `evil-bro`
blocked by handle -> process conversion it seems:
`No method or function named .asm.sllv.r0 for type (pointer process-tree)`
* so close! SLLV issue (not a simple handle scenario)
* decomp: finalize `evilbro` blocked by func return type
* decomp: finish `evilbro`
* add support for non virtual states
* typecheck go
* start on virtual states
* more support for virtual states
* offline passes
* fix tests
* use behavior shortcut instead of lambda
* final cleanup of virtual go
* unused var warnings and fix inconsistent enum decompile order on win vs linux
* fix thread safety bug with goal symbol table and vif1 interrupt handler
* fix type mistake
* started process on `camera`
* the bulk of `camera` is finished, with the exception of a few
* decomp: Confirmed function ret value
* decomp: `cam-combiner` dead code issue
* stash
* decomp: finish `cam-combiner`
* decomp: finish `cam-start`
* decomp: mostly finish `cam-update`, just needs polish now
* `cam-standard-event-handler` still causing issues
* `cam-combiner` issues with top-level `s6` usage
* decomp: update reference tests
* decomp: finalize `cam-start`
* decomp: `cam-update` cleaned up everything except the array of planes issue
* label correction
* still blocked in cam-combiner over casting issues
* decomp: resolve issues in `cam-start`
* decomp: finalize `cam-update`
* stash
* decomp: finalize `cam-combiner`
* decomp: finalize `camera`
* decomp: address feedback
* clean up
* before int to float stuff
* before trying to eliminate the separate read and write maps
* partial fix for register issues
* add missing include
* [decomp] `loader` prelim work + some cleanup
* more things
* more
* even more
* yet even more
* minor fixes
* decompiler fix + use behaviors for two funcs
* last functions
* Create loader_REF.gc
* more work
* change vag tool config format + unrelated farmer and yakow stuff
* update some things
* fix some decomp
* cleanup things i came across + make dgo compileable
* fix consistency test
* update refs
* offline test: skip buggy `external-art-buffer` method
* fix test
* decomp: mostly done `hint-control`
* decomp: Started and decent chunk of `menu` done
* temp stash
* decomp: escape from `menu` hell
* decomp: starting on `default-menu`
* decomp: As much as i can do in `default-menu` at this time
* decomp: clean up `hint-control`
* decomp: fix reference tests
* temp stash
* decomp: finalize `menu`
* decomp: add `menu` to goal_src
* decomp: finalize `hint-control`
* decomp: Resolve TypeConsistency issues
* and fix reference tests
* address feedback
* format and lint
* fix tests
* stash
* decompiler: support symbols in arrays
* decomp: a ton of lambda casts
* finish task control?
* add lambda label casts
* decomp: resolve some issues, add enums
* decomp: One last decompiler issue remains
* decomp: finalize `task-control`
* linting
* revert change to test change, didn't mean to commit
* decomp: comment out function call for now