Commit graph

261 commits

Author SHA1 Message Date
water111 e2b7e5c001
[goalc] reduce compiler memory usage (#2247) 2023-02-24 18:32:30 -05:00
Tyler Wilding 39658dfd71
docs: Automatically generate documentation from goal_src code (#2214)
This automatically generates documentation from goal_src docstrings,
think doxygen/java-docs/rust docs/etc. It mostly supports everything
already, but here are the following things that aren't yet complete:
- file descriptions
- high-level documentation to go along with this (think pure markdown
docs describing overall systems that would be co-located in goal_src for
organizational purposes)
- enums
- states
- std-lib functions (all have empty strings right now for docs anyway)

The job of the new `gen-docs` function is solely to generate a bunch of
JSON data which should give you everything you need to generate some
decent documentation (outputting markdown/html/pdf/etc). It is not it's
responsibility to do that nice formatting -- this is by design to
intentionally delegate that responsibility elsewhere. Side-note, this is
about 12-15MB of minified json for jak 2 so far :)

In our normal "goal_src has changed" action -- we will generate this
data, and the website can download it -- use the information to generate
the documentation at build time -- and it will be included in the site.
Likewise, if we wanted to include docs along with releases for offline
viewing, we could do so in a similar fashion (just write a formatting
script to generate said documentation).

Lastly this work somewhat paves the way for doing more interesting
things in the LSP like:
- whats the docstring for this symbol?
- autocompleting function arguments
- type checking function arguments
- where is this symbol defined?
- etc

Fixes #2215
2023-02-20 19:49:37 -05:00
ManDude 324def1303
split new pc features in some files into their own code files + address some old issues + ripple graphics improvements (#2216)
Moves PC-specific entity and debug menu things to `entity-debug.gc` and
`default-menu-pc.gc` respectively and makes `(declare-file (debug))`
work as it should (no need to wrap the entire file in `(when
*debug-segment*` now!).

Also changes the DGO descriptor format so that it's less verbose. It
might break custom levels, but the format change is very simple so it
should not be difficult for anyone to update to the new format. Sadly,
you lose the completely useless ability to use DGO object names that
don't match the source file name. The horror!

I've also gone ahead and expanded the force envmap option to also force
the ripple effect to be active. I did not notice any performance or
visual drawbacks from this. Gets rid of some distracting LOD and some
water pools appearing super flat (and pitch back for dark eco).

Fixes #1424
2023-02-13 21:39:14 +00:00
water111 45f7770ca2
[decomp] finish up nav-graph, traffic-engine, traffic-manager, height-map (#2191)
Some of these were almost done, just needed a few more types.
2023-02-05 18:16:58 -05:00
Tyler Wilding e6f55b9b55
d/jak2: decompile all *-texture files except castle-texture (#2149)
Fixes https://github.com/open-goal/jak-project/issues/2051
2023-01-28 17:37:37 -05:00
Tyler Wilding 98393c6f4c
goalc: support static arrays of types (#2140)
draft because using the array is a little weird still, don't feel like
dealing with window's slow debugging builds today.

I get the following weird error:
```clj
(define test-array (new 'static 'boxed-array :type type vector))
gr> (-> test-array 0)
1538004        #x1777d4              0.0000        vector
gr> (type? (-> test-array 0) type)
1342757        #x147d25              0.0000        #t
gr> (new 'static (-> test-array 0))
-- Compilation Error! --
Got 3 arguments, but expected 2
Form:
(-> test-array 0)
Location:
Program string:1
(new 'static (-> test-array 0))
^
Code:
(new 'static (-> test-array 0))
```

Maybe this is expected though and the `new` method wants a symbol, not a
type?

Fixes #2060

Co-authored-by: water <awaterford111445@gmail.com>
2023-01-21 21:40:39 -05:00
water111 d65a1c365a
some bug fixes (#2125) 2023-01-14 11:04:15 -05:00
Tyler Wilding 00ac12094e
goalc/repl: cleanup of goalc/REPL code and some QoL improvements (#2104)
- lets you split up your `startup.gc` file into two sections
  - one that runs on initial startup / reloads
  - the other that runs when you listen to a target
- allows for customization of the keybinds added a month or so ago
- removes a useless flag (--startup-cmd) and marks others for
deprecation.
- added another help prompt that lists all the keybinds and what they do

Co-authored-by: water <awaterford111445@gmail.com>
2023-01-07 11:24:02 -05:00
Tyler Wilding 2f4146d469
tests: make the offline tests aware of the current terminals row count (#2105)
This fixes the hideous output when your terminal would be too small to
hold all the threads.
2023-01-07 10:35:12 -05:00
Tyler Wilding 284fb09997
d/jak2: decompile fodder and nav-enemy (#2080)
fixes #1983 

First functioning enemy!


https://user-images.githubusercontent.com/13153231/210184845-f1d93491-11f5-4a64-b69c-41e535992ebc.mp4
2023-01-01 21:03:07 -05:00
Tyler Wilding 9c631e11fe
offline-test: Partition by DGO and colorize/condense output (#2045)
This solves two main problems:
- the looming threat of running out of memory since every thread would
consume duplicate (and probably not needed) resources
- though I will point out, jak 2's offline tests seem to hardly use any
memory even with 400+ files, duplicated across many threads. Where as
jak 1 does indeed use tons more memory. So I think there is something
going on besides just the source files
- condense the output so it's much easier to see what is happening / how
close the test is to completing.
- one annoying thing about the multiple thread change was errors were
typically buried far in the middle of the output, this fixes that
- refactors the offline test code in general to be a lot more modular

The pretty printing is not enabled by default, run with `-p` or
`--pretty-print` if you want to use it


https://user-images.githubusercontent.com/13153231/205513212-a65c20d4-ce36-44f6-826a-cd475505dbf9.mp4
2022-12-22 13:41:33 -05:00
Tyler Wilding ac3c4e59b0
goalc/repl: Allow hot-loading files via ml with just the object name (#2036)
This allows you to not have to define the entire file path to a source
file to re-compile and load it. Technically a stop-gap until editor
tools are developed around writing OpenGOAL.


![image](https://user-images.githubusercontent.com/13153231/203196148-de61cf4b-42c8-43dc-a7fd-80e6ba6f5ac2.png)

As opposed to `(ml "goal_src/jak2/engine/game/main.gc")` (which still
works)

This is accomplished via the following config (connection attempts is
irrelevant):
```json
{
  "numConnectToTargetAttempts": 1,
  "jak2": {
    "asmFileSearchDirs": [
      "goal_src/jak2"
    ]
  }
}
```

This also provides a way to make game-specific configurations for the
REPL fairly easily.
2022-11-29 19:22:22 -05:00
Tyler Wilding a0a85eb60a
repl: Add a few new quality of life improvements (#2030)
- You can define a `startup.gc` in your user folder, each line will be
executed on startup (deprecates the usefulness of some cli flags)
- You can define a `repl-config.json` file to override REPL settings.
Long-term this is a better approach than a bunch of CLI flags as well
- Via this, you can override the amount of time the repl will attempt to
listen for the target
- At the same time, I think i may have found why on Windows it can
sometimes take forever to timeout when the game dies, will dig into this
later
- Added some keybinds for common operations, shown here


https://user-images.githubusercontent.com/13153231/202890278-1ff2bb06-dddf-4bde-9178-aa0883799167.mp4
> builds the game, connects to it, attaches a debugger and continues,
launches it, gets the backtrace, stops the target -- all with only
keybinds.

If you want these keybinds to work inside VSCode's integrated terminal,
you need to add the following to your settings file
```json
"terminal.integrated.commandsToSkipShell": [
    "-workbench.action.quickOpen",
    "-workbench.action.quickOpenView"
  ]
```
2022-11-20 14:28:41 -05:00
water111 811b5de6cb
[decomp] collide-mesh (#1999) 2022-10-30 15:54:10 -04:00
water111 b6f0ef52b3
[decomp] fixes for loader and game-info (#1968)
Fixes for the functions that weren't decompiled. `initialize` in
`game-info.gc` was fixed manually.
2022-10-14 20:47:59 -04:00
Tyler Wilding e3473c1902
d/jak2: finish cty-guard-turret-button | race-h | height-map-h and a lot of rigid-body (#1957)
Also cleaned up `data_decompiler.cpp` to make it a lot less verbose to
add a special case for an array field.
2022-10-11 23:20:36 -04:00
Tyler Wilding f6bdc07990
d/jak2: finish progress menu code and initialize the camera (#1945)
This PR does a few main things:
- finish decompiling the progress related code
- implemented changes necessary to load the text files end-to-end
   - japanese/korean character encodings were not added
- finish more camera code, which is required to spawn the progress menu
/ init the default language settings needed for text
  - initialized the camera as well

Still havn't opened the menu as there are a lot of checks around
`*target*` which I havn't yet gone through and attempted to comment out.
2022-10-11 18:30:26 -04:00
Tyler Wilding 4d751af38e
logs: replace every fmt::print with a lg call instead (#1368)
Favors the `lg` namespace over `fmt` directly, as this will output the
logs to a file / has log levels.

I also made assertion errors go to a file, this unfortunately means
importing `lg` and hence `fmt` which was attempted to be avoided before.
But I'm not sure how else to do this aspect without re-inventing the
file logging.

We have a lot of commented out prints as well that we should probably
cleanup at some point / switch them to trace level and default to `info`
level.

I noticed the pattern of disabling debug logs behind some boolean,
something to consider cleaning up in the future -- if our logs were more
structured (knowing where they are coming from) then a lot this
boilerplate could be eliminated.

Closes #1358
2022-10-01 11:58:36 -04:00
ManDude 9351bf782e
[decomp2] game-info, game-task and task-control (#1884)
And everything else needed for them!

A couple functions are bad currently.

- fixes #1929 - untested on linux
- fixes #1924 - now you need to type `,` before a lambda you want to put
in a pair.
- fix debugger symbol table in jak 2
- made the decompiler output `(meters 2)` instead of `(meters 2.0)`
- fixed a bug with the bitfield enum special -1 case
- made bad game text decomp not exit the decompiler
- added `editable-player` and `script`
2022-09-27 19:44:20 -04:00
Hat Kid a53c06fe2b
decomp: target, target-board, board-states (#1915)
Just missing `target-board-handler` because it errored with `Bad delay
slot in clean_up_cond_no_else_final` and I wasn't able to figure out
where the asm branch was.

Commented out `target-board-clone-anim` because it crashes even after
adding `clone-anim` and `clone-anim-once`.
2022-09-25 12:07:37 -04:00
water111 edecac9f09
[decomp] add shrubbery renderer (#1914)
as with last time, fog and time of day are still garbage, but it still
works:


![image](https://user-images.githubusercontent.com/48171810/192119423-91b80500-a161-42ce-b734-0c528cc721cf.png)

The shrub near render is just falling back to generic again.
2022-09-24 17:46:13 -04:00
water111 6227c6d6a8
More array special cases (#1913)
also make jak 1 and jak 2 behave the same way, to reduce confusion. It
wasn't too bad to update jak 1.
2022-09-24 16:10:13 -04:00
water111 1b45aab3cc
[decompile] subdivide, wind-work, tie-work, bsp, focus (#1897)
- decompile `subdivide`, `wind-work`, `tie-work`, `bsp`, `focus`
- support `ppacb` in compiler
- don't assert when bitfield stuff fails due to constant propgataion
weirdness
- finish up history
- div/mod unsigned assert fix in decompiler
- empty assert fix in decompiler for failed `add` type prop
- make jak 1 performance counters "work" (just measure time)
- fix cast/typos on pcgtb/vftoi15
2022-09-17 14:58:25 -04:00
Tyler Wilding 82e0517275
d/jak2: get script decompiling, no ref tests yet (#1877)
Has boxed array accessing that prevents me from adding anything to ref
tests (the entire file is lambdas so the access pattern that i would
like to ignore happens at the top-level, can't ignore it.

This code actually already has quite a bit of original docstrings so
it's not too bad in that regard considering a `script-context` can have
16 arbitrary objects. It seems they rarely put more than a single object
in the context and the types are usually obvious / are actually type
checked!
2022-09-14 19:37:12 -04:00
water111 b5d21be9c5
W/misc fixes (#1838)
* temp

* temp

* before cleaning up

* cleanup merge

* fix warnings

* merge fix

* clang format
2022-09-05 20:29:12 -04:00
Tyler Wilding 0896bef2bf
jak1/speedruns: Some final touches for speedrunning in jak 1 (#1830)
* jak1: put common speedrunning code into it's own file

* jak1: enforce `60` fps while in speedrunning mode

* jak1: when speedrunning, display the version until you get the first powercell

* jak1: add an explicit option for skipping cutscenes

* jak1: extend `game-option` to allow any menu option to be disabled

* tests/jak1: allow whitelisting types to be redefined to satisfy typeconsistency checks

* jak1: add file headers

* jak1: cleanup bool checking

* test: delete the es substitle file

* test: add it back

* jak1: missed one cleanup spot related to bool comparisons
2022-09-02 18:15:42 -04:00
water111 bc3cd2cf72
[goalc] Fix error when putting #f in an array of symbols (#1804) 2022-08-26 15:54:29 -04:00
Tyler Wilding 090b2984a2
decomp: add docstring support to relevant places in all-types (#1753)
* decomp: add `docstring` support to relevant places in `all-types`

* decomp: output method docstring into the `defmethod` instead

* goalc: handle docstrings in `define[-extern]` (gracefully ignore for now)

* decomp: output docstrings for bitfield deftypes too

* goalc: fix `defenum` parsing when coming from the compiler (no symbol metadata to store to)

* lsp/tests: fix ups

* lint: formatting

* goalc: handle edge-case of defining a string constant

* cleanup leftovers, fix codacy issues, rename struct
2022-08-23 00:32:07 -04:00
ManDude dee8feb003
fix a game bug and add a new compiler feature (#1758)
* dont crash on inline static value array

* [goalc] add support for non-static inline arrays of values

* add a kernel group to jak1

* move and cleanup pc debug code

* random cleanup in `hud-classes`

* pc port fix for gondola spools

* format

* address feedback (wow it was actually just changing 1 line all along...)

* minor decomp cleanup

* ref test
2022-08-15 18:46:29 -04:00
water111 dc652d10c5
[jak2] decomp gkernel, setup offline tests (#1638)
* add comments

* oops

* format'

* spelling is hard
2022-07-12 18:50:18 -04:00
water111 5e23057ed1
[goalc] compile/run code for jak2 ckernel, set up dummy KERNEL.CGO (#1625)
[goalc] start can compile and run code for jak2 ckernel, set up dummy KERNEL.CGO
2022-07-08 19:23:49 -04:00
water111 28a2ecdfd3
[jak2] goalc supports multiple projects (#1619)
* [jak2] goalc supports multiple projects

* disable deci2 server if not debugging
2022-07-06 21:18:08 -04:00
Tyler Wilding 6446389263
extractor: cleanup, support unicode properly, and add multi-game support (#1609)
* extractor: refactor and cleanup for multi-game support

* deps: switch to `ghc::filesystem` as it is utf-8 everywhere by default

* extractor: finally working with unicode

* unicode: fix unicode cli args on windows in all `main` functions
2022-07-05 20:38:13 -04:00
ManDude 9676100039
finish out english subtitles (#1586)
* put some duplicated code in a func

* make jak 2 text "work"

* group up all subtitles c++ code into one folder

* compact single-line subtitles

* fix a couple compiler crashes

* Update game_subtitle_en.gd

* `rolling` and `sunken`

* `swamp`

* `ogre`

* `village3`

* `maincave`

* `snow`

* `lavatube`

* `citadel`

* Update .gitignore

* clang

* fix encoding and decoding for quote

* properly fix quotes

* subtitle deserialize: sort by kind, ID and name

* sub editor: fix line speaker not being converted

* cleanup game text ids 1

* update text ids 2

* update source

* update refs
2022-07-03 17:25:28 -04:00
water111 f763eb6bf0
[jak2] support for multiple out/ directories (#1585)
* [jak2] support for multiple out/ directories

* windows
2022-06-30 21:11:58 -04:00
Tyler Wilding 36dc015d62
jak2: Create goal_src skeleton (#1576) 2022-06-29 22:20:09 -04:00
ManDude 2649fd17ec
subtitle editor fixes + other smaller fixes (#1572)
* [extractor] validate files when extracted as folder

* jp text fixes

* move game text version to the text file and fix subtitle editor escape chars

* make bad subtitles not crash the game

* fix texscroll in lag

* fix mood, fix decomp of other versions, fix text decomp

* clang

* fix tests

* oops dammit

* new fixes

* shut up codacy

* fix nonexistant subtitles crashing the game

* fix text hacks and extractor re-use on folders
2022-06-29 19:43:23 -04:00
water111 628ce47b2e
compiler speed improvement (#1547) 2022-06-24 18:21:24 -04:00
ManDude 85bef15364
full ntsc-j mode + greatest hits support (#1525)
* remove str files from inputs

* removed unused verbose flag

* allow some kind of conditional game building for JP extras

* make `PrintBankInfo` do nothing

* clang

* Update gltf_mesh_extract.cpp

* fix `*jak1-full-game*` in extractor

* use json library to build json file

* screw it red label support as well
2022-06-23 18:44:02 -04:00
Tyler Wilding 2d595c1ac0
lint: add include sorting config to clang-format (#1517) 2022-06-22 23:37:46 -04:00
Tyler Wilding 9fdf0bbc2f
tools: Add cutscene player / subtitle editor window (#1429)
* stash

* temp

* tools: subtitle tool works! just gotta fill out the db / polish UX

* tools: added configuration for every subtitle we have so far

* tools: add some colors to the editor, time for repl controls and make it run the code!

* tools: continuing polish of tool, getting very close

* tools: finished UX polish, just need to write deserializers

* tools: added deserializer for subtitle data

* tools: exported subtitle files, all data appears intact

* tools: more UX polish and test all the cutscenes, majority work

* assets: update subtitle files

* lint: formatting and cleanup

* lint: codacy lints
2022-06-11 16:32:27 +01:00
water111 ecb2781a89
[decompiler] cleanup to get atomic ops working on jak2 inputs (#1426)
* [decompiler] cleanup to get atomic ops working on jak2 inputs

* clang format
2022-06-08 18:34:52 -04:00
ManDude d01c16f451
[game] music player (#1416)
* music flava hack toggle

* fix hint subtitles not being considered offscreen

* music player works

* fixes + citadel, maincave, lavatube music

* add remaining tracks

* fix test

* fixes

* fix lib
2022-06-04 14:53:49 -04:00
ManDude 0212aa10c9
[decomp] better handling of animation code and art files (#1352)
* update refs

* [decompiler] read and process art groups

* finish decompiler art group selection & detect in `ja-group?`

* make art stuff work on offline tests!

* [decompiler] detect `ja-group!` (primitive)

* corrections.

* more

* use new feature on skel groups!

* find `loop!` as well

* fully fledged `ja` macro & decomp + `loop` detect

* fancy fixed point printing!

* update source

* `:num! max` (i knew i should've done this)

* Update jak1_ntsc_black_label.jsonc

* hi imports

* make compiling the game work

* fix `defskelgroup`

* clang

* update refs

* fix chan

* fix seek and finalboss

* fix tests

* delete unused function

* track let rewrite stats

* reorder `rewrite_let`

* Update .gitattributes

* fix bug with `:num! max`

* Update robotboss-part.gc

* Update goal-lib.gc

* document `ja`

* get rid of pc fixes thing

* use std::abs
2022-05-20 02:30:14 +01:00
water111 be976d2e69
[merc] Add merc extraction (#1356)
* docs for ee merc code

* wip

* more extraction stuff

* partial mat1 working

* mat1

* cleanup

* partial mat2 and mat3 support

* merc extraction seems to work
2022-05-11 22:53:53 -04:00
Tyler Wilding 7b6d732a77
goalc: Add TCP server socket in REPL process (#1335)
* goalc: cleanup goalc's main method and add nrepl listener socket

* deps: add standalone ASIO for sockets

* lint: formatting

* common: make a common interface for creating a server socket

* goalc: setup new repl server

* deps: remove asio

* goalc: debug issues, nrepl is working again

* git: rename files

* attempt to fix linux function call

* test

* scripts: make the error message even more obvious....

* goalc: make suggested changes, still can't reconnect properly

* game: pull out single-client logic from XSocketServer

* nrepl: supports multiple clients and disconnection/reconnects

* goalc: some minor fixes for tests

* goalc: save repl history when the compiler reloads

* common: add include for linux networking

* a few small changes to fix tests

* is it the assert?

* change thread start order and add a print to an assert

Co-authored-by: water <awaterford111445@gmail.com>
2022-05-06 18:19:37 -04:00
ManDude 8eb0796f8e
subtitle editing and text project compiler rework (#1331)
* add a `subtitle` bucket and rename buckets 67 and 68

* automatic `heap-base`

* subtitles editor

* adjust default params

* bug

* Update subtitle.gc

* fix blue color inconsistency in progress

* void.

* Update subtitle.gc

* Update subtitle.gc

* Extend `heap-base` detection to check for errors

* sometimes it's actually manually set!

* remove broken error check

* rework how compiling text files works

* try out subtitles source v2

* bugfix

* oops

* Update game_text_common.cpp
2022-04-24 15:15:16 -04:00
ManDude 0cdeed9be7
minor windows fixes (#1311)
* [pp2] put `define` dest on a single line

* update source!

* Update type_analysis.cpp

* update old credits & racer code

* change clang-cl args (REALLY force avx)

* Update credits_REF.gc

* comment small unused code

* add timer to decompiler

* fix unnecessary copy-constructors (no speed increase)

* fixes

* Update expression_build.cpp

* wtf is this thing anyway

* im bored.

* clang

* fix!

* Revert "fix!"

This reverts commit 5b1ce6c718.

* Revert "clang"

This reverts commit 5e67d9ccd1.

* Revert "im bored."

This reverts commit 070e957ce8.

* Revert "Update expression_build.cpp"

This reverts commit b94d092fc5.

* Revert "fixes"

This reverts commit f3d871f60a.

* Revert "fix unnecessary copy-constructors (no speed increase)"

This reverts commit 9100725802.

* Keep the random inoffensive changes

* Revert "Update type_analysis.cpp"

This reverts commit d2456a5c75.

* Update type_analysis.cpp
2022-04-17 21:11:09 -04:00
water111 b263e33e94
[goalc] fix mod bug and add div tests (#1296)
* fix mod bug and add div tests

* update changelog
2022-04-11 20:53:24 -04:00
ManDude a7eee4fdc9
[game] pc port progress menu (#1281)
* fix typo

* more typo

* shorten discord rpc text

* allow expanding enums after the fact (untested)

* make `game_text` work similar to subtitles

* update progress decomp

* update some types + `do-not-decompile` in bitfield

* fixes and fall back to original progress code

* update `progress` decomp with new enums

* update config files

* fix enums and debug menu

* always allocate (but not use) a lot of particles

* small rework to display mode options

* revert resolution/aspect-ratio symbol mess

* begin the override stuff

* make `progress-draw` more readable

* more fixes

* codacy good boy points

* first step overriding code

* finish progress overrides, game options menu fully functional!

* minor fixes

* Update game.gp

* Update sparticle-launcher.gc

* clang

* change camera controls text

* oops

* some cleanup

* derp

* nice job

* implement menu scrolling lol

* make scrollable menus less cramped, fix arrows

* make some carousell things i guess

* add msaa carousell to test

* oops

* Update progress-pc.gc

* make `pc-get-screen-size` (untested)

* resolution menu

* input fixes

* return when selecting resolution

* scroll fixes

* Update progress-pc.gc

* add "fit to screen" button

* bug

* complete resolutions menu

* aspect ratio menu

* subtitles language

* subtitle speaker

* final adjustments

* ref test

* fix tests

* fix ref!

* reduce redundancy a bit

* fix mem leaks?

* save settings on progress exit

* fix init reorder

* remove unused code

* rename goal project-like files to the project extension

* sha display toggle

* aspect ratio settings fixes

* dont store text db's in compiler

* properly save+load native aspect stuff
2022-04-11 18:38:54 -04:00