I noticed that jak 3's compilation was spending a lot of time accessing
the `unordered_map`s we use to store constants and symbol types.
I repurposed the `EnvironmentMap` originally made for GOOS for this. It
turns out that we were copying the entire constant map whenever we
encountered a `deftype`, and fixed that too.
This speeds up jak3 compiles from ~16 to 11 seconds for me.
This updates `fmt` to the latest version and moves to just being a copy
of their repo to make updating easier (no editing their cmake / figuring
out which files to minimally include).
The motivation for this is now that we switched to C++ 20, there were a
ton of deprecated function usages that is going away in future compiler
versions. This gets rid of all those warnings.
The logger used in `goalc` tries to print an already-formatted string
`message` using `fmt::print(message);` Usually this doesn't cause
problems, but if you try to print, for example, an exception that has
special characters (notably `{`) it will try to do another round of
formatting/replacements, despite not having any args to replace with,
which ends up throwing another exception. This is why errors when
parsing custom level JSON cause the REPL to exit.
I've hopefully identified all the various instances of this across the
codebase
Major change to how `deftype` shows up in our code:
- the decompiler will no longer emit the `offset-assert`,
`method-count-assert`, `size-assert` and `flag-assert` parameters. There
are extremely few cases where having this in the decompiled code is
helpful, as the types there come from `all-types` which already has
those parameters. This also doesn't break type consistency because:
- the asserts aren't compared.
- the first step of the test uses `all-types`, which has the asserts,
which will throw an error if they're bad.
- the decompiler won't emit the `heap-base` parameter unless necessary
now.
- the decompiler will try its hardest to turn a fixed-offset field into
an `overlay-at` field. It falls back to the old offset if all else
fails.
- `overlay-at` now supports field "dereferencing" to specify the offset
that's within a field that's a structure, e.g.:
```lisp
(deftype foobar (structure)
((vec vector :inline)
(flags int32 :overlay-at (-> vec w))
)
)
```
in this structure, the offset of `flags` will be 12 because that is the
final offset of `vec`'s `w` field within this structure.
- **removed ID from all method declarations.** IDs are only ever
automatically assigned now. Fixes#3068.
- added an `:overlay` parameter to method declarations, in order to
declare a new method that goes on top of a previously-defined method.
Syntax is `:overlay <method-name>`. Please do not ever use this.
- added `state-methods` list parameter. This lets you quickly specify a
list of states to be put in the method table. Same syntax as the
`states` list parameter. The decompiler will try to put as many states
in this as it can without messing with the method ID order.
Also changes `defmethod` to make the first type definition (before the
arguments) optional. The type can now be inferred from the first
argument. Fixes#3093.
---------
Co-authored-by: Hat Kid <6624576+Hat-Kid@users.noreply.github.com>
Started at 349,880,038 allocations and 42s
- Switched to making `Symbol` in GOOS be a "fixed type", just a wrapper
around a `const char*` pointing to the string in the symbol table. This
is a step toward making a lot of things better, but by itself not a huge
improvement. Some things may be worse due to more temp `std::string`
allocations, but one day all these can be removed. On linux it saved
allocations (347,685,429), and saved a second or two (41 s).
- cache `#t` and `#f` in interpreter, better lookup for special
forms/builtins (hashtable of pointers instead of strings, vector for the
small special form list). Dropped time to 38s.
- special-case in quasiquote when splicing is the last thing in a list.
Allocation dropped to 340,603,082
- custom hash table for environment lookups (lexical vars). Dropped to
36s and 314,637,194
- less allocation in `read_list` 311,613,616. Time about the same.
- `let` and `let*` in Interpreter.cpp 191,988,083, time down to 28s.
Normally, when they allocate a VagCmd, they do a bunch of stuff to clear
all the status bits and reset things
in particular the InitVAGCmd function does a lot
but for the stereo command, they do a lot less:
Which means that the new_stereo_command can just have random status bits
left over from whatever the last user had.
we seem to end up in a state where byte21 is set, and this causes
everything else to be wrong and off-by-one dma transfer. My guess is
that the original game avoided this bug due to lucky timing that I don't
understand.
I think the fix of just clearing byte21 is ok because there's no way
that the old value of the byte is useful after the command is
repurposed.
Adds sprite distort, fixes buggy sprite rendering in progress, adds
scissoring support (used in various scrolling menus) and a very basic
implementation of `blit-displays`. This is enough to make the fade
effect in the progress menu work, along with all the menus working
properly without needing to use the REPL. This does not make screen
flipping and the filter when failing a mission work.
Added support in the decompiler for detecting `dma-buffer-add-gs-set`
and `dma-buffer-add-gs-set-flusha` and updated all of the Jak 2 code to
use it. Readability improved!
Fixes decompiler issue with `with-dma-buffer-add-bucket` not inlining
forms which broke syntax. Fixes store error warnings showing up for
non-existent stores, there is now a dedicated pass for this at the end.
I started work on making `BITBLTBUF` stuff work in the DirectRenderer,
but stopped for now because it wasn't strictly necessary. It will still
assert like before.
Slight change to float divide operations (again). Now it only turns into
inverse multiplication if the float is a power of 2 (positive or
negative). Non-zero immediate divisors will be compiled as regular float
divisions but will forgo the extra branches and checks for divide by
zero.
Also fixes#2584
Updates the decompiler for the new format and there's new macros. This
new format should be easier to read/parse.
Also rewrote `sp-init-fields!` (both jak 1 and 2) from assembly to GOAL.
Hopefully I did not miss any regressions in Jak 1/2 while updating the
files, it's a lot.
This automatically generates documentation from goal_src docstrings,
think doxygen/java-docs/rust docs/etc. It mostly supports everything
already, but here are the following things that aren't yet complete:
- file descriptions
- high-level documentation to go along with this (think pure markdown
docs describing overall systems that would be co-located in goal_src for
organizational purposes)
- enums
- states
- std-lib functions (all have empty strings right now for docs anyway)
The job of the new `gen-docs` function is solely to generate a bunch of
JSON data which should give you everything you need to generate some
decent documentation (outputting markdown/html/pdf/etc). It is not it's
responsibility to do that nice formatting -- this is by design to
intentionally delegate that responsibility elsewhere. Side-note, this is
about 12-15MB of minified json for jak 2 so far :)
In our normal "goal_src has changed" action -- we will generate this
data, and the website can download it -- use the information to generate
the documentation at build time -- and it will be included in the site.
Likewise, if we wanted to include docs along with releases for offline
viewing, we could do so in a similar fashion (just write a formatting
script to generate said documentation).
Lastly this work somewhat paves the way for doing more interesting
things in the LSP like:
- whats the docstring for this symbol?
- autocompleting function arguments
- type checking function arguments
- where is this symbol defined?
- etc
Fixes#2215
- lets you split up your `startup.gc` file into two sections
- one that runs on initial startup / reloads
- the other that runs when you listen to a target
- allows for customization of the keybinds added a month or so ago
- removes a useless flag (--startup-cmd) and marks others for
deprecation.
- added another help prompt that lists all the keybinds and what they do
Co-authored-by: water <awaterford111445@gmail.com>
Running reference tests/decompiler should now be possible on macos
(arm). Most of the changes were just cleaning up places where we were
sloppy with ifdefs, but there were two interesting ones:
- `Printer.cpp` was updated to not use a recursive function for printing
lists, to avoid stack overflow
- I replaced xxhash with another version of the same library that
supports arm (the one that comes in zstd). The interface is C instead of
C++ but it's not bad to use. I confirmed that the extractor succeeds on
jak 1 iso so it looks like this gives us the same results as the old
library.
This solves two main problems:
- the looming threat of running out of memory since every thread would
consume duplicate (and probably not needed) resources
- though I will point out, jak 2's offline tests seem to hardly use any
memory even with 400+ files, duplicated across many threads. Where as
jak 1 does indeed use tons more memory. So I think there is something
going on besides just the source files
- condense the output so it's much easier to see what is happening / how
close the test is to completing.
- one annoying thing about the multiple thread change was errors were
typically buried far in the middle of the output, this fixes that
- refactors the offline test code in general to be a lot more modular
The pretty printing is not enabled by default, run with `-p` or
`--pretty-print` if you want to use it
https://user-images.githubusercontent.com/13153231/205513212-a65c20d4-ce36-44f6-826a-cd475505dbf9.mp4
This allows you to not have to define the entire file path to a source
file to re-compile and load it. Technically a stop-gap until editor
tools are developed around writing OpenGOAL.
As opposed to `(ml "goal_src/jak2/engine/game/main.gc")` (which still
works)
This is accomplished via the following config (connection attempts is
irrelevant):
```json
{
"numConnectToTargetAttempts": 1,
"jak2": {
"asmFileSearchDirs": [
"goal_src/jak2"
]
}
}
```
This also provides a way to make game-specific configurations for the
REPL fairly easily.
- You can define a `startup.gc` in your user folder, each line will be
executed on startup (deprecates the usefulness of some cli flags)
- You can define a `repl-config.json` file to override REPL settings.
Long-term this is a better approach than a bunch of CLI flags as well
- Via this, you can override the amount of time the repl will attempt to
listen for the target
- At the same time, I think i may have found why on Windows it can
sometimes take forever to timeout when the game dies, will dig into this
later
- Added some keybinds for common operations, shown here
https://user-images.githubusercontent.com/13153231/202890278-1ff2bb06-dddf-4bde-9178-aa0883799167.mp4
> builds the game, connects to it, attaches a debugger and continues,
launches it, gets the backtrace, stops the target -- all with only
keybinds.
If you want these keybinds to work inside VSCode's integrated terminal,
you need to add the following to your settings file
```json
"terminal.integrated.commandsToSkipShell": [
"-workbench.action.quickOpen",
"-workbench.action.quickOpenView"
]
```
The offline-tests are going to end up taking too long for jak 2, I did
some rough math and by the end of it we'll be spending almost 2 minutes
for a full offline test on my machine.
These changes allow us to throw hardware at the problem
Still some work to do to make the output nicer, but seems to be fairly
reliable. By default it still uses 1 thread, use `num_threads` to change
this.
Favors the `lg` namespace over `fmt` directly, as this will output the
logs to a file / has log levels.
I also made assertion errors go to a file, this unfortunately means
importing `lg` and hence `fmt` which was attempted to be avoided before.
But I'm not sure how else to do this aspect without re-inventing the
file logging.
We have a lot of commented out prints as well that we should probably
cleanup at some point / switch them to trace level and default to `info`
level.
I noticed the pattern of disabling debug logs behind some boolean,
something to consider cleaning up in the future -- if our logs were more
structured (knowing where they are coming from) then a lot this
boilerplate could be eliminated.
Closes#1358
And everything else needed for them!
A couple functions are bad currently.
- fixes#1929 - untested on linux
- fixes#1924 - now you need to type `,` before a lambda you want to put
in a pair.
- fix debugger symbol table in jak 2
- made the decompiler output `(meters 2)` instead of `(meters 2.0)`
- fixed a bug with the bitfield enum special -1 case
- made bad game text decomp not exit the decompiler
- added `editable-player` and `script`
Fixes https://github.com/open-goal/jak-project/issues/1821 by adding a
special case for `new` method calls where the argument with type
`symbol` is actually an address to uninitialized structure on the stack.
Fixes https://github.com/open-goal/jak-project/issues/1849 (or at least
the cause of the issue Vaser gave in chat, and one random one I found in
`debug-sphere`)
Fixes https://github.com/open-goal/jak-project/issues/1853
Fixes https://github.com/open-goal/jak-project/issues/1857 by moving the
cast into the cond if the body is a single form and the destination type
is a bitfield/enum which is likely to work well. Seems to work on the
examples we could find in jak 1 and jak 2.
Also fixes an issue with casts on the result of `handle->process` (a
common place to use casts)
the output of process->handle is a plain process. Most of the time, you
end up casting this to a more specific. If you add a cast on every use
of the variable, the decompiler will decide to change the type of that
variable to the more specific type, and this breaks the handle cast.
so previously it was impossible to get code like
```
(let* ((s2-0 (the-as swingpole (handle->process (-> self control hack))))
(gp-0 (-> s2-0 dir))
)
```
But now it will work
* lsp: json-rpc example is working, a decent place to start...
* lsp: vendor library
* lsp: cleanup and time to get started
* lsp: commit what i got so far
* lsp: example `initialize` payload
* lsp: switch to `stdio`
* stash
* modularize the lsp implementation
* lsp: implement first actual LSP feature - function names in outline
* lsp: produce document diagnostics
* lsp: remove unused third-party lib
* lsp: support hovering MIPS instructions in IR files
* lsp: basic go-to all-types definition
* stash
* lsp: cleanup code, just need to add it to the release artifacts
* fix some project configuration
* fix linux build
* lsp: add lsp to PR artifacts and release assets
* lsp: address feedback
* fix utf-8 handling around env-vars
* fix file opening errors related to unicode
* add uncaught exception handler in `gk` to ensure something is logged
* gracefully fail if window icon cant be loaded and work with unicode
* linux fix and add changes to vendor file
* extractor: refactor and cleanup for multi-game support
* deps: switch to `ghc::filesystem` as it is utf-8 everywhere by default
* extractor: finally working with unicode
* unicode: fix unicode cli args on windows in all `main` functions
* fix repl-history saving
* subtitles: fix `\"` edge-case
* subtitles: add almost every remaining (current) cutscene
* merge: master
* subtitles: fix some crashes and disable `Remove Line` as it is broken
* some jp support to fix some errors in the original game
* music fade toggle
* recognize `process-new` macros!!
* strip casts in this macro
* rename macro
* fix cast typecheck
* update source 1
* detect kernel stack case
* less boilerplate
* `manipy-spawn` special case
* pretty printer improvements
* revert dumb thing from earlier
* use shell detection on `send-event`
* fix some events
* remove unused argument
* detect `static-attack-info` and add `CondNoElse` to shell detect
* better `attack-info` detect
* support `process-spawn` in multi-lets
* detect `rand-float-gen` pt 1
* detect as return value
* detect in `countdown` and `dotimes`
* oops this wasnt working
* fancier `send-event`s
* clang
* update source!!
* fix tests
* fine jeez
* uh okay
* fix some accidental regressions
* fix more regressions
* regression fixes
* fix big bug...
* extra safety!
