readme fixes (#985)

* Create logo-text-colored-new.png

* move image

* Update logo

* Update README.md

* add screenshots

* fix images...

* Update README.md

* fix `code_progress.png` as well

* playing around lol
This commit is contained in:
ManDude 2021-12-04 17:32:53 +00:00 committed by GitHub
parent b394b7e687
commit 0d7b7da116
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
5 changed files with 82 additions and 84 deletions

166
README.md
View file

@ -1,5 +1,5 @@
<p align="center"> <p align="center">
<img width="500" height="100%" src="./docs/img/logo-text-colored.5cda006b.png"> <img width="500" height="100%" src="./docs/markdown/imgs/logo-text-colored-new.png">
</p> </p>
<p align="center"> <p align="center">
@ -25,7 +25,7 @@
- [Directory Layout](#directory-layout) - [Directory Layout](#directory-layout)
- [More Documentation](#more-documentation) - [More Documentation](#more-documentation)
- [ASan Build](#asan-build) - [ASan Build](#asan-build)
- [On Windows / Visual Studio](#on-windows--visual-studio) - [On Windows / Visual Studio](#on-windows--visual-studio)
<!-- tocstop --> <!-- tocstop -->
## Project Description ## Project Description
@ -45,24 +45,22 @@ Our objectives are:
We support both Linux and Windows on x86-64. We support both Linux and Windows on x86-64.
## Current Status ## Current Status
So far, we've decompiled around 341,233 lines of GOAL code, out of an estimated 500,000 total lines and we've started work on an OpenGL renderer. Currently, the main display process (`*dproc*`) runs and sends data to our renderer. We can load textures, text files, and level files. Using keyboard controls, we can open the debug menu and turn on some simple debug visualizations. So far, we've decompiled around 341,233 lines of GOAL code, out of an estimated 500,000 total lines and we've started work on an OpenGL renderer. Currently, the main display process (`*dproc*`) runs and sends data to our renderer. We can load textures, text files, and level files. Using keyboard controls, we can open the debug menu and turn on some simple debug visualizations.
Here are some screenshots of the renderer: Here are some screenshots of the renderer:
![](./docs/img/screenshot_hut.jpg) ![](./docs/markdown/imgs/screenshot_hut.png)
![](./docs/img/screenshot_vi1.jpg) ![](./docs/markdown/imgs/screenshot_vi1.png)
YouTube playlist: YouTube playlist:
https://www.youtube.com/playlist?list=PLWx9T30aAT50cLnCTY1SAbt2TtWQzKfXX https://www.youtube.com/playlist?list=PLWx9T30aAT50cLnCTY1SAbt2TtWQzKfXX
To help with decompiling, we've built a decompiler that can process GOAL code and unpack game assets. We manually specify function types and locations where the original code had type casts until the decompiler succeeds, then we clean up the output of the decompiled code by adding comments and adjusting formatting, then save it in `goal_src`. Our decompiler is designed specifically for processing the output of the original GOAL compiler. As a result, when given correct casts, it often produces code that can be directly fed into a compiler and works perfectly. This is tested as part of our unit tests, and so far we have around 130,000 lines (220 files) that pass. To help with decompiling, we've built a decompiler that can process GOAL code and unpack game assets. We manually specify function types and locations where the original code had type casts until the decompiler succeeds, then we clean up the output of the decompiled code by adding comments and adjusting formatting, then save it in `goal_src`. Our decompiler is designed specifically for processing the output of the original GOAL compiler. As a result, when given correct casts, it often produces code that can be directly fed into a compiler and works perfectly. This is tested as part of our unit tests, and so far we have around 300,000 lines (380 files) that pass.
We don't save any assets from the game - you must bring your own copy of the game and use the decompiler to extract assets. We don't save any assets from the game - you must bring your own copy of the game and use the decompiler to extract assets.
## What's Next ## What's Next
- Continue decompilation of GOAL code. We've made huge progress recently in decompiling gameplay code. We're finishing that up and also working on the some of the rendering and collision code. Here's our decompilation progress over the past year: ![](./docs/markdown/imgs/code_progress.png)
- Continue decompilation of GOAL code. We've made huge progress recently in decompiling gameplay code. We're finishing that up and also working on the some of the rendering and collision code. Here's our decompilation progress over the past year:
![](./docs/img/code_progress.png)
- Start running gameplay code! We can now spawn simple actors like particle effects and collectables. - Start running gameplay code! We can now spawn simple actors like particle effects and collectables.
- Improve the decompiler. We are always finding new features and macros in the GOAL language. - Improve the decompiler. We are always finding new features and macros in the GOAL language.
- Investigate more complicated renderers. We have an in-progress port of the "tfrag" background renderer, shown in the screenshots above. - Investigate more complicated renderers. We have an in-progress port of the "tfrag" background renderer, shown in the screenshots above.
@ -88,12 +86,12 @@ Run tests:
./test.sh ./test.sh
``` ```
Note: we have found that `clang` and `lld` are significantly faster to compile and link than `gcc`, generate faster code, and have better warning messages. To install these: Note: we have found that `clang` and `lld` are significantly faster to compile and link than `gcc`, generate faster code, and have better warning messages. To install these:
``` ```sh
sudo apt install lld clang sudo apt install lld clang
``` ```
and run `cmake` (in a fresh build directory) with: and run `cmake` (in a fresh build directory) with:
``` ```sh
cmake -DCMAKE_SHARED_LINKER_FLAGS="-fuse-ld=lld" -DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld" -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ .. cmake -DCMAKE_SHARED_LINKER_FLAGS="-fuse-ld=lld" -DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld" -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ ..
``` ```
@ -138,7 +136,6 @@ nix-shell -A packages.x86_64-linux.jak-asan-dev # development environment with C
nix-build -A packages.x86_64-linux.jak-asan # package with Clang ASan build nix-build -A packages.x86_64-linux.jak-asan # package with Clang ASan build
``` ```
## Getting Started - Windows ## Getting Started - Windows
Install Visual Studio 2019 and get the C++ and CMake tools via the Visual Studio Installer Install Visual Studio 2019 and get the C++ and CMake tools via the Visual Studio Installer
@ -183,13 +180,13 @@ Running the decompiler on the entire game is slow and not needed, so it is recom
"decompile_code": false, // change this to false, don't decompile code "decompile_code": false, // change this to false, don't decompile code
``` ```
Place a copy of the game's files in `iso_data`, then run the decompiler with the `scripts/decomp.sh` script. Place a copy of the game's files in `iso_data` (on Windows, make a `jak1` subfolder and place the files there), then run the decompiler with the `scripts/shell/decomp.sh` (Linux) or `scripts/batch/decomp-jak1.bat` (Windows) script.
### Build Game ### Build Game
Run the OpenGOAL compiler `build/goalc/goalc`. Enter `(mi)` to build the `"iso"` target, which contains everything we have so far. Run the OpenGOAL compiler `build/goalc/goalc`, or use one of the `gc` scripts. Enter `(mi)` to build the `"iso"` target, which contains everything we have placed in the build system so far.
### Run Game ### Run Game
In a separate terminal, start the runtime with `build/game/gk -fakeiso -debug`. Then, in the OpenGOAL window, run `(mi)` to create the data for the game and give the REPL information for running code, `(lt)` to connect, `(lg)` to load the game engine and `(test-play)` to start the game engine. If it all works right, it will look something like this: In a separate terminal, start the runtime with `build/game/gk -fakeiso -debug`. Then, in the OpenGOAL window, run `(mi)` to create the data for the game and give the REPL information for running code, `(lt)` to connect, `(lg)` to load the game engine and `(test-play)` to start the game engine. If it all works right, it will look something like this:
``` ```
g > (lt) g > (lt)
[Listener] Socket connected established! (took 0 tries). Waiting for version... [Listener] Socket connected established! (took 0 tries). Waiting for version...
@ -205,99 +202,100 @@ gc> (test-play)
gc> gc>
``` ```
Then, in the graphics window, you can use the period key to bring up the debug menu. Then, in the graphics window, you can use the period key to bring up the debug menu. Controllers also work, using the same mapping as the original game.
Check out the `pc_debug` and `examples` folder under `goal_src` for some examples of GOAL code we wrote. They have instructions for how to run them. Check out the `pc_debug` and `examples` folder under `goal_src` for some examples of GOAL code we wrote. They have instructions for how to run them.
## Project Layout ## Project Layout
There are four main components to the project. There are four main components to the project.
The first is `goalc`, which is a GOAL compiler for x86-64. Our implementation of GOAL is called OpenGOAL. All of the compiler source code is in `goalc`. To run the compiler on Linux, there is a script `gc.sh`. The compiler is controlled through a prompt which can be used to enter commands to compile, connect to a running GOAL program for interaction, run the OpenGOAL debugger, or, if you are connected to a running GOAL program, can be used as a REPL to run code interactively. In addition to compiling code files, the compiler has features to pack and build data files. The first is `goalc`, which is a GOAL compiler for x86-64. Our implementation of GOAL is called OpenGOAL. All of the compiler source code is in `goalc`. To run the compiler on Linux, there is a script `gc.sh`. On Windows, there is a `gc.bat` scripts and a `gc-no-lt.bat` script, the latter of which will not attempt to automatically attach to a running target. The compiler is controlled through a prompt which can be used to enter commands to compile, connect to a running GOAL program for interaction, run the OpenGOAL debugger, or, if you are connected to a running GOAL program, can be used as a REPL to run code interactively. In addition to compiling code files, the compiler has features to pack and build data files.
The second component to the project is the decompiler. You must have a copy of the PS2 game and place all files from the DVD into the `iso_data` folder. Then run `decomp.sh` (Linux) to run the decompiler. For Windows, it is the `decomp-jak1.bat` file, and it wants your game's DVD files in a `jak1` folder inside `iso_data`. The decompile will extract assets to the `assets` folder. These assets will be used by the compiler when building the port, and you may want to turn asset extraction off after running it once. The decompiler will output code and other data intended to be inspected by humans in the `decompiler_out` folder. Stuff in this folder will not be used by the compiler. The second component to the project is the decompiler. You must have a copy of the PS2 game and place all files from the DVD into the `iso_data` folder. Then run `decomp.sh` (Linux) to run the decompiler. For Windows, it is the `decomp-jak1.bat` file, and it wants your game's DVD files in a `jak1` folder inside `iso_data`. The decompile will extract assets to the `assets` folder. These assets will be used by the compiler when building the port, and you may want to turn asset extraction off after running it once. The decompiler will output code and other data intended to be inspected by humans in the `decompiler_out` folder. Stuff in this folder will not be used by the compiler.
The third is the game source code, written in OpenGOAL. This is located in `goal_src`. All GOAL and GOOS code should be in this folder. Right now most of this is placeholders, but you can take a look at `kernel/gcommon.gc` or `goal-lib.gc` to see some in-progress source code. The third is the game source code, written in OpenGOAL. This is located in `goal_src`. All GOAL and GOOS code should be in this folder. Right now most of this is placeholders or incomplete, but you can take a look at `kernel/gcommon.gc` or `goal-lib.gc` to see some in-progress source code.
The final component is the "runtime", located in `game`. This is the part of the game that's written in C++. In the port, that includes: The final component is the "runtime", located in `game`. This is the part of the game that's written in C++. In the port, that includes:
- The "C Kernel", which contains the GOAL linker and some low-level GOAL language features. GOAL has a completely custom dynamically linked object file format so in order to load the first GOAL code, you need a linker written in C++. Some low-level functions for memory allocation, communicating with the I/O Processor, symbol table, strings, and the type system are also implemented in C, as these are required for the linker. It also listens for incoming messages from the compiler and passes them to the running game. This also initializes the game, by initializing the PS2 hardware, allocating the GOAL heaps, loading the GOAL kernel off of the DVD, and executing the kernel dispatcher function. This is in the `game/kernel` folder. This should be as close as possible to the game, and all differences should be noted with a comment. - The "C Kernel", which contains the GOAL linker and some low-level GOAL language features. GOAL has a completely custom dynamically linked object file format so in order to load the first GOAL code, you need a linker written in C++. Some low-level functions for memory allocation, communicating with the I/O Processor, symbol table, strings, and the type system are also implemented in C, as these are required for the linker. It also listens for incoming messages from the compiler and passes them to the running game. This also initializes the game, by initializing the PS2 hardware, allocating the GOAL heaps, loading the GOAL kernel off of the DVD, and executing the kernel dispatcher function. This is in the `game/kernel` folder. This should be as close as possible to the game, and all differences should be noted with a comment.
- Implementation of Sony's standard library. GOAL code can call C library functions, and Naughty Dog used some Sony library functions to access files, memory cards, controllers, and communicate with the separate I/O Processor. The library functions are in `game/sce`. Implementations of library features specific to the PC port are located in `game/system`. - Implementation of Sony's standard library. GOAL code can call C library functions, and Naughty Dog used some Sony library functions to access files, memory cards, controllers, and communicate with the separate I/O Processor. The library functions are in `game/sce`. Implementations of library features specific to the PC port are located in `game/system`.
- The I/O Processor driver, Overlord. The PS2 had a separate CPU called the I/O Processor (IOP) that was directly connected to the DVD drive hardware and the sound hardware. Naughty Dog created a custom driver for the IOP that handled streaming data off of the DVD. It is much more complicated than I first expected. It's located in `game/overlord`. Like the C kernel, we try to keep this as close as possible to the actual game. - The I/O Processor driver, Overlord. The PS2 had a separate CPU called the I/O Processor (IOP) that was directly connected to the DVD drive hardware and the sound hardware. Naughty Dog created a custom driver for the IOP that handled streaming data off of the DVD. It is much more complicated than I first expected. It's located in `game/overlord`. Like the C kernel, we try to keep this as close as possible to the actual game.
- Sound Code. Naughty Dog used a third party library for sound. We have not started on this yet. - Sound Code. Naughty Dog used a third party library for sound. We have not started on this yet.
- PC specific graphics stuff. We have a functional OpenGL renderer and context that can create a game window and display graphics on it. The specific renderers used by the game however are mostly unimplemented. We have a debug, sprite and tfrag renderers, which is enough to render most of the game world, on-screen text and sprites like the progress menu. This is located in `game/graphics`. Many liberties will be taken to make this work, but the end result should closely match the actual game.
- PC specific graphics stuff. We have not started on this yet.
## Directory Layout ## Directory Layout
- `assets`: extracted assets (textures, translated game text, audio) generated by the decompiler. Not included in the repository. This data is used when building the PC port. - `assets`: extracted assets (textures, translated game text, audio) generated by the decompiler. Not included in the repository. This data is used when building the PC port.
- `build`: C++ CMake build folder - `build`: C++ CMake build folder.
- `common`: common C++ code shared between the compiler, decompiler, and game. - `common`: common C++ code shared between the compiler, decompiler, and game.
- `audio`: tools for decoding the audio files - `audio`: tools for decoding the audio files.
- `cross_os_debug`: platform-independent library for implementing the OpenGOAL debugger. Linux-only currently - `cross_os_debug`: platform-independent library for implementing the OpenGOAL debugger. Linux and Windows.
- `cross_sockets`: platform-independent library for sockets. Used to connect the compiler to a running game. Linux and Windows. - `cross_sockets`: platform-independent library for sockets. Used to connect the compiler to a running game. Linux and Windows.
- `goos`: the compiler-time macro language and parser for OpenGOAL. - `goos`: the compiler-time macro language and parser for OpenGOAL.
- `type_system`: the OpenGOAL type system - `type_system`: the OpenGOAL type system.
- `texture`: texture unpacking and format conversion - `texture`: texture unpacking and format conversion.
- `util`, `math`, `log`: Random utility functions for accessing files, timers, etc. - `dma`: utilities for handling the transfer of PS2 DMA packets.
- `decompiler`: Source code for the decompiler - `util`, `math`, `log`: Random utility functions for accessing files, timers, etc.
- `analysis`: analysis algorithms - `decompiler`: Source code for the decompiler.
- `config`: JSON config files for the decompiler and type definition file. - `analysis`: analysis algorithms.
- `data`: utilities to extract assets from the game - `config`: JSON config files for the decompiler and type definition file.
- `Disasm`: MIPS disassembler - `data`: utilities to extract assets from the game.
- `Function`: Tools for analyzing GOAL functions - `Disasm`: MIPS disassembler.
- `gui`: an early prototype of a Python GUI for reading the output of the decompiler - `Function`: Tools for analyzing GOAL functions.
- `IR2`: the "Intermediate Representation" for GOAL functions and expressions - `gui`: an early prototype of a Python GUI for reading the output of the decompiler.
- `ObjectFile`: Utilities for processing the GOAL object file format. - `IR2`: the "Intermediate Representation" for GOAL functions and expressions.
- `scripts`: Useful scripts for setting up the decompilation - `ObjectFile`: Utilities for processing the GOAL object file format.
- `util`: random utilities - `scripts`: Useful scripts for setting up the decompilation.
- `VuDisasm`: disassembler for VU code - `util`: random utilities.
- `VuDisasm`: disassembler for VU code.
- `decompiler_out`: output of the decompiler that's not automatically used by the compiler. This is for humans to read and use. Not included in the repository. - `decompiler_out`: output of the decompiler that's not automatically used by the compiler. This is for humans to read and use. Not included in the repository.
- `docs`: more documentation! - `docs`: more documentation!
- `game`: the source code for the game executable - `game`: the source code for the game executable.
- `common`: shared stuff between the `kernel` (EE) and `overlord` (IOP) - `common`: shared stuff between the `kernel` (EE) and `overlord` (IOP).
- `graphic`: PC Port graphics - `graphics`: PC Port graphics.
- `kernel`: the part of the GOAL kernel written in C. The entry point for the game is in `kboot.cpp`. - `kernel`: the part of the GOAL kernel written in C. The entry point for the game is in `kboot.cpp`.
- `overlord`: the I/O processor driver used to get data off of the DVD - `overlord`: the I/O processor driver used to get data off of the DVD.
- `sce`: the Sony library implementation - `sce`: the Sony library implementation.
- `system`: PC-port specific OS-level stuff, like file I/O, threads, controllers, debug network connection - `system`: PC-port specific OS-level stuff, like file I/O, threads, controllers, debug network connection.
- `goal_src`: The GOAL code for the game. - `goal_src`: The GOAL code for the game.
- `build`: info related to the GOAL build system. - `build`: info related to the GOAL build system.
- `engine`: the game engine - `engine`: the game engine.
- `kernel`: The GOAL kernel - `kernel`: The GOAL kernel.
- `levels`: Level specific code. - `levels`: Level specific code.
- `goalc`: The OpenGOAL compiler - `old`: Unused or deleted files from the original game.
- `compiler`: The implementation of the OpenGOAL language - `dgos`: File lists for DGOs in the build system.
- `data_compiler`: Tools for packing data - `examples`: Example GOAL code we have written.
- `debugger`: The OpenGOAL debugger (part of the compiler) - `test`: Test code we have written.
- `emitter`: x86-64 emitter and object file generator - `pc_debug`: GOAL code for PC-specific debugging. Also written by us.
- `listener`: The OpenGOAL listener, which connects the compiler to a running GOAL program for the interactive REPL - `user`: User sandboxes. Written by you! And anyone else that wishes to share their user code.
- `make`: The OpenGOAL build system, builds both code and data files - `goalc`: The OpenGOAL compiler.
- `regalloc`: Register allocator - `compiler`: The implementation of the OpenGOAL language.
- `data_compiler`: Tools for packing data.
- `debugger`: The OpenGOAL debugger (part of the compiler).
- `emitter`: x86-64 emitter and object file generator.
- `listener`: The OpenGOAL listener, which connects the compiler to a running GOAL program for the interactive REPL.
- `make`: The OpenGOAL build system, builds both code and data files.
- `regalloc`: Register allocator.
- `iso_data`: - `iso_data`:
- `out`: Outputs from the build process. Only the `iso` subfolder should contain assets used by the game. - `out`: Outputs from the build process. Only the `iso` subfolder should contain assets used by the game.
- `iso`: Final outputs that are used by the game. - `iso`: Final outputs that are used by the game.
- `obj`: Object files generated by the compiler. - `obj`: Object files generated by the compiler.
- `resources`: To be removed. Contains fake versions of some files required to get things booting. - `resources`: To be removed. Contains fake versions of some files required to get things booting.
- `scripts`: Utility scripts. Windows-specific batch files are in a `batch` folder while Unix shell scripts are in a `shell` folder. - `scripts`: Utility scripts. Windows-specific batch files are in a `batch` folder while Unix shell scripts are in a `shell` folder.
- `test`: Unit tests (run on GitHub Actions) - `test`: Unit tests (run on GitHub Actions).
- `third-party`: Third party libraries - `third-party`: Third party libraries.
- CMake Code Coverage. For code coverage statistics on GitHub builds - CMake Code Coverage. For code coverage statistics on GitHub builds.
- `fmt`. String formatting library - `fmt`. String formatting library.
- `googletest`: Test framework - `googletest`: Test framework.
- `inja`: templating library used for generating test code for compiler tests - `inja`: templating library used for generating test code for compiler tests.
- `lzokay`: decompression code for Jak 2 and later DGOs - `lzokay`: decompression code for Jak 2 and later DGOs.
- `mman`: Windows library used to emulate `mmap` on Linux - `mman`: Windows library used to emulate `mmap` on Linux.
- `run-clang-format`: Utility to check and enforce code formatting - `run-clang-format`: Utility to check and enforce code formatting.
- `run-clang-tidy` - `run-clang-tidy`
- `zydis`: x86-64 disassembler used in the OpenGOAL debugger - `zydis`: x86-64 disassembler used in the OpenGOAL debugger.
- `json`: A JSON library - `json`: A JSON library.
- `replxx`: Used for the REPL input. Supports history and useful editing shortcuts. - `replxx`: Used for the REPL input. Supports history and useful editing shortcuts.
- `svpng`: Save a PNG file - `svpng`: Save a PNG file.
### On Windows / Visual Studio ### On Windows / Visual Studio
Until 16.9 Preview 4, when attaching a debugger to the ASan build, you must disable breaking on Win32 Access Violation exceptions. See the relevant section `Debugging - Exceptions` here https://devblogs.microsoft.com/cppblog/asan-for-windows-x64-and-debug-build-support/#known-issues Until 16.9 Preview 4, when attaching a debugger to the ASan build, you must disable breaking on Win32 Access Violation exceptions. See the relevant section `Debugging - Exceptions` here https://devblogs.microsoft.com/cppblog/asan-for-windows-x64-and-debug-build-support/#known-issues

Binary file not shown.

After

Width:  |  Height:  |  Size: 11 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 104 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 741 KiB