update readme and fix unused function issue (#821)

README.md

@@ -43,6 +43,24 @@ Our objectives are:
 
 We support both Linux and Windows on x86-64.
 
+## Current Status
+So far, we've decompiled around 130,000 lines of GOAL code, out of an estimated 500,000 total lines and we've started work on an OpenGL renderer.  Currently, the main display process (`*dproc*`) runs and sends data to our renderer.  We can load textures, text files, and level files. Using keyboard controls, we can open the debug menu and turn on some simple debug visualizations.
+
+
+Video of debug menu and actor visibility:
+https://www.youtube.com/watch?v=0AoiYY4S7nI
+
+To help with decompiling, we've built a decompiler that can process GOAL code and unpack game assets. We manually specify function types and locations where the original code had type casts until the decompiler succeeds, then we clean up the output of the decompiled code by adding comments and adjusting formatting, then save it in `goal_src`. Our decompiler is designed specifically for processing the output of the original GOAL compiler.  As a result, when given correct casts, it often produces code that can be directly fed into a compiler and works perfectly. This is tested as part of our unit tests, and so far we have around 130,000 lines (220 files) that pass.
+
+We don't save any assets from the game - you must bring your own copy of the game and use the decompiler to extract assets.
+
+## What's Next
+
+We are focusing on three areas:
+
+- Continue decompilation of GOAL code. Recently we have started on the "gameplay" code for creatures and objects in the game world, which is going pretty fast.
+- Improve the decompiler. We are always finding new features in the GOAL language.
+- Investigate more complicated renderers. The font and debug rendering is much simpler than the highly optimized renderers used for characters and backgrounds. We are starting to look at some of these renderers in detail.
 
 ## Getting Started - Linux (Ubuntu)
 
@@ -147,6 +165,45 @@ You may also wish to view the files that pertain to each CMake target, rather th
 
 ![](./docs/markdown/imgs/cmake-target-view.png)
 
+## Building and Running the Game
+
+Getting a running game involves 4 steps:
+1. Build C++ tools (follow steps above)
+2. Extract assets from game
+3. Build game
+4. Run game
+
+### Extract Assets
+Running the decompiler on the entire game is slow and not needed, so it is recommended to just run it on data. Edit `decompiler/config/jak1-ntsc_black_label.jsonc` and disable `decompile_code`.
+```json
+  "decompile_code": false, // change this to false, don't decompile code
+```
+
+Place a copy of the game's files in `iso_data`, then run the decompiler with the `scripts/decomp.sh` script.
+
+### Build Game
+Run the OpenGOAL compiler `build/goalc/goalc`.  Enter `(mi)` to build the `"iso"` target, which contains everything we have so far.
+
+### Run Game
+In a separate terminal, start the runtime with `build/game/gk -fakeiso -debug`.  Then, in the OpenGOAL window, run `(lt)` to connect, `(lg)` to load the game engine `(test-play)` to start the game engine.  If it all works right, it will look something like this:
+```
+g > (lt)
+[Listener] Socket connected established! (took 0 tries). Waiting for version...
+Got version 0.8 OK!
+[Debugger] Context: valid = true, s7 = 0x147d24, base = 0x2123000000, tid = 2438049
+
+gc> (lg)
+10836466        #xa559f2              0.0000        ("game" "kernel")
+
+gc> (test-play)
+(play :use-vis #t :init-game #f) has been called!
+0        #x0              0.0000        0
+
+gc> 
+```
+Then, in the graphics window, you can use the period key to bring up the debug menu.
+
+Check out the `pc_debug` and `examples` folder under `goal_src` for some examples of GOAL code we wrote.  They have instructions for how to run them.
 
 ## Project Layout
 There are four main components to the project.
@@ -175,29 +232,34 @@ The final component is the "runtime", located in `game`. This is the part of the
 - `assets`: extracted assets (textures, translated game text) generated by the decompiler. Not included in the repository. To be used when building the PC port.
 - `build`: C++ CMake build folder
 - `common`: common C++ code shared between the compiler, decompiler, and game.
+  - `audio`: tools for decoding the audio files
   - `cross_os_debug`: platform-independent library for implementing the OpenGOAL debugger. Linux-only currently
   - `cross_sockets`: platform-independent library for sockets. Used to connect the compiler to a running game. Linux and Windows.
   - `goos`: the compiler-time macro language and parser for OpenGOAL.
   - `type_system`: the OpenGOAL type system
-  - `util`: Random utility functions for accessing files, timers, etc.
+  - `texture`: texture unpacking and format conversion
+  - `util`, `math`, `log`: Random utility functions for accessing files, timers, etc.
 - `decompiler`: Source code for the decompiler
+  - `analysis`: analysis algorithms
   - `config`: JSON config files for the decompiler and type definition file.
   - `data`: utilities to extract assets from the game
   - `Disasm`: MIPS disassembler
   - `Function`: Tools for analyzing GOAL functions
   - `gui`: an early prototype of a Python GUI for reading the output of the decompiler
-  - `IR`: the "Intermediate Representation" for GOAL functions
+  - `IR2`: the "Intermediate Representation" for GOAL functions and expressions
   - `ObjectFile`: Utilities for processing the GOAL object file format.
   - `scripts`: Useful scripts for setting up the decompilation
   - `util`: random utilities
+  - `VuDisasm`: disassembler for VU code
 - `decompiler_out`: output of the decompiler that's not automatically used by the compiler. This is for humans to read and use. Not included in the repository.
-- `doc`: more documentation!
+- `docs`: more documentation!
 - `game`: the source code for the game executable
-  - `common`: shared stuff between the `kernel` and `overlord`
+  - `common`: shared stuff between the `kernel` (EE) and `overlord` (IOP)
+  - `graphic`: PC Port graphics
   - `kernel`: the part of the GOAL kernel written in C. The entry point for the game is in `kboot.cpp`.
   - `overlord`: the I/O processor driver used to get data off of the DVD
   - `sce`: the Sony library implementation
-  - `system`: PC-port specific stuff
+  - `system`: PC-port specific OS-level stuff, like file I/O, threads, controllers, debug network connection
 - `goal_src`: The GOAL code for the game. It's mostly empty now.
   - `build`: info related to the GOAL build system.
   - `engine`: the game engine
@@ -209,6 +271,7 @@ The final component is the "runtime", located in `game`. This is the part of the
   - `debugger`: The OpenGOAL debugger (part of the compiler)
   - `emitter`: x86-64 emitter and object file generator
   - `listener`: The OpenGOAL listener, which connects the compiler to a running GOAL program for the interactive REPL
+  - `make`: The OpenGOAL build system, builds both code and data files
   - `regalloc`: Register allocator
 - `iso_data`: 
 - `out`: Outputs from the build process. Only the `iso` subfolder should contain assets used by the game.
@@ -232,30 +295,6 @@ The final component is the "runtime", located in `game`. This is the part of the
   - `svpng`: Save a PNG file
 
 
-## More Documentation
-Check out these files for more documentation. Some of it is still in progress
-- `doc/goal_dbg_doc.md`: OpenGOAL debugger
-- `doc/goal_doc.md`: OpenGOAL language
-- `doc/reader.md`: OpenGOAL "reader" documentation (OpenGOAL syntax)
-- `doc/type_system.md`: OpenGOAL type system documentation
-- `doc/porting_to_x86.md`: Summary of changes we're making to port to x86-64
-- `doc/goos.md`: GOOS macro language
-
-## ASan Build
-The project supports building with Address Sanitizer (https://github.com/google/sanitizers/wiki/AddressSanitizer) in Linux.
-```sh
-export CXX=clang++
-cmake .. -DASAN_BUILD=TRUE
-```
-You will have to delete the build folder when changing compilers.  When running `cmake`, you should see something like:
-```
--- The CXX compiler identification is Clang 10.0.0
-...
--- Doing ASAN build
-```
-
-Then you can run the tests, runtime, and compiler as normal and they will abort if ASan finds an error.
-
 ### On Windows / Visual Studio
 
 Until 16.9 Preview 4, when attaching a debugger to the ASan build, you must disable breaking on Win32 Access Violation exceptions.  See the relevant section `Debugging - Exceptions` here https://devblogs.microsoft.com/cppblog/asan-for-windows-x64-and-debug-build-support/#known-issues

decompiler/Disasm/InstructionMatching.cpp

@@ -288,7 +288,8 @@ bool is_gpr_2_imm_int(const Instruction& instr,
                       MatchParam<Register> src,
                       MatchParam<int32_t> imm) {
   return kind == instr.kind && dst == instr.get_dst(0).get_reg() &&
-         src == instr.get_src(0).get_reg() && imm == instr.get_src(1).get_imm();
+         src == instr.get_src(0).get_reg() && instr.get_src(1).is_imm() &&
+         imm == instr.get_src(1).get_imm();
 }
 
 /*!

decompiler/IR2/LabelDB.cpp

@@ -102,6 +102,15 @@ int LabelDB::get_index_by_offset(int seg, int offset) const {
   return m_labels_by_offset_into_seg.at(seg).at(offset);
 }
 
+std::optional<int> LabelDB::try_get_index_by_offset(int seg, int offset) const {
+  auto it = m_labels_by_offset_into_seg.at(seg).find(offset);
+  if (it == m_labels_by_offset_into_seg.at(seg).end()) {
+    return {};
+  } else {
+    return it->second;
+  }
+}
+
 int LabelDB::get_index_by_name(const std::string& name) const {
   return m_labels_by_name.at(name);
 }

decompiler/IR2/LabelDB.h

@@ -31,6 +31,7 @@ class LabelDB {
   const LabelInfo& lookup(int idx) const;
   const LabelInfo& lookup(const std::string& name) const;
   int get_index_by_offset(int seg, int offset) const;
+  std::optional<int> try_get_index_by_offset(int seg, int offset) const;
   int get_index_by_name(const std::string& name) const;
 
   LabelInfo set_and_get_previous(int idx,

decompiler/ObjectFile/ObjectFileDB_IR2.cpp

@@ -639,7 +639,14 @@ void ObjectFileDB::ir2_insert_lets(int seg) {
   for_each_function_in_seg(seg, [&](Function& func, ObjectFileData&) {
     if (func.ir2.expressions_succeeded) {
       attempted++;
-      combined_stats += insert_lets(func, func.ir2.env, *func.ir2.form_pool, func.ir2.top_form);
+      try {
+        combined_stats += insert_lets(func, func.ir2.env, *func.ir2.form_pool, func.ir2.top_form);
+      } catch (const std::exception& e) {
+        func.warnings.general_warning(
+            fmt::format("Error while inserting lets: {}\n Make sure that the return type is not "
+                        "none if something is actually returned.",
+                        e.what()));
+      }
     }
   });
 

decompiler/analysis/label_types.cpp

@@ -53,12 +53,17 @@ void find_functions(LabelDB* db, LinkedObjectFile* file) {
         // first, the label for the start of the function
         // + 4 bytes for the type tag.
         int offset_of_function = func.start_word * 4 + 4;
-        int idx_of_label = db->get_index_by_offset(seg, offset_of_function);
-        auto old = db->set_and_get_previous(idx_of_label, func.type, false, {});
-        if (old.known) {
-          throw std::runtime_error(fmt::format(
-              "There is a config entry for label {}, but it's a function. Remove the config.",
-              old.name));
+        auto idx_of_label = db->try_get_index_by_offset(seg, offset_of_function);
+        if (!idx_of_label) {
+          func.warnings.general_warning("Could not find any references to this function: {}",
+                                        func.guessed_name.to_string());
+        } else {
+          auto old = db->set_and_get_previous(*idx_of_label, func.type, false, {});
+          if (old.known) {
+            throw std::runtime_error(fmt::format(
+                "There is a config entry for label {}, but it's a function. Remove the config.",
+                old.name));
+          }
         }
       }
 
@@ -73,7 +78,9 @@ void find_boxed(LabelDB* db, LinkedObjectFile* file) {
     if ((lab.offset & 7) == BASIC_OFFSET) {
       // it's a basic! probably.
       const auto& word = file->words_by_seg.at(lab.target_segment).at((lab.offset - 4) / 4);
-      if (word.kind == LinkedWord::TYPE_PTR) {
+      // the snowball-bank is a weird basic with no fields other than the built-in type.
+      // so it can actually share a label with something else.
+      if (word.kind == LinkedWord::TYPE_PTR && word.symbol_name != "snowball-bank") {
         TypeSpec basic_type(word.symbol_name);
         const auto& existing = db->lookup(lab.name);
         if (existing.known) {

decompiler/util/data_decompile.cpp

@@ -351,7 +351,12 @@ goos::Object decomp_ref_to_inline_array_guess_size(
   const auto& start_label = labels.at(pointer_to_data.label_id);
   int end_label_idx =
       index_of_closest_following_label_in_segment(start_label.offset, my_seg, labels);
-  assert(end_label_idx >= 0);
+
+  if (end_label_idx < 0) {
+    throw std::runtime_error(
+        "Failed to find label: likely just an unimplemented case for when the data is the last "
+        "thing in the file.");
+  }
   const auto& end_label = labels.at(end_label_idx);
   // fmt::print("Data is from {} to {}\n", start_label.name, end_label.name);