jak-project/docs/scratch/shrub.md

Shrub EE code notes

Looking through village1-vis.asm

The "prototype" main class is prototype-bucket-shrub. These are stored in a prototype-inline-array-shrub.

Example of a prototype-bucket-shrub:

    .type prototype-bucket-shrub
    .word L22404
    .word 0x0
    .word 0x20025
    .word L4630
    .word L4604
    .word L4617
    .word L4641
    .word 0x48a00000
    .word 0x48200000

The .word L22404 is a string "palmplant-top.mb". The four L4630, L4604, L4617, L4641 are the 4 geometry. In this case, it looks like each is a different kind, but I don't think we can make any assumptions about this.

Highest Detail prototype-generic-shrub

For now, let's ignore this because we don't have a generic renderer. It also really looks like no shrubs ever draw with generic in-game, but I could be wrong. It looks like every single shrub has this as their first geomery.

2nd Highest prototype-shrubbery

I think this is the "normal" shrub. Each prototype is made up of one or more shrubbery. Again, every shrub has this.

3rd Highest prototype-trans-shrubbery

Based on PCSX2, it looks like these are usually lower-res models that fade out as you move away. Like the 2nd, they are made of shrubbery, and every shrub has one.

4th Highest billboard

These are the lowest res, always face toward the camera. Each billboard is just a single thing - no sub-parts. I suspect that each prototype can become a single rectangle "billboard", no matter how complicated the original shrub is.

I think we should work on prototype-shrubbery (and possibly the transparent one) first. Maybe we can leave out the billboard and claim that we're "increasing the level of detail" :).

Looking through shrubbery.gc decomp

Doing reverse order in the ir2.asm file.

login billboard

No surprise here, billboard is a single rectangle, it can have only one texture, and there's exactly one adgif-shader-login.

mem-usage-shrub-walk

This function recursively iterates over a draw-node tree, computing memory usage, and is a good example of how these draw-node trees work. Each draw-node has a sphere. Inside this sphere are between 1 and 8 "children", which are in an inline-array. The children are usually draw-node, except for the leaf nodes, which are some other drawable type. In this case, they are instance-shrubbery. This is used for frustum culling. If a draw-node sphere is outside of the view of the camera, you can just skip all instances in that draw-node.

Unlike a normal tree, they rarely refer to a single draw-node, but instead a group of between 1 and 8 draw-nodes. These are passed around as a draw-node (the first in the group) and an integer (the number in the group). This might seem weird, but simplifies the iteration logic. (if you don't know the type/size of the leaf node, you can't iterate over an inline-array of them, but you could call some virtual function that expects the object to be the first one in an inline-array with a size argument.)

This is how to recursively iterate over draw-node tree:

    (dotimes (s1-0 arg1)
      (let ((a1-2 (-> s2-0 child-count)))
        (cond
          ((logtest? (-> s2-0 flags) 1) ;; flag means that we aren't a leaf.
           (mem-usage-shrub-walk (the-as draw-node (-> s2-0 child)) (the-as int a1-2) arg2 arg3)
           )
          (else
          	;; you are at an inline-array of leaves.
          	;; (-> s2-0 child) is an inline-array of a1-2 leaves
          	;; a1-2 is between 1 and 8.
          	)
          )
        )
        ;; move on to the next child draw node.
        (&+! s2-0 32)
      )

NOTE: arg1 should be an int here.

We learned here that shrub does use the draw-node BVH system. (this was unclear because shrub does not appear to use the compressed visibility string based vis system, which is often used simultaneously with draw-node).

login generic-shrub-fragment

generic-shrub-fragment's have a variable number of textures. The cnt-qwc field refers to the total size of the textures array, as each adgif-shader is 5 qw.

login prototype-shrubbery

From village1-vis, we know this is just logging in shrubberys.

asize-of prototype-shrubbery

The size calculation makes sense for an inline-array of shrubbery, with 1 shrubbery being inside the type, and the rest running off the end of the array.

login prototype-generic-shrub

No surprises. Confirms that these aren't inline arrays (and we could tell from village1-vis anyway)

login shrubbery

Gives us some idea of the layout of the "header". The first u32 is the number of textures / 2. Each shrubbery may have multiple textures.

login drawable-tree-instance-shrub

Nothing interesting

shrub-num-tris

This tells us the number of triangles in a shrubbery is header[2] - 2*header[1]. One of the most popular GS formats is "triangle strip". The number of triangles in a triangle strip is num_verts - 2. So, very likely header[2] is the total number of vertices, and header[1] is the number of strips

shrub-make-perspective-matrix

This probably makes the matrix used to transform shrubbery. I'm not sure why yet, but they take the normal transformation (camera-temp) and modify it a bit. We may be able to get away with ignoring this. Ideally we just get the shrub positions (in the world coordinate system, just like with TIE/TFRAG), and then use the same logic we used for TIE/TFRAG.

shrub-init-view-data.

Note: change texture-giftag in shrub-view-data to a gs-gif-tag.

This populates fields of a shrub-view-data, which will probably end up in VU memory.

There's two things here that are suspicious. First:

  (set! (-> arg0 texture-giftag tag) (new 'static 'gif-tag64 :nloop #x1 :nreg #x4))
  (set! (-> arg0 texture-giftag regs) (new 'static 'gif-tag-regs
                                        :regs0 (gif-reg-id a+d)
                                        :regs1 (gif-reg-id a+d)
                                        :regs2 (gif-reg-id a+d)
                                        :regs3 (gif-reg-id a+d)
                                        )
        )

usually an adgif-shader is 5 qw's of a+d data. This sets up 4. The fifth one is usually gs-alpha, and it seems reasonable that they could set this once, then not bother setting it again.

Second, they put #x40a00000 in the 3rd word of the giftag:

(set! (-> arg0 texture-giftag word 3) (the-as uint #x40a00000))

as far as I can tell the GIF will ignore this. They might use it as a constant 5.0. But it seems weird to put it here.

If you change the score so it uses the named fields, the rest of the constants are:

  (set! (-> arg0 tex-start-ptr) (the-as int 25167696.0))
  (set! (-> arg0 mtx-buf-ptr) (the-as int 8388608.0))
  (set! (-> arg0 fog-0) (-> *math-camera* pfog0))
  (set! (-> arg0 fog-1) (-> *math-camera* pfog1))
  (set! (-> arg0 fog-clamp x) (-> *math-camera* fog-min))
  (set! (-> arg0 fog-clamp y) (-> *math-camera* fog-max))

the fog stuff seems reasonable to me (it's involved in computing the perspective division and the GS fog coefficient, don't worry for now).

The buf-ptr stuff is very likely this awful trick they do. The VU doesn't have good instructions for common integer operations (like you'd use to mainpulate memory addresses), so they find floating point values, then when added/subtracted in certain ways, have the lower 16 bits of the float equal to the VU memory address. Gross. Maybe the 5.0 is also part of this.

shrub-upload-view-data

Note: I changed the type casts:

  "shrub-upload-view-data": [
    [[3, 16], "a0", "dma-packet"]
  ],

and got a very typical upload.

(defun shrub-upload-view-data ((arg0 dma-buffer))
  (let ((s5-0 3))
    (let* ((v1-0 arg0)
           (a0-1 (the-as object (-> v1-0 base)))
           )
      (set! (-> (the-as dma-packet a0-1) dma) (new 'static 'dma-tag :id (dma-tag-id cnt) :qwc s5-0))
      (set! (-> (the-as dma-packet a0-1) vif0) (new 'static 'vif-tag :imm #x404 :cmd (vif-cmd stcycl)))
      (set! (-> (the-as dma-packet a0-1) vif1) (new 'static 'vif-tag :cmd (vif-cmd unpack-v4-32) :num s5-0))
      (set! (-> v1-0 base) (&+ (the-as pointer a0-1) 16))
      )
    (shrub-init-view-data (the-as shrub-view-data (-> arg0 base)))
    (&+! (-> arg0 base) (* s5-0 16))
    )
  #f
  )

• s5-0 this is the number of quadwords they will upload.
• dma-tag: cnt, means the data will come after the tag (and the next thing comes after that). qwc makes sense
• As part of a DMA tag, you get 2 VIF tags "for free". These are sent to the VIF.
• vif0: the stcycl sets the cl/wl register of VIF to 0x404 which is the mode for just copying stuff like normal (doesn't skip data)
• vif1: the unpack command means copy data to data memory. The v4-32 is the mode for copying quadwords.

The 0x404 stcycl and v4-32 format is used if you just want to copy memory exactly, so it shows up a lot. The other formats can do weird unpacking stuff, but are less common.

shrub-time

Not really sure what this is. I noticed in PCSX2 there's a time: that shows up in instance info. Maybe it computes the number of VU cycles.

shrub-do-init-frame

This adds stuff to the DMA list to get the hardware ready for shrub.

Note:

  "shrub-do-init-frame": [
    [[10, 21], "a0", "dma-packet"],
    [[24, 29], "a0", "dma-packet"],
    [33, "v1", "(pointer vif-tag)"],
    [[35, 41], "v1", "(pointer uint32)"],
    [42, "v1", "(pointer vif-tag)"],
    [[43, 51], "v1", "(pointer uint32)"],
    [52, "v1", "(pointer vif-tag)"],
    [54, "v1", "(pointer uint32)"]
  ],

All the data gets sent to the VIF. You can check the VIFcode reference part of the manual to see how long the data for each vifcode is.

upload the shrub program. In the PC port we usually set the VU program definitions to have a size of 0, then this function will do nothing.

(dma-buffer-add-vu-function arg0 shrub-vu1-block 1)

upload the shrub view data:

(shrub-upload-view-data arg0)

Add a DMA tag. Note that qwc is zero so this one just sends the two vif tags that are part of the dma tag:

    (set! (-> (the-as dma-packet a0-3) dma) (new 'static 'dma-tag :id (dma-tag-id cnt)))
    (set! (-> (the-as dma-packet a0-3) vif0) (new 'static 'vif-tag :cmd (vif-cmd mscalf) :msk #x1 :imm #x0))
    (set! (-> (the-as dma-packet a0-3) vif1) (new 'static 'vif-tag :cmd (vif-cmd flushe) :msk #x1))
    (set! (-> v1-0 base) (&+ (the-as pointer a0-3) 16))

The mscalf runs a VU1 program. The imm is 0, so it runs starting at the beginning of VU1 program memory. Let's ignore this for now. It's probably an initialization program that sets up the VU memory and registers. We'll look at it when we get to the actual drawing.

The flushe waits for that program to end.

Next it sets the strow/stcol/stmask registers of the VIF. These are used for unpacking settings.

shrub-init-frame

Calls shrub-do-init-frame then sets the gs-test register.

shrub-upload-model

Seems to upload a model

(function shrubbery dma-buffer int symbol)

and


  "shrub-upload-model": [
    [[17, 26], "a3", "dma-packet"],
    [[33, 41], "a0", "dma-packet"],
    [[47, 55], "a0", "dma-packet"]
  ],

The model data upload:

    (set! (-> (the-as dma-packet a3-0) dma)
          (new 'static 'dma-tag
            :id (dma-tag-id ref)
            :addr (-> arg0 obj)
            :qwc (+ (-> arg0 obj-qwc) (-> arg0 vtx-qwc) (-> arg0 col-qwc) (-> arg0 stq-qwc))
            )
          )
    (set! (-> (the-as dma-packet a3-0) vif0) (new 'static 'vif-tag :cmd (vif-cmd base) :imm *shrub-state*))
    (set! (-> (the-as dma-packet a3-0) vif1) (new 'static 'vif-tag :cmd (vif-cmd offset)))
    (set! (-> v1-0 base) (&+ (the-as pointer a3-0) 16))
    )

this tells us that there is a static DMA chain, starting at the obj field of the shrubbery. This includes obj, vtx, col, stq. The fact that it's a ref likely means the DMA will do one big transfer of all that data, then come back to the DMA buffer we're filling.

The use of the base with offset = 0 indicates they aren't using the normal VU double buffering (the weird float tricker earlier was also a clue). They have insane buffering schemes so you can have one model uploading, another model generating GIF tags, and a third model (no longer in VU1 memory) having its GIF tags being processed by the GS. Luckily we can mostly ignore this and just make non-buffered versions because there's no performance benefit to doing this on a PC.

Next, is:

  (cond
    ((= arg2 1)
     (let* ((v1-2 arg1)
            (a0-9 (the-as object (-> v1-2 base)))
            )
       (set! (-> (the-as dma-packet a0-9) dma) (new 'static 'dma-tag :id (dma-tag-id cnt)))
       (set! (-> (the-as dma-packet a0-9) vif0) (new 'static 'vif-tag))
       (set! (-> (the-as dma-packet a0-9) vif1) (new 'static 'vif-tag :cmd (vif-cmd mscal) :msk #x1 :imm #x11))
       (set! (-> v1-2 base) (&+ (the-as pointer a0-9) 16))
       )
     )
    (else
      (let* ((v1-3 arg1)
             (a0-11 (the-as object (-> v1-3 base)))
             )
        (set! (-> (the-as dma-packet a0-11) dma) (new 'static 'dma-tag :id (dma-tag-id cnt)))
        (set! (-> (the-as dma-packet a0-11) vif0) (new 'static 'vif-tag))
        (set! (-> (the-as dma-packet a0-11) vif1) (new 'static 'vif-tag :cmd (vif-cmd mscal) :msk #x1 :imm #x15))
        (set! (-> v1-3 base) (&+ (the-as pointer a0-11) 16))
        )
      )
    )

which adds a tag that runs a VU1 program. Either program starting at 0x11 or 0x15, depending on the argument.

Then

(set! *shrub-state* (- 164 *shrub-state*))

Based on the use of *shrub-state*, I think it is a VU buffer management thing.

draw-inline-array-instance-shrub

asm. I'll come back to this later. I assume this runs before the prototype draw, and that it will build lists of instances in each bucket, to be processed by the next function.

draw-prototype-inline-array-shrub

This one is tricky too, I'll come back later.

draw-drawable-tree-instance-shrub

This is the main draw. It does:

• setup instance-shrub-work (likely used in the instance asm function)
• set the next-clear stuff to 0. This is just a shortcut fast way to reset lists of instances from the previous frame. Will make more sense once we see the previous two functions, I think.
• call draw-inline-array-instance-shrub
• call draw-prototype-inline-array-shrub
• a whole bunch of performance counters we can skip

draw drawable-tree-instance-shrub

This connects the background system and the drawable/bsp system. When the draw engine executes, it calls this function, adding the drawable tree to background-work. When finish-background is called, it calls draw-drawable-tree-instance-shrub that does the real work

unpack-vis

Having this do nothing makes the shrub system not part of the occlusion culling system.

collect-stats

This shows that geometry 0 is shrub-near, geometry 1 is shrub, 2 is trans-shrub, and 3 is billboard.

It appears that near shrubs are drawn with the generic renderer (also looking at the prototype draw to confirm).

If this is the case, we can probably get away with ignoring geometry 0 and making a single shrub renderer that can handle the near case. From playing in PCSX2, "near" doesn't mean higher resolution, it just means the shrub is partially behind the camera, or close to that. If we port the rendering to opengl properly, we get this "for free".

Shrubbery Header

• [0] u32 - number of textures divided by 2.
• [4] u32 - number of vertices to draw.
• [8] u32 - number of triangle strips.