pouët.net

Up until working with Primordial Soup, I had actually been minifying my shaders manually. As this is obviously absolute idiocy, it was time to move on.

Unfortunately, as good as LLB's Shader Minifier is, it was not suitable for crunching the shaders in this particular intro. The main problem is, Shader Minifier works with single source files. The shader file listing for Primordial Soup look like this:

This is because there is no 1quad -shader, but multiple different shader paths. In particular, the final shaders are combined as follows:

• header.glsl, default.vert.glsl, default.geom.glsl, default.frag.glsl - bacteria and plants
  
• header.glsl, default.vert.glsl, worm.geom.glsl, default.frag.glsl - worms
  
• header.glsl, quad.vert.glsl, quad.frag.glsl - post-processing
  

I can't exactly start bugging herr. Laurent because of some special corner case I want but no-one else does. I momentarily thought of extending his work, but to be absolutely honest, I have no idea how F# works at all. Options are to learn F#, continue crunching shaders manually, or write a shader minifier of my own (from scratch).

Naturally, option 3) is the most satisfying one.

So, even if the intro was already released at Assembly '17, the shader minification tool was quite not up to snuff. For the last two weeks or so, I've been working on cleaning up the codebase and fixing the remaining bugs. First of all, I'd like to thank noby, rimina and visy for the sources of Waillee, Back to the Roots and Quadtripophobia respectively, which, along Ghosts of Mars and forementioned Primordial Soup have served as the test set.

Current test results (listing does not concern Primordial Soup, as Shader Minifier obviously cannot crunch it):

Here, the numbers on the left show the flat, uncompressed shader payload. Since I'm not on Windows, I can't exactly try Crinkler compression, so I'm using xz LZMA mode to approximate entropy. This is fine, since LZMA is the go-to filedump compression on *nix anyway.

The results would imply I'm already beating Shader Minifier in flat size on every example, and on entropy in every example besides the latest Paraguay intro (have to look at that later).

So, what does dnload currently do to crunch the shaders then?

• Apply all constant operations in the code, preserve higher precision to the result.
  
• Select swizzles that have more character matches on locked (non-renameable) names within the code.
  
• For each name (variable, function name), starting from most references, rename to currently most common letter that is not conflicting in its scope(s).
  
• Combine all declaration statements with commas.
  
• Exploit comma operator to combine all statements and possibly eliminate scopes in the process.
  
• etc.
  

Or, as the minifier is about 5000 lines of Python, it's probably easier to explain what does dnload do that Shader Minifier does not do:

• Get rid of const - directives for variables - you already tested those without minification, we know they're not going to get written.
  
• Exploit the comma operator in all possible places.
  
• Handle in/out struct blocks between different shader files, handle multiple shader files to begin with (our motivation in the first place).
  

And what does Shader Minifier do that dnload does not do:

• Change constants to integers in situations where it's not wanted (warranting float(44100), etc).
  
• Correctly rename variables that are reusing a name that already is a GLSL reserved word.
  
• Possibly handle cases where one-letter namespace [a-zA-Z] runs out. Haven't tested.
  
• Print nicer parse errors. I.e. actually point out the offending line, not just Python stack traces.
  

And what dnload not yet do that it should (TODO):

• Change constants to integers in every single location where the conversion cannot change semantics (e.g. vec(1.0), perhaps others).
  

I've also seen no reason to re-invent the wheel regarding the conventions herr. Laurent already set in place. Think of his implementation as gcc, mine as clang (with shittier error messages):

• Use i_ -prefix to mark variables to be inlined.
  

You're free to try this out and post angry replies or send angry /msg's on IRC. To do so, first clone the repo:
https://github.com/faemiyah/dnload
Then run crunch:

Pipe to header file as needed.

NOTE: Tests with noby's Waillee revealed that smaller is not necessarily better. In particular, exploiting the comma operator seemed to consistently decrease size but increase entropy as perceived by Crinkler. To avoid this, use the command line option --glsl-mode=nosquash to prevent any squashing of statements.

NOTE: Additional thanks to cupe and urs for their Mercury List of Reserved Words for GLSL. Some parts of that are already in, I have to do the rest later.

Really cool. I need to run some comparisons with this and minifier as well :)

Yup, this is good :)

Based on what I tried this is pretty much on par with LLB's minifier, or better noticeably better in some cases. Some differences are from minute almost unpredictable changes though.

Recommending for others to try it out too!

It would be interesting if you could use Crinkler for checking how the minifier performs. Has anyone managed to get Crinkler to work under Wine? I can kind of get it to at least start in wine-1.6.2, but it crashes at "Estimating models for code".

yzi: Blueberry said it should work on Wine 1.8:

http://www.pouet.net/prod.php?which=18158#c749608

Thanks! I got the newer 1.8 version from jessie-backports and it does indeed work. Crinkler produces the same exe as in Windows. The produced exe doesn't actually run under Wine, but it does work in Windows. Compofiller Studio now seems to work under Linux so that it's at least possible to make a Windows 4k intro in Linux without actually having Windows ... though the final exe still has to be tested in actual Windows. ;)

Case Back to the Roots bothered me, so I fixed some issues:

• Squash all preceding statements into return correctly.
  
• Allow integrifying floating point values inside vecN(float).
  

With this update:
Changes can be pulled from master.

Would also be very interested about yzi's potential crinkler findings.

Nice to see some competition :)
I haven't checked your tool yet, but it would be interesting to do a comparison of the outputs.

I'll encourage everyone to report any bug / feature request you have for Shader Minifier. In particular, if any limitation is blocking you, please let me know. I've made a release today, although there is no major change.

https://github.com/laurentlb/Shader_Minifier

I'm just thinking that "0.14285714" is "1./7."