Writing PNG's in partial bits on weak hardware in C#
category: code [glöplog]
Hi all,
I am facing a challenge for a program I am working on. One feature of the program is, that it can export little posters which are made to be printed on 2 pieces of A4 paper and stuck on a wall.
I am working in C# in Unity3D, targeting iOS, and memory is very limited. I would love to generate a PDF (for example with iTextSharp), but I don't want to buy a license or release my code as GPL. The second best thing would be to write a PostScript file, but since there is no out-of-the-box viewer for Windows 7, and I want our output to be accessible on ALL machines, so PostScript is not the way to go either...
So what I would like to do is export a large PNG file (4096 x 5792 for each A4 page) from my program. I could also do BMP, but it would take way too much space (like 200 megabytes). Since I cannot hold a 4096 x 5792 texture in memory in iOS, I have to render it in slices, and then output them 4096x128 at the time.
So what I need is some C# library that can output 24-bit PNG's by the slice, while still compressing them (at least lightly). The reports have a lot of solid-color backgrounds, so they are super easy to compress. Does any of you know something like that?
Something like:
(should I take this to StackOverflow instead?)
I am facing a challenge for a program I am working on. One feature of the program is, that it can export little posters which are made to be printed on 2 pieces of A4 paper and stuck on a wall.
I am working in C# in Unity3D, targeting iOS, and memory is very limited. I would love to generate a PDF (for example with iTextSharp), but I don't want to buy a license or release my code as GPL. The second best thing would be to write a PostScript file, but since there is no out-of-the-box viewer for Windows 7, and I want our output to be accessible on ALL machines, so PostScript is not the way to go either...
So what I would like to do is export a large PNG file (4096 x 5792 for each A4 page) from my program. I could also do BMP, but it would take way too much space (like 200 megabytes). Since I cannot hold a 4096 x 5792 texture in memory in iOS, I have to render it in slices, and then output them 4096x128 at the time.
So what I need is some C# library that can output 24-bit PNG's by the slice, while still compressing them (at least lightly). The reports have a lot of solid-color backgrounds, so they are super easy to compress. Does any of you know something like that?
Something like:
Code:
PNGFile png = new PNGFile("left_page.png", 4096, 5792);
for(y = 0; y < 5792; y+= 128) {
// ... render partial report into imageArray ...
png.writeSlice(imageArray, (y + 128 > 5792) ? (5792 - y) : 128); // write 128 lines, or whatever is the remainder
}
png.Close();
(should I take this to StackOverflow instead?)
I've used this for generating PDF files in C# in the past. Only ever used it for Windows, but might be worth a look for you as it's open source: PDFSharp.
As for a library generating PNG files in chunks/slices, I've never heard of such a thing I'm afraid. Would be interested to know how you do overcome the problem though.
As for a library generating PNG files in chunks/slices, I've never heard of such a thing I'm afraid. Would be interested to know how you do overcome the problem though.
libpng can also write rows, not only whole images, maybe that can help you...
If you're working at the per-scanline level, you're probably going low-level enough that you might as well handle the details of the PNG format yourself... just hand off to zlib to do the DEFLATE compression and CRC calculation. I had to dig into the PNG format for this Javascript compression hack, and it's a pretty sane format. The data stream is just the raw RGB data, with each pixel row prefixed by 0x00 (other byte values will enable filters, for improving compression, which you don't need), all piped through DEFLATE. Wrap that in an 'IDAT' chunk, add a couple of header blocks and you're done.
(The ruby script linked above might be useful in a pseudocode sort of way... lines 64-68 are where it kicks off)
(The ruby script linked above might be useful in a pseudocode sort of way... lines 64-68 are where it kicks off)
Yeah, I'd normally go with what gasman said because it's really simple like that - write a header, then the 32 bit data compressed with Deflate which you normally get for free in C# because there's DeflateStream...
... but before you make a mistake there, be warned: DeflateStream doesn't work in Unity3D clients. Or perhaps it does on iOS but certainly not in the web player which I tested. So try to use it before you go down that route.
(Tech details: Unity3D is based on Mono and in Mono the DeflateStream internals are not part of the core libraries but reside in an external DLL named PosixHelper.dll which isn't linked to web player builds. I spent half a day porting the inflate from stb_image to C# because I only needed to decompress but it's possibly harder in your case)
... but before you make a mistake there, be warned: DeflateStream doesn't work in Unity3D clients. Or perhaps it does on iOS but certainly not in the web player which I tested. So try to use it before you go down that route.
(Tech details: Unity3D is based on Mono and in Mono the DeflateStream internals are not part of the core libraries but reside in an external DLL named PosixHelper.dll which isn't linked to web player builds. I spent half a day porting the inflate from stb_image to C# because I only needed to decompress but it's possibly harder in your case)
kb_: Thanks for the heads up on that! I just tested DeflateStream, and it doesn't even work on my current editor settings (Unity 4.0).
Now we're actually contemplating laying out the report in HTML, and then embedding whatever images we need directly as base64 encoded PNGs. With some custom css for printing, MAYBE we can make it work.
Now we're actually contemplating laying out the report in HTML, and then embedding whatever images we need directly as base64 encoded PNGs. With some custom css for printing, MAYBE we can make it work.
I know this is stupid, but I bet most app developers would simply have a server-side that does all the heavy labour, such as generate PDFs and whatnot.
yeah skrebbel, I know. But some of the devices that we will be working with likely does not have a 3G connection at all, and only sporadic Wi-Fi connections.
About PNG: So, I just thought about this, I know it's crazy, but here's an idea, and it could work, maybe.
So, seeing that Unity already has PNG compression built into it, I could in theory I could take my texture slices and compress them with PNG, and then extract the IDAT blocks and stitch them together like another Frankenstein's monster, without ever touching any DEFLATE algorithms myself. Is it insane?
About PNG: So, I just thought about this, I know it's crazy, but here's an idea, and it could work, maybe.
So, seeing that Unity already has PNG compression built into it, I could in theory I could take my texture slices and compress them with PNG, and then extract the IDAT blocks and stitch them together like another Frankenstein's monster, without ever touching any DEFLATE algorithms myself. Is it insane?
Why not save n images and lay them out on the page properly, if you're already going for the HTML approach?!
From the PNG spec
So, no you can't do it because the chunks represent parts of one compressed stream, not individually compressed parts of the image.
Quote:
There can be multiple IDAT chunks; if so, they must appear consecutively with no other intervening chunks. The compressed datastream is then the concatenation of the contents of all the IDAT chunks.
So, no you can't do it because the chunks represent parts of one compressed stream, not individually compressed parts of the image.
But let's get even more insane...er:
The deflate specification supports uncompressed blocks. Just saying. :)
The deflate specification supports uncompressed blocks. Just saying. :)
Converting PostScript to PDF on all platforms is really trivial. ps2pdf is part of GhostScript. Imagemagick can also do ps to image conversion.
One idea would also be to save the images as slices and generate a HTML file that stitches them together for printing to a PDF printer or straight printing from the web browser.
kb_: Thanks for looking that up. Damn. It makes perfect sense from a compression point of view though.
Graga: Ok, forgot half of what I've been saying. Looked at the deflate spec some more (perhaps I should start working instead :/), and your crazy idea might just work with a slight modification: Deflate is block based. There's almost nothing that keeps you from concatenating blocks of deflated data (and of course each of the single IDAT chunks you'll try to lump together will contain whole blocks); the only thing that you might need to adjust are the header flags of each block, notably the BFINAL flag that specifies the last block in the stream.
So go to through all the IDAT chunks except the last one, and zero out the LSB of the first byte of each block in each chunk. Simple as that! :D
So go to through all the IDAT chunks except the last one, and zero out the LSB of the first byte of each block in each chunk. Simple as that! :D
Graga: I get what you're saying about there being no internet connection a lot of the time... but just how the fuck are they going to share/print the images without a net connection? :D
If they have a connection at that point, just do the server thing right there.
If they have a connection at that point, just do the server thing right there.
kb_: IT WORKED :D :D :D
fuck I was calling the old function.
:D
what's that BFINAL flag?
check the link above. There's a few bits of header in front of each block.
Yeah I got what you said after I asked (typical...). I checked out the LSB's of the first byte on all my input data, and there seemed to be no correlation between its value and whether it was from the last IDAT block or not (each PNG from Unity already consists of several). I guess that is a good thing, since it makes the blocks less dependent on each other.
Changing the data would mean that it changed the CRC32 too, so I would have to recalculate that.
There seems to be some fowl wizardry at play here. If anyone wanna double check, it would be cool. I am going to try and write a python-script that concatenates two identical PNG's without modifying their IDAT content, some time later this week.
Changing the data would mean that it changed the CRC32 too, so I would have to recalculate that.
There seems to be some fowl wizardry at play here. If anyone wanna double check, it would be cool. I am going to try and write a python-script that concatenates two identical PNG's without modifying their IDAT content, some time later this week.
The IDAT chunk has a few bytes of ZLib header before the compressed data: http://www.w3.org/TR/PNG-Compression.html. I think you'll need to parse the whole thing and recalculate both the zlib and the chunk CRC.
Quote:
It is important to emphasize that the boundaries between IDAT chunks are arbitrary and can fall anywhere in the zlib datastream. There is not necessarily any correlation between IDAT chunk boundaries and deflate block boundaries or any other feature of the zlib data. For example, it is entirely possible for the terminating zlib check value to be split across IDAT chunks.
You're right about your final kb_, I'm gonna drop the idea and either accept a low resolution result (max 2048x2048 per. page) or change over to some HTML based solution (since most of our report is just text and solid color background).
Thanks a lot for your help everyone.
You're right about your final observation....