Support grayscale displays

[twasilczyk]

modm::ui::display either supports monochrome or color displays. The two displays I've been working with (SSD1322 and ST7586S) are 16- and 4-level grayscale.

One option would be to make ColorGraphicDisplay a template allowing other color profiles than Rgb565; then add 16/4 level color profile. Then maybe even MonochromeGraphicDisplay could just use the same template with 1-bit color profile. There's a challenge though: in 16-level grayscale displays, two pixels are encoded in a single byte.

While we're at modm::ui::display, there's also one benefit of using these controllers that modm doesn't utilize: you can select a fragment of the display and not update the whole thing.

[salkinium]

@TomSaw is our display expert now, his #665 is addressing some of these issues.

[TomSaw]

Yepp I've also had my concerns with the display implementations in the beginning of this year.
The #665 rewrite considers any type of color including grayscale of any depth.

However, not all cases where multiple pixels are stuffed into one byte are covered yet:

• modm::color::Grayscale<> needs implementation for n-bit grayscale-depths to cover f.e. 4 and 16 level Grayscale.

• modm::graphic::Buffer needs according implementation.

I've added this to the TODOs of #665 and also should have some SSD1322 around so I can test it. Being on a journey until next week but I'll implement this as next. Stay tuned!

[twasilczyk]

This is awesome and I'm glad to hear you're considering multi-pixel-per-byte cases. There's one extra complication with SSD1322 that you need to be aware of: addressing is per word, not per byte. So a single 16-bit cell in memory represents four 4-bit pixels - this affects the widths of partial drawing sprites.

Did you consider extending support to drawing just parts of the display? This speeds up most of the use cases a lot. In my own project I easily hit north of 100 FPS (that's where my counter ends) even with a very inefficient SPI driver. No wonder why - each frame was sending just a handful of bytes. Added benefit is that you don't need much RAM - you can drive a large OLED with something as small as atmega8.

[TomSaw]

a single 16-bit cell in memory represents four 4-bit pixels - this affects the widths of partial drawing sprites.
Ok, good to know! This should only affect the transfer-algorithm of the display-driver. I'll keep this in mind when debugging ;).

Did you consider extending support to drawing just parts of the display
That's already in the #665 Box 😋. You can define arbitrary graphic-buffer with arbitrary dimensions and color-type and send it to any other arbitrary graphic-buffer or display.

• One can fragment the display into several areas and update each area asynchroniously.
• Another interesting scenario is writing a buffer with low color-depth (f.e. 2-bit/pixel) to display with higher color-depth. The display-driver converts the buffer-pixels to it's predefined color on demand using a small local conversion-buffer. This spares a ton of RAM when f.e. writing just text where no muti-colors are required.

Spend some time today on the generic colors / buffers and will add some self-explaining examples for the new APIs to #665 as soon as the generic colors and buffers are good enough...

... In short, this is the new workflow:

1. Declare your color-type

using Mono = GrayT<1>; // Monochrome 1 bit/pixel
using Gray2 = GrayT<2>; // Grayscale 2 bits/pixel
using Gray4 = GrayT<4>; // Grayscale: 4 bits/pixel
using Gray3 = GrayT<3>; // Grayscale: 3 bits/pixel for whatever phantasy application
using Rgb565 = RgbT<5,6,5>; // Rgb565-Color: 5bits red, 6bits green, 5bits blue/pixel
using Rgb888 = RgbT<8,8,8>; // Rgb888-Color: 8bits per red, green,blue / pixel
using Hsv888 = HsvT<8,8,8>; // Hsv888-Color: 8bits per hue, sat, value / pixel

2. Declare your buffer-type

When using Mono, Gray2 or Gray4 as color-type, the buffers internal data-scheme already stacks the pixels, hence memory is saved and can be transfered to the display without additional conversion.

using MyBuffer = modm::Buffer<Gray16, 100, 123> // Graphic-buffer with x=100, y=123

3. Draw and write into the buffer (similar like before)

MyBuffer.draw(Line(...));
MyBuffer.draw(Circle(...));
MyBuffer << "hello Buffer";
MyBuffer.write(AnotherBuffer, {20, 30});

4. Finally send the buffer to a display with optional position. When the buffer overflows the display, it's clipped. Therefore you can move buffers around on the display to create nice animations.

display.write(MyBuffer); // Position defaults to {0, 0}
display.write(MyBuffer, {-12, 34}); // specific position

[twasilczyk]

This is awesome, exactly (or more than) what I expected from embedded display library.

In my own personal project I did one more thing with buffers and embedded images. Writable Buffer inherits from read-only Image, so that you can use the same code to display RAM-backed buffers and Flash-backed images (with no ram usage). And I have a Python converter script from png to header files that also embeds all the metadata like size.

[TomSaw]

Hey twasilczyk, sry for my late answer. I'm running too much projects X-)

Please check #665 -> modm/ui/graphic/decoder.hpp. It's work in progress but shows how image data can be read from any storage in any format and written to any target. To cover a new storage - f.e. SD-Card - you only have to code an appropriate modm::accessor::SDCard and plug it into the right modm::graphic::decoder as template argument. There will be a Factory, so the application-designer doesn't have to worry about the details.
Gonna document this in more detail soon...

Support for more colormodels

To support more than monochrome images, the header - of course - has to be extended with some signature for the colormodel.
It currently only holds the image-size, see modm/ui/display/image/*

I didn't even review the conversion-script that makes *.hpp and *.css out of *.pbm.
The workflow is very nice: conversion is triggered by scons on project-compilation!

Would love to fix that as well but my priority is on finishing these generic colors and followup modm::Buffer- support. Afterwards I want to concentrate on 2D hardware acceleration.

💥 Would be awesome, if you could spend some time on the converter to support more colormodels and png! Do you want to?

[TomSaw]

PS: I can't wait getting back to my project(s) and enjoy the fruits of this "little subproject" of a rewrite 🥳.

[twasilczyk]

I can’t promise anything, since modm image APIs are just a prerequisite to a side project (Smart Response XE BSP) to my other hobby project that I temporarily gave a break :D.

Having said that, a converter script with png and multi-color mode support shouldn’t be far from what I currently have. I’ll try to give it a shot.

[TomSaw]

No problem, just had the feeling you're willing to bring yourself in.
For me there's no prio so don't worry and have fun with programming!

[TomSaw]

#665 now supports stacked monochrome, gray1, gray2 and gray4 Buffers.

I have slept a night over the ST7586 pixel-format you've mentioned in #673 - where one byte just contains 3 2bit-pixels - and came out with this:

Facts

• Buffer-manipulation (drawing shapes, copying etc.) is more expensive than buffer-conversion.
• Algorithms working on symetric patterns optimize much better than for fuzzy patterns. Looking at you ST7586-pixelformat 👀
• One does not want to waste 2bits in every byte of a potentially big buffer.

Outcome

1. A regular generic modm::Buffer<Gray2, Resolution> should be used.
2. Implement the modm::St7586::write(BufferInterface<Gray2> &buffer) with integrated conversion!
   You can convert pixel by pixel but i absolutely recommend a small local conversion-buffer and do it in chunks. That's also a must have for Direct memory access.

Please check Ili9341::writeImage(Decoder<Monochrome, Accessor>, Point) in #665 witch is very similar. It converts a Monochrome buffer to Rgb565-color on demand.

From my side, issue can be closed.

[twasilczyk]

In general, I agree, but there's one important counter-point: in its current form, pixel conversion from monochrome to st7586s monochrome format takes 3x more time than transmission itself (checked with logic analyzer, but I don't remember exact numbers), what brings FPS down significantly. If the implementation supported both Gray2 and Gray332 formats, one could pick former for fast buffer manipulation and the latter for fast drawing directly from flash.

Having said that, I'm happy with the performance with on-the-fly conversion.

For closing this (or any other) issue, I would wait for merging the PR. Up to you.

[TomSaw]

I see your point: There's nothing that restricts you from storing your image in whatever colorformat in flash and copy it byte by byte. It's mainly the Display drivers business who takes a modm::accessor::Flash into account.

#665 concentrates on the fundamental architecture and of course port existing features. Thus i've just migrated the good old monochrome Buffer<Gray<1>, ...> flash->display functionality for now and just kept future support for more colormodels in mind.

What's left to make Gray332-image flash->display work:

1. Improve this python image conversion script
2. Implement the 'writeImage(...)' method for your display.
3. Optional: develop some reusable supporting classes around modm::accessor::* to grab the metadata for example.

If you're in a hurry, you may implement 1. and 2. 🙄 somehow and we'll migrate everything later!?

You may boost The Gray2222 -> Gray332 conversion by a factor using a LUT.. This converts 12 pixels within a handful of ticks. Costs you 64bytes for the LUT but spares 1/4 of your flash used for images. Here's a sketch of a the whole algorithm:

True, issue is not solved. Let's wait for #665 and keep it until things run as expected.