1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
|
+++
title = "Terminal Renderer Mk. II - Rendering to Text with Compute"
date = "2025-10-02"
+++
<section>
<div class="text-section">
<p>This week I brought my terminal renderer to the next level by performing text rendering on the GPU.
</p>
</div>
<figure class="cover-image">
<img src="cover.png" alt="" style="width:100%;">
<figcaption>The Stanford Dragon, outlined and rendered as Braille characters in a terminal emulator. <a href="https://tv.soaos.dev/w/fBnDAUPsTPHaoPeNNxBGch" target="_blank">
Full video</a>
</figcaption>
</figure>
</section>
<section class="text-section">
<h2>Context</h2>
<h3>Unicode Braille</h3>
<p>
I first messed around with rendering images to the terminal with Braille characters in like 2022 I
think? I wrote a simple CLI tool
that applied a threshold to an input image and output it as Braille characters in the terminal. <a
href="https://tv.soaos.dev/w/twpHAu4Jv8LJc9YjZbfw5g" target="_blank">Here's a recording I took back
when I did it.</a>
</p>
<p>
<figure class="fig fig-right">
<div class="centered">
<table class="schema-table">
<tbody>
<tr>
<td>0</td>
<td>3</td>
</tr>
<tr>
<td>1</td>
<td>4</td>
</tr>
<tr>
<td>2</td>
<td>5</td>
</tr>
<tr>
<td>6</td>
<td>7</td>
</tr>
</tbody>
</table>
</div>
<figcaption>The corresponding bit position for each braille dot.</figcaption>
</figure>
This effect is pretty cool, and it was pretty easy to implement as well. The trick lies in how the
<a href="https://en.wikipedia.org/wiki/Braille_Patterns#Block" target="_blank">Unicode Braille block</a>
is laid out. Every 8-dot Braille combination happens to add up to 256 combinations, the perfect amount to
fit in the range between <code>0x2800</code> (⠀) and <code>0x28FF</code> (⣿). In other words, every
character
within the block can be represented by changing the value of a single byte.
</p>
<p>
The lowest 6 bits of the pattern map on to a 6-dot braille pattern. However, due
to historical reasons the 8-dot values were tacked on after the fact, which adds
a slightly annoying mapping to the conversion process. Either way, it's a lot easier
than it could be to just read a pixel value, check its brightness, and then use a
bitwise operation to set/clear a dot.
</p>
<h3>Ordered Dithering</h3>
<p>
Comparing the brightnes of a pixel against a constant threshold is a fine way to
display black and white images, but it's far from ideal and often results in the loss
of a lot of detail from the original image.
</p>
<figure class="fig fig-horizontal">
<div class="horizontal-container">
<img src="david.png" alt="" />
<img src="davidthreshold.png" alt="" />
<img src="davidbayer.png" alt="" />
</div>
<figcaption>From left to right: Original image, threshold, and ordered dither. <a
href="https://en.wikipedia.org/wiki/Dither" target="_blank">Wikipedia</a></figcaption>
</figure>
<p>By using <a href="https://en.wikipedia.org/wiki/Ordered_dithering" target="_blank">ordered dithering</a>,
we
can preserve much more of the subtleties of the original image. While not the "truest" version of
dithering possible,
ordered dithering (and <i>Bayer</i> dithering in particular) provides a few advantages that make it very
well suited to realtime computer graphics:
<ul>
<li>Each pixel is dithered independent of any other pixel in the image, making it extremely
parallelizable and good for shaders.</li>
<li>It's visually stable, changes to one part of the image won't disturb other areas.</li>
<li>It's dead simple.</li>
</ul>
Feel free to read up on the specifics of threshold maps and stuff, but for the purposes of this little
explanation it's
enough to know that it's basically just a matrix of 𝓃⨉𝓃 values between 0 and 1, and then to determine
whether a pixel (𝓍,𝓎)
is white or black, you check the brightness against the threshold value at (𝓍%𝓃,𝓎%𝓃) in the map.
</p>
</section>
<section class="text-section">
<h2>The old way™</h2>
<p>
My first attempt at <i>realtime</i> terminal graphics with ordered dithering
(<a href="https://tv.soaos.dev/w/dzHBnPJXtDBwtSvirgwTvY" target="_blank">I put a video up at the time</a>)
ran entirely on the CPU. I pre-calculated the threshold map at the beginning of execution and ran each
frame
through a sequential function to dither it and convert it to Braille characters.
</p>
<p>
To be honest, I never noticed
any significant performance issues doing this, as you can imagine the image size required to fill a
terminal
screen is signficantly smaller than a normal window. However, I knew I could easily perform the
dithering on the GPU
as a post-processing effect, so I eventually wrote a shader to do that. In combination with another
effect I used to
add outlines to objects, I was able to significantly improve the visual fidelity of the experience. A
good example of
where the renderer was at until like a week ago can be seen in <a
href="https://tv.soaos.dev/w/9Pf2tP3PYY5pJ3Cimhqs9x" target="_blank">this video</a>.
</p>
<p>
Until now I hadn't really considered moving the text conversion to the GPU. I mean, <i>G</i>PU is for
graphics,
right? I just copied the <i>entire framebuffer</i> back onto the CPU after dithering
and used the same sequential conversion algorithm. Then I had an idea that would drastically reduce the
amount
of copying necessary.
</p>
</section>
<section class="text-section">
<h2>Compute post-processing</h2>
<p>
What if, instead of extracting and copying the framebuffer every single frame, we "rendered" the text on
the GPU
and read <i>that</i> back instead? Assuming each pixel in a texture is 32 bits (RGBA8), and knowing that
each braille
character is a block of 8 pixels, could we not theoretically shave off <i>at least</i> 7/8 of the bytes
copied?
</p>
<p>
As it turns out, it's remarkably easy to do. I'm using the <a href="https://bevy.org"
target="_blank">Bevy engine</a>,
and hooking in a compute node to my existing post-processing render pipeline worked right out of the
box.
I allocated a storage buffer large enough to hold the necessary amount of characters, read it back each
frame, and dumped
the contents into the terminal.
</p>
<p>
I used UTF-32 encoding on the storage buffer because I knew I could easily convert a "wide string" into
UTF-8 before printing it, and
32 bits provides a consistent space to fill for each workgroup in the shader versus a variable-length
encoding like UTF-8. <a href="https://tv.soaos.dev/w/fBnDAUPsTPHaoPeNNxBGch" target="_blank">Here's a video of the new renderer working</a>.
Although now that I think about it, I could probably switch to using UTF-16 since all the Braille
characters could be represented
in 2 bytes, and that would be half the size of the UTF-32 text, which is half empty bytes anyways.
</p>
<p>
Okay so I went and tried that but remembered that shaders only accept 32-bit primitive types, so it doesn't matter anyways. This little side quest has been a part of my
broader efforts to revive a project I
spent a lot of time on. I'm taking the opportunity to really dig in and rework some of the stuff I'm not
totally happy with. So there might be quite a few of this kind of post in the near future. Stay tuned.
</p>
</section>
|
