Rendered at 16:57:35 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
mikewarot 10 hours ago [-]
Back in the MS-DOS days, my friend Ward Christensen worked at IBM in Technical Sales Support. They had a file informally maintained that listed every single product they had for sale, in a format that could be read on a PC. He was really good at searching it with his text editor, but he wanted a MS-DOS based stand alone program to make it easy to search for his co-workers. Thus was born, PCxREF which was a Turbo Pascal based program I wrote, that included search.
My first pass at it was a naïve search using the built in string matching. Because of the size of the file, and the speed of PCs, it would sometimes take more than 30 seconds to do a search. Ward wanted it faster, much faster.
I eventually came up with an algorithm that took the text you wanted to search for, and changed it into a special kind of bit mask, table. It was a table 256 bytes long, initialized to zeros. If the first character of the string you wanted to search for was a D, then the first bit of the $44 and $64 mask would be set, to handle case sensitivity. A wildcard * was only 1 character long, but would set all bits for that character. I then used the 8088 XLAT instruction and a very small loop ingest a character of the text being searched, use that as an index into the mask table, and AND that with the existing flag in a different register, then shifted it, checking for CARRY (a match!) and looping if not zero.
It took search down to less than 3 seconds if the whole file was loaded in RAM. The thing we used as a benchmark was searching for "DOS TECH REF", the IBM DOS Technical Reference Manual.
I can't find the code, lost it over the years, but it was one of my coolest hacks.
gchamonlive 7 hours ago [-]
These kinds of things are still important these days. In my previous employment we had to create security guardrails for content distribution resources in AWS (API gateways, s3 buckets, instances and cloudfront distros) for a standalone CDN. For cloudfront you'd be right to question why put a CDN behind a CDN and the answer is technical debt, so we had to deal with it.
The problem is that cloudfront functions are very limited in code size and execution cycles, and the list of cidr blocks from the external CDN was so long it meant we couldn't do a simple comparison, so we did something similar, representing all cidr blocks and the incoming source IP to a binary representation and used binary search to see if the source IP matched any cidr block range. It's also very hard to estimate correctly if the resulting function would in fact fit the compute budget, because AWS isn't very clear about what a cf function cycle is in practice, so we tested it extensively, with real and synthetic cases to simulate worse conditions.
kragen 1 hours ago [-]
This is a very nice algorithm, used by agrep for fuzzy string search. Wikipedia says:
> The bitap algorithm for exact string searching was invented by Bálint Dömölki in 1964 and extended by R. K. Shyamasundar in 1977, before being reinvented by Ricardo Baeza-Yates and Gaston Gonnet in 1989 (one chapter of first author's PhD thesis) which also extended it to handle classes of characters, wildcards, and mismatches. In 1991, it was extended by Manber and Wu to handle also insertions and deletions (full fuzzy string searching). This algorithm was later improved by Baeza-Yates and Navarro in 1996.
But you know what all those guys didn't invent? They didn't invent BBSes and XMODEM. Ward did.
taspeotis 14 hours ago [-]
Somewhat related [1]:
> Years ago, I was working at Video Seven, a now-vanished video adapter manufacturer, helping to develop a VGA clone. The fellow who was designing Video Seven’s VGA chip, Tom Wilson, had worked around the clock for months to make his VGA run as fast as possible, and was confident he had pretty much maxed out its performance. As Tom was putting the finishing touches on his chip design, however, news came fourth-hand that a competitor, Paradise, had juiced up the performance of the clone they were developing by putting in a FIFO.
> That was all he knew; there was no information about what sort of FIFO, or how much it helped, or anything else. Nonetheless, Tom, normally an affable, laid-back sort, took on the wide-awake, haunted look of a man with too much caffeine in him and no answers to show for it, as he tried to figure out, from hopelessly thin information, what Paradise had done. Finally, he concluded that Paradise must have put a write FIFO between the system bus and the VGA, so that when the CPU wrote to video memory, the write immediately went into the FIFO, allowing the CPU to keep on processing instead of stalling each time it wrote to display memory.
> Tom couldn’t spare the gates or the time to do a full FIFO, but he could implement a one-deep FIFO, allowing the CPU to get one write ahead of the VGA. He wasn’t sure how well it would work, but it was all he could do, so he put it in and taped out the chip.
> The one-deep FIFO turned out to work astonishingly well; for a time, Video Seven’s VGAs were the fastest around, a testament to Tom’s ingenuity and creativity under pressure. However, the truly remarkable part of this story is that Paradise’s FIFO design turned out to bear not the slightest resemblance to Tom’s, and didn’t work as well. Paradise had stuck a read FIFO between display memory and the video output stage of the VGA, allowing the video output to read ahead, so that when the CPU wanted to access display memory, pixels could come from the FIFO while the CPU was serviced immediately. That did indeed help performance—but not as much as Tom’s write FIFO.
this alleged fifo sounds like a register with a two-phase handshake bit, but they lived in jargon deserts in those days
whizzter 5 hours ago [-]
It's exact nature is kinda irrelevant, the important part is that it was a cheap solution that gave a damn good return on investment just by pondering on possible improvements based on a rumor.
green_wheel 11 hours ago [-]
Awesome article. I guess grass is always greener but coding in those days seems really cool. Today it's all package managers this and toolchains that. We're so far abstracted away from what's actually running our code. Articles like this make me here the CPU calling "come back to meeeeee"
Agentlien 9 hours ago [-]
It depends a lot on what you work with. I am a much younger generation of game programmer (only 15 years of professional experience), but a lot of the work I do has a similar feel.
Not saying that most game programming is like that, but if you specialize in graphics and performance, preferably in large productions, you get to work close to the hardware and solve a lot of really fun problems in interesting ways.
kragen 56 minutes ago [-]
I thought a lot of games just use Godot, Unreal Engine, or that other one that decided they were going to change their licensing terms to demand per-copy fees from free-to-play games that their customers had already written with it. Does your company have an in-house Godot alternative, or are you having to do things like this on top of one of the existing engines, or what?
Agentlien 33 minutes ago [-]
It varies a lot. I've worked at a few different companies throughout the years (EA, Thunderful, 505 Games) and with a bunch of different engines (Frostbite, Unity, Unreal, and a few custom engines). For most games beyond basic indie titles there's enough complexity that you have to make at least some changes to rendering systems - especially when working on performance against weaker consoles.
I have also spent a lot of time at 505 Games porting a high-end custom game engine from PC to consoles. This requires a lot of work directly against graphics hardware in a way which is closer to the metal than you typically need for PC development.
Krei-se 8 hours ago [-]
You can do that now - and id argue its even easier to get into it.
WebGPU in a browser is worth the learning curve - and gives you the same interface then in Rust!
When you sniffed that sweet sweet shader code and memory layouting - why not get into WebAssembly?
You can have sth very similar to an OS in your browser if you want. It's a lot of fun and knowledge actually useful instead of the next hypecycle of frameworks and dialects.
pjc50 7 hours ago [-]
These days if you want to get esoteric you can do it in GPU shaders; https://www.shadertoy.com/ provides lots of fun examples with code.
Or there's MCUs with programmable IO for bit-bashing protocols.
ivanjermakov 7 hours ago [-]
GPU shading languages are about as abstract as CPU programming languages. I would rather suggest bare-metal programming, preferably on some obscure hardware.
choult 7 hours ago [-]
I guess my equivalent these days is optimizing distributed services - a small change here, a realization there and you can save whole percentages of performance in affected services when effectively DDoSing yourself.
BrtByte 8 hours ago [-]
[flagged]
sehugg 6 hours ago [-]
The 6502 has a more narrow and diabolical version of the 6809's PSHS trick. The BRK instruction pushes the PC and status register to the stack in three consecutive cycles, and is pretty much the only non-interrupt instruction that writes more than twice.
The Atari 2600 has horizontal object counters that you can't set directly, but they latch when you write to them, and they set the corresponding object to the X position of the beam at that cycle. Pole Position requires three of these objects to draw the left/right side of the road and center line, and they are supposed to meet at the horizon.
But the beam moves to the right 3 pixels for every CPU cycle, and writing with STA would result in each being separated by 9 pixels. What's the fix? Move the stack pointer to where the object pointers live (the 2600 maps the stack to zero page, where the RAM and registers live) and invoke BRK, making sure the BRK vector returns you to the next instruction. The object pointers are adjacent, so BRK writes to each object pointer on successive cycles, and now each of them is 1 cycle = 3 pixels apart.
adonovan 5 hours ago [-]
Great story! We may write little assembler (or, increasingly, code) these days but, optimization always has this same feeling of adventure, serial discovery, and of learning what parts it’s ok to throw away.
pjc50 7 hours ago [-]
Trying to work out which computer actually had the 6509 in; candidates seem to be Dragon or Tandy?
Each of which had limited but dedicated functionality for moving and transforming bytes in RAM. The blitter in particular was dedicated to moving memory around for tile and sprite rendering.
> Here’s the quick version: The graphics hardware didn’t do jack squat.
On the PC in question, if you wanted to draw something onto the screen, you had to do that yourself, byte by byte. No texture mappers, no blitters. Just bytes. Bytes that you had to move yourself.
That's not entirely correct: the graphics hardware took the video memory and built an image from that (frame buffer graphics). Which is already more than the earliest game consoles did.
RetroTechie 3 hours ago [-]
A ZX81 works like one of those pre-framebuffer machines:
At some point, the Z80 cpu jumps into an array that holds character info for each line on the screen. Its ULA chip intercepts that instruction fetch, and feeds the Z80 a NOP instruction.
Character itself is read by the ULA, and output onto the address bus. The ROM then outputs the corresponding bit pattern. ULA reads that, and clocks it into a shift register: 8 bits parallel in, serial out produces the pixels on screen.
But here's the nifty bit: that ULA stuff is timed to take exactly 8 cycles of a 6.5 MHz pixel clock. The Z80 runs on 1/2 that, so it takes the exact same time to execute the NOP (4 cycles). Done -> next instruction fetch, ULA ready to grab bit pattern for the next 8 pixels.
That's how screen lines are built on a ZX81. 50/60 Hz screen refresh is done by varying the # of lines displayed. Downside is that the (already 'slow') CPU is executing NOPs ~3/4th of the time, making for an effective CPU speed of ~0.8 MHz.
But the scheme was somewhat flexible, and programmers found ways to produce high-resolution graphics by (ab)using ROM data for non-text graphics. And with a RAM pack you could enjoy full hi-res graphics. All of this in b&w only.
The follow-up ZX Spectrum added a regular framebuffer with color (and even hardware flashing!) in 8x8 pixel blocks. This required a much more complex ULA chip though.
jimbobbam 14 hours ago [-]
This process is what’s missing in the ai era.
whizzter 5 hours ago [-]
That's the secret, everyone who's done nasty tricks have become donors of tricks to be used.
That said, modern machines have enough oomph that tricks of that kind haven't really been the major contributions on the table for the past 25 years (sure, there's some tricks with PS2 VU's/memory, register swaps,etc but they're small in comparison to the total product sizes).
jfyi 10 minutes ago [-]
[delayed]
nunez 11 hours ago [-]
idk about that. people will always find a way of working together to solve problems. that "solving" the AI era means "pulling the lever on the claude machine" is sad, I agree
msephton 2 hours ago [-]
Such sweeping generalisations make me smile, as they can’t possibly be true.
There’s a guy who has got Command & Conquer running on Atari ST at ~6fps (it started out at ~2fps). Months and months of thinking up new tricks and approaches by himself and with his agents. It’s only missing of you choose not to do it.
Dwedit 14 hours ago [-]
Monster Max on Game Boy happens to use unrolled "ld de,NNNN / push de" loops constructed in main memory to write to video ram.
For those unfamiliar with Monster Max (1994, Game Boy), it's basically the last of the British Isometric Platformers, a genre started by Knight Lore on ZX spectrum.
whizzter 5 hours ago [-]
Ooh, might steal that for my gameboy experiments if I get stuck in some perf-problem (right now I have asynchronous upload queueus that works most of the time since things are queued well ahead of time so it's updates outside of the screen).
For reference, unlike the PC,C64,etc f.ex. the Gameboy (and NES) only allows users to write memory during VBlanks/HBlanks (or when the screen is turned off), so instead of delaying and adding complicated logic for that due to memory usage contention the CPU is simply cut-off from the memory (thus reads/writes at the wrong time are just nonsense/discarded iirc).
BrtByte 7 hours ago [-]
[flagged]
Agentlien 9 hours ago [-]
This type of story is always so cool to read. In this type of community there's often a bit of wistful dreaming back to these days and the type of challenges they presented.
I'm young enough that I wasn't there, but I've been programming since I was a kid and working with graphics for the last 15 years.
This type of excitement is the feeling I got from learning C and C++ to begin with. Trying to really understand how things were working under the hood.
I still get that sense when working on low level graphics and performance in my current job. In fact, when the texture memory corruption was introduced the article felt eerily similar to the events which inspired me to write my own recent article about texture memory layouts ( https://agentlien.github.io/texture )
mrandish 12 hours ago [-]
The clever 6809 PSHU/PULS memory copy trick was probably independently discovered several times before Tom Moertel discovered it. The game Tom worked on was for the Coco 3, which was released in 1986.
My first computer was the original 4K Color Computer (1981). I learned 6809 assembler on it and worked full time programming it for several years, while also running one of the largest users groups for it. Traveling to Coco trade shows around the country, I met many of the devs writing the most popular Coco games and the trick was definitely known by around Summer 1983 as that's when the first Rainbowfest was held and I heard about there.
Around 1983, Steve Bjork (MegaBug / Zaxxon Coco) told me the devs at DataSoft, the company behind some of the popular 1st party cartridge games Radio Shack sold, were using it. I also heard about it in 1984 from Tim Jenison (CocoMax). The earliest usage of the trick in a commercial game was probably in a Williams arcade game, either Defender (1981) or Joust (1982). A few years ago at the Hackers Conference Tim told me he'd once asked Eugene Jarvis (Joust, Defender) about it and Eugene told him after discovering it they'd nicknamed the trick "PUSH-U / PUL-Me." I confirmed this with Sam Dicker (Sinistar (1983), Defender) last year during the Amiga 040th anniversary at the Computer History Museum.
While the basic trick to do a fast software block image transfer was discovered separately by several 6809 programmers, I can't recall ever seeing the Direct Page (DP) register used with it before today. That and some of the other refinements in TFA are quite clever additions. I think the Coco's lack of any hardware sprites, tiles, scrolling or acceleration made Coco devs especially desperate, so we had to become masters of software blits with PSHU/PULS and software sprites with XOR. And since all this was happening before Usenet and dial-up BBSes were commonly available, there was no way to share cool tricks widely except printed books and magazines.
Fortunately, the 6809 made up for Radio Shack's penny-pinching design by being much faster than the 6502 and Z80, which were microcoded while the 6809 was implemented directly in logic gates. It's widely considered the most advanced of all the 8-bit CPUs with an especially elegant, orthogonal ISA supporting position independent, re-entrant, multitasking code and powerful indirect addressing modes. One little known bit of history is that Jef Raskin's team at Apple used the 6809 for the initial Macintosh prototypes before Steve Jobs managed to cut a steep discount deal with Motorola on the workstation-priced 68000 CPU. By 1981, Moto was probably starting to realize they'd been pricing their CPUs far too high, having lost the Apple II, Commodore PET, VIC-20, C64 and then IBM PC designs, despite having a better chip. Waking up just in time to win, arguably, the LAST big desktop computer CPU design... ever? Way to go Moto!
> A few years ago at the Hackers Conference Tim told me he'd once asked Eugene Jarvis (Joust, Defender) about it and Eugene told him after discovering it they'd nicknamed the trick "PUSH-U / PUL-Me."
It's interesting to think about the traditional attitude towards documentation. Back in the day, people could discover incredibly clever things, but when it came to writing stuff down and distributing the information in an organized way, it seems like you only got the "reference" documentation. The books you could get would only list the opcodes; to learn a trick like this, you'd either have to talk to someone who knew it, or figure it out again.
kragen 44 minutes ago [-]
People wrote magazine articles and books with stuff like this. Dr. Dobb's, early BYTE, Don Lancaster's books, etc. I even remember typing in DOS TSR .COM files in DEBUG.COM from PC Magazine. The problem was, writing magazine articles was a much more limited channel than Usenet, BBSes, blogging, Stack Overflow, or GitHub. Dr. Dobb's published less than one article per day, for the entire world. And things like CACM weren't interested in things like the PUSH-U/PUL-Me.
So orality was more central.
mrandish 15 minutes ago [-]
> to learn a trick like this, you'd either have to talk to someone who knew it, or figure it out again.
Yes, personal computers were a new thing and before the internet, people doing it were pretty much isolated. When I got my 4K 8-bit in 1981 no one in my entire extended family's social network knew anyone who had a computer at home OR used one at work. I'd excitedly tell people "I got a personal computer!", and they'd respond like I'd said "I got a personal cement truck!" Just a long pause, followed by a puzzled,"Why? What can you do with that?" :-)
The only computer books the local library had were about programming COBOL on a mainframe with punch cards. So I started a local computer club just to find people who knew more than I did. And it quickly grew to hundreds of people with some driving more than an hour to attend. Users found each other via word of mouth or flyers at a computer store and formed into computer clubs, each was basically like an isolated island community and we'd travel to connect with each other. Early BBSes helped and when FidoNet got going, as primitive as it was, it could relay forum messages across the country in "just" three days! I connected with people in the New York and Chicago microcomputer 'islands' and later traveled to meet them at regional shows.
Around late 83 I started to get tantalizing bits of info passed along by a guy who occasionally came to the club which he'd heard from a magical thing called 'Arpanet', because he worked for a big defense contractor. It wasn't until after 86 those outside large universities and mega-corps could even get expensive dial-up access to Usenet. In 1987 I was still driving over an hour once a month to a guy's house, who'd driven an hour from from him to another guy's house, who had free, unmetered 1200 baud access and downloaded every month's set of "Fish Disks", collections of open source utilities, games and source code curated by a guy named Fred Fish and posted to Usenet. We'd chat for half an hour while the disks copied onto the blanks I'd brought and then the next day at the users group meeting, people would bring their computers and set up an ad hoc assembly line to copy over a hundred sets of disks during the meeting - and I know at least three of those disk sets were feeding other users groups.
In 1988 it was still a long distance call and high per-hour fee when I first got Usenet access but I simply couldn't wait any longer. Reading COMP.SYS.* was just too awesome. Each of those steps from BBSes to Fidonet to monthly disks to Usenet to Internet felt like a huge leap for those of us who started out waiting for monthly newsletters and hobbyist zines to arrive in the mail. The lack of specialized information was frustrating but the silver-lining was it forced everyone to work a lot out for themselves. Then when little trickles of info began to intermittently flow, we'd pore over each and every one. These days the problem is completely flipped to being buried under low value 'content' and having to sift through it to find value and extract meaning.
djmips 10 hours ago [-]
The 6502 was most definitely not microcoded! It's way simple.
On the Vectrex, also a 6809 machine, the fastest vector draw routines use the stack to thread draw commands, not quite the same thing but those sixteen bit stack pointers sure are useful.
kragen 43 minutes ago [-]
The PLA is sort of like microcode?
mrandish 10 hours ago [-]
Doh! You're correct - and I should have known that since the 6502 design was based on a greatly cost-reduced Motorola 6800, the 6809's dad. Somehow I bit-flipped between the 6502/Z80 (not microcoded) and the 8088 (microcoded). Thanks for the correction.
Re: the Vectrex, I never knew that but it makes sense since chasing the beam requires feeding a massive queue of vector pairs.
fferen 13 hours ago [-]
I started programming by making games in TI-BASIC, for the TI-83+ graphing calculators. There were some weird tricks like omitting all ending parentheses. Good preparation for weird hacks in real languages.
__del__ 5 hours ago [-]
oh that's a miserable environment. i remember discovering that the label search starts from the beginning of your program text each time you branch (there's no table of label positions) so branching is much slower in later parts of your code.
the ti-81 was somehow worse. the If conditional only had the implicit else form (no Then/Else/End), there were no looping constructs (just Lbl, Goto, DS< and IS>), you had two lists for array-like data, and i can't recall ever figuring out how to manipulate strings—it may not have been possible.
baud9600 14 hours ago [-]
Wonderful. What an extraordinary effort and great results. Makes my 6502 efforts (using page boundaries) seem rather simple. Those were the days!
Terry Davis in the comments plugging Temple OS is a bonus
__del__ 5 hours ago [-]
i've written a bit of HolyC and at one point knew how to get around in that environment. it's pretty neat actually. it's easy to make fun of his clunky demos and mental illness, but he built something incredibly entertaining to work with.
as he emphasized endlessly, you can modify the running kernel, so it's easy to rip out his fanatical religious stuff.
Agentlien 9 hours ago [-]
Seeing that actually shocked me a bit. I remember when I was first on HN I'd see occasional messages of him either promoting his TempleOS or going off on crazy aggressive rants completely out of context. A lot of comments were saying never mind Terry, he isn't well. It's very sad how his story ended.
nigehban 8 hours ago [-]
this feels like precursor of SIMD mechanism, instead of byte processing, process the capacity (max). I remember my image processing class with Ian R. Greenshild. We had to dig a lot of tricks for matrix calculations.
But the most important thing is that you have to rule the hardware (IRQs, Registers, PCs, etc). Thank you for the article, it made me smile and remember my 80s...
whizzter 5 hours ago [-]
I remember doing software rendered truecolor 3d graphics in the mid/later 90s on my computer (that was probably among the last Intel machines without MMX).
Shading was expensive since you had 3 color components to scale, but one trick if you had monochrome color was that you could do it with 1(16bit) or 2 multiplications(32bit) by moving bits around.
If you had RGB, you could do red and blue at once since you'd easily mask by 0xff00ff , then multiply by the intensity (0-0x100), since the result of 0xff*0x100 is at most 0xff00 you could just mask the result by 0xff00ff00 and shift it down 8 bits (the shifting could be with the green component combined back in).
1 multiply for 16bit color (5R 6G 5B) was basically the same thing by shifting green upwards, or'ing, masking and multiply a 5bit intensity once on the 32bit pattern 00000gggg gggg00000 rrrrr000 000bbbbb before shifting down the green part again and combining the result.
I'm sure others had even more similarly clever tricks back in that era.
Panzerschrek 12 hours ago [-]
On modern CPUs such tricks are irrelevant. They are able to process multiple instructions in the same time, so that the main bottleneck is no longer executing instructions, but accessing memory.
Rohansi 11 hours ago [-]
Kind of, but not really? At its core this article is basically a hack to do a bit of SIMD on an instruction set that doesn't have SIMD. Obviously SIMD is still highly beneficial on modern CPUs otherwise compilers would stop emitting it.
pjc50 7 hours ago [-]
Eh, you can always move the performance frontier into something else, and the SIMD capabilities give you lots of strangely shaped instructions to play with. We have absolutely not seen the end of clever hand written code, it's just very rare.
4gotunameagain 8 hours ago [-]
Those specific tricks are irrelevant mostly due to compiler optimisations, even in compute heavy workloads.
Pannoniae 51 minutes ago [-]
how are they irrelevant? Compilers don't even do non-standard calling conventions (unless inlined ofc), very basic optimisations are just not done because the source language doesn't give enough info on memory layout, aliasing and range of inputs....
abrookewood 12 hours ago [-]
I love reading about this stuff, even if I understand very little of it. Similar to the demo scene and code golf stuff, I'm amazed at the knowledge that some people have.
BrtByte 8 hours ago [-]
I was already impressed by the stack pointer trick, but then the interrupts started corrupting memory and I thought, well, that's the end of that. Somehow the solution was to deliberately corrupt a different part of memory and change the drawing order. That's the kind of thing you'd never believe if you hadn't seen it work.
brunooliv 5 hours ago [-]
Man seeing Terry Davis commenting and promoting TempleOS "13 years ago"... Absolutely crazy stuff! This was an awesome read!
It is definitely in the spirit of the story of Mel, since Tom was exploiting an intimate knowledge of the machine to achieve the impossible by using instructions in unintended ways, but always remember that Tom is not a real programmer:
Real Programmers wrote in machine code.
Not FORTRAN. Not RATFOR. Not, even, assembly language.
Machine Code.
Raw, unadorned, inscrutable hexadecimal numbers.
Directly.
That said, I prefer Tom's story. It's more about the joy of understanding the machine than it is about the framing of a legend.
My first pass at it was a naïve search using the built in string matching. Because of the size of the file, and the speed of PCs, it would sometimes take more than 30 seconds to do a search. Ward wanted it faster, much faster.
I eventually came up with an algorithm that took the text you wanted to search for, and changed it into a special kind of bit mask, table. It was a table 256 bytes long, initialized to zeros. If the first character of the string you wanted to search for was a D, then the first bit of the $44 and $64 mask would be set, to handle case sensitivity. A wildcard * was only 1 character long, but would set all bits for that character. I then used the 8088 XLAT instruction and a very small loop ingest a character of the text being searched, use that as an index into the mask table, and AND that with the existing flag in a different register, then shifted it, checking for CARRY (a match!) and looping if not zero.
It took search down to less than 3 seconds if the whole file was loaded in RAM. The thing we used as a benchmark was searching for "DOS TECH REF", the IBM DOS Technical Reference Manual.
I can't find the code, lost it over the years, but it was one of my coolest hacks.
The problem is that cloudfront functions are very limited in code size and execution cycles, and the list of cidr blocks from the external CDN was so long it meant we couldn't do a simple comparison, so we did something similar, representing all cidr blocks and the incoming source IP to a binary representation and used binary search to see if the source IP matched any cidr block range. It's also very hard to estimate correctly if the resulting function would in fact fit the compute budget, because AWS isn't very clear about what a cf function cycle is in practice, so we tested it extensively, with real and synthetic cases to simulate worse conditions.
> The bitap algorithm for exact string searching was invented by Bálint Dömölki in 1964 and extended by R. K. Shyamasundar in 1977, before being reinvented by Ricardo Baeza-Yates and Gaston Gonnet in 1989 (one chapter of first author's PhD thesis) which also extended it to handle classes of characters, wildcards, and mismatches. In 1991, it was extended by Manber and Wu to handle also insertions and deletions (full fuzzy string searching). This algorithm was later improved by Baeza-Yates and Navarro in 1996.
But you know what all those guys didn't invent? They didn't invent BBSes and XMODEM. Ward did.
> Years ago, I was working at Video Seven, a now-vanished video adapter manufacturer, helping to develop a VGA clone. The fellow who was designing Video Seven’s VGA chip, Tom Wilson, had worked around the clock for months to make his VGA run as fast as possible, and was confident he had pretty much maxed out its performance. As Tom was putting the finishing touches on his chip design, however, news came fourth-hand that a competitor, Paradise, had juiced up the performance of the clone they were developing by putting in a FIFO.
> That was all he knew; there was no information about what sort of FIFO, or how much it helped, or anything else. Nonetheless, Tom, normally an affable, laid-back sort, took on the wide-awake, haunted look of a man with too much caffeine in him and no answers to show for it, as he tried to figure out, from hopelessly thin information, what Paradise had done. Finally, he concluded that Paradise must have put a write FIFO between the system bus and the VGA, so that when the CPU wrote to video memory, the write immediately went into the FIFO, allowing the CPU to keep on processing instead of stalling each time it wrote to display memory.
> Tom couldn’t spare the gates or the time to do a full FIFO, but he could implement a one-deep FIFO, allowing the CPU to get one write ahead of the VGA. He wasn’t sure how well it would work, but it was all he could do, so he put it in and taped out the chip.
> The one-deep FIFO turned out to work astonishingly well; for a time, Video Seven’s VGAs were the fastest around, a testament to Tom’s ingenuity and creativity under pressure. However, the truly remarkable part of this story is that Paradise’s FIFO design turned out to bear not the slightest resemblance to Tom’s, and didn’t work as well. Paradise had stuck a read FIFO between display memory and the video output stage of the VGA, allowing the video output to read ahead, so that when the CPU wanted to access display memory, pixels could come from the FIFO while the CPU was serviced immediately. That did indeed help performance—but not as much as Tom’s write FIFO.
[1] https://www.jagregory.com/abrash-black-book/#the-challenge-o...
Not saying that most game programming is like that, but if you specialize in graphics and performance, preferably in large productions, you get to work close to the hardware and solve a lot of really fun problems in interesting ways.
I have also spent a lot of time at 505 Games porting a high-end custom game engine from PC to consoles. This requires a lot of work directly against graphics hardware in a way which is closer to the metal than you typically need for PC development.
WebGPU in a browser is worth the learning curve - and gives you the same interface then in Rust!
When you sniffed that sweet sweet shader code and memory layouting - why not get into WebAssembly?
You can have sth very similar to an OS in your browser if you want. It's a lot of fun and knowledge actually useful instead of the next hypecycle of frameworks and dialects.
Or there's MCUs with programmable IO for bit-bashing protocols.
The Atari 2600 has horizontal object counters that you can't set directly, but they latch when you write to them, and they set the corresponding object to the X position of the beam at that cycle. Pole Position requires three of these objects to draw the left/right side of the road and center line, and they are supposed to meet at the horizon.
But the beam moves to the right 3 pixels for every CPU cycle, and writing with STA would result in each being separated by 9 pixels. What's the fix? Move the stack pointer to where the object pointers live (the 2600 maps the stack to zero page, where the RAM and registers live) and invoke BRK, making sure the BRK vector returns you to the next instruction. The object pointers are adjacent, so BRK writes to each object pointer on successive cycles, and now each of them is 1 cycle = 3 pixels apart.
This sort of thing is why the Amiga was so massively successful by adding a collection of co-processors: https://en.wikipedia.org/wiki/Amiga_Original_Chip_Set
Each of which had limited but dedicated functionality for moving and transforming bytes in RAM. The blitter in particular was dedicated to moving memory around for tile and sprite rendering.
On the PC in question, if you wanted to draw something onto the screen, you had to do that yourself, byte by byte. No texture mappers, no blitters. Just bytes. Bytes that you had to move yourself.
That's not entirely correct: the graphics hardware took the video memory and built an image from that (frame buffer graphics). Which is already more than the earliest game consoles did.
At some point, the Z80 cpu jumps into an array that holds character info for each line on the screen. Its ULA chip intercepts that instruction fetch, and feeds the Z80 a NOP instruction.
Character itself is read by the ULA, and output onto the address bus. The ROM then outputs the corresponding bit pattern. ULA reads that, and clocks it into a shift register: 8 bits parallel in, serial out produces the pixels on screen.
But here's the nifty bit: that ULA stuff is timed to take exactly 8 cycles of a 6.5 MHz pixel clock. The Z80 runs on 1/2 that, so it takes the exact same time to execute the NOP (4 cycles). Done -> next instruction fetch, ULA ready to grab bit pattern for the next 8 pixels.
That's how screen lines are built on a ZX81. 50/60 Hz screen refresh is done by varying the # of lines displayed. Downside is that the (already 'slow') CPU is executing NOPs ~3/4th of the time, making for an effective CPU speed of ~0.8 MHz.
But the scheme was somewhat flexible, and programmers found ways to produce high-resolution graphics by (ab)using ROM data for non-text graphics. And with a RAM pack you could enjoy full hi-res graphics. All of this in b&w only.
The follow-up ZX Spectrum added a regular framebuffer with color (and even hardware flashing!) in 8x8 pixel blocks. This required a much more complex ULA chip though.
That said, modern machines have enough oomph that tricks of that kind haven't really been the major contributions on the table for the past 25 years (sure, there's some tricks with PS2 VU's/memory, register swaps,etc but they're small in comparison to the total product sizes).
There’s a guy who has got Command & Conquer running on Atari ST at ~6fps (it started out at ~2fps). Months and months of thinking up new tricks and approaches by himself and with his agents. It’s only missing of you choose not to do it.
For those unfamiliar with Monster Max (1994, Game Boy), it's basically the last of the British Isometric Platformers, a genre started by Knight Lore on ZX spectrum.
For reference, unlike the PC,C64,etc f.ex. the Gameboy (and NES) only allows users to write memory during VBlanks/HBlanks (or when the screen is turned off), so instead of delaying and adding complicated logic for that due to memory usage contention the CPU is simply cut-off from the memory (thus reads/writes at the wrong time are just nonsense/discarded iirc).
I'm young enough that I wasn't there, but I've been programming since I was a kid and working with graphics for the last 15 years.
This type of excitement is the feeling I got from learning C and C++ to begin with. Trying to really understand how things were working under the hood.
I still get that sense when working on low level graphics and performance in my current job. In fact, when the texture memory corruption was introduced the article felt eerily similar to the events which inspired me to write my own recent article about texture memory layouts ( https://agentlien.github.io/texture )
My first computer was the original 4K Color Computer (1981). I learned 6809 assembler on it and worked full time programming it for several years, while also running one of the largest users groups for it. Traveling to Coco trade shows around the country, I met many of the devs writing the most popular Coco games and the trick was definitely known by around Summer 1983 as that's when the first Rainbowfest was held and I heard about there.
Around 1983, Steve Bjork (MegaBug / Zaxxon Coco) told me the devs at DataSoft, the company behind some of the popular 1st party cartridge games Radio Shack sold, were using it. I also heard about it in 1984 from Tim Jenison (CocoMax). The earliest usage of the trick in a commercial game was probably in a Williams arcade game, either Defender (1981) or Joust (1982). A few years ago at the Hackers Conference Tim told me he'd once asked Eugene Jarvis (Joust, Defender) about it and Eugene told him after discovering it they'd nicknamed the trick "PUSH-U / PUL-Me." I confirmed this with Sam Dicker (Sinistar (1983), Defender) last year during the Amiga 040th anniversary at the Computer History Museum.
While the basic trick to do a fast software block image transfer was discovered separately by several 6809 programmers, I can't recall ever seeing the Direct Page (DP) register used with it before today. That and some of the other refinements in TFA are quite clever additions. I think the Coco's lack of any hardware sprites, tiles, scrolling or acceleration made Coco devs especially desperate, so we had to become masters of software blits with PSHU/PULS and software sprites with XOR. And since all this was happening before Usenet and dial-up BBSes were commonly available, there was no way to share cool tricks widely except printed books and magazines.
Fortunately, the 6809 made up for Radio Shack's penny-pinching design by being much faster than the 6502 and Z80, which were microcoded while the 6809 was implemented directly in logic gates. It's widely considered the most advanced of all the 8-bit CPUs with an especially elegant, orthogonal ISA supporting position independent, re-entrant, multitasking code and powerful indirect addressing modes. One little known bit of history is that Jef Raskin's team at Apple used the 6809 for the initial Macintosh prototypes before Steve Jobs managed to cut a steep discount deal with Motorola on the workstation-priced 68000 CPU. By 1981, Moto was probably starting to realize they'd been pricing their CPUs far too high, having lost the Apple II, Commodore PET, VIC-20, C64 and then IBM PC designs, despite having a better chip. Waking up just in time to win, arguably, the LAST big desktop computer CPU design... ever? Way to go Moto!
2013 Hacker News comments here: https://news.ycombinator.com/item?id=6913467
It's interesting to think about the traditional attitude towards documentation. Back in the day, people could discover incredibly clever things, but when it came to writing stuff down and distributing the information in an organized way, it seems like you only got the "reference" documentation. The books you could get would only list the opcodes; to learn a trick like this, you'd either have to talk to someone who knew it, or figure it out again.
So orality was more central.
Yes, personal computers were a new thing and before the internet, people doing it were pretty much isolated. When I got my 4K 8-bit in 1981 no one in my entire extended family's social network knew anyone who had a computer at home OR used one at work. I'd excitedly tell people "I got a personal computer!", and they'd respond like I'd said "I got a personal cement truck!" Just a long pause, followed by a puzzled,"Why? What can you do with that?" :-)
The only computer books the local library had were about programming COBOL on a mainframe with punch cards. So I started a local computer club just to find people who knew more than I did. And it quickly grew to hundreds of people with some driving more than an hour to attend. Users found each other via word of mouth or flyers at a computer store and formed into computer clubs, each was basically like an isolated island community and we'd travel to connect with each other. Early BBSes helped and when FidoNet got going, as primitive as it was, it could relay forum messages across the country in "just" three days! I connected with people in the New York and Chicago microcomputer 'islands' and later traveled to meet them at regional shows.
Around late 83 I started to get tantalizing bits of info passed along by a guy who occasionally came to the club which he'd heard from a magical thing called 'Arpanet', because he worked for a big defense contractor. It wasn't until after 86 those outside large universities and mega-corps could even get expensive dial-up access to Usenet. In 1987 I was still driving over an hour once a month to a guy's house, who'd driven an hour from from him to another guy's house, who had free, unmetered 1200 baud access and downloaded every month's set of "Fish Disks", collections of open source utilities, games and source code curated by a guy named Fred Fish and posted to Usenet. We'd chat for half an hour while the disks copied onto the blanks I'd brought and then the next day at the users group meeting, people would bring their computers and set up an ad hoc assembly line to copy over a hundred sets of disks during the meeting - and I know at least three of those disk sets were feeding other users groups.
In 1988 it was still a long distance call and high per-hour fee when I first got Usenet access but I simply couldn't wait any longer. Reading COMP.SYS.* was just too awesome. Each of those steps from BBSes to Fidonet to monthly disks to Usenet to Internet felt like a huge leap for those of us who started out waiting for monthly newsletters and hobbyist zines to arrive in the mail. The lack of specialized information was frustrating but the silver-lining was it forced everyone to work a lot out for themselves. Then when little trickles of info began to intermittently flow, we'd pore over each and every one. These days the problem is completely flipped to being buried under low value 'content' and having to sift through it to find value and extract meaning.
On the Vectrex, also a 6809 machine, the fastest vector draw routines use the stack to thread draw commands, not quite the same thing but those sixteen bit stack pointers sure are useful.
Re: the Vectrex, I never knew that but it makes sense since chasing the beam requires feeding a massive queue of vector pairs.
the ti-81 was somehow worse. the If conditional only had the implicit else form (no Then/Else/End), there were no looping constructs (just Lbl, Goto, DS< and IS>), you had two lists for array-like data, and i can't recall ever figuring out how to manipulate strings—it may not have been possible.
as he emphasized endlessly, you can modify the running kernel, so it's easy to rip out his fanatical religious stuff.
Shading was expensive since you had 3 color components to scale, but one trick if you had monochrome color was that you could do it with 1(16bit) or 2 multiplications(32bit) by moving bits around.
If you had RGB, you could do red and blue at once since you'd easily mask by 0xff00ff , then multiply by the intensity (0-0x100), since the result of 0xff*0x100 is at most 0xff00 you could just mask the result by 0xff00ff00 and shift it down 8 bits (the shifting could be with the green component combined back in).
1 multiply for 16bit color (5R 6G 5B) was basically the same thing by shifting green upwards, or'ing, masking and multiply a 5bit intensity once on the 32bit pattern 00000gggg gggg00000 rrrrr000 000bbbbb before shifting down the green part again and combining the result.
I'm sure others had even more similarly clever tricks back in that era.
https://users.cs.utah.edu/~elb/folklore/mel.html