If you look at the documentation for the GEM AES Scrap Library, two things soon become very obvious.

First, it’s clear that the designers of GEM at Digital Research recognized the need for and utility of a well-designed system clipboard that applications could use to quickly and easily exchange data.

Second, it’s even more clear that within no more than a couple of minutes after sitting down to map out what sort of library functions were needed and what sort of predefined data exchange formats should be used, something distracted them to such a degree that they put down their engineering comp books, got up from their desks, and left the office.

I’m guessing the distraction was lunch time. Maybe the food truck rolled into the parking lot and tooted its horn.

A third thing becomes obvious soon after. After lunch was over, they never came back to finish what they’d started.

Three Functions

The GEM AES scrap library consists of three functions.

You heard me. Three functions.  Here they are:

  • scrp_write allowed an application to specify a directory which would contain file(s) containing the data being placed on the clipboard.
  • scrp_read allowed an application to retrieve the directory containing clipboard files.
  • scrp_clear function deleted all files in the clipboard directory.

Actually, I lied. The scrp_clear function was only on PC, not Atari. I can only imagine that two or three minutes into the five minute task of bringing the PC code over, the guy doing it said, “screw it — nobody’s gonna use this anyway” and went off for a long weekend.

And Yet, Not Very Useful Functions

The first problem with this setup is that it makes individual applications responsible for deciding where the clipboard resides, rather than the system. That’s simply a recipe for disaster, especially on a hard drive where you could easily end up with a separate clipboard folder for each and every application.

The next problem is that it doesn’t provide a definitive way for an application to find out if there is data on the clipboard or not.  It has to read the directory returned by scrp_read to see if there are any files there or not.  That doesn’t sound terrible at first, but there’s no mechanism for an application to be notified when the clipboard status is changed by another application.  So, as a practical matter, an application has to check again every time it loses and then regains focus.  On a hard drive system, with a fixed system-defined directory, this would be moderately painful.  With the possibility of the directory changing every single time you check, it’s much less moderate.  And that’s on a hard drive system.

On a floppy drive system… well, shit, that set of problems deserves its own section of the article.

Not Remotely Practical Without Hard Drive

Another problem can be neatly summed up in three words: floppy disk system.

Well, not so much a problem in its own right as a problem magnifier.  Trying to use this clipboard setup on a floppy disk system basically takes every problem and makes it worse. When the user has to use one disk to boot the system, another to launch an application, and yet another to load documents, there’s a lot of disk swapping going on.  So, let’s make it worse by having the clipboard on floppy disk also.  We could use the document disk for that, or we could use a separate disk just for the clipboard.  Let’s assume that last option.

Now imagine that you need to copy some formatted text from your word processor into your favorite graphics program.  Let’s step through it.

Boot system on one disk.  Switch disks to launch word processor.  Switch disks to load the document you want.  Switch disks again to clipboard floppy.  Copy/cut the text, meaning the word processor writes out one or more files with the currently selected text.  Now, maybe, switch back to the document floppy to save changes to the file.  Exit the word processor.  Switch disks to the graphics program floppy and launch that program.  Switch to the floppy with your graphics files and load your graphics file.  Once that’s loaded, switch disks to the clipboard floppy.

Now, remember what we said earlier about an application having to check the clipboard folder ever time it loses and regains focus?  That didn’t actually happen here.  The application stayed in focus the whole time, but we switched floppy disks.  How is the program supposed to know this so it can, or should, check to see if clipboard data is now available?  Should the program have to monitor floppy drive media change just to use the clipboard? Apparently so, even though that’s just ridiculous.

Anyway, let’s just skip past the details and assume the program has checked again for clipboard files.  Maybe there’s a menu item for “refresh clipboard” or something equally inelegant but functional.  So now the graphics program finds the clipboard data with the text saved earlier by the word processor.

But now there’s a new problem.  The word processor saved out plain ASCII into CLPBOARD.ASC and formatted text into CLPBOARD.WUP, because the word processor is WordUp and WUP is that program’s native file format.  But the graphics program, like every other application in the known universe other than WordUp, has no clue whatsoever how to read a WUP file and extract anything useful from it.

Data Formats

There was no problem with the idea of WordUp saving its own native WUP format to the clipboard.  The problem was that it didn’t also save the information using a more universally recognized file format. There was zero documentation to indicate anything about what kind of data could (or should) be exchanged via this whole process, or what file formats should be used.

I’m not exaggerating. When I say “zero” I mean exactly that.  The official GEM 1.0 documentation says nothing whatsoever about clipboard data formats. I would imagine the guys at DRI assumed GEM metafiles would be used for vector graphics and IMG files for bitmapped graphics, but they didn’t actually put those ideas into the docs.

And what about other data? How should formatted text be transferred? What about a block of cells from a spreadsheet?  How about MIDI sequencer data?

The answer ultimately was, your guess is as good as mine. Is it any wonder that application programmers basically ignored the scrap library?

The real tragedy of this situation is that it’s entirely a documentation problem. There’s got to be probably no more than 8 or 10 basic, simple types of data that would cover probably 95% of all clipboard requirements. If the guys at DRI, or at Atari for that matter, had written a couple of pages of documentation saying “use this format for this kind of data” then maybe applications would have supported the clipboard.

As with many other GEM shortcomings, there would be 3rd party attempts to fix the broken situation. The PC version of GEM eventually included some documentation about data formats, but it was way too late, and limited mostly to very obvious things like TXT, IMG, IMG, and CSV.  And for whatever reason this information never really circulated to the Atari side of things at the time.

The Simple Fix, Part 1

Making the clipboard genuinely useful would have been fairly simple, had anybody been paying attention at the time. The first fix would have been to define a reasonable list of standard data exchange formats. and to make sure that list wasn’t completely GEM-centric.   For example, here’s a few just off the top of my head:

  • GEM – GEM metafile vector graphics
  • IMG – Bitmapped graphics
  • ASC or TXT – Plain ASCII text.
  • CSV – Comma separated values, text file containing one or more data records with fields separated by commas.
  • RTF – Rich Text Format formatted text.
  • MID – MIDI Data

Each “standard” format would be assigned a 4-byte value like “_ASC” or “_IMG” that will be used as an identifier.  In most cases the code would correspond roughly to the filename extension commonly used.

Keep in mind that there are some file formats in common use today that weren’t around in 1984. For example, the TIFF format for graphics files was introduced in 1986.  JPEG came out in the early 90’s just barely in time for the creation of the WWW.

The choice of IMG files for bitmapped graphics seems unavoidable, yet there is a critical flaw in that IMG doesn’t allow color palette information to be saved with the bitmap.  DRI seemed to think you’d save out a .GEM metafile defining the palette, but that conflicts with another likely scenario: if your source data is vector graphics, then saving out a rendered bitmap version on the clipboard is entirely reasonable.

This is just a short simple list that serves to illustrate the point.

The Simple Fix, Part 2

After defining the basic data formats, the main thing would be to allow for the clipboard to be either disk or RAM-based.  Most copy/paste operations could be done with a fairly small RAM-based clipboard, but it would have been easily to accommodate disk-based clipboard data when needed.

The existing functions scrp_read and scrp_write are history in this scenario as far as I’m concerned.  In their place, I would have suggested a pair of functions for saving & retrieving RAM-based clipboard data.  Something like this:

int resultcode = scrp_copy( CLIPBOARDDATA *cDataSaved )
 
int resultcode = scrp_paste( CLIPBOARDDATA *cDataRequested )

Yeah, I know. We started with two functions and I’m proposing replacing them with two functions. Kind of ironic, but keep reading and I think you’ll agree I get a lot more mileage out of my two functions than AES got out of the original scrap library functions.

The CLIPBOARDDATA Structure

The CLIPBOARDDATA structure defines the information being saved to the clipboard or retrieved from it. It looks like this:

typedef struct
{
    WORD    formatCount;
    DWORD   *formats;
    UCHAR   **data;
    DWORD   *lengths;
} CLIPBOARDDATA;

The formatCount field indicates specifies the size of the arrays pointed to by the formats, data, and lengths parameters.   When writing data to the clipboard, it specifies how many types of data the application is saving.  When requesting data from the clipboard, it specifies the formats that the application knows how to process.  When the system returns clipboard data to the application, it indicates which formats were actually returned. Note that an application can request one format at a time, or many.

Each of the formats specified should represent the same basic data in different formats. For example, a word processor might save formatted text in its own native file format like WUP, but also using both RTF (Rich Text Format) and plain ASCII TXT.  A graphics application might save vector graphics as EPS and GEM, but also a rendered bitmap version as IMG and TIFF.

The formats parameter is a pointer to an array of 32-bit ASCII format identifier codes (like “_IMG” or “_GEM”)  as mentioned earlier.  Each code represents a single type of data, like ASCII text, IMG bitmap, etc.  The list we defined earlier would form the core of this.

The data parameter is a pointer to an array of UCHAR pointers (WORD aligned), with formatCount elements,  that point to the data.  This field is ignored as input to the scrp_paste function.

Finally, the lengths parameter points to an array, with formatCount elements, where each element specifies the length of the data items pointed to by the data parameter.   This field is ignored as input to the scrp_paste function.

The API Functions

The scrp_copy function would save data, as defined in the CLIPBOARDDATA structure, to the clipboard, in each of the specified formats. Depending on the length of the data, it will be copied from the application space to a buffer allocated and maintained by AES, or saved out as a file to the system clipboard directory.  Or possibly both.

Calling scrp_copy would cause AES to send a message to all open applications indicating that the clipboard had been updated.  (Some later 3rd party revision of GEM added a message named SC_CHANGED for this purpose but it was not part of GEM’s original specification.)

The scrp_paste function would retrieve the current data on the clipboard, if any is available in the requested formats.  On input, the CLIPBOARDDATA structure specifies the data formats the application can accept.  This allows the AES to ignore clipboard formats the program won’t be using.  On output, the structure will be updated to indicate the data formats being returned and the actual data.

The resultcode value returned is 0 for no error,  or various negative values to indicate different errors.

Do We Need More Than That?

We could stop right there and the result would be a billion times more useful than the original scrap library.  There is at least one more function which would be nice to have, though.

  • scrp_bufferinfo – Retrieve information about the RAM-based clipboard data buffer, like maximum size.

Something like this would have been easy to create, fairly compact, and it would have made the GEM clipboard a genuinely useful tool from day one.

Next Time

I think we’ll be headed back to VDI topics for next time around but it’s still up in the air.   Don’t be afraid to comment and let me know what topics you’d like!

Other Articles In This Series

I had originally planned to cover some things about GEM AES in this installment but I’ve recently seen a variety of questions or comments about a couple of VDI related things lately that I’d like to address.

I’m also gonna cut down a bit on the exposition at the beginning that I’ve included in previous posts in this series.  If you don’t know, at least generally, what GEM is, or what VDI is, then why are you reading this? If you need to, go look at earlier posts in the series and then come back to this one.

A disclaimer. Much of what I’m going to discuss here pertains to developments at Atari that occurred long before my employment began. Some of the information was obtained while I was a third-party developer, and some from having conversations with guys who worked in the TOS group at Atari in the years before I started working there. Finally, quite a lot of it comes from conversations with coworkers after I started working there as the ST developer support guy.

GDOS In ROM.

One of the biggest questions people have about GDOS, to this day, is why didn’t Atari put it into the TOS ROM?  Even it wasn’t ready for v1.0, why not v1.02 or v1.04?  Why not 2.0x or 3.0x when the Mega STE and TT030 came out?

Ok, here’s a LITTLE exposition.  GDOS was the part of GEM VDI which was responsible for reading device drivers into memory on request and hooking them into the system so that VDI requests went to the right place.  It also handled loading fonts into memory and hooking them in so that they were available to be used by GEM VDI.

The PC version of GEM had GDOS in its own separate executable file, loaded separately from the rest of GEM. Since GEM was never in ROM at all on the PC this largely escaped attention, but the Atari simply followed this example.

You can download the PC GEM/3 source code here if you’re interested in having a look. The “GDOS (GEMVDI.EXE) source” file contains what corresponds to GDOS.PRG on the Atari.  The “Screen Drivers” file corresponds to the Atari’s ROM portion of VDI, which was really just the screen device driver plus a basic trap handler that routed VDI commands to it.

Beyond following the PC’s example, perhaps the main reason GDOS was not included in ROM on the Atari was that there wasn’t room.

When the ST was designed, it included a ROM space of 192 kilobytes. The problem was, the total size of early versions of TOS came in at a little over 200 kb, not even including GDOS, so it simply didn’t fit within the available ROM space. So instead of having everything in ROM, the operating system was loaded into RAM from disk.  So, the first problem that needed solving was getting everything else squeezed down to 192k.  This took another few months after the ST was put on the market, but finally at the end of 1985, Atari started including TOS 1.0 ROMs in new 520ST shipments, and made ROM chips available to existing owners.

But the TOS 1.0 ROMs still didn’t include GDOS.  It remained a separate executable file of about 9 kb that you loaded by placing into your boot disk’s AUTO folder. Not that big, in the overall scheme of things, but it was big enough that there was no room in ROM even if they had wanted to include it.

OK, so no room in the early 192kb ROM.  Later machines had bigger ROM space, so why didn’t it make its way into those?  Well, at about the same time new machines like the Mega STE and TT030 came out with bigger ROM spaces, Atari was also working on FSMGDOS, which included an outline font scaler as well as new VDI functions for things like drawing bezier curves.  FSMGDOS was too big to fit even with the larger 256kb ROM of the Mega STE.

It might have fit in the TT030’s 512kb space, but by that point, most serious users had hard drives and plenty of RAM. I don’t remember the idea ever even coming up.  Plus, realistically it was too volatile.  There was a new version every week or so for quite awhile and it simply wouldn’t have made sense to put it into ROM. And before the TT030 shipped in significant quantities, FSMGDOS was pulled in favor of replacing it with SpeedoGDOS.

Why Didn’t Atari Make The ST’s ROM Space Bigger?

Not really an expert on the history of ROM chips, but from what I recall from those days, I’m reasonably sure that 32kb was the biggest available ROM chip at the time the ST was first designed.  Or, possibly larger capacities were available but only at significantly greater cost, and maybe they were physically larger.  Either way, larger chips either weren’t available or weren’t practical.

Reasonably, the only way that Atari could have made the ROM space bigger than 192kb would have been to put more sockets onto the motherboard.  Two more sockets would have bumped up the capacity to 256kb, but it also would have required another few square inches of space on the motherboard, which was already pretty much jam-packed.  Look at the picture of the 520ST motherboard below.  Aside from the area at the top center where which was reserved for the RF modulator that is not included in this particular example, there was simply nowhere you could possibly put two more ROM sockets.

520st-motherboard

The other thing to consider was that the basic design of the motherboard was done long before TOS needed to be finalized.  When they decided to include six ROM sockets they very may well have thought they were being generous.  It’s very likely nobody ever even considered the possibility that 192kb wouldn’t have been enough space.

Why Didn’t Atari Put As Much As Possible Into ROM & Disk Load The Rest?

This refers, of course, to the fact that the early 520ST shipments didn’t have TOS in ROM.  Instead, you loaded it from an included floppy disk into RAM, meaning it took up 200K (~40%) of your available memory.  So if the problem was that everything didn’t fit, why didn’t Atari put as much possible into ROM and only soft-load what didn’t fit?

The answer is, they did put some stuff in ROM right from the beginning.  The early 520ST that loaded TOS from disk had two ROM chips instead of six, with the other four sockets left empty.  That means there was as much as 64kb of code in ROM already.

There are essentially 7 components to TOS that ultimately had to fit into the ROM:

  • Bootstrap code – the code that gets the system up and running at power-on, and tries to read a boot sector from floppy disk or hard disk.
  • XBIOS – A variety of low-level functions for accessing hardware resources.
  • BIOS – Low-level functions for accessing hardware devices (i.e. serial ports, printer port, TTY screen driver, disk drives)
  • GEMDOS – Disk operating system & buffered device access
  • GEM VDI – Graphics library
  • GEM AES – Application Environment Services
  • GEM Desktop – Shell application

The preliminary ROMs that were shipped in early machines included the first four items in this list, albeit perhaps not in a finalized form. If you remember the early pre-ROM days, the disk loaded version of TOS was based on having a file named TOS.IMG on your boot disk.  There was nothing else special about the disk. It wasn’t specially formatted or anything.

If you think about what was necessary to read that disk at all, you’ll realize that some version of GEMDOS had to be in ROM, or else the machine wouldn’t have been able to read the disk’s directory, find the TOS.IMG file, and load it. In order for GEMDOS to work, that means a pretty good chunk of the BIOS had to be there.  And that means that certain XBIOS functions had to be there.  And of course, if the bootstrap code wasn’t in place, then the whole system would have been a paperweight when you turned the power on.

So if some of this stuff was in ROM already, then why was TOS.IMG around 200kb in size? Clearly, the TOS.IMG file included new versions of all of the TOS components, not just the GEM stuff.  The main answer to that is, the versions of the components that were in the 64kb ROM were neither complete nor finalized.  They really only included what was necessary to read the TOS.IMG file into RAM and get it started.

While writing these posts revisiting the old days at Atari Corp., it’s occurred to me to note how many things we take for granted in modern operating systems that we just didn’t have back in those days.

For example, I can’t help but think how useful it would have been to have some sort of a system-wide database similar to the registry in Windows. That might have some of you who don’t care much for Windows doing something of a spit take, but when you get down to it, both Linux and Macintosh OS X have similar constructs.

It could have been something relatively simple, like a .INI file, for example. Microsoft Windows got away with a simple setup like that for many years.

The DESKTOP.INF file was arguably an embryonic version of a system registry. This file was used strictly by the desktop to store desktop settings and a few bits of information from the control panel. It could have been used for much, much more, but there was no system API to access it.

How much easier, might it have been, to have all of the information for things like the DESKTOP.INF file, ASSIGN.SYS file, and so forth, all in one place and managed by a common API, rather than in a variety of separate files? And since these files were used to store configuration information even in the early days when they were loaded from floppy disk, I can’t help think that some attempt at making it into a more general purpose system registry would have been a huge advantage.

Not long after I started working there, Atari decided to come out with a new Control Panel which supported the idea of plug-in modules. This way individual applications, utilities, or new system features could have a nice centralized location where users could go to set various options. However, one of the problems we had early on was figuring out a method to allow the modules to retrieve or store their information. We didn’t want them creating individual configuration files everywhere, but what was the alternative? It was finally decided to have them write any info they wanted to persist between sessions into the data segment of their own executable file.

I know, I know. But this was 1990-1991 or so and it seemed like a good idea at the time. But imagine how much easier that situation would have been if we had a system-wide registry.

As goofy as that may sound, the way that programs had to read Control Panel information at run time was arguably even goofier. In order to make it accessible to applications, TSR programs and Control Panel modules could store information in a little block of memory which we called a cookie (no relation to modern web browser cookies, though the concept is arguably similar).

To find the cookie information that was available, a program would access a predefined memory location to get a pointer to the start of the cookie list. Then it would simply walk the link list of cookie blocks to find all of the information available.

This was 20 years ago, remember?

The cookie idea worked, but it was kind of a kludge and programs had to do a lot of work to support it. They were expected to be able to do things like allocate new memory for cookies as needed and link that into the existing list. The bottom line is that it was overly difficult to maintain, especially considering there really should just have been a couple of API calls to handle the whole mess. A system-wide registry would have been able to easily manage something like this.

Oh well, you know what they say about hindsight.

Note: I commonly use the term “we” when referring to Atari in general, but unless I otherwise provide particular details, I don’t intend to imply anything about how much I may have had to do with the decision making process on any particular thing. I had a lot of input on some things, little on others. In some cases, i wrote code myself, and in others my influence was largely based on having lunch regularly with the guys doing the actual coding and throwing around ideas while we were eating. In many cases, I’d discuss ideas with our third party developers and then bring their feedback into the mix. Anyway, my point is that “we” usually translates into about a dozen people, at least, and they all deserve proper credit. Or blame.

« Previous Entries