hyperlook

http://www.art.net/~hopkins/Don/hyperlook/

via;: http://mailman.laptop.org/pipermail/sugar/2006-December/001022.html

I love the ideas behind Smalltalk, EToys and HyperCard, and would like
to combine them with ideas from visual programming languages like Robot
Odyssey, KidSim, Klik-and-Play, SimAntics, Body Electric/Bounce,
Max/MSP/Jitter, etc.

Here are some ideas about HyperLook and other systems, that could be applied to Sugar:

HyperLook was a PostScript-based user interface development environment for the NeWS window system, which Arthur van Hoff created at the Turing Institute in Glasgow.

http://www.donhopkins.com/home/catalog/hyperlook/

I helped develop HyperLook into a commercial product, with a editable user interface development environment, as well as a redistributable non-editable runtime, and I used it to port SimCity to Unix, and develop other components and applications .

http://www.donhopkins.com/home/catalog/hyperlook/

HyperLook was inspired by HyperCard, but it additionally provided a client/server programming model, and more powerful graphics and scripting based on NeWS's object oriented dialect of PostScript. http://www.donhopkins.com/home/catalog/hyperlook/TalkInterfacing.gif

The NeWS window system was like AJAX, but with: 1) PostScript code instead of JavaScript code 2) PostScript graphics instead of DHTML graphics, and 3) PostScript data instead of XML data.

It had a unified programming/graphics/data/networking model based on NeWS's extended multi-threaded object-oriented dialect of PostScript, instead of a hodge-podge of accidental technologies. (Although I will be the first to admit the X11/NeWS merge was quite a hodge-podge and huge-kludge!)

NeWS had an object system based on the simple dynamic ideas of Smalltalk, implemented with the PostScript dictionary stack, supporting multiple inheritance and runtime modification of objects and classes.

HyperLook was a gui development framework and desktop environment, that extended NeWS with a user-editable structured PostScript graphics format, a persistence system, a HyperCard-like delegation model using a network based client/server library, that passed messages from button to page to background to stack, and finally over the network to the application (and back), and an entire user interface toolkit, window manager, gui editor, clipboard and other desktop tools and services, all designed around the PostScript graphics format and message passing system.

Users could create their own integrated applications, task oriented interfaces, presentations and journals, by cutting and pasting high level components and graphics together, configuring them with property sheets and graphical editors, scripting them with PostScript message handlers, sending and receiving messages between other stacks and applications, and customizing applications to suit their needs.

It provided a user-editable window manager, that allowed you to draw and shape the window frame with arbitrary PostScript graphics, cut and paste you own resize corners, close boxes, menus, buttons, pie menus and other controls, as well as combining and scripting together multiple application components and custom graphics.

Because everything was based on PostScript, you could print any window to a PostScript printer or copy it to the clipboard, and iconified windows were just live miniaturized views.

Developers could create back-end services (like audio mixing) and applications (like SimCity) that could send messages back and forth, use shared memory for efficient bitmap animation, and share services like audio mixing with other applications.

I developed SimCity in parallel with HyperLook, so there was a powerful synergy, SimCity drove the development of many of HyperLook's features, and the other way around.

SimCity extended HyperLook with client/server based components like the city editor, map view, graph display, which could be copied and pasted and placed anywhere in the interface.

It was great to have a demanding application like SimCity as an acid test, to shake out bugs and limitations of the platform, and prove the abilities of general purpose components like bitmap animation and audio mixing.

HyperLook was a commercial product, with a WYSIWYG interface development environment for developers, and a freely redistributable non-editable runtime.

We released it at the same time as SimCity (using SimCity as bait to entice people to try out the included HyperLook runtime).

http://www.donhopkins.com/home/catalog/hyperlook/HyperLook.README

http://www.donhopkins.com/home/catalog/hyperlook/SimCity.README

The user interface development environment could be stripped out of the system to create a non-editable binary-encoded runtime version for shipping turn-key commercial products, which I delivered with SimCity. But if you had the development version, you could create your own stacks, put the interface into edit mode, cut and paste user interface components around, and edit their scripts (enabling user interface vandalism).

Personally, I think all users should have a user-editable development environment, but it's important to be able to lock it down and constrain it, so casual user's can't accidentally break the system, or get confused with unnecessary details.

HyperLook included a “warehouse” of pre-configured object templates, which you could cut and paste into your own stacks, and configure and script to create your own HyperLook applications, custom interfaces, interactive presentations, etc.

Like HyperCard, you could copy and paste components around (including per-object customizations like scripts, properties and graphics) to construct and edit your own interfaces.

Like HyperCard, you could open up a property sheet or script editor on any object.

Unlike HyperCard, the scripts were written in object oriented PostScript. (Arthur van Hoff also wrote PdB, an object oriented C to PostScript compiler, and later went on to write the first Java compiler in Java at Sun!)

Unlike HyperCard, the property sheets were implemented as HyperLook stacks themselves, which made it easy to develop new components or customize property sheets for existing components (i.e. simplified for kids, graphically oriented for designers, advanced for developers), and the user interface editor itself was even a plug-in component that could be removed or replaced (to make a non-editable runtime, or to plug in simpler or more advanced gui editors).

HyperLook was able to implement its own property sheets as stacks, because it had a sufficiently rich set of built-in components including text and graphics editors, and stacks could be scripted to import and export properties to control settings (to convert between data types and control values, implement apply/cancel, change detection, and selecting previously edited objects).

HyperLook included a nice little PostScript graphics editor component that you could integrate into your own applications and property sheets. For example, there is a user-customizable clock that lets you edit its face and hands (by incorporating three graphics editors in its property sheet), and there are some example clocks, pre-configured in the warehouse. You can copy them into your own stacks, place and stretch them (since they're scalable PostScript graphics), modify their appearance with the clock editor property sheet, and paste your original creations back into the warehouse to use again:

Here's a cellular automata laboratory that uses the shared memory raster animation library to integrate a Toffoli/Margolis CAM-6 simulator I wrote in C with the PostScript graphics editor (so you can cut and paste PostScript graphics into live running cellular automata, and copy the cells into the graphics editor, and generate garish but seamlessly tiled screen backgrounds, and place a live bubbling cellular automata view component clipped into a lava-lamp shaped window!):

The nice thing about having a standard PostScript based structured graphics format, is that the entire system supports it, so you are free to do fun stuff like making a clock face out of a cellular automata or SimCity map, copying an entire window including its user interface components as structured graphics, clipping and stretching it in in the graphics editor, and using it as a clock hand, or whatever else you can think of.

Here's a transcript and video demo of HyperLook SimCity, cellular automata, user interface and graphics editing, and cutting and pasting graphics between various HyperLook applications.

http://www.donhopkins.com/home/catalog/simcity/hyperlook-demo.html

http://www.donhopkins.com/home/movies/HyperLookDemo.mov

It is extremely important that the base system includes a standard user-editable structured graphics format (higher level that raw PostScript, but including Encapsulated PostScript or modern equivalents like PDF, SVG, PNG, etc).

It's also essential to have a reusable structured graphics editor component, and also that all of the property sheets and applications take full advantage of it, so you can edit every visual aspect of the user interface, and copy any graphics to the clipboard with their structure intact.

HyperLook's graphics editor (HyperDraw) was not too fancy, but it was really easy to use for the stuff you want to do 90% of the time. A bitmap image editor would have been really nice too, but that was hard to build into the NeWS server without a C client, unfortunately, so we never got around to that (although the CAM stack has some simple drawing tools for playing with the live cellular automata pixels through shared memory).

Something like Photoshop or GIMP would be too complicated and monolithic, unless it could be stripped down and re-packaged as an light-weight, easy-to-use, plug-in component.

I've worked on and programmed in some interesting visual programming
languages like Bounce (aka Body Electric, VPL's VR scripting language),
SimAntics (The Sims behavior scripting language) and PSIBER (visual
PostScript and NeWS debugger), and played around with others like
Max/MSP/Jitter, ProGraph and KidSim (aka Cocoa, Stagecraft Creator,
which was at once point actually implemented in ProGraph!).

Visual programming languages are great fun, and can be extremely productive at their finest, but they tend to be quirky, nonstandard, have more differences than similarities, and extremely weird ways of looking at the world.

There is no one best way to design the ultimate visual programming language, but there are many good ideas to be tried (and bad ideas to be avoided) from previous systems, that could be recombined in interesting ways.

There will never be one true visual programming language, just like there will never be one true text programming language, so I think the system should be designed to accommodate different languages and skill levels. In the same way Lisp is excellent for meta-programming application specific mini-languages, I would like a similarly enlightened system for creating application specific visual programming languages (like the Eclipse Visual Editor project, but for a nice dynamic language like Python, instead of Java).

Microsoft's IScriptingEngine interface lets you plug different scripting languages like Visual Basic Script, JScript, Perl and Python into the web browser and other applications. But COM/ActiveX/OLE Automation isn't the only way to do that. I am quite happy that Sugar is using Python, because it makes a great common runtime and “virtual machine language,” for implementing visual programming languages and integrating software modules (especially with tools like SWIG for integrating native libraries and C++ code).

My ultimate fantasy gui environment would support plugging in different script editors, with not only easy-to-use but also advanced visual programming interfaces, tailored to specific tasks and skill levels (like handling and sending events, solving constraints, processing signals, playing music, controlling robots, blinking lights, dancing bears, etc).

Python will serve well as the internal “machine language” that you compile the visual graph notations into (or interpret the visual languages directly in Python, if practical).

So kids could easily create a high-level graphical scripts like The
Learning Company's “Robot Odyssey” or Maxis's “Klik-and-Play”, and
developers could meticulously create low-level behind-the-scenes scripts
in visual languages like “Body Electric” and “Max”, or text language
like Python (or a hybrid of the visual and text languages – check out
how PSIBER implements a new visual interface to the existing PostScript
language, and how Cycling 74's Max/Jitter integrates JavaScript with the
visual programming language: JS code inside icons, JS objects on wires!):
http://en.wikipedia.org/wiki/Klik_&_Play
http://www.cycling74.com/products/jitter
http://www.cycling74.com/download/JavascriptInMax.pdf

Just imagine a visually scriptable version of SimCity, that lets you clone and edit the monster, and reprogram him to tend the forests instead of stomping on buildings!

  1. Don

Some notes on alternatives to Python:

I love Python, and it's my primary scripting language of choice, but I'm not a monolinguist or strict interpretationalist, so I think some other languages are worth considering and learning from, in the long term:

Lua:

Lua is an excellent, well designed and implemented scripting language, that's extremely light weight and efficient, truly open source, and is also nicely supported by SWIG. It's well worth considering if you're starting from a clean slate, and don't need all of Python's modules, and do care a lot about speed and size, and want to tightly integrate the scripting language with the application, without requiring the installation of a bunch of external files, resources and libraries. It's very popular in the game industry, widely used in World of Warcraft, inspired by Lisp but without the macros, easy to learn and to read, has a pretty good JIT that works on x86 systems, and has a bunch of useful extensions (but not nearly as many as Python). Lua has an extremely flexible roll-your-own object system (not one standard oop paradigm, but the basis for making all different kinds, but very simple, not nearly as powerful and general purpose as CLOS, not quite as elegant and minimal (nor as bloated a runtime) as Self).

SWIG supports Lua objects at a kind of low level, but can be extended to nicely support higher level objects, given a particular model. But SWIG does not work as well out of the box with Lua as it does with Python (since Python's object system is well known and not a moving target subject to the whims of the application programmer). The fact that Lua doesn't have Python's huge set of modules is both a blessing and a curse, but Lua actually a bit faster and much smaller than Python, vastly simpler, has a few weird but tolerable quirks (like one-based array indexing), and is conceptually cleaner and Lispier than Python in some nice ways. According to the computer language shoot-out, it's one of the fastest and smallest interpreted scripting languages (and presumably even faster with the JIT, which is not included in that benchmark). Lua's ratio to C's speed is 6.6, compared to Python's 7.3, which comes in right behind Lua.

http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=all

JavaScript:

JavaScript is a horribly designed and implemented language, although it's ubiquitous, never going away, well worth learning, and good enough to get a lot of stuff done in (Lisp with C syntax, without macros). However, SpiderMonkey (the original JavaScript interpreter, and the one in Firefox) is extremely inefficient and wasteful of memory. According to the computer language shoot-out, it's absolutely the worst language benchmarked, by a long shot. SpiderMonkey's ratio to C's speed is 31, compared to Ruby's 16, which is the next slowest language. Incredibly, its memory usage is even worse compared to other languages (ratio to C's size is 32, compared to the next worst Smalltalk VisualWork's 13), so it's not like it's making a memory-vs-speed trade-offs to be that slow.

http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=all

I have a hard time believing that JavaScript will ever run as fast or small as Lua, simply because Lua was well designed to be fast and small on purpose (and thus avoids inefficient pitfalls by design), and JavaScript wasn't designed at all, it just happened accidentally. It would be better to take all the cleverness that you'd have to apply to JavaScript to make it fast, and apply it to Lua instead – you'd be way ahead of the game, you wouldn't be fighting against but remaining compatible with bad designs, and you'd end up with a much nicer language. Another important thing that would make me choose Python or Lua over SpiderMonkey/JavaScript, regardless of the speed and size differences, is that SWIG supports Python and Lua well, and it doesn't currently support SpiderMonkey.

However, there is an interesting recent development in JavaScript's favor: Adobe has donated the source code for the ActionScript VM that's in the Flash 9 (AVM2)!

It turns out that AVM2 does not totally suck. (There are no benchmarks available comparing it to other JavaScript VM's, but you can easily see and feel the huge difference between Flash 8 and Flash 9). The previous versions of Flash were based on an absolutely horrible and inefficient virtual machine, which has finally been entirely rewritten by grown-ups, and includes a JIT, so it has drastically better performance. They have not made the entire Flash player open source, just the ActionScript byte code VM, which does not include an actual JavaScript source code to AVM2 byte code compiler. (Although they are developing a prototype JavaScript compiler in JavaScript, it is not fully complete and tested, although it's a great ambitious idea.) Adobe has their own ActionScript to VM byte code compiler, but it's not open source. Of course a web browser needs a JavaScript compiler, since it downloads JavaScript in source code form. But it also totally doesn't suck to be able to download pre-compiled binary JavaScript byte code (i.e. SWF files).

OpenLaszlo is developing its own open source JavaScript to AMV2 byte code compiler (the Legals project), so OpenLaszlo will be able to compile code that it will run in AVM2. The Mozilla project plans on incorporating AVM2 into the browser to accelerate JavaScript, but there's still a lot of work to be done. But I think that AVM2 is a great long term strategy to solve the performance problems of SpiderMonkey in the web browser, that will dovetail with OpenLaszlo, and be useful for developing efficient AJAX applications, using a completely open source software stack.

http://www.openlaszlo.org

Here are some references to work I've done in the past, describing some ideas that might be applied to Sugar.

NeWS is James Gosling's “Network extensible Window System”, which was based on a multithreaded, networked, object oriented dialect of PostScript. I developed pie menus for NeWS at the UMCP HCIL, worked on the Open Look NeWS Toolkit at Sun, and various NeWS tools and applications for other companies.

http://www.donhopkins.com/drupal/node/92

I developed lots of other fun stuff with NeWS, including various implementations of pie menus and tab windows:

http://www.donhopkins.com/drupal/node/92 http://www.piemenu.com/DDJPieMenuArticle.html http://www.piemenu.com/

The NeWS Toolkit was an Open Look user interface toolkit written entirely in NeWS PostScript, which I helped develop at Sun, and integrated with HyperLook, and used to develop pie menus and tab windows.

http://www.donhopkins.com/home/catalog/images/tnt.gif

Pizza Tool was a graphical Open Look gui for editing and ordering pizzas via FAX, that I wrote as a programming demo for The NeWS Toolkit:

http://www.donhopkins.com/home/catalog/images/pizzatool.gif

The HyperTIES hypermedia browser, with pie menus and “applets” scripted in PostScript (for Ben Shneiderman at the UMCP HCIL):

http://donhopkins.com/drupal/taxonomy/term/61

Emacs “thin wire” NeWS display driver for UniPress Emacs (and later Gnu Emacs), with pie menus and multiple tab windows, and an authoring tool for HyperTIES:

http://donhopkins.com/drupal/node/101

HyperLook (aka HyperNeWS and GoodNeWS) was a HyperCard-like user interface development environment for the NeWS window system. I worked on HyperLook with Arthur van Hoff at the Turing Institute, and used it for various applications.

http://www.donhopkins.com/home/catalog/hyperlook/

SimCity is a constructive simulation game from Maxis. I ported SimCity to HyperLook and X11, and designed a multi player scriptable user interface using TCL/Tk.

http://www.donhopkins.com/home/catalog/simcity/index.html

PSIBER is a visual interface to PostScript and interactive debugger for NeWS. I developed it at the UMCP HCIL and Grasshopper Group.

http://www.donhopkins.com/drupal/node/97

Bounce (aka Body Electric) is a real time data flow visual programming language, developed at VPL by Chuck Blanchard and used by Jaron Lanier and others for integrating midi and various i/o devices, driving 3d rendering engines, scripting VR, scientific, training and artistic simulations. I programmed multimedia and character simulations in Bounce, ported it to PowerPC, developed its user interface, a COM plug-in and type extension system, and multimedia support, with David Levitt at Levity and Interval Research Corporation.

http://www.donhopkins.com/home/catalog/lang/bounce/bounce.html

SimAntics is the visual programming language for scripting AI and behavior in The Sims, pie menus (aka marking menus, compass menus) are an efficient Fitts'-law-friendly menu selection technique. I programmed in SimAntics, ported it from the Mac to Windows, and developed its user interface (Edith) and primitives (character animation, etc), with Will Wright at Maxis:

http://www.donhopkins.com/drupal/taxonomy/term/35

The Dumbold Voting Machine is an educational agit-prop plug-in for The Sims, that I programmed in SimAntics to educate people about real voting machine problems – check out the “high concept” stuff at the end of the page: It's meant to stimulate discussion, illustrate how computer programming is free speech, and why all election software should be open source and publicly inspectable.

http://www.dumbold.com/dumbold

This is a movie that demonstrates the character animation system, pie menus, direct manipulation object placement and architectural editing tools that I developed for The Sims, as well as an in-depth demo of SimAntics visual programming with the Edith tool (stepping through the Satan Generator code):

http://www.donhopkins.com/home/movies/TheSimsPieMenus.mov

category animation