Like many developers I enjoy the addition of code completion (or insight or intellisense etc) in an IDE. Jbuilder, Delphi, VB and various other major IDE's have it.
The ability to see a list of possible members (properties & methods) in OOP is valuable, especially in huge languages like Java. The ability to see syntax & argument reminders is handy too - especially where the programmer needs to (or wants to) use various tools. Back in the days of Turbo C etc is was plausible to study and know all commonly used library functions & syntactic requirements, and reference a book for the rest. Nowadays, if you switch between 3 or 4 tools in the hugely extended languages we have it's nigh on impossible, and the books these days have to be enormous!
So I'm quite interested in the history of code completion in IDE's & code editors. Does anyone have any info or personal recollections, or even opinions on the merits/demerits of code completion?
> So I'm quite interested in the history of code completion in IDE's & > code editors. Does anyone have any info or personal recollections, > or even opinions on the merits/demerits of code completion?
It used to be much biger than it is nowadays, as I understand it. One name for the general area is "syntax-directed editting". I'm a little curious, myself, why it hasn't caught on more than it has. I wonder how much it's due to a still-prevailing attitude that Real Programmers can do it themselves and don't need the help?
Two styles are available for syntax-directed editting. First, the editor can force you to only generate syntactic forms. To enter a statement, you select from some kind of menu or keyboard command what kind of statement you want, and the editor will fill in defaults. Now, the editor can have a compiled version of the program at all times. If you want to change the name of a function, you can select that function and then bring up a menu -- the editor will know that (a) you have selected a function name, and (b) what functions are in scope at that point. Squeak's end-user programming tool is a current system that works like this, but it's not practical for general programming. (Though there is talk....)
As I understand, programmers who have used traditional tools tend to get irritated in systems of this first style. That's too bad; it actually sounds to me like it could be a very smooth programming experience! Imagine, no syntax errors.... But, I've never actually tried it.
The other style is to leave the programmer with a general text editor, and then have the editor maintain a parse tree for what the user has typed -- essentially compiling while the user types. There are algorithms for incremental parsing and incremental analysis, e.g.:
Thomas Reps, Tim Teitelbaum, Alan Demers. Incremental Context-Dependent Analysis for Language-Based Editors. ACM Transactions on Programming Languages and Systems, Vol. 5, No. 3, July 1983, 449-477.
It's tricky, though -- a lot of the intermediate stages aren't going to be legal code!
This style is used in VB and Smalltalk, and it's used in syntax highlighters in, e.g., Emacs. The syntax highlighters in particular often rely on heuristics instead of doing strictly correct parses, at least if the current buffer has unsyntactic code in it. I honestly don't know whether this is due to theoretical limitations or just due to a lack of effort by the authors.
In either style, note that you *have* to have a language-specific editor or editor mode for this kind of tool to work. For it to work well, you furthermore need the editor to be integrated with the compiler (or at least, the editor needs to know about the compiled codebase.) For some reason, there is a lot of resistance to such integrated development environments, partially because they tend to not cooperate well with other languages.
In my more cynical moments, I sometimes wonder if software-development tools are driven by a desire to make programming remain "fun". What a crazy bunch we are!
-Lex [I've used both kinds of editors. The first kind is incredibly annoying, because there are lots of common and simple text changes that are large changes in the parse tree, e.g, move a close brace down a few lines past some other statements. The second kind is still around. Teitelbaum et al started a company and turned their stuff into products. See http://www.grammatech.com/ -John]
On 21 Oct 2001 21:30:13 -0400, Lex Spoon <l...@cc.gatech.edu> wrote:
> This style is used in VB and Smalltalk, and it's used in syntax > highlighters in, e.g., Emacs. The syntax highlighters in particular > often rely on heuristics instead of doing strictly correct parses, at > least if the current buffer has unsyntactic code in it. I honestly > don't know whether this is due to theoretical limitations or just due > to a lack of effort by the authors.
Tim Wagner's PhD thesis at Berkeley in 1997 was the design of an incremental GLR parser that could identify and efficiently manage syntactically incorrect regions in parts of the parse tree:
This was done as part of the Ensemble (now Harmonia) project for building extensible development environments. There was also work on incremental lexing done, of course.
Once you have the parse tree, the next question is what kinds of semantic analysis you can offer the user. Online typechecking is obvious, and even highlighting which compiler optimizations are feasible with a given code structure is possible.
I've only really seen two example of this. First is the refactoring browser. The other one is the Dylan compiler from Functional Objects, which has an IDE that highlights method calls to identify which method dispatches the compiler is able to optimize away. This lets you visually identify whether you have a lots of dynamic dispatching in a critical inner loop, for example. It's very neat.
> So I'm quite interested in the history of code completion in IDE's & > code editors. Does anyone have any info or personal recollections, > or even opinions on the merits/demerits of code completion?
Looking Glass Software's Alice Pascal, a syntax directed editor, interpreter, and debugger for teaching Pascal, had context sensitive code completion. It shipped for DOS PCs and CEMCORP ICONs in 1985, and for Atari STs in 1986. You can read about it, and download the PC and ST versions of Alice, source code and all, from Brad Templeton's Alice Pascal page at http://www.templetons.com/brad/alice.html. The PC version is a DOS exe that uses a curses layer, writing directly to VGA/EGA, and it still seems to run properly on my Windows 2000 machine.
Here are some of the things you could do / can do in Alice Pascal.
1. Go to any placeholder, press ALT+T, and get a popup menu of syntax elements and/or identifiers that are valid there.
2. Declare a new symbol (variable, type, etc). Go to a placeholder that expects that kind of symbol. Press the END key to obtain a popup menu of all current symbols that are valid there. Or, type the first letter or two of the identifier, press END, and its name is completed (or, if more than one is possible, a popup list of (context valid) names with that prefix appears). Ditto for prompting for / completing the names of valid fields of a record on the RHS of a '.' field expression.
3. Debug, single-step, etc. even partially completed programs. Or, enter a statement in the 'immediate mode' window, press ENTER, and run it.
4. Hide and reveal code blocks (source code outlining).
5. Edit the declaration of a symbol to rename all uses of it.
One thing you couldn't do was enter a syntactically incorrect program. (After the addition of a text-editor-expression-parser, you could enter syntactically incorrect *expressions*.) And if an edit introduced a semantic error, (e.g. a type error), Alice would often diagnose the error immediately.
Alice Pascal demonstrated many helpful uses of an AST. But it also showed that syntax directed editing was a dead end. The programming community was not willing to unlearn programming (by gradually refining syntactically incorrect program snippets) in a text editor, in favor of power-assisted programming (by entering program tree elements in prefix order), even if you provide a lot of nice conveniences and benefits.
That's OK. You don't need syntax directed editing to obtain an AST. You can always recover an AST with a background compilation process from within the IDE. On modern machines you can probably do this after every keystroke or pause, parsing (as best you can) what the user has entered, in order to recover as much partial AST as you can, in order to provide context sensitive assistance, instant cross-reference browsing, etc.
Alice influenced my later work on various VC++ features (unrelated to IntelliSense), but that was long years ago. These days I love the IntelliSense code prompting and completion in VS.NET beta 2. It makes exploring the vast surface area of the .NET Framework Class Library easy and fun.
I was lucky to work for Brad on Alice Pascal. I learned so much from him. True story: once we were up late, testing some new AST manipulation code. Editing worked -- you could load, save, edit your program just fine. Execution worked -- press F1 (RUN) and your program ran. Oops! Hide/reveal was broken. If you selected a block of code and tried to HIDE it (collapse it in the program outline) and then REVEAL it again, the AST could be corrupted. Without missing a beat, we both looked up and exclaimed "You can RUN but you can't HIDE!"
Lex Spoon <l...@cc.gatech.edu> wrote: >The second kind is still >around. Teitelbaum et al started a company and turned their stuff >into products. See http://www.grammatech.com/ -John]
The once very popular THINK Pascal compiler for the Macintosh also worked in that fashion. The program text was parsed until the first syntax error, after which text was rendered in outline font. Correct text would be formatted nicely, with keywords in boldface. The AppleScript editor also works mostly this way.
Having spent my teenage years hacking in COMPAS (which later became Turbo) Pascal, I appreciate the combination of a fast compiler with the simplicity of always only having to deal with the first occuring syntax error. I still tend to program that way, and would love to have a compiler that could work as a coroutine to for example vim. To this day I still don't really understand why a compiler should waste time trying to second-guess the programmer's intent after the first occuring syntax error.
I have worked a bit with Sun's Forte for Java IDE, which also has completion features - it works quite well.
> The other style is to leave the programmer with a general text editor, > and then have the editor maintain a parse tree for what the user has > typed -- essentially compiling while the user types.
I'm currently working on something for (X)Emacs that does this in a better fashion than the current syntax highlighters do (the current syntax highlighters basically work on regular expressions and don't actually offer symbol completion).
I have yet to try it out (I'm still pondering a couple design decisions related to C), but instead of keeping track of everything all the time, you can simplify it immensely if you just parse the input every time a completion is requested. You still need error recovery, of course, because you have no idea what the input might be.
Admittedly, this is somewhat wasteful, but if parsing speeds are acceptable, the system wouldn't be all that bad (as far as I can tell, based on what I know). If speeds are not quite up to par, you can either force the programmer to manually force a parse or cache some information and not get all the symbols every time. Acceptable losses, I suppose, but I guess I'll have to find out.
-- Geoff(rey) Wozniak, Limited Duties Instructor University of Western Ontario Computer Science Department London, Ontario, Canada http://woz.killdash9.org/
> In either style, note that you *have* to have a language-specific > editor or editor mode for this kind of tool to work. For it to work > well, you furthermore need the editor to be integrated with the > compiler (or at least, the editor needs to know about the compiled > codebase.) For some reason, there is a lot of resistance to such > integrated development environments, partially because they tend to > not cooperate well with other languages.
I think that resistance is something from the C crowd. E.g. Pascal (Delphi, Turbo Pascal) has been doing this for quite a long while now.
> In my more cynical moments, I sometimes wonder if software-development > tools are driven by a desire to make programming remain "fun". What a > crazy bunch we are!
> -Lex > [I've used both kinds of editors. The first kind is incredibly > annoying, because there are lots of common and simple text changes > that are large changes in the parse tree, e.g, move a close brace > down a few lines past some other statements. The second kind is still > around. Teitelbaum et al started a company and turned their stuff > into products. See http://www.grammatech.com/ -John]
I somehow have the feeling that this is the same battle as an entirely free form texteditor vs something with modes like VI.
Lex Spoon <l...@cc.gatech.edu> wrote: >> So I'm quite interested in the history of code completion in IDE's & >> code editors. Does anyone have any info or personal recollections, >> or even opinions on the merits/demerits of code completion? >It used to be much biger than it is nowadays, as I understand it. One >name for the general area is "syntax-directed editting". I'm a little >curious, myself, why it hasn't caught on more than it has. I wonder >how much it's due to a still-prevailing attitude that Real Programmers >can do it themselves and don't need the help?
I can do it myself. When I know what I want, I don't want programs "helping" because they often get it wrong. Word is infamous for helping (read "sabotaging") in this way. I finally have all of its nonsense shut off, I think.
Microsoft Word has the setting for handling automatically numbering paragraphs under AutoCorrect. Considering that I was doing what I wanted correctly, I didn't look there until someone else pointed me to it. I think Microsoft has lots of gall doing that.
Most of us can type well. I'd rather just type than also have to worry that a program is going to change what I type.
>Two styles are available for syntax-directed editting. First, the >editor can force you to only generate syntactic forms. To enter a >statement, you select from some kind of menu or keyboard command what >kind of statement you want, and the editor will fill in defaults.
I'd just as soon type than select from a menu. It's generally much faster. If the selection involving mousing, it's even worse. Taking ones hands off the home row is expensive for a touch typist.
>Now, the editor can have a compiled version of the program at all >times. If you want to change the name of a function, you can select >that function and then bring up a menu -- the editor will know that >(a) you have selected a function name, and (b) what functions are in >scope at that point. Squeak's end-user programming tool is a current >system that works like this, but it's not practical for general >programming. (Though there is talk....)
Will it change all calls, including in other programs? No? Then, I had better keep track of things. Will it adjust comments to line them up if the new name is a different length than the old name? How do I specify this? Whichever way it works, it's unlikely to do it my way exactly.
>As I understand, programmers who have used traditional tools tend to >get irritated in systems of this first style. That's too bad; it >actually sounds to me like it could be a very smooth programming >experience! Imagine, no syntax errors.... But, I've never actually >tried it.
No syntax errors is easy. No semantic errors is the problem. I'd just as soon look up how a function works if I'm uncertain than to rely on a program's suggestion.
>The other style is to leave the programmer with a general text editor, >and then have the editor maintain a parse tree for what the user has >typed -- essentially compiling while the user types. There are >algorithms for incremental parsing and incremental analysis, e.g.:
> Thomas Reps, Tim Teitelbaum, Alan Demers. > Incremental Context-Dependent Analysis for Language-Based Editors. > ACM Transactions on Programming Languages and Systems, > Vol. 5, No. 3, July 1983, 449-477.
>It's tricky, though -- a lot of the intermediate stages aren't going >to be legal code!
It depends on the language. C won't work when there are fancy compile-time substitutions.
>This style is used in VB and Smalltalk, and it's used in syntax >highlighters in, e.g., Emacs. The syntax highlighters in particular >often rely on heuristics instead of doing strictly correct parses, at >least if the current buffer has unsyntactic code in it. I honestly >don't know whether this is due to theoretical limitations or just due >to a lack of effort by the authors.
It depends on the language, but I expect theoretical limitations bite often. Compiler preprocessors probably raise much havoc with such editors.
>In either style, note that you *have* to have a language-specific >editor or editor mode for this kind of tool to work. For it to work >well, you furthermore need the editor to be integrated with the >compiler (or at least, the editor needs to know about the compiled >codebase.) For some reason, there is a lot of resistance to such >integrated development environments, partially because they tend to >not cooperate well with other languages.
A language-specific editor means that I may have to work with a number of editors rather than just one that I prefer.
>In my more cynical moments, I sometimes wonder if software-development >tools are driven by a desire to make programming remain "fun". What a >crazy bunch we are!
I agree with that.
>[I've used both kinds of editors. The first kind is incredibly >annoying, because there are lots of common and simple text changes >that are large changes in the parse tree, e.g, move a close brace >down a few lines past some other statements. The second kind is still
Quite.
>around. Teitelbaum et al started a company and turned their stuff >into products. See http://www.grammatech.com/ -John]
> >> So I'm quite interested in the history of code completion in IDE's & > >> code editors. Does anyone have any info or personal recollections, > >> or even opinions on the merits/demerits of code completion?
> >It used to be much biger than it is nowadays, as I understand it. One > >name for the general area is "syntax-directed editting". I'm a little > >curious, myself, why it hasn't caught on more than it has. I wonder > >how much it's due to a still-prevailing attitude that Real Programmers > >can do it themselves and don't need the help?
> I can do it myself. When I know what I want, I don't want > programs "helping" because they often get it wrong.
Ah, but the people who write this stuff actually have to use it too, so they take some pains to get it right. MSVC's code completion, for example, is perhaps not perfect but is nevertheless pretty darn good.
> Word is infamous for helping (read "sabotaging") in this way. I > finally have all of its nonsense shut off, I think.
EEEEEKKKKK!!!!!
Last time I used Word (which was about two months ago on-site), it took me 30 minutes to write a very short document. This was partly because Word objected to my attacking its Auto-Correct settings (yes, I was trying to shut them all down too), and hung the OS. Not just the program, but the OS. (And this was NT4, which isn't supposed to let that sort of thing happen.)
God bless vi.
> Microsoft Word has the setting for handling automatically > numbering paragraphs under AutoCorrect. Considering that I was doing > what I wanted correctly, I didn't look there until someone else > pointed me to it. I think Microsoft has lots of gall doing that.
You may want to have a look at Lotus WordPro, which is mildly less irritating in this regard. But vi is *much* less annoying. :-)
> Most of us can type well. I'd rather just type than also have to > worry that a program is going to change what I type.
Well, I wouldn't buy that editor either. But shortcuts are fair enough, as long as they are *consciously* invoked.
> >Two styles are available for syntax-directed editting. First, the > >editor can force you to only generate syntactic forms. To enter a > >statement, you select from some kind of menu or keyboard command what > >kind of statement you want, and the editor will fill in defaults.
> I'd just as soon type than select from a menu. It's generally > much faster.
Sure. Code completion tends to come into its own when you have long, descriptive variable names. I find it pleasant to be able to type OutPacket.Msg.P<tab>.F<tab> rather than OutPacket.Msg.Packet.FromUser. I could have typed M<tab> in there too, but whether I can type <tab> faster than sg is pretty debatable so I don't bother with the short cut.
In vi (or vim, rather, but I can't help calling it vi), it's even easier. ^P cycles through hits. It's a big time saver, but of course you don't /have/ to use it.
> If the selection involving mousing, it's even worse. > Taking ones hands off the home row is expensive for a touch typist.
Agreed. Fortunately, the selection more normally involves <tab>, which isn't too bad.
<snipped a whole bunch of stuff where I agree with Gene>
> A language-specific editor means that I may have to work with a > number of editors rather than just one that I prefer.
But a well-written one will let you select the language you want it to be specific about. For example, Brief has support for a whole bunch of languages, and it looks at the file extension to distinguish which language filter to use. -- Richard Heathfield : bin...@eton.powernet.co.uk "Usenet is a strange place." - Dennis M Ritchie, 29 July 1999. C FAQ: http://www.eskimo.com/~scs/C-faq/top.html K&R answers, C books, etc: http://users.powernet.co.uk/eton [Enough about MS Word, please. -John]
ge...@mail.ocis.net (Gene Wirchenko) writes: > I can do it myself. ... > It depends on the language, but I expect theoretical limitations > bite often. Compiler preprocessors probably raise much havoc with > such editors.
Good point -- C is hard to support in a system like this, in the fully general case. Then again, I'm sure there is a large subset of preprocessor usage that *could* be supported, if only people bothered.
> >Now, the editor can have a compiled version of the program at all > >times. If you want to change the name of a function, you can select > >that function and then bring up a menu -- the editor will know that > >(a) you have selected a function name, and (b) what functions are in > >scope at that point. Squeak's end-user programming tool is a current > >system that works like this, but it's not practical for general > >programming. (Though there is talk....)
> Will it change all calls, including in other programs? No? > Then, I had better keep track of things. Will it adjust comments to > line them up if the new name is a different length than the old name? > How do I specify this? Whichever way it works, it's unlikely to do it > my way exactly.
Yes, that's the idea. Of course, your codebase must be visible to the editor and the editor must understand the language, both of which seem unappealing to you.
Regarding code layout, it is only a small step forward to have it point you to all the changes it made. In fact, a tool is likely to have "show references to this function" well before it has "change the name of this function".
On 8 Nov 2001 23:21:44 -0500, Lex Spoon <l...@cc.gatech.edu> wrote:
>ge...@mail.ocis.net (Gene Wirchenko) writes: >> I can do it myself. ... >> It depends on the language, but I expect theoretical limitations >> bite often. Compiler preprocessors probably raise much havoc with >> such editors.
> Good point -- C is hard to support in a system like this, in the > fully general case. Then again, I'm sure there is a large subset of > preprocessor usage that *could* be supported, if only people > bothered.
Lisp systems are pretty good about tracking macros and macro expansions, so I don't think that it's an insurmountable problem. About the only major annoyance is that teaching Emacs to indent new macro-created syntax is a PITA. The <bigwig> language has an interesting solution to this gripe: they specify indentation hints (via special comments) as part of the macro definition, so that editors and pretty-printers can correctly format macros.
Neel [C macros are hard because you can (and people do) mess around with statement and expression bracketing. In one infamous case, Steve Bourne originally wrote the Bourne shell with a bunch of macros to make C look just like Algol68, with definitions like THEN mapping to ") {" which is perfectly valid but kind of hard to teach to a syntax editor. -John]
> The once very popular THINK Pascal compiler for the Macintosh also > worked in that fashion. The program text was parsed until the first > syntax error, after which text was rendered in outline font. > ....snip .... > Having spent my teenage years hacking in COMPAS (which later became > Turbo) Pascal, I appreciate the combination of a fast compiler with > the simplicity of always only having to deal with the first occuring > syntax error.
Well, I hated Turbo Pascal for exactly the same reason. I found correcting only one CT error at a time took far longer that doing them in larger batches. Also, I sometimes prefer to choose an order of work other than source file order. It was especially tedious when porting existing code.
The total compile time to get a clean compile was O(N**2), because every time you fixed one problem, it started over at the beginning, recompiling everything it had done the previous time, before getting to the new part between the last and the next error. It also took more keystrokes than just moving to the next error. Of course, that was on a very slow machine, by today's standards.
My view was that the claims that this was a wonderful feature were just rationalization for compiler writers' not wanting to do the work of error recovery. I do know that's a lot of work and it never performs nearly as well as one would like, but I still prefer it. -- Rodney M. Bates
> > Having spent my teenage years hacking in COMPAS (which later became > > Turbo) Pascal, I appreciate the combination of a fast compiler with > > the simplicity of always only having to deal with the first occuring > > syntax error.
> Well, I hated Turbo Pascal for exactly the same reason. I found > correcting only one CT error at a time took far longer that doing them > in larger batches. Also, I sometimes prefer to choose an order of > work other than source file order. It was especially tedious when > porting existing code.
I didn't like this myself, and I would have preferred an error-recovering compiler.
But that's about the only point that I agree with:
> The total compile time to get a clean compile was O(N**2), because > every time you fixed one problem, it started over at the beginning, > recompiling everything it had done the previous time, before getting > to the new part between the last and the next error.
It was (very little time)^2, which was fast enough. Even on machines that were considered slow in the times of Turbo Pascal. Of course, this worked only because the compiler was smart enough to recompile only those modules that needed recompilation. Early versions of Turbo Pascal that did not have modules might have suffered from this N^2 problem.
-> Conclusion for compiler and language designers: if you don't have error recovery, make sure that you have separately compilable modules, and make sure that the compilation is guided by 'make' or a similar dependency analysis algorithm.
> It also took more keystrokes than just moving to > the next error.
Not really. Recompilation was never more than a function key away. I remember that starting recompilation was an entirely subconscious and automatic process, essentially the same as jumping to the next error. Of course, if you insist on going through the menu for a recompilation, then it's many more keystrokes. But jumping to the next error can be just as many or few keystrokes as starting recompilation; that's just a question of appropriate key bindings, totally unrelated to the question whether the next error is located by recompilation or by consulting an error list.
> Of course, that was on a very slow machine, by today's standards.
These machines were really fast enough. Compilation never disrupted my flow of thought, and that means it was fast enough for me, even though I didn't have the fastest available hardware.
> My view was that the claims that this was a wonderful feature were > just rationalization for compiler writers' not wanting to do the > work of error recovery.
Agreed. But it would have been really stupid marketing if they had emphasized that it's a limitation of the product.
I have been involved in some compiler work, and the main issue isn't whether the compiler manufacturer does or does not do some work - you can be pretty sure that the available manpower is dedicated to improving the product.
The real question is where this manpower is directed. If introducing error recovery would have delayed the introduction of modules by half a year, I would have preferred modules. Similar for the object stuff (which wasn't great but very, very workable).
In a similar vein, I didn't like the generated object code, but I preferred a rather bug-free, surprise-free, and quickly released compiler to the highly optimizing but late, unstable, and unreliable compilers that were available elsewhere.
> I do know that's a lot of work and it never performs nearly as well > as one would like, but I still prefer it.
|
