Sunday, February 15, 2009

The Future of the Delphi Compiler

The Future of the Delphi Compiler

By: Nick Hodges

Abstract: This article provides an overview on the status and future of the Delphi compiler.

Introduction

This article is intended to give all you Delphi users some insights into what we are working on and considering for the Delphi compiler. We here at the Embarcadero World-wide Headquarters know all too well how you all love your Delphi compiler, and since there are some cool language and compiler technologies brewing in the R&D labs, and we wanted to make sure you were up to date on the future of your favorite compiler.

Some History

But first, let me give you a little history about the compiler. Most of this is probably not news to you all, but it is important to remember it. First, the Delphi compiler has been around for quite a while. It has proven itself time and again as a fast, modern tool, and it has provided you folks with lots of cool and modern features like generics, anonymous methods, inlining, and more. However, because of its rich heritage, there actually is still some code in it from the old Turbo Pascal days, and those days were, well, a long time ago in a galaxy far, far away. There are still language features and code constructs hanging around that many of you probably didn’t even know existed. For instance, how many of you know that you can still create comments with (*...*)? Or how many of you know that this still compiles:

var
WeirdLookingArray: array(.1..10.) of string;

Come on, be honest. Not too many of you knew that.

At any rate, the compiler maintains these constructs, and many others, for backwards compatibility. Backwards compatibility is really, really important to you and it is really, really important to us. Let me repeat that: Backwards compatibility is really, really important to you and it is really, really important to us. (Hopefully the italics help emphasize the point. Okay, now we can move on.) However, many of these features are not used by a large portion of you anymore. Nevertheless, we still maintain them, test them, and certify them for each release of the compiler. That’s a lot of work. But since you all are probably not using all the stuff that is in there. (Come on, don’t even try to tell me you knew about that “parentheses and periods” thing....), it ends up being work that most of you don’t care about. Supporting less used language features costs time that we could spend working on new capabilities.

The above examples are just that, examples. But many of these code semantics are features that actually can prevent us from making changes and modernizing the language. Let me be clear before I go any further: I’m not going to be talking about eliminating features; I’m merely talking about an issue that the language currently has. We have no plans to remove any language features or break your code. Again: we have no plans to do that. We do have some interesting new plans, though.

Therefore....

Therefore, we’ve been thinking long and hard about this. Specifically: How can we move our compiler forward to enable it to keep doing all the things that modern, powerful, compilers do? How do we move the compiler forward adding cool new stuff while at the same time preserving your existing code?

Well, the answer to that question is neither short nor easy. It also isn’t something that we take lightly. We could just rewrite the whole compiler front end to create a new Delphi language that does what we want and which then abandons a lot of the code out there that you all have. But that would be bad. Really bad. We definitely don’t want to do that. Let me repeat that: we have no plans, no intentions, and no reason to make your existing code no longer work.

What we do want to do, though, is to be able to continue to move Delphi forward in order to keep it at the cutting edge of native code development (and shoot, general development, for that matter). But sometimes, moving forward is hard because of things that are hanging around from the past. Delphi is a very type-safe language, but not completely type-safe. You can still do things like call GetMem or create an unbounded array (array[0..0] of integer) and do all kinds of crazy things with raw memory using such a reference. That’s a powerful thing to be able to do, but it also means that you can be a lot less descriptive about what you are doing, and so the compiler lets you go on your merry way. In many cases, this may be exactly what you want, but this is a primary example of opening things up so that you can “shoot yourself in the foot”. Delphi lets you ride without a seatbelt, but the fact that you can do that means that there are things Delphi itself cannot do. Wouldn’t it be nice if you could still drive your cool car, but with a nice three-point harness and a full set of airbags. So what’s a compiler to do, then?

The Compiler Front End

Compilers generally have what is commonly called a “front end”. The front end take code, parses it, and creates a tree structure that represents the semantic intent of the program. It’s the part that understands the language syntax and ultimately defines what the language is. In Delphi, the front end knows how to read units, for loops, begin...end pairs, variable declarations, generic classes, all that “Delphi Stuff”. It is also the front end that creates the DCU files that currently are compiler-specific. It’s the part, too, that deals with all the historical syntax above. So the question becomes: How to modify the Delphi front end to enable new and modern language features while still ensuring that your valuable code works?

What if we created a new compiler front end that gives you the choice between a new, non-legacy syntax and the older way of coding? (Let me be clear again: we are not talking about removing language features. If you are firing up the comments to ask “So, what code constructs are you going to deprecate?,” I answer that now by saying “None. All you code will still work.” We should be pretty clear on that by now, right?) Or better yet, how about a new way of viewing existing code and “new” code so that they can work together, allowing you to put new code in new modules while keeping your old code working as it always has? Existing code would all still be able to be injected into the new way of doing things, but if you wanted to use the “new and improved” stuff, you’d have to do it in the new type of code module? And how about if, while we are at it, we end up creating a DCU (or something akin to what DCUs are today) format that aren’t compiler version specific. How does all that sound?

That sounds pretty cool to me, but of course, doing something like that isn’t trivial, and would require a serious amount of time and effort. But something like that would be worth it, no?

But the front end is only half the battle. There is also the compiler’s back end.

The Compiler Back End

For you non-compiler mavens, the “back end” end of a compiler is the part that reads what the front end produces and actually writes out the machine code to produce usable binaries or something that is useful to some type of runtime somewhere. The back ends of our Delphi and C++ compilers currently produce 32-bit binaries for Windows. But of course, back ends don’t have to be limited to doing 32-bits and Windows, now, do they.

One of the new things we want the back end to do is produce 64-bit binaries. And so producing 64-bit binaries will require a new back end. We could just take our existing 32-bit back end and hack around it to be a 64-bit back end and toss that over the fence to you guys. But that wouldn’t be the right way to do it, especially considering that our C++ and Delphi compilers currently have different back ends. Why do the work twice when you can, well, do it once? And of course, a 64-bit project is far more than the compiler. We want to make sure that the debugger, the editor, the file system, refactoring, modeling, documentation – everything – is done right and totally ready to be able to let you develop 64-bit applications with no limitations. That will take some time.

Instead, the right way to do it would be to write a new back end. And if you are going to write a new back end, the right thing to do would be to write a single back end that can be used for both Delphi and C++Builder. And if you are going to do that, you’d want it to be a back end that is architected in such a way so that it could be a bit more flexible with respect to what architecture it actually targets. Interesting thought, eh?

So it makes sense to do a whole new, unified back end that will work for both Delphi and C++Builder. But doing that takes time. More time than we’d like, frankly, but these things cannot be rushed. As a result, doing it right, making it work the way it should, and creating a proper compiler architecture to benefit both Delphi and C++Builder will be a non-trivial endeavor. Doing it right is important to us, and I suspect it is important to you as well.

Interestingly, the time needed to create this new back end and build a 64-bit product creates a window of opportunity for us to do the new Delphi front end. So, we’ve made the decision to take the time to do all the backend work needed, allowing us to do the interesting front end stuff as well. We aren’t going to hack something together and throw it over the fence. We want to create a new compiler that will be really cool. We assume you do, too.

The Delphi compiler is at an inflection point. And it seems like this inflection point affords us (and, ultimately, you) the opportunity to make that inflection point really mean something.

Bottom Line

What, then, is the bottom line here? Well, to get right to the point (finally!) – with some major compiler work being done on the new backend and the 64-bit compiler, this seems like a good time to “make a move” and transition to this new compiler architecture. We expect the 64-bit version of Delphi to be ready in mid-2010. Because the compiler is the first step in a complete 64-bit product, we are planning to release a preview of the 64-bit compiler in mid-2009 so that those of you who are extremely eager to see a 64-bit Delphi can get started early.

We know this is a change, but we believe – and we hope you agree – that it is worth taking the time to get it right and to make the very cool changes that will keep Delphi on the cutting edge language technology. Once we take the time to build the new compiler, there should be a lot of cool and new stuff we can add from there. And you guys love cool and new stuff, don’t you?

Conclusion

Delphi has to move forward. C++Builder has to move forward. Our compiler has to move forward. To move forward, the compilers have to embrace new sets of features that modern developers demand. Both languages have to provide the solutions that you are looking for now and in the future. In order to produce these new features we are going to have to, as the saying goes, break a few eggs to make an omelet.

But you guys have about ten bazillion lines of code out there that you want to continue to work. We totally get that. So, we are working to create a “new Delphi” and a new compiler architecture, to keep your existing code working, to emit 64-bit binaries using both Delphi and C++Builder, and maybe a few other kind of binaries while we are at it. And it all has to be done right so that it all works for you.

And you know what? That is exactly what we plan to do.

What is Delphi 2009 ?

Delphi 2009 The fastest way to build native Windows applications
  • Make your team more productive with Delphi’s time saving IDE and tools
  • Take advantage of drag-and-drop visual development and a powerful component library to speed development
  • Easily connect with all your data across multiple sources and tiers
  • Get power and speed with the powerful Delphi language, a fast compiler and full Windows API access
  • Visualize your database structures with database design and modeling capabilities in the new Delphi Architect edition

New Delphi 2009 Opens a World of Opportunity

Delphi 2009 helps you meet the challenges of the rapidly expanding global market, and expand the reach of your applications to more users. But that’s just the beginning.
  • Give your user interfaces a fresh, updated look with new and enhanced VCL controls.
  • Build applications that handle global data with ease using new Unicode support throughout the IDE and VCL
  • Move your code into the future with the addition of generics and anonymous methods to the Delphi language
  • Build powerful, efficient and secure database applications by separating data and business logic with new DataSnap
  • Get new insight into your database structures with data modeling and visualization in the all new Delphi Architect edition
See more of What's new in Delphi 2009 and How to Upgrade

Delphi is your solution for rapidly building high performance Windows applications

Delphi® is the premier development environment for software developers and database application developers who need to rapidly deliver high performance and easy to maintain software applications. Use the powerful Delphi IDE with its visual design surface, extensive component universe and powerful heterogeneous database framework to visually design and deliver applications in a fraction of the time. The powerful Delphi language and compiler deliver high performance and access to all the power and speed of native Windows development. Delphi also provides a comprehensive set of editing, refactoring, and debugging tools in a complete solution focused on making development faster and easier.

Delphi 2009 Editions

Delphi 2009 Professional
Delphi 2009 Professional is the high performance Windows visual IDE and framework designed for building packaged, desktop, and workstation applications quickly and easily. Delphi 2009 Professional combines the productivity of visual development, an extensive component universe, powerful object oriented programming, an ultra-fast optimizing native code compiler, complete Windows API support, and flexible local database connectivity. Delphi Professional features include:
  • Local database connectivity to InterBase®, Blackfish™ SQL and MySQL
  • Blackfish SQL deployment on systems with 1 user, 512MB database size
  • VCL for the Web with limit of 5 connections
Broad support for Delphi 2009 from tool and component vendors
Video: What's new in the Delphi 2009 IDE
Video: What's new in the VCL in Delphi and C++Builder 2009

Delphi 2009 Enterprise
Delphi Enterprise edition includes everything in the Professional edition, plus the following for building client/server and multi-tier database and web applications:

  • Database server connectivity to InterBase, Blackfish SQL, MySQL, Microsoft SQL Server, Oracle, DB2, Informix and Sybase
  • DataSnap multi-tier database application development
  • Blackfish SQL deployment on systems with 5 users, 2GB database size
  • VCL for the Web with no connection limit
  • Additional UML modeling capabilities

Delphi 2009 Architect
Delphi Architect edition includes everything in the Enterprise edition, plus powerful database modeling and design capabilities:

  • Reverse-engineer, analyze, and optimize databases
  • Create logical and physical models based on information extracted from databases and script files
  • Work with highly readable and navigable diagrams
  • Automatically generate database code from models for forward engineering
  • Advanced compare and merge with bidirectional comparisons and merges of model and database structures

For more detailed differences between the editions, see the Datasheet or Feature Matrix

What's New in Delphi 2009 ?

Delphi 2009 What's New

Build powerful, efficient and secure database applications by separating data and business logic with new DataSnap

Managing data is the heart of any business application. Delphi 2009 provides DataSnap, a complete set of tools for building scalable, robust multi-tier applications that manage data safely, securely, and effectively. In Delphi 2009, DataSnap has been overhauled to provide a powerful, flexible, and COM-free solution that enables you to write Server Methods (i.e. Stored Procedures on the middle-tier server) in Delphi. DataSnap now can create ultra-thin clients that access connections and datasets directly from your applications servers as well as call Server Methods just like they were part of the client code – all the while supporting existing IAppServer-based remote datamodules.

Take advantage of new Delphi language enhancements including generics and anonymous methods

Developers demand ever more powerful programming languages, and Delphi 2009 raises the bar by including support for generics and anonymous methods. Utilize generics to create code that will work for any data type. Anonymous methods allow you to inject code blocks where ever they are needed, either inline or as parameters to other code blocks. These two features and other enhancements keep Delphi 2009 at the forefront of native language technology.

Create great looking user interfaces with new VCL components, Ribbon controls, PNG Support, enhancements to existing components, and continued support for the Vista interface

Delphi has always made building vibrant user interfaces as easy as a few mouse clicks, and Delphi 2009 continues this trend. By adding support for the Microsoft Office UI features, or Ribbon controls, Delphi 2009 enables developers to build modern, powerful user interfaces for desktop applications. The VCL continues to advance as well, with the addition of new components and enhancements to existing components that ensure that VCL developers can take advantage of the latest features of the Windows platform.

Expand your business to global markets - Internationalize your applications with new Unicode support throughout the Delphi environment and localization tools that make it easier to translate to multiple languages

In Delphi 2009, the entire development environment -- including the IDE, the Code Editor, the RTL, and the VCL framework -- is fully Unicode-enabled, allowing your applications to run on any Windows locale and to accept virtually any form of Unicode data. Business is being done today with Unicode data. Applications, no matter where they are run, need to be able to accept, process, and display Unicode information. Delphi 2009 makes that possible. Further, if you need to localize your applications to other spoken languages, Delphi 2009 provides a powerful set of translation tools to allow you to target emerging markets with your products.

Work more efficiently with IDE enhancements like the all new Class Browser, robust build configurations, and improved resource management

Developer productivity has been an obsession for the Delphi team, and Delphi 2009 continues this fine tradition. New productivity features include a customizable Class Browser that provides insight into class structures and hierarchies, a new Resource Manager that manages the addition of Windows resources to an application, an enhanced Build Configuration management system that allows for easy selection of build configurations, and many usability improvements to the Project Manager. The Delphi 2009 IDE sets a new standard for development speed and productivity.

Greater visibility into your COM and ActiveX source code gives you greater flexibility and complete control

Delphi 2009 has completely reworked the support for the key Windows technologies of COM and ActiveX. Now, Delphi developers have a transparent, fully source code control friendly solution. Developers can create COM and ActiveX objects, automation objects, and ActiveForms, as well as wrap existing VCL components as ActiveX controls. Delphi 2009 is the complete COM and ActiveX development solution.

Build AJAX enabled web apps with VCL for the Web. Create great looking Web 2.0 applications in the same way you build Delphi applications

Web development can be a difficult process of melding together numerous technologies and skill sets. Delphi 2009 provides VCL for the Web, a framework that allows Delphi developers to leverage their existing skills to build robust, full-featured, AJAX-enabled web applications – including support for Microsoft Silverlight 2.0 -- without having to master the variety of web-based technologies. VCL for the Web includes a large collection of familiar components that work exactly like their Windows counterparts. Drag and drop components, hook events, and write Delphi code to produce browserbased applications. Getting an application to the web has never been simpler.

Welcome to Delphi... Delphi... Delphi

I have been a Delphi Developer since Delphi 3 when I finally decided on Delphi in 1996 as my programming language of choice for the Windows 32 environment. So what have I created with Delphi ?

Would you believe that I had single-handedly created a full ERP2 system comprising ERP+CRM where ERP=Sales Distribution+MRP+ Procurement Management+Planning & Production +Finacial Management + Human Resources Management System.

Since 15th February 2009, we have visitors from more than 60 countries including Malaysia, United States, Brazil, Italy, Australia, India, Turkey, Russian Federation, Spain, Indonesia, Hungary, South Africa, Germany, Mexico, Argentina, Singapore, Saudi Arabia, Colombia, Czech Republic, Canada, France, Croatia,Thailand, Bulgaria, Slovenia, Hong Kong, Poland, Sri Lanka, Chile, Japan, Austria, Ukraine, Azerbaijan, Ireland, Tunisia, Greece, Taiwan, Egypt, Bolivia, Paraguay, Iran, Islamic Republic , Morocco, Angola, Belgium, Portugal, Norway, Venezuela, United Arab Emirates, Algeria, Korea, Republic Of, Slovakia, Georgia, Lebanon, Macedonia, Sweden, Philippines, Vietnam, Dominican Republic