Linq to Sql: the 'what' and 'why' of it

The following comments concern 'what' is Linq (to Sql)/EF and the 'why'
of it, what motivated MS to develop it. What does MS mean by 'beyond 
'relational'? I also explore in what ways Linq, sql and the relational
model are related to each other. How these technologies are connected
to each other is quite a fascinating picture:)
My following 5 comments orginally appeared on www.sqlblog.com
in the post:

'Beyond Relational ???' 10/29/07
by Paul Nielsen 
http://tinyurl.com/686z6h

Comment #1

There is the association of relational to mathemetics (set theory). So
people criticize sql based on this point of view. Sql allows duplicates
rows, doesn't require a table to have a key, dependencies based on
ordinal position, is a poorly designed language etc. etc. These things
really are critical but the real problem is the prevailing idea that
relational is just a question of mathemetics. If it's just mathemetics
then allowing duplicate rows is perceived as 'close enough'. All the
objections from the set theory point view are not perceived as
compelling enough to really question the validity of sql. IMO the real
holes of sql have nothing to do with mathemetics. Rather it's the
foundation, the computer science if you will, that set theory and
relational algebra are embedded in. This point of view is unfortunately
not prevalent in IT. What the hell do I mean by the computer science of
the relational model? Well first, the set theory that relational depends
on is not some special kind of set theory. There is only one set theory.
In the same way there is only one computer science, there is no special
kind of computer science. But sql has invented such a special computer
science and this is the biggest flaw. What am I talking about?:)
Consider this, here is a table variable:

DECLARE @MyTableVar table(
   EmpID int NOT NULL primary key,
   OldVacationHours int,
   NewVacationHours int,
   ModifiedDate datetime);

Here is a server table:

create MyTable
      EmpID int NOT NULL primary,
      OldVacationHours int,
      NewVacationHours int,
      ModifiedDate datetime);
 
Here's the key question. If @MyTableVar really is a variable then what
is MyTable? In other words, @MyTableVar is to variable as MyTable is to
?????. If MyTable is persisted in the database what is it persisted as?
What computer science term describes it? Well whatever the hell it is (a
constant?) it certainly isn't a variable. And if it isn't a variable
then end of ballgame, end of relational model. And what of @MyTableVar?
Bol says 'A table variable behaves like a local variable.' and at the
same time says 'Assignment operation between table variables is not
supported.'. When is a door not a door?..when it's ajar:) Who the hell
ever heard of a variable that doesn't support assignment? Who ever heard
of a variable that doesn't support comparison? No one. Whatever
@MyTableVar really is it sure as hell ain't a variable. In a relational
db I should be able to assign the table @MyTableVar, all its rows, to
MyTable: 
      
MyTable=@MyTableVar

And I should be able to compare them.

if MyTable=@MyTableVar
 then print 'All rows in MyTable are in @MyTableVar and all rows in   
  @MyTableVar are in MyTable'
   else print 'Nope they're not equal'      
   
A relational db demands a table be a variable just like an integer
variable. Sql simply does not support basic computer science for tables.
Whatever a table is in sql it doesn't have a 'type' because computer
science is computer science and a variable must be typed. The only way
sql can recognize a table is by its name, not its type. This is why sql
doesn't support relational division and why dynamic sql must be used so
much. A table as a variable is a completely different animal than a
table in sql. This is why the expressive power of a relational db is
orders of magnitude greater than an sql db. Sql views and constraints
are redefined relationally. The 'types' in Dates work:   
Databases, Types and the Relational Model, The Third Manifesto' 2007
is about the central importance of variables of a particular type (a
table as one of many types) in a relational db. What a table as a
variable means and its significance. It is really a basic computer
science book. Ripping out the mathematics of relational theory (at least
trying to copy it), ie. the syntax to join, union tables, without the
computer science of relational has done all the damage. MS can't change
sql server because they are caught in an crazy computer science. The
difference in computer science between sql and net is the impedance
mismatch they're trying address. But I'm afraid they still don't get the
idea of a table as a variable. This is different than a table as a
class. The anonymous type in Linq is distinctly different
than a table type. So MS is doing the same thing IBM did forty years ago
with the sql System-R. The damage is the difference between a pickup
game in a playground and organized sports. You can draw up plays in the
dirt but they don't quite work the same as those run in a stadium. We're
still doing application development in the playground. Sometimes it
works, sometimes it doesn't but we're not basing it on the science of
any model. Sql is not a model of anything, it's an invention all its own. 
Close enough is only for horsehoes:) Maybe my blog will make more sense now:)

Comment #2

Wherever you see the word 'relational' just substitute 'fog'. As in fog of war:) 
> But when you have guys like Don Chamberlin (co-inventor of SQL and  
> co-designer of XQuery) on your staff, I guess you can afford to 
> boast your XML prowess.
He is revered in the sql world and reviled in the relational one. He was
a lead designer of System-R, the prototype of all sql database systems.
Those guys created a query language based on Codds description of basic
relational operators like projection, union and join. But they did NOT
implement the relational model Codd described. They just ripped out
these constructs without regard for their meaningfulness in the entire
relational model. So what you have today is nothing like the relational
model as it was envisioned. (IT successfully marginalizes the huge
difference and those that point it out:) And now comes 'beyond
relational'. What does this phrase really mean to MS? They are more than
willing to tell us. Aside from Jim Grays article/presentation, everyone
should read the articles on this site, the 'Comega language':
http://research.microsoft.com/Comega/
Especially this article:
'Unifying Tables, Objects and Documents'
http://tinyurl.com/yq7c4f

Here you'll find history repeating itself. MS, just like IBM did with
System-R, has extracted relational operators out of the relational model
and put them in an imperative object environment without any regard to
relational theory. The great irony is that the extensions that MS added
to net to realize projections of columns and xml within net is the
foundation for a true relational model! But the compiler generated
anonymous type of Linq while a variable is a different beast than the
explicit variable that is a table type in the relational model. It's the
relational variable that supports assignment and comparison as opposed
to the Linq variable that's no where near as smart:) But each supports a
'variable' which is a major step up from sql. Had MS any idea of the
friggin true relational model they would make a different kind of
history. Talk about dumbing down. Talk about of only academic interest.
Talk about relational fog (I should add that Alphora (Dataphor)
recognized the ability of the object imperative environment to support
the D relational language and implemented it. And it works:) 
Here is what Anders Hejlsberg, MS VS guru, and now the head of database
technology has to say about the disconnect:
InfoWorld
Interview of Microsoft Distinguished Engineer Anders Hejlsberg
'Microsoft's Hejlsberg touts .Net, C-Omega technologies'
June 10, 2005
http://www.infoworld.com/article/05/06/10/HNhejlsberg_1.html

"So what we're looking at is really trying to much more deeply integrate
the capabilities of query languages and data into the C# programming
language. And I don't specifically mean SQL, and I emphatically don't
mean just take SQL and slap it into C# and have SQL in there. But rather
try to understand what is it expressively that you can do in SQL and add
those same capabilities to C#."

Anders Hejlsberg is microsofts version of Don Chamberlin at IBM. So what
they have done is replace one flawed implementation of sql with another.
And this is how they achieve efficiency in application development. Now
that is unfriggin believable:) Well there's no free lunches. And I await
to be enlightened on just how this environment will replace the concept
of the logical relational model in solving business problems. I would
say the real meaning of beyond relational is sideways.

Comment #3

Consider the MS whitepaper:
'Microsoft SQL Server 2008 and Microsoft Data Platform Development'
http://www.microsoft.com/sql/techinfo/whitepapers/sql_2008_dp.mspx

Does anyone find it the least bit odd that an sql server technical article
is all about VS, LINQ and the entity framework? At the expense of the logical
relational model and the sql language.

What MS means by 'beyond relational' is 'forget relational already':)
Looking at sql server as if it was somehow an embodiment of relational
theory is every bit a form of dumbing down as some silly utterance by
some poor nitwit at MS. There never was and never will be any 'intent'
by MS to offer a 'relational' database. Sql servers only intent now is
to be responsive to its biggest customer, visual studio. And that team
is as knowledgeable in relational databases as the server team. Not. Why
does the community still view sql server thru an imaginary lense? Did
you ever hear of somewhat pregnant? If you open the dumbing down door be
prepared to greet all those who come thru:)

Comment #4

There is no longer a great divide, a debate, an impedance mismatch. MS 
has issued their own Emancipation Proclamation. And as a result they no
longer support the relation model as it is know to developers today.  
'A Call to Arms'
by Jim Gray, Microsoft
Mark Compton, Consultant
April 2005
http://www.acmqueue.org/modules.php?name=Content&pa=showpage&pid=293

This paper is an invitation to embrace a new model. It's just as much
'A Farewell to Arms', an emancipation from the relational model which 
they are leaving behind. What does sql server look like in this new model?

'Interview with David Campbell'
General Manager of Strategy, Infrastructure and Architecture of Microsoft SQL Server.
May 14, 2007
http://tinyurl.com/6maseb
Campbell says:
"I believe the next major advance in Information Technology will come 
from addressing the gap between people and information."

That gap is the relational logical model itself.

Campbell continues:
"The focus will move to the data itself rather than on the machinery
used to manipulate it. We'll be less concerned with the plumbing and
more concerned with data quality, data protection, and information
production."
"Most of the data services provided by SQL Server will be driven from
a common data model. Whether you're creating a report, building an
information cube, or integrating data from another system, you will be
able to start from a common model of the key data entities such as
'customer', 'order', or 'prospect'."
"Finally, fewer and fewer people will miss, (or remember), the 'open
databases' sp_configure option..."

The class replaces the table as the basic unit of work.  VS replaces
QA/SSMS as the interface for application development. There is no
concept of relational anything in this object world. Sql constructs are
independent of application development. The language of the relational
model is replaced with the language of entities. There is no concept of
a dba.
MS is no longer in the database wars as we know it. They are trading 3rd
place in that world for 1st place in another. And they now have the
freedom to talk about this new world. It just sounds silly to those who
have not left the old one.
Ironically some were hoping for a new sub-language to further AD. Perhaps
the lesson here is to be careful of what you wish for. I too was hoping
they'd enter a new world but not the one they have chosen.

Comment #5

> should we be concerned staying in the DB world long with the fear 
> that we become obsolete one day?
Although I'm not an expert I can understand where you're coming from. It
would be nice to get a clear and concise answer to where MS is going and
what you should do about it. But there is no Oracle when it comes to MS.
There is no one position paper, no one person that clearly lays out
their five year plan and what it means to you. The experts here have
enormous importance and influence in the db community. But they also
have an enormous investment. How far can they be reasonably expected to
go without putting themselves in an awkward position should they take a
position that is not currently in line with company thinking? In the end
it's a question of connecting the dots. You get a dot here a dot there.
You have to do your homework. Study what they say and write and study
what they offer. Sql server pros shouldn't neglect what's going on in VS
and it's impact. If you study the company and the various technologies
enough you should be able to draw your own picture. Think of it as the
MS X-files:)