Click here to monitor SSC

Tony Davis is an Editor with Red Gate Software, based in Cambridge (UK), specializing in databases, and especially SQL Server. He edits articles and writes editorials for both the Simple-talk.com and SQLServerCentral.com websites and newsletters, with a combined audience of over 1.5 million subscribers. You can sample his short-form writing at either his Simple-Talk.com blog or his SQLServerCentral.com author page. As the editor behind most of the SQL Server books published by Red Gate, he spends much of his time helping others express what they know about SQL Server. He is also the lead author of the book, SQL Server Transaction Log Management. In his spare time, he enjoys running, football, contemporary fiction and real ale.

Going with the Flow

Published 30 August 2013 2:27 pm

Every so often, a new programming model or framework emerges, promising to tackle the burgeoning complexity of the software development process. This week, the potential savior of the sanity of web developers everywhere is an “Arcane Coding Method from 1970s Banking Software“. The arcane method in question is Flow-based Programming (FBP) and, at its core, the piece aims to promote a new framework called NoFlo, an implementation of FBP for Node.js, which allows the programmer to construct their data flow logic by connecting components in a visual, network-style diagram.

To call FBP arcane is a bit unfair. FBP never went away. The ideas became part of the established conventions of multitasking event-driven asynchronous systems. However, it didn’t take over the mainstream from the more intuitive, brute-force method of problem solving via procedural languages.

Any attempt to simplify and visualize the software architecture and design process is laudable, but I doubt that FBP is, as the author suggests, a “revolution still waiting to happen”. It is certainly a good way of tackling a specific range of problems but is it a general solution? Fred Brooks set out most of the reasons it hasn’t, and probably won’t, gain mainstream support in his No Silver Bullets paper. Firstly, visual/graphical programming attacks only the accidental complexity that creeps into many projects, not the essential complexity of the problem. Can the visual approach really make the design process accessible to the input of non-programmers? As complexity increases, so the “slide towards an unreadable spider web of lines and hundreds of boxes” will deter even the most dogged.

Likewise, data flow programming itself doesn’t really attack the intrinsic complexity of the process being automated. It only changes the way we express the problem. However, it is attractive because it gives clarity to the nature of the process and the current state of each instance of it. It can control the flow of data between components. This helps with the tricky requirements of tracking state in parallel processing. In dataflow programming, we define data declaratively rather than writing procedural code and we need look no further than Excel to understand the potential power of this approach. Having defined the relationships between the various cells, tasks proceed with no need to control how and when certain data values update in response to other changes. Various libraries already allow us to incorporate elements of dataflow programming into our existing languages (for example, Underscore.js and Knockout.js brings declarative data binding to JavaScript).

Database developers have lived happily in a strange non-procedural, multithreaded alternative universe that can baffle the procedural programmer. Could it be that the rise in popularity of event-driven platforms like Node.js, and asynchronous I/O, might yet jolt procedural programming from its current dominance amongst application programmers? I’d welcome that. Would you?

9 Responses to “Going with the Flow”

  1. lyynx says:

    Nothing ever truly goes away.

    It’s funny, whenever I see someone mention procedural programming, I have to go and look up what that means. I then think to myself that all programming is procedural programming if you want to get nit picky. Code lives in a method or function. Code calls it from somewhere. OOP just puts a rule in place to say that these functions should only operate on their own data. Methods on a class or object are still functions you call. Programming in its purest form is a bunch of functions with some conditional statements. Its all just text. Event driven programming only changes how these functions are called or rather from where. You can’t jolt procedural programming because that all we have. Each thread has a block of code to run in order until it hits the end. What else is there?

  2. AngryArchitect says:

    Flow based programming looks very much like ETL.
    Certain enterprise grade ETL tools have the concept of continuous flows which are real-time ETL.
    As anyone who has done any significant SSIS work will tell you a simple looking dataflow can hide a huge degree of complexity. On the plus side it provides a medium/high level overview of the system.

    SSIS contains a “DataReaderDestination”. Shame about the name but it exposes the output from an SSIS dataflow in a way that allows it to be consumed as a source for a .NET DataReader.

  3. Keith Rowley says:

    Once again I would say this is a situation where you need to use the right tool for the job at hand, not just your favorite tool. So FBP might very well be the best tool for solving a lot of programming problems but I don’t think it will be for all problems.
    Just like SQL isn’t the right language to use for all programming sad as it makes me to say this.

  4. Lee Dise says:

    The Teacher in Ecclesiastes laments, “…what has been done will be done again; there is nothing new under the sun. Is there anything of which one can say, ‘Look! This is something new”? It was here already, long ago; it was here before our time.’”

    Nothing new? Nothing at all? Well, on the one hand, I’m pretty sure King Solomon never had to write T-SQL code; but on the other hand, there is still some truth here to be gleaned from the ancient wisdom — namely, that people don’t remember what is old and so it gets recycled as new.

    It even gets confusing for some of us oldsters who were there and at least trying to pay attention. What we once denigrated as “stovepipes” are marched forth today with pomp and circumstance as “data marts.” What was once derided as “unnormalized” is now cheerfully “denormalized”. Those who don’t remember the past may be condemned to repeat it, but seldom does that happen with diminishing displays of sanctimony.

    > “Arcane Coding Method from 1970s Banking Software“.

    Well, thank the Lord that at least 1970s military software is still resting safely in its sarcophagus. I remember when, if you handed an 80-column card to an officer, he thought it was a ticket to a parade.

    When I was starting my first programming job, I wanted to do something sexy, like applications. However, I found myself somewhat disappointingly slotted for the more pruriently-challenged database shop. My tech school instructor wrinkled his nose when he handed my assignment to me. “Database. Boring.”

    But it was ostensibly even worse than he thought: I wound up in the so-called “data moving” shop. We didn’t have canned database backup commands or BCP, but just many thousands of lines of vivid, homegrown, COBOLish ugliness that were used for populating databases and stripping them clean. Learned plenty about COBOL programming and network database navigation, but very little about either applications or the internals of the database itself.

    However, at some point it did finally hit me: “just moving data” is the essence of programming. Move that here, calculate this, reformat it, send the results way over there. Doing the job required a high-level basic understanding of what was going where. And doing the job right required a detailed low-level knowledge of data representation.

    So, since all programming revolves around moving data, why *not* conceptualize projects in terms of data flow?

    I’m sure that’s fine, as long as everyone understands that the details are best left to the folks who understand programming.

    But as for me, I’ve never been an enthusiastic top-down designer. I tend to start with the smaller pieces that I know will have to be built regardless of how things go. By the time I’m done with the little pieces, the remaining integration structures become obvious. (I suppose this means there are some tasks to which I’m ill-suited, but I’ve never wanted for tasks to do.)

    One thing that does actually seem new under the sun is that software is the closest man has come so far to reflecting the infinite creativity of God. The Book of Genesis has God simply speaking all of existence into being. We can now speak — or rather, painstakingly write — whole new virtual worlds into existence, provided you spot us an operating system and several layers of programming infrastructure.

    Admittedly, “Let there be a rollup by department of all invoices during the month of August!” doesn’t have quite the magisterial zing.

  5. Robert Young says:

    I knew it sounded familiar. Surfing through the Wiki, and such leads me to the forlorn conclusion that FBP will have the same fate as OOD/OOP. That is, action object (FORTRAN functions) passing around FORTRAN data “objects”.

    It also sounds a lot like Executable UML, which appears to be copyright/trademark, but has been implemented in various ways over the years. Just as COBOL was intended to promote “programming” by SMEs (before the term was invented).

    However, as I’ve mused often, to the irritation of many, code is just verbose data compares, so why not skip the middleman and do the “logic” in the data the way Codd intended? To the extent that FBP avoids the ActionObject/DataObject death spiral, it may lift the scales from the eyes of some.

  6. jpaulmorrison says:

    I must have run into the exactly same arguments so many times over the last 40 years. Not sure if Tony or any of the people leaving comments here have actually read my book – but, if they had, they would have realized that FBP is a paradigm shift. Its roots actually go back over 40 years – in fact, for the real old-timers among you – it has affinities with Unit Record (what’s that, you ask!). lyynx above does not seem to be able to break out of the von Neumann paradigm – but that is exactly the reason why conventional applications are so complex and difficult to maintain.

    We never claimed that FBP was good for absolutely any programming application, but some programs written using this paradigm have been in continuous production use for almost 40 years, while undergoing on-going maintenance to cope with hardware and software changes, and new application requirements – often being maintained by people who hired on decades after the original code was written. So that’s not a bad record!

    Finally, I totally agree with Lee Dise: “So, since all programming revolves around moving data, why *not* conceptualize projects in terms of data flow?” Couldn’t have said it better myself!

  7. Robert Young says:

    – but some programs written using this paradigm have been in continuous production use for almost 40 years

    Well, I’m not sure that deserves a medal, or not. I’ve had the sad experience of working with COBOL-ers patching applications that old (and older). The sad truth is that the CS community, and not just the academic branch, chose to invert the prime principle: data is forever, code is ephemeral. Since code doesn’t “wear out”, any sufficiently complex application will continue to accrete more code so long as the compiler is available.

    By tying data (VSAM, in mainframe terms) to its code, one ends up deep in the silo. Properly used, the RM (even in SQL implementations) puts data in a totally language/engine/hardware agnostic world.

    And for those who’ve not looked it up, Unit Record was/is IBM’s term for the 80-column keypunch card (1890), unless there is a proprietary definition of which I’m unaware. ( http://en.wikipedia.org/wiki/Unit_record_equipment )

    What Codd envisioned is far superior to any FOTM language syntax, even if the month stretches to decades. Bang on a nail long enough, and it submits.

    With modern hardware, Xeon/oodles of RAM/SSD (or even the au courant in-memory implementations), even the lowly SQL engines can run rings around code-centric approaches.

  8. GedByrne says:

    Hi Robert,

    But underneath those lowly SQL engines you find flow graphs based on the same principles as FBP.

    You only need to look at the execution plan.

    https://www.simple-talk.com/sql/performance/execution-plan-basics/

    Generative FBP programming, with flow graphs being generated from a language like SQL, is a whole area ripe for exploration.

    You could, for example, provide a set of components analogous to Darwen’s ALGEBRA [1]: AND, OR, NOT, REMOVE, RENAME, REMOVE, COMPOSE andTCLOSE. The data packets would simply require a suitable representation of the tuples.

    You could wire up a graph for any transformation you wanted, either by hand or automatically.

    For example: Converting Tutorial-D [2] into flow graphs would be trivial.

    If Date and Darwen are to be believe this would allow us to data as Codd intended.

    [1] http://www.dcs.warwick.ac.uk/~hugh/TTM/APPXA.pdf
    [2] http://en.wikipedia.org/wiki/D_(data_language_specification)#Tutorial_D

  9. Robert Young says:

    – Generative FBP programming, with flow graphs being generated from a language like SQL, is a whole area ripe for exploration.

    generating from the catalog has been around for years, although not widely adopted. change the schema, push a button, get a new client.

Leave a Reply