June 10, 2020

Tale: ETL component

In this occasion we will see a situation that unexpectedly generated a great design, by purpose and by accident as well.

Problem
Here we've had a customer with a legacy system that stored all its data in a local data source, this is a very old system, written in Fortran, that handles huge amounts of data and it's rarely updated.

As we can already expect, this is not a simple system to integrate with external participants, and that was exactly what they needed. A partner had a website that was well done and wanted to integrated with the legacy system data, meaning that data could be fetch from that system and updated as well.

Proposed solution - draft 1

Our team was asked to provide a solution for this scenario, and here comes out handy the figure of an Architect, we were a young team, with very few years of experience and almost no insight on what to do next.

The Architect took the lead, he designed an ETL (Extract/Transform/Load) component that would act as a mediator between the systems, it would read data from the legacy system and synchronize it to the partner web system, and consequently it would take the data from the partner web system and synchronize it back to the legacy system.

The behavior was designed in this way because the legacy system couldn't be disrupted, a single misbehavior could take the system down, synchronous communication was out of the picture, everything needed to be asynchronous to ensure a stable integration.

Solution - state 1
Our team worked well and delivered the first ETL to production with some small issues, but worked well, the synchronization process between the Legacy and the External partner was performed every two hours and worked really well.

Problems in legacy

But after a few weeks in production the customer and owner of the Legacy system saw a problem, the amount of data changed by the External partner was huge compared to their normal changes rate, the Legacy system was getting slower because of this integration and data synchronization.

Their Legacy system had a great feature for this scenario, they've created a Replica environment that worked for read-only data (which amounted to 90% of the operations) and the original system where data could be written. The new schema solved the problem on their side, but the ETL needed to be updated, it needed to look now at 2 different data sources, with different behaviors.

Our team's Architect took the lead again, he said: "the ETL remains unchanged, we will fix this in the wire". His decision was to put queues in front of the ETL to handle the data traffic, and install agents in the Legacy systems and the External partner system, this agent would monitor the data and communicate with the queues with structured messages, asking for data (read) or performing a data modification (write).

The changes were done in two weeks, the performance remained the same, the ETL performed consistently without changing a line of code, the Legacy system's load was controlled and the External system worked really well and fast.

Final setup

Backwards Analysis

Thinking back, the project went really well, the development process was smooth, integration was simple and a lot of things that usually go wrong were avoided. Many of these benefits are related to some key decision made by the Architect for the beginning:

  • Functional programming as main development paradigm
  • Automated test covering critical cases
  • Periodical automated build
  • Asynchronous communication 
  • Ensured communication channel

June 4, 2020

About software development methodologies

A while ago I've sent a question to James Coplien, one of the loudest voices of software development processes, methodologies and design.
Hello mister Coplien.

After some meetings regarding working processes and working methodologies for software, we've reached a point on asking what are the alternatives to the most know approaches (waterfall and agile processes).

Our group consists of people actively working on software development, testers, students, and college teachers. So, from our limited experience, we couldn't list too many approaches; we ended up with a commitment of digging into the topic and share our findings in our next meetings.

That's why I would be super thankful if you could recommend some resources (articles/books) on software development processes/methodologies.
To my surprise, he replied back with a great explanation:

Hej, Juan,Thanks for writing.

First: just to pick nits, “methodology” is a branch of philosophy which is the study of method. Technically, things like waterfall and XP are methods and not methodologies. Alistair Cockburn, who has developed a theory of software engineering, playfully likes to call himself a “methodist” rather than a “methodologist.”

A method is therefore a pre-packaged set of practices and processes suitable to design and implementation. One of the great learnings of the 20th century is that method doesn’t work. The architect Christopher Alexander has been one of the leaders of the Design Movement since about 1968, and is very familiar with the notion of method. He writes:

No one will become a better designer by ... following any method blindly... if you try to understand the idea that you can create abstract patterns by studying the implication of limited systems of forces, and can create new forms by free combination of these patterns—and realize that this will only work if the patterns which you define deal with systems of forces whose internal interaction is very dense, and whose interaction with the other forces in the world is very weak—then, in the process of trying to create such diagrams or patterns for yourself, you will reach the central idea of which this book is all about. — Christopher Alexander, “Notes on Synthesis of Form,” 1974

So it is not a matter of choosing between alternative potential approaches, but rather of transcending the whole framework of reducing design to method. No more than Michelangelo followed any method in making the statue of David or of painting the roof of the Sistine Chapel will method help any technical design activity.

Even within the framework of agile, there are those that believe that Man can triumph over the complexity of nature and that they can prevail using some method. XP is a very old example. SaFE is a newer example. The things that work dismiss this claim and open the doors to self-organization and feedback. That is true agile. True agile is incompatible with the fundamental notions of method.

This is why I am an advocate of Scrum. Scrum is not a method but rather a framework. If you want to explore the method space I’d recommend exploring Crystal and the works of Alistair Cockburn. Scrum and Crystal aren’t the only ones. But to find anything effective for complex development you must abandon method. Method is good for simple and merely complicated work — not for complex work.

What you have read is aa OK start, but I think you are exploring noise and need to dig deeper into some of the underlying theory before looking at experience reports like Scrum from the Trenches. If you want to know Scrum in depth, then get the new “A Scrum Book” (https://pragprog.com/book/jcscrum/a-scrum-book). That will help you understand Scrum. But Scrum is not a method, and I think you need to come to grips with the scope within which method works. Maybe you even need to come to grips with the taxonomy of design within which method is historically a relatively small part. Therefore, I recommend getting these books as well:

Design after Modernism: Beyond the Object, by John Thackara (https://www.amazon.com/gp/offer-listing/0500234833/)


Notes on the Synthesis of Form, by Christopher Alexander, 1974 printing (https://www.amazon.com/Notes-Synthesis-Form-Harvard-Paperbacks/dp/0674627512/ref=sr_1_fkmrnull_1)

There are many more, but this is a start. From where you seem to be at, you have many years of work to do before coming to any conclusion.
At this point, Alistair Cockburn (participant on the Agile Manifesto) got involved, and provided a lengthy response:
hi, Juan, Cope, 
I rarely like to disagree with Cope, but he is wrong twice here, and putting words in my mouth, so I feel somewhat obliged to step in. I'm afraid this is a long discussion, but one in which I have invested considerable research. Nothing of what I am about to say is "off the cuff". 
First because it is easiest : I am a methodologist, not a methodist. The "-ologist" refers unambiguously to "person who studies --".?? I study methods and methodologies (explanation of the 'methodology' in long form below).?? I am not a methodist, and never refer to myself as such. A methodist is not a studier of, he is a person who follows the rules of the method, in particular, a very rigid form of Christianity. So if I am anything, it is not a methodist :).
For those who wish to be pure, that "-ology" means "study of", I then insist that I am a "technologist" because my field is "technology", the study of techniques. Until the users of the English language repair the word "technology" to mean, as they imply from the suffix "-ology", study of techniques, then I feel no obligation to honor the false appeal to ancient Greek etymology for "methodology". But more on that in a moment. I am a "technologist" (studying techniques") or a "methodologist" (studying methods and methodologies), but not a "methodist" (a particular, very formulaic Christian religion)

For the second, "Methodology", Cope makes the usual vague reference to a false etymology based on ancient greek. For my PhD dissertation, I had to take on this question, I researched and tracked this word over a period of years and using different dictionaries. 
It is best that I don't try to re-write that research, but rather, show you that section of my PhD dissertation. 


1.1 Clarification of Words
With the word methodology being used in different ways by various writers, and often interchanged with the word method, I must clarify my use of the words in order for the rest of the thesis to make sense. 

Methodology

My use of the word comes from two sources: the Merriam-Webster Dictionaries and common usage among large contract software houses.
?????????????? The (U.S.-based) Merriam-Webster dictionaries list as their first definition: "A series of related methods or techniques." 
?????????????? When I sat in a discussion of methodology with a group of senior consultants from Coopers & Lybrand, Ernst & Young, Andersen Consulting, and the IBM Consulting Group, one of them stood up and said (words to the effect of): "Let's get clear what we mean by methodology. It is the roles, teams, skills, processes, techniques, tools, and standards used by the project team." The others in the room nodded their heads, and the meeting proceeded. As one of the senior consultants later told me, "If it is what one or two people do while working on a task, it is technique. If it is what the team uses to coordinate their work, then it's methodology."
Those two meanings of the word are congruent and match the common use of the word. 
Many British and continental European speakers dislike the above definition, preferring to use the word method instead. They cite the (U.K.-based) Oxford English Dictionary, which in the early 1990s contained only one entry for methodology: "study of methods." This explains the cultural difference in usage. However, the 1995 Oxford English Reference Dictionary includes in the entry for methodology: "2. a body of methods used in a particular branch of activity." Both American and British dictionaries now agree on the subject.

Method

The Merriam-Webster dictionaries define method as a "systematic procedure"; that is, similar to a technique. This matches the way many European speakers and European research literature uses the word method. An example, from Mathiassen (1998), is the following:
Lesson 2: Methods are primarily frameworks for learning.
The intention behind Mathiassen's word method is quite different from my intention for the word methodology. Mathiassen writes method to connote what one or a few people are learning to use, usually as a technique. This learning eventually disappears into the background skill set of the practitioner. 
In contrast, I write methodology to connote the conventions people agree to across the team (including development techniques). While the techniques disappear into each person's skill set, the conventions in play do not disappear into skill sets. 
We shall need both words to complete the discussion. This thesis primarily addresses methodology rather than method.

Shifts in Meaning

As the research took place over a ten-year period, there were several minor shifts in the way the word methodology got used, as well as some shifts in the way the practitioner community used the word. I deliberately leave in place some of those shifts, because readers will come to this thesis with their own personal interpretations, and part of the thesis topic is dealing with those personal interpretations. 
The first research question, for example, asks: "Question 1: Do we need yet another software development methodology, or can we expect a convergence and reduction at some point in time?" In this question, I do not mean just, "Do we need yet another set of team conventions, or can we expect a convergence and reduction at some point in time?" Rather, I accept the fluid way in which people ask the question, which often includes, "Do we need yet another software development technique, or can we expect a convergence and reduction at some point in time?" The answer that I develop responds to both versions of the question. Research question #2 is handled in a similarly liberal way.
However, by the time we reach the third question, "Question 3: How does the methodology relate to the people on the project?" the meaning narrows. It is quite a different question to ask, "How does a techniquerelate to the people on the project?" That question has to do with introducing techniques into groups, as well as the relation between technique-in-theory and techniques-in-practice (the latter topic being well discussed in the Scandinavian literature, see Ehn (1988) and Mathiassen (1998)). In the context of this inquiry, it is relevant to use the narrower meaning, "How do the team conventions relate to the people on the project and other aspects of life on the project?"??

I found these replies and explanations extremely good, especially to someone trying to get more information about software development processes, so, I wanted to make them public because they might be useful for someone else.

Additionally, I've never expected a reply, it's great to know that knowledge can be shared and guidance can be provided despite the name, profession, location or expectations, thank you James and Alistair for your time.
 
My recommendations:

August 29, 2019

Bitbucket and Mercurial


It has been recently announced that Bitbucket will be leaving Mercurial support in 2020 [1], defaulting to Git from then on.

A lot of things come to mind when reading the announcement, mostly because Mercurial has been my first distributed revision control system or at least the first one I understood. It's fairly simple to use, concise and with a simple interface, but sadly has fallen in popularity.

Git has won in other fields where popularity is more important that correctness and simplicity. If I would go back in time and choose between Git and Mercurial, I would still choose Mercurial, it's still a great tool.


[1] https://bitbucket.org/blog/sunsetting-mercurial-support-in-bitbucket