Art of Clean Code - Logging

 "In some ways, programming is like painting. You start with a blank canvas and certain basic raw materials. You use a combination of science, art, and craft to determine what to do with them." - Andrew Hunt

My first lesson for importance of logging was in 2001-02, from my manager and coach Puneet Agarwal. We were building a framework for Data Bound Swing Components which can be used to design any application using a custom Net Beans kind of IDE, to enable rapid application development.

Design was to extend Swing Components, attach metadata for data source/s in context and pass it across component hierarchy using component life cycle events like add/removeNotify, and extend paint/print functions to render the components using data. This seems like a simple design flow, but had complex implementation due to hierarchical nature of UI elements. Data needs to be passed (and cleaned too) from parent to children and vice versa. Which means, any break in this recursive flow or any small miss (ex: even mishandling of data loading in repeated rendering events trigger by Swing) could cause issue in data loading, in rendering, memory management, and on focus kind of user facing features also.

The only savior to debug in such complex recursive programming environment was 'art of logging'. This is when my manager taught me the importance of logs, including right level of logs with right level of information and easy to understand format. I still remember that lesson and follow it that,

Logging is an Art

Fast forward today, we have much advanced features in IDE, ex: breakpoints, debug mode etc. These are making debugging much easier with live runtime data and control on start, restart and stop of events. Having said that, when there is issue in server or distributed environments, logs are very useful to understand the issue in one go by looking at logged data. Even in local dev environment also, it is much easier to see the whole system state by reading the properly crafted logs in one go.

Hence an effective logging is very important for clean code, a code which can be maintained easily for long, a code which can run in production without causing distress in nights for developers / ops / users and is easy to debug and fix quickly if there is any issue.

Let's talk about few of the important basics to make logging effective, and hence to write a clean effective code.

Have a common understanding of 'Format of Logs'

Yes, it is important. Refer to previous blog here about clean code, where we discussed about having standard formatting for codebase in a team, as it makes reading code easy. Same principle applies to logs also. If there are different kinds of log formatting, and patterns, it takes time to understand, and makes sense of it.

We should be able to read logs like a Story . A story narrating the state of system in simple words

Art of Clean Code

 If someone ask me who is my Guru for Java, I would say 'Sun Microsystem' and 'Apache Foundation'.

I am from a generation, where most of the programming learning was by reading Java source code. Internet was not a big store of many rich blogs or forums, and hence reading through code was the best resource. Moreover, kind of components we built, were not possible without understanding the behavior of system we are extending. Hence reading through code was almost mandatory for building right extension, and the products.

One notable thing, I still remember and admire about Java and Apache kind of products source code is, clarity of code constructs and rich documentation, code which speaks its intention very clearly. That made the code reading experience enjoyable and easy like a story book. I remember reading tons of code of Swing, Concurrent library, Collections and Apache Axis etc like a novel. Much of the credit of reading such large codebase was to that effective coding and documentation style, which developers followed to write such code.

Fast forward today, we are in era where doing more by writing lesser code is sought after style. With availability of functional language, many abstract language constructs and lambada expression kind of support, it makes all the sense and makes code less bulky too.

Yesterday or today, then or now, basic principles of writing clean code are same and equally required. Ultimately it is all about art of writing readable code, clean code, a code which speaks of its intention clearly, where it is hard for bugs to hide, a code which can tell about any dysfunction loudly.

What is clean code:

Any fool can write code that a computer can understand. Good programmers write code that humans can understand - Martin Fowler

• A code which can be read easily by anyone and everyone other than its original author.
• A code which speaks of its intention clearly, either with write naming convention or with documentation as relevant
• A code where it is hard for bugs to hide
• A code which can tell about any dysfunction in system loudly itself
• Let us talk in brief about some of key enabler principles for clean code.

Meaningful Names - Names which reveals intention

If I need to choose one most important thing for clean code, I will pick this. That is its importance for code readability and maintainability.

Picking right name, which can reveal its intention of existence is very important. It takes time to pick right name, however, that effort and time are worth it as it could save hours of efforts later to understand the code or any issue.

Story of Story Points

"Let's use Story Points

Why?

We are following agile now

Yayy.. story points!

How many hours these epics, stories will take?

But we are using story points..

Yeah, but we still need to plan. Hours are better for that. 1 SP - 5 hours. ok?"

Heard this story of story points many times.

-----

It is important to understand why do we need Story Points? Is it mandatory for being agile? Are hours prohibited?

The key motive for story points or hours is that we need to estimate the work, so we can tell what can be achieved in given time. Hence, we try to estimate. Hours are used since long because time is easy measure for us, and we can quantify easily, 'accurately'.

However, it become challenging when we found that even after best try, it is hard to estimate the time. It is more and more true in non-mechanical activities, where outcome depends on many intangible factors like skills, information available, mood, motivation, environment and so on.

If estimates can not be done accurately, then why to estimate. Let us just keep working and ship whatever is done. And this is also one of the philosophy and may work in specific setup. However, most still need a roadmap with understanding of possibilities which feed into overall plan of business.

'Possibility' is the key word here. We gradually understood that accurate estimate is not semantically and practically right. However, we can try to have best guess based on available information, which reflects the possibilities. As long as, we accept this fact, that we are talking about possibilities, hours or story points or tshirt size or anything else works.

If we consider this, ultimately, it boils down to 'common understanding'. Common understanding that when we are estimating, we are talking about possibility. If we all are good with this, any unit can be used for estimation.

Building up on this understanding, let us talk about common understanding. How easy or difficult is it to have common understanding with many people of different skills set, experiences, background, motives, different stakes and responsibilities. There are so many psychological factors, which makes it difficult to have this common understanding, more so if unit of measurement (hours) is traditionally used for accurate outcome. As soon as, we say x hours, it triggers a mental understanding of accurate number and then we want to see that getting materialized. People usually understand that estimates are just estimates. But as soon as we attach it with Hours kind of definitive measure, it build up expectations of accuracy. This is the key challenge to address.

Daily Stand-Up - Not Just a Ritual

Here is a brief write-up about Daily Stand-up ritual of 'Being Agile' journey. 

Motive is to give quick overview about this ceremony, sharing some of the learning. For detail reading, I would recommend the link given at the end of this article.  

What is the purpose of daily stand-up:

• Shared understanding of goals and current status 
• Plan your day (self-management)
• Coordinate efforts, ask for help, offer help
• Ensure team communication
• Identify as a Team

What are daily stand-up basics, and these are designed for important purpose (refer to links given at the end):

• Fix time
• Fix place
• Must not be more than 15 min
• Whole team gather themselves
• Everyone derive her/his status following three scrum magical questions. Important: It is all about sharing plan, not the task names. Focus should be, what I am doing to contribute to delivery.
• Breakout session at the end or later, if anyone need to discuss more
• Share plan which makes sense to team in bigger context for delivery, It is not about status sharing.
• Only PIGs are allowed to speak. Chickens can join as listener although (refer to scrum pig/chicken story please)

NOT to do:

• Not being regular
• Coming without preparation
• Story telling 
• Problem solving
• Using it as reporting or tracking meeting, 
• Reporting to a Leader or to SM or to Manager. It is all about team communication.
• Work assignment - Mostly, it is done during sprint plan and weekly scrum. There could be few exceptions like helping anyone or any critical work, which team can take call.
• Running across locations (Soln: break in co-located teams. Share the brief in Teams space – only if day to day work is too interdependent - which is neither ideal, nor productive)

What Great Leaders Do - Continuing

Covering rest of the two points which I promised in previous post. 

4. Create a sense of Belonging and Purpose

An important feeling for all of us is feeling of 'I belong here'. This feeling is everything. It defines the comfort for us, for team. It defines the efforts, real contributions, and the results. It defines how much team is connected with the Organization.  

Fostering this feeling is not easy. Leaders helps to create, nourish, and empower the environment which can help in fostering this feeling.

'Promote Environment for Being thyself' - This is the big contributor to feeling of Belonging. If people feels, they are accepted for what they are and need not to change just to fit in, they feel comfortable. It is not about being rigid, or averse to change or learning. Rather it is more helpful to promote learning. Learning comes from adaptability and adaptability starts with being comfortable with oneself, without being defensive or faking any of the trait just to fit in. Once people are comfortable in being what they are, they are actually better aligned to learn and improve to next level, while being connected to the Organization. 

One of the mistake we can do is to have one mold to make everyone perfect and fit in our own world. That is a trap which seems like working at first, and is very well doable with all the great powers, but it will eventually stop people from learning and to use their genuine talent. Eventually they either just 'fit in' or don't belongs to the place. In both the cases, we loose the talent.

Leaders accept their team for who they are, however, have genuine interest to help them to learn, grow and become better version of themselves.

What Great Leaders Do

'What Great Leaders Do' - I don't buy the greatness of 'Great' word, it is simply a search keyword :) 

Whenever I think what leaders do, few flashbacks always help me, when I heard following advice from my mentors.

• 'Let us see how we can solve this problem yours way'
• 'Show people what you think, so they know what you think, and your management style'
• 'Understanding context, Respecting everyone, Integrity in words and actions are important ingredients to manage the talent'

Adding some observations and learning, I experienced so far. Promising, will not make it a long list, but explanation for each point could be long. 

1. Ensure that People feel safe

This is the first priority for the leaders. They reflect this in all of their actions, and words. They work on it day in, day out. 

Why does this has to be a day in, day out job. Because once you are in leadership role, every action impact the masses. The way you communicate, listen and respond, write emails, body language, how you respond to conflicts, how you react to mistakes and failures, all these are followed closely by the team. Every single action is a clue for team, to deduce that how safe they are around you.

Why does feeling safe so important. Let us understand it from neurological perspective. 

When Human being faces threat (physical or mental), our self-defense mechanism kicks in and make us ready for 'fight or flight'. Result, arteries get tightened and tensed to pump more blood to body elements. Heart works rapidly and hard. When body is working so hard to manage the threat, naturally our brain can not focus on creativity, executive functions or social engagements. 

Hence, if team is not feeling safe, we loose Creativity, Executive thought process and most importantly Social Engagements which is the basis of team work. Loosing these means, loosing the team and organizational goals as well.

Next, important point is how to ensure this safety net.

Monolithic vs Decoupled Schema

Let us understand what we mean by Monolithic and Decoupled architecture, Schema with an example.

Setting up a library

Let us take example of library. A new library is opening up. It would have many books, may be on various subjects. First task for librarian is, to store the books in library.

Librarian started with obvious approach first, i.e. start putting all books in shelf, one after other. It was quick. She puts all the books quickly in the shelves; and created an index to search any book.  

Benefits:

• It was quick, easy, and less work. Just one simple storing strategy, every new book will go to next available space.
• You have all sort of books at one place, which means just keep updating one index for every new book – no need to maintain separate indexes.
• Readers can find all sort of books at one place for all of their tastes. They need not to go to multiple places. Just iterate over and you will find what you need.
• For librarian, access management is easy. Anyone having access to library, can access any book easily. Less time spent on management.
• It is easy for readers to refer any other related book also. For ex: if you reading astronomy book, and need to refer a math book. It might just be somewhere in same row.

All setup done. Things started rolling. All good so far. Eventually library become popular and started getting more readers and requirements for new books. Librarian started ordering more and more books on various subjects.

Challenges:

However, as volume of books increased, Librarian started observing few issues. Like:

Own the Problem and Solve It

Problems are problematic and sometime so much problematic that this demand last pint of efforts from us and still challenge us more. But, problems are part of life and will continue to be there. Hence only way to deal with problems is to 'Deal' with these.

One of the common phoenomenon I found is, we are more worried by thinking about problem than actually by the problem. We keep on thinking about problem, by doing all permutation and combination for all possible scenarios, outcomes, what if situations, and get defeated multiple times even before facing the actual problem. Seems, this is more problematic than actual problem. This mentality or habbit (quite common, you are not special) creates more pain for us than what actual Problem can do. We keep on struggling with these problematic thoughts for days,  months or years sometime, but won't take a step to solve it. Result, many a times, we are making a Mountain out of a Molehill or a Bigger Mountain from a small mountain, or pushing our head against that Mountain rather than to take a step to side and walk around this.

Why we do this. What is the solution.

Let us address it step wise. We do this because many a times, we stuck in a loop of thinking, anticipating, visualizing, fearing and the repeating all again. We don't take even a single step considering that result won't be good. We have high inertia.

What is the solution. Solution is to take first step towards solution. We all have solution. It may not be perfect, or may not be the right one.  But we all have capacity to think about solution. So taking step towards solution is the only way out. It is perfectly fine to make a wrong decision. However, it is good as long as we have courage to review it and change our course of action. But what absolutely wrong is, if we don't even take a step. So the solution is to own the problem, think about solution, pick the best from your options, and start executing. If it is right, your problem will be solved. If it is not right, at least you will learn to take next step.

Problems are part of life, and will continue to be there till we exist. Problem free life is not a reality and is not going to be happen. However, a life with solutoins for problem is feasible. So strive for that. And only way out is, Own the problem and take first step to solve it.

What is In-Memory Database Management System

In Memory Database is a fascinating field for a programmer, for an architect, and for technology leaders. Let us try to understand in this article, what is In-Memory DB and why are these so useful. 

As a programmer, designer or architect, whenever I work with traditional databases, one major area to be conscious about is, that database access should be designed carefully. This is because traditional database system store everything on disk, and hence any access or update to this data could be one of the slowest operations in application flow. Undoubtedly, a lot of research and improvements have been done to improve the performance of disk based database systems with improved storing, searching and retrieving logic. Still, disk access, being mechanical operation, always have limitations. And this is one of the important reasons to be fascinated about in-memory database systems. 

Visualize a system where all data (or all data required for curernt context) is sitting in memory. Which means, we can expect a lot more efficient and faster access to data. Database software can have much simpler logic to manage the data, as there is no need to manage the loading/unloading of pages in memory. There is no need to work on time taking locking mechanism to safeguard the data on disk, as access in memory is much faster and chances of conflicts are extremly small which change the way we design database system algorithms. It also means that we can exploit the best available CPU power (which is increasing continuously) to process the data available in memory, having no limitation of disk access. This could also mean that there is no need of another layer of caching system. Rather, database itself will be working like in-memory cache. There are many more benefits if we design the system considering in-memory database system. 

Let us understand what is In-Memory Database System (IMDS). It is a DBMS which maintain all data primarily in Main Memory. Data will be loaded in memory even if it is in GB or TB. A few highlights:

•    With 64 bits computer architecture, systems are capable to address 16 EB of data (1 TB * 1000 * 1000). 82% of the enterprise application’s databases are below 1 TB, growing with an average speed of 10% per year, which means that In-Memory database systems can cater to most of the applications now and in coming future as well.  

•    Does not need write or read to Disk, hence no dependency on mechanical parts and their performance limitations. When all data is stored in single address space, it reduces the complexity of storage algorithms, no need now of loading and unloading the pages in memory

•    Much faster than Disk based traditional DBMS. Having all data in memory means that data is available at our finger tips, only microseconds or nanoseconds away.

•    And it supports ACID properties of Database, including D (durability) 

Attitude Builds the Great Products

During starting years of my career, I was interviewed by a start-up company. Recruitment manager (read CTO) explained me some complex product idea and asked, "Can you implement it". I was not able to understand those complex pieces completely (being slow by nature in new writings :) ). I replied, "I am not very through with whole thing. However, if it is logically feasible, we can do it for sure." And I was hired.

The product concept and the technologies were very new for the team. It was a uphill learning task. Team average experience was around 5 years (excluding the manager). Still team had done wonderful job. With a team of 10 people, we developed a beautiful product with many complex and feature rich components that also in months’ time frame. Whole system was developed on self-validating test workflows, following TDD kind of advance methodologies which were very new and challenging for that 'junior' team. We challenged our every existing belief and drew new lines on learning graph every day, while producing some very interesting product pieces. Definitely, there were many environmental factors which contributed to this wonderful work. However, one of the main factors was the right attitude and willingness to accomplish.

Later, I asked my manager that how he selected me without even asking much technical question. He replied, "I have looked for right attitude in all the candidates and that paid us well".

Other experience: I was working with another project team, a very experienced team having much bigger resource pool. Team was enhancing and managing an accounting product. Product was having some issues in core implementation, due to which we were getting lot of bugs around the years. Course of action was to patch the data or correct the current broken flow of application by temporary fixes. Team was spending many hours in this exercise. Although team had great skills and experience, however, nobody was willing to fix the core issue. Reason, that would need complete rewrite of that component; a critical piece of the product. And it was comparatively easy and less risky to make fixes. Hence, due to lack of right attitude and driving force, we kept on putting hours of efforts in fixing the bugs only.

By now, we already know the difference between first and second example. It is not (only) about skills and experience, but, it is more about right attitude. In second example, team was much experienced and bigger. Many of the members were skilled enough to do right things and do the things right. However, differentiating factor was the right 'attitude'. It needs a high level of 'can do' and risk taking attitude to say, "Let’s build something better even if it mean scrapping existing piece of code. Let’s write a good product, where team can spend time adding value to the product instead of fixing issues and doing plumbing work around the year. Let sit and figure out, how we can improve the things".

Finding skills and experience could be easy, however, finding and developing right attitude needs a lot of efforts. Also it does not end with the attitude of individuals, rather attitude of leaders also play a big role in developing this whole eco-system. In first example, it was the leader who has given weightage to the right attitude over skills, education and experience. It was the leader who has promoted the attitude to take the risk and to challenge the stagnant beliefs to improve the product.

When right attitude is mixed with right set of environment, people take action. They become passionate for the goals, for the things which they are doing. They go extra mile to build what they believe in. A very important element in producing great products is the attitude of people working on it.

It takes whole village to build a good product and a successful Organization, however, it takes a right set of attitude of team and the leader to build and groom that village.

What is your experience for role of 'attitude' in building great products, what are your examples?