Mark Erikson

Follow this blog

Coding Career Advice: Using Git for Version Control Effectively

Introduction Git has become the standard tool for software development version control. Other VCS tools exist, and some work better than Git for certain scenarios, but most of today's development world relies on using Git. So, becoming comfortable with Git and knowing how to use it effectively is a key skill for any software developer. I'd like to pass along some of the most useful Git concepts and tips that I've learned over the last few years. In addition, I've covered background info on how Git works and common operations, and there's some specific usage patterns I've found to be especially valuable when working with a team and trying to understand a codebase. As usual, none of the info or advice in this post is completely new or original, and there's many other sites that cover the same topics (and probably explain them better). I'm just trying to provide an overview of the relevant material and provide enough details that you can do further research and learning from there. This post is largely based on my slideset Git Under the Hood: Internals, Techniques, and Rewriting History, and I talked about rewriting repo history in my post Rewriting Your Git History and JS Source for Fun and Profit Table of Contents Introduction Table of Contents Git Fundamentals Git Terms and Concepts Overview Git Basics Sharing Data Between Repositories Branches Understanding Git Internals Git Tools Git Techniques Improving CLI Logging Preparing Commits in Pieces Stashing Changes Working with Branches Creating and Switching Branches Fetching, Pushing, and Pulling Branches Merging Branches Feature Branch Strategies Pull Requests Updating Branches in the Background Rewriting Git History Amending Commits Resetting Branches Rebasing Branches Reverting Commits Cherry-Picking Interactive Rebasing Reflog Advanced History Rewriting Git Patterns and Best Practices Write Good Commit Messages Make Small, Focused Commits Clean Up Commit History Before Pushing Only Rewrite Unpushed History Keep Feature Branches Short-Lived Code Archeology with Git Displaying Historical File Changes Bisecting Bugs Final Thoughts Further Information Git Fundamentals Git is notoriously difficult to work with, especially using the command line. The CLI commands and options are confusing, mismatched, and hard to remember. There's phrases and warnings like "detached HEAD". Git, frankly, is not easy to learn and kinda scary. The good news is that once you understand how Git works, it becomes an extremely powerful tool that offers a lot of flexibility. Git Terms and Concepts Overview While I'm not going to turn this into a complete "Git tutorial from scratch", it's worth reviewing some of the key concepts. Git Basics Git is a tool for tracking changes to file content over time. A Git repository is a folder that has a .git folder inside. The .git folder contains all the metadata and stored history of the project's changes. The working copy is all other folders and files in the repository folder that Git is storing and tracking. Any newly created files start out untracked. Git knows that the files are there, but you haven't told Git to save them. To tell Git to start tracking a file, you add the file (git add some-file). Git then saves a copy of the file in an internal section called the staging area. Staged files are not being saved permanently, yet. Instead, they represent the set of files and contents that will be saved when you actually tell Git to save them. Once you've added one or more files to the staging area, you can save them by committing them. "Commit" is both a verb and a noun here: we "commit" files to save them, and every time we save them, we make a "commit". Git commits contain a certain set of files and their contents, at a specific point in time. They also contain metadata, including the author's name and email address, and a commit message that you write to describe the changes that were saved. After a file has been added at least once, making further changes to that file will cause Git to mark it as modified. That means that Git knows the contents are different, but you haven't told Git to save the new changes yet. Once you add that file to the staging area again, Git sees that its latest copy of the file is the same as what's on disk, so it describes the file as unchanged. Sharing Data Between Repositories Each Git repository folder is standalone. However, Git repositories can be shared across folders, computers, and networks, allowing developers to collaborate on the same codebase. A Git repo can be configured with the URL of another repo, allowing the two repos to send commits back and forth. Each URL entry is called a remote. Downloading commit data from a remote repo is a fetch or a pull (with slight differences in behavior), and uploading commit data from local to remote is a push. Downloading a complete repo from scratch is making a clone of that repo. Repositories normally have a default remote repo they point to, called the origin. Whenever you clone a repo, the new local repo points to the remote source as the origin, but that entry can be changed later. Repos can be configured to talk to many other repos at once, and can push and pull data from any remote. Branches Git commits are tracked using branches. A branch is like a pointer to the latest commit in a specific series of commits. Any time you make a new commit, Git bumps that branch pointer to point to the newest commit. You can make many branches within a repo, and most devs create a new branch for each task they work on. You can also make tags, which also point to a specific commit, but don't get moved or changed automatically. Tags are normally used to identify checkpoints and releases, so you can easily jump back and see how the code was at that point in time. Changes from multiple branches can be brought together using a merge process. If some of the changes apply to the same lines of code, there is a merge conflict, and it's up to you as the developer to look at the mismatched changes and resolve the conflict by picking out what the right contents are. Historically, most repos use a branch called master as the primary development branch. More recently, the community has started switching to use a primary branch named main instead. But, you can configure Git to use any branch name as the "default development branch" if you want. Git uses the term checking out to refer to updating the working copy files on disk, based on previously committed values. Typically you check out a branch, which overwrites the files on disk to match the files as they exist in the latest commit of the branch. However, you can check out other versions of files as well Uncommitted changes can be copied and saved for later by creating a stash. A stash is kind of like an unnamed commit - it again points to specific files at a certain point in time, but it doesn't exist in a branch. Stashed changes can later be applied on top of your working copy. Overall, the Git data workflow looks like this: Understanding Git Internals I really feel that understanding Git's internal data structures is critical to understanding how Git works and how to use it correctly. Git tracks all content using SHA1 hashes of byte data. Running any specific sequence of bytes through the hashing function calculates a specific hex string as a result: from hashlib import sha1 sha1("abcd").hexdigest() '81fe8bfe87576c3ecb22426f8e57847382917acf' sha1("abce").hexdigest() '0a431a7631cabf6b11b984a943127b5e0aa9d687' readme = open("README.md", "rt").read() sha1(readme).hexdigest() '45257c0245c56a4d5990827b044f897c674c8512' Git hashes files and data structures, then stores them inside the .git folder based on the hash: /my-project /.git /objects /00 /01 ... /81 81fe8bfe87576c3ecb22426f8e57847382917acf /82 ... /fe /ff Git has three primary internal data structures: blobs are file contents, and identified by a hash of the file's bytes file trees associate folder and file names with file blobs, and are identified by a hash of the file tree data structure commits contain metadata (author, timestamp, message), point to a specific file tree, and are identified by a hash of the commit data structure Type Contains Identified By Blob File contents Hash of the file's bytes File tree Associates names and folder definitions with file blobs Hash of the file tree data structure Commit Metadata for author, commit timestamps, and message Hash of the commit data structure A file tree may point to multiple other file trees for subfolders: Commit objects themselves form a linked list, which points backwards to earlier commits based on their hashes: A <- B <- C <- D. A Git "ref" is a name label that points to a specific commit. Branches are names associated with a given ref, where each time a new commit is made, the ref is updated to point to that latest commit. So, you can start from the branch ref pointer, then walk backwards through the chain of commits to see the history. HEAD is a ref that points to "whatever the current active commit" is. Normally this is the same as the current branch pointer, but if you check out a specific earlier commit, you get the ever-popular warning about a "detached HEAD". This just means that HEAD is pointing to a specific commit instead of a branch, and if you make any new commits, they won't be part of any branch. Because commits are a linked list based on hashes, and the hashes are based on byte contents of files and other structures, changing any one bit in an earlier commit would have a ripple effect - every hash of each commit after that would be different. Git commit objects are immutable - once created, they cannot actually be changed. This means that you can't change history, exactly - you can only create an alternate history. Git Tools I've seen a lot of arguments about whether it's better to use a Git GUI tool, or use Git from the command line. To those people, I say: why not both? :) I find that having a Git GUI tool is absolutely invaluable. It makes visualizing the state of the repository and its branches much easier, and many operations are way simpler via a GUI. For example, I can view the diffs for many pieces of a file at once, and selectively add specific changes to the staging area by clicking "Add Hunk" or CTRL-clicking a few lines to select them and clicking "Add Lines". This is much simpler and more intuitive than trying to use Git's "patch editing" text UI to manipulate pieces of changes. Interactive rebasing is also much easier to do via a GUI. I can't remember what the different options like "pick" mean, but it's straightforward to use a GUI listview with arrow buttons that lets you reorder commits and squash them together. On the other hand, it's often faster to create or switch branches from the CLI. You can add all changed files to the staging area with a single command of git add -u. And of course, if you are using a remote system via SSH, you probably do only have the Git CLI available. So, I use both a Git GUI, and the CLI, based on what tasks I'm doing. I primarily use Atlassian SourceTree (Win, Mac). It's very powerful, with a lot of options, and has a good built-in UI for interactive rebasing. It also happens to be free. The biggest downside is that it doesn't have a way to view the contents of the repo file tree as of a given commit. Other Git tools I've used in some form include: Git Extensions for Windows (Win): integrates with Windows Explorer to let you perform Git operations from the filesystem. I mostly use this to do a quick view of a given file's history if I happen to be browsing the folder contents of the repo. Git Fork (Win, Mac): excellent UI design, and does have an interactive rebase UI. Recently switched from being free to $50, but likely worth paying for. Sublime Merge (Win, Mac, Linux): from the makers of Sublime Text. Fewer options and tool integrations, but very snappy. Tells you what CLI operations it's doing when you try to push or pull, so it expects familiarity with the CLI. $100, but will run for a while with nag messages. There's also Tower (Win, Mac) and Git Kraken (Win, Mac, Linux), which have slick UIs but require yearly subscriptions, and a laundry list of other smaller Git GUIs. There's even "text-based UI" tools like lazygit, gitui, and bit. All major IDEs have Git integration. JetBrains IDEs like IntelliJ and WebStorm have excellent Git capabilities. VS Code has adequate Git integration, but really needs additional extensions like Git History and GitLens to be useful. I also really prefer using external diff tools for comparing complete files, or fixing merge conflicts. I personally use Beyond Compare as my external diff tool, and DiffMerge as my conflict resolution diffing tool. Git Techniques Improving CLI Logging The default git log output is ugly and hard to read. Whenever I start using Git on a new machine, the very first thing I do is browse to https://coderwall.com/p/euwpig/a-better-git-log and copy-paste the instructions for creating a git lg alias to set up a much pretter CLI logging view that shows the branch and commit message history: git config --global alias.lg "log --color --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit" That gives us this view whenever we run git lg: Note that git log accepts a variety of filtering options, including text strings, dates, branches, etc. Preparing Commits in Pieces I've seen comments that complain that the Git staging area is confusing. To me, the staging area is one of the most valuable features of Git - it lets me carefully craft commits that contain only the code that belongs together. When I work on a task, I frequently end up modifying multiple files before I'm ready to make a commit. However, the changes might logically belong in several smaller commits instead of one big commit. If I do git add some-file, it adds all the current changes in the file to the staging area. Instead, I often want to stage just a couple sections from file A, and a couple sections from file B, and maybe all of file C, because those are the changes that should go together in one commit. You can do this from the commandline using the git add -p flag, which brings up a text UI that lets you view each "hunk" of changes in a file, and decide whether to stage that hunk or not. However, I strongly recommend using a Git GUI tool like SourceTree for adding pieces of files, because it's easier to click "Add Hunk" or CTRL-click a couple lines and click "Add Lines" than it is try to decipher what the command abbreviations in the text UI actually mean: Once you've got these pieces added, you can make a commit with just chose changes, and repeat the process for the next commit. This is a key part of the "making small commits" practice that I cover below. On the flip side, sometimes you do just want to add everything that's been changed at once. In that case, the fast way is to run git add -u from the command line, which adds all modified files to the staging area. Stashing Changes Stashes are most useful when you've got some modified files that aren't committed, and need to set those aside to work on a different branch for a while. Git's list of stashes acts like a stack data structure, but you can also supply names for stash entries when you create them. Creating stash entries normally resets the modified files back to the latest commit, but you can choose to leave the modifications in place. From the CLI, the main options are: git stash: save a copy of local changes for later reuse, and clears the working directory/index git stash push: creates a new stash entry git stash pop applies changes from the top stash entry and removes it git stash apply stash@{2}: applies changes from the third stash entry git stash -p: choose specific pieces to stash git checkout stash@{2} -- someFile: retrieve a specific file contents from the stash But, this is another situation where it's particularly useful to use a GUI instead. It's easier to just click a "Stash" button in a toolbar and type in a name for the entry to create one, or to expand a "Stashes" section of a treeview, right-click an entry, and "Apply Stash" to apply a stash. Working with Branches Creating and Switching Branches Git has a bunch of different commands for working with branches. The most common way to create a branch is actually with git checkout -b NAME_OF_NEW_BRANCH. That creates a new branch, starting from the latest commit on the current branch, and switches to it. You can also use git checkout NAME_OF_EXISTING_BRANCH (without the -b flag) to switch to an existing branch. There's many other branching commands - see the Git docs and other pages like this Git branching cheatsheet for lists of commands and options. Fetching, Pushing, and Pulling Branches Most Git network operation commands accept the name of the remote repo to talk to, but assume that you want to talk to the origin remote repo by default if you don't specify a remote name. git fetch tells Git to contact another repo, and download copies of all commits that the local repo doesn't have stored. This includes information on branches in the remote repo as well. Once your repo has downloaded the list of remote branches, you can create a local branch based on the remote branch's name, with git checkout NAME_OF_REMOTE_BRANCH. Git will create a new branch that points to the same commit. It also sets up the local branch to track the remote branch, which means that any pushes from the local branch will update the remote branch. Later, you can update the remote branch with the new commits you made locally, with git push. You can also push local branches taht the remote repo doesn't know about yet. If the remote branch has commits you don't have in your local branch, git pull will both fetch the set of new commits into your local repo, and update your local branch to contain those commits. If you rewrite history on your local branch so that it's different than the remote branch, a git push attempt will fail with an error. You can force push , which will hard-update the remote branch to use these commits instead. Force pushing is semi-dangerous, depending on workflow. If someone else pulled the old history, and you force-push, now they have a conflict to deal with. Force pushing is a valuable tool if you need it, and can be a legitimate solution to fixing problems or repeatedly updating a PR, but should be used cautiously. Think of it as a chainsaw - if you need it, you need it, you just have to be very careful when using it :) Merging Branches Merging allows you to take changes and history that exist on branch B, and combine them into the changes on your current branch A. The assumption is that both branches have a common set of ancestor commits, and two different sets of changes (either to different files, or even the same files). Merging creates a new "merge commit" on the current branch that has all of the changes together in one spot. This is used to let developers collaborate by writing code separately, but combine their changes together. Merging is done with git merge OTHER_BRANCH_NAME, which tells Git to merge from the other branch into the current branch. If the changes on the two branches interfere with each other, there's a merge conflict. Git will mark the file with text strings indicating the two mismatched sections. It's up to you to fix the problem, save the corrected file, add it, and finish the merge commit. I like using SourceGear DiffMerge as a GUI tool for fixing conflicts, but VS Code also does a nice job of highlighting conflict markers in files and offering hover buttons to pick one side or the other. Feature Branch Strategies Most teams use some kind of a "feature branch" strategy for development. They have a primary development branch such as main, master, or develop. Any time a developer starts work on a new task, they create a new branch based on the primary branch, and often using the name and ID of a task/issue as the branch name: git checkout -b feature/myapp-123-build-todos-list. The developer works on their feature for a while. Once the work is complete, they push the branch up to the team's central repository, other team members review the changes, the developer makes any needed fixes from the review, and then the feature branch is merged back into the primary development branch. Developers may need to pull down changes that have been added to the primary branch, then "merge down" from the primary branch into their feature branch. Merging the feature branch back into the primary branch is referred to as "merging up". Pull Requests If you've worked with Git at all, you've probably heard the term "pull request" (also know as a "PR" for short, or occasionally "merge request") before. Strictly speaking, a "pull request" isn't even a Git concept - it's a merging workflow that is built on top of Git by repository hosting sites and tools like Github, Gitlab, and Bitbucket. Pull Requests are an approach to doing code reviews and handling merging at the central Git repo/server level. This is typically associated with using feature branches. A developer pushes up their completed feature branch, and creates a PR that will merge some-feature into main. Other devs can look at the page for the PR, see the file diffs, and leave comments on specific lines suggesting changes. The feature dev makes more commits based on those suggestions, pushes them up, and the PR is updated to reflect the changes. After other team members approve, the PR can be merged and the feature branch can be deleted. Updating Branches in the Background Normally, the main way to update a local copy of a branch is to git checkout some-branch and then git pull. But, if I'm working on a feature branch, I often have unsaved changes and don't want to switch over to the main branch just to do a pull. There's a really useful trick for doing a "background pull" of a branch without checking it out: git fetch <remote> <remoteBranch>:<localBranch> So, say I'm on features/some-feature, and I want to update my main branch without switching to it. Typically the local branch and remote branch have the same name. So, I can run: git fetch origin main:main and Git will download any new commits on the remote origin/main branch, then update my local main branch to have those commits too. Rewriting Git History There's a variety of ways to alter the history in a Git repository. Each technique is useful in different situations, and these are often valuable for fixing earlier problems. As mentioned earlier, Git commits are immutable, so you can never actually modify them - you can only replace commits with new ones. So, when we "rewrite history", we're actually creating an "alternate history" instead. It's critical that you should only ever rewrite history that is still local to your own repository and has never been pushed up to another repository! As long as commits haven't been pushed, no one else cares about them, and you can rewrite them to your heart's content. But, once they've been pushed, someone else's Git repo clone may be relying on the old history, and changing that history will likely cause conflicts for them. Amending Commits The easiest technique for rewriting history is to "amend" the latest commit. Amending a commit really means replacing it with a slightly different one. This can be done via git commit --amend, or a corresponding option in a GUI tool: Technically, the old commit still exists in Git's storage, but the current branch ref now points to the newly created commit instead. Resetting Branches Since branch refs are pointers to a given commit, we can reset a branch by updating the ref to point to an earlier commit. This is typically used to roll back some of the commits you made. When you reset a branch, you have three options for what happens to the files on disk and in the staging area: git reset: move a branch pointer to point to a different commit --soft: keep the current files on disk and in the staging area --mixed: keep the current files on disk, but clear the staging area --hard: clear the staging area and make the working directory look exactly like this specific commit So, git reset --soft is fairly "safe" to do, because it doesn't change any files on disk. git reset --hard is "dangerous", because it will wipe out any files that were changed during these commits or that haven't been committed yet, and replace them all with the files from this exact commit. git reset requires a commit identifier as an argument. This could be a specific commit hash ( git reset ABCD1234 ), or some other revision identifier. You can even update your current branch to point to the same commit as a different branch ( git reset --hard some-other-branch ). Rebasing Branches "Rebasing" is a technique that is an alternative to merging for updating one branch with another's changes. Instead of combining the two sets of changes directly, rebasing rewrites history to act as if the current branch was created now, off the latest commits on the source branch, instead of starting from the earlier commits. Similar to merging, this is done with git rebase OTHER_BRANCH_NAME. Imagine that the main branch has commits A <- B to start with, and we make a feature branch starting from commit B. Now, someone else merges some more work into main, giving it commits A <- B <- C <- D. If we rebase our feature branch against main, it's kind of like cutting off the line of our feature branch, transplanting it to the end, and pretending we really started this branch after commit D instead of B: Reverting Commits Resetting a branch effectively throws away the newer commits. What if we want to undo the changes in an earlier commit, but keep the history since then? Reverting a commit with git revert creates a new commit that has the opposite changes of the commit you specified. It doesn't remove the original commit, so the history isn't actually modified - it just inverts the changes. Cherry-Picking Cherry-picking allows you to copy the changes in specific commits, and apply those as new commits onto a different branch. For example, maybe there's an urgent patch that has to be created directly onto a hotfix branch and deployed to production, but you need to also make sure that main has that commit as well. You can cherry-pick the individual commit from the hotfix branch over onto main. git cherry-pick accepts either a single commit reference, or a commit range. Note that the range excludes the first commit you list. if I run git cherry-pick A..E, then it will copy commits B,C,D,E over onto this branch. This creates new commits with new hashes (because the timestamps and parent commits are different), but preserves the diffs and commit metadata. Interactive Rebasing "Rebasing" involves rewriting the entire history of a branch. There is a variation on this called "interactive rebasing", which allows you to selectively modify earlier commits on a branch. This is done with git rebase -i STARTING_COMMIT. Interactive rebasing lets you perform several different types of modifications. You can: Edit the message for a commit Reorder commits Squash multiple commits together Remove commits After you specify the desired changes to the commit history, Git will execute the modifications you listed, and update all commits after the starting point accordingly. As with other history rewriting operations, this always produces a new set of commits after any changed commit, with new hashes even if the rest of the contents haven't changed due to the parent commits changing. Running an interactive rebase from the CLI brings up a list of all commits after the starting commit in your text editor, along with a column of odd command names like "pick" and "squash". You rework the commits by actually modifying the text in the file, and then saving and exiting. For example, if you want to swap a couple commits, you'd cut one of the text lines and paste it in a different location. I find this very unintuitive to work with, so I strongly recommend using a Git GUI for any interactive rebase operations. SourceTree and Fork have pretty good UIs for performing interactive rebasing. Reflog It's actually very hard to completely wipe out your Git commits and permanently lose work. Even if you do a git reset --hard and the commits appear to have vanished, Git still has a copy of those commits saved internally. If you do end up in a situation where you can't see those commits referenced from any tag or branch, you can use the Git reflog to look back and find them again. The reflog shows all commits, no matter what branch they're on or whether there's still a meaningful pointer to that commit. That way you can check them out again, create a new tag or branch pointing to those commits, or at least see the diffs. Advanced History Rewriting Finally, Git supports some very advanced tools for rewriting history at the whole repository level. In particular, git filter-branch lets you perform tasks like: rewriting file names and paths in the history (example: changing files in ./src so that they now appear to be in the repo root) creating a new repo that contains just certain folders from the original, but with all their history rewriting actual file contents in many historical commits git filter-branch is notoriously slow, so there's other external tools that can perform similar tasks. While I haven't used it, https://github.com/newren/git-filter-repo claims to be able to run the same kinds of operations much more quickly, and is apparently now even recommended by the actual Git docs. Sometimes repos end up with very large files cluttering the history, and you want to rewrite the history to pretend those files never existed. A tool called the BFG Repo Cleaner does a good job of that. If these existing tools don't do what you need, you can always write your own. I once wrote a set of Python-based tools to rewrite the JS source for for an entire repository with multiple years of history, including optimizing it to run in just a few hours. These tools are very powerful and should not be something you use for day-to-day tasks. Think of them as fire extinguishers. You hope you never need to use them, but it's good to have it sitting around in case something happens. Git Patterns and Best Practices So now that we've covered a bunch of commands and technical details, how do you actually use Git well? Here's the things that I've found to be most helpful: Write Good Commit Messages It's critical to write good commit messages. It's not just a chore to satisfy the Git tools. You're leaving notes to any future developers on this project as to what changes were made, or even more importantly, why those changes were made. Anyone can look at a set of diffs from a commit and see the changed lines, but without a good commit message, you may have no idea what the reason was to make those changes in the first place. There's lots of good articles out there discussing rules for writing commit messages, with plenty of good advice. I personally don't care so much about things like "max 72 characters per line" or "use present tense for the top line and past tense for other lines", although there's valid reasons to do those things. To me, the critical rules are: Always start the commit message with a relevant issue tracker ID number if there is one. That way you can always go back to the issue tracker later to see more details on what the specific task was supposed to be. First line should be a short high-level summary of the intent of the changes. This line of the message is what will be shown in any Git history log display, so it needs to both fit in one line, and clearly describe the commit overall. Aim more for the purpose of the changes than a specific list of "what changed" in this line. If you have any further details, add a blank line, then write additional paragraphs or bullet points. Write as much as you want here! This section will usually be collapsed by default in a Git UI, but can be expanded to show more details. I've seen some excellent commit messages that were multiple paragraphs long, and they provided important context for why changes were being made. A typical example of this format would look like: MYAPP-123: Rewrite todos slice logic for clarity - Added Redux Toolkit - Replaced handwritten reducer with `createSlice`. This simplified the logic considerably, because we can now use Immer and it auto-generates the action creators for us. - Updated `TodosList` to use the selectors generated by the slice. Make Small, Focused Commits This goes hand-in-hand with the advice to write good commit messages. Commits should be relatively small and self-contained, conceptually. One commit might touch several files, but the changes in those files should be closely related to each other. There's multiple reasons for this: It makes it easier to describe the changes in the commit It's easier to look at that one commit and see what changes it includes If the commit needs to be reverted later, there's fewer other changes that would be affected When someone looks at the line-by-line history, there will be more specific comments associated with each line ("Fixed bug X" instead of "Made a bunch of changes", etc) It's easier to bisect the commit history and narrow down what changes might have caused a particular bug For example, say I'm adding a new JS library to a project. I would make one commit that just updates package.json and yarn.lock, then put the initial code changes using that library into a separate commit. You can see an example of this commit approach in the commits for the "Redux Fundamentals" tutorial example app I wrote. To me, the commit history should "tell a story" of how a given task was accomplished. Someone should be able to read through the series of commits, whether it be during the PR review process or years down the road, and be able to understand my thought process for what changes I made and why I made them. Clean Up Commit History Before Pushing I frequently have to make "WIP" commits as I'm working on a task. Maybe I've just made a bunch of edits, the code is now mostly working, and I want to record a checkpoint before I keep going. Or, maybe I forgot to commit a particular bit of code, added it in another commit later, but it doesn't really belong as part of the "story" that I'm telling. I often use interactive rebase to clean up my commits before I push a branch for a PR. Just because I have some junk commits in my history locally doesn't mean that the rest of the world needs to know or care that was part of my actual progress for this task. The "story" that I'm telling with my commits is sort of the idealized version - ie, "let's pretend that I did this task perfectly without any mistakes along the way". Only Rewrite Unpushed History As mentioned earlier: as long as a branch is still local and hasn't been pushed, it's fair game - rewrite it all you want! Once it's been pushed, though, you should avoid rewriting it. The one main exception to that is if the branch is still up for PR, and you redo the history. At that point, most likely no one depends on it yet, so you can get away with force-pushing the branch to update the PR. (The React team does this frequently.) Keep Feature Branches Short-Lived There's no hard rule about how many lines of code or commits can be in a branch. In general, though, try to keep feature branches relatively short-lived. That way the size of the changes to merge in a PR is smaller, and it's less likely that you'll need to pull down changes from someone else. Some people argue about whether it's better to merge feature branches back into the primary branch, or rebase them when they're done to keep the main branch history "clean and linear". I kinda like having merge commits, personally - I prefer seeing when things got merged in. The important thing is to pick a convention as a team and stick with it. Code Archeology with Git So why do all these good commit practices matter? Say you're working in a codebase with multiple years of history. One day, you're assigned a task to work on some portion of the codebase. Maybe it's fixing a bug that just popped up, or adding a new feature. You open up a file, and there's hundreds of lines of code inside. You read through it, and it's kind of ugly - there's a bunch of extra conditions in the logic, and you're really not sure how it ended up this way. Reading through that file tells you what the code does, now. Unless the file has a lot of good comments, there may not be much information for why the code is like that, or how it got that way. We naturally have a tendency to assume that "whatever code is there currently must be correct", but that's not always true :) That's where having a good Git history is critical. Digging through a file's history can show you: Who wrote each line of code When that code was changed, and what other code changed at the same time What task the changes were part of What the intent was behind the change What the author was thinking at the time When a bug was introduced These can all be extremely valuable pieces of information when tracking down a bug or working on a feature. Displaying Historical File Changes There's a variety of ways to view the history of changes to a file. git log lets you look at the commits that affected a specific file. IDEs and Git GUIs let you explore the history of a file as well, showing each commit its diffs, often including the ability to diff two arbitrary versions of a file. Some Git GUIs also let you explore the entire repo file tree as of a specific commit. Git has a feature called git blame, which prints the commit ID, author, and timestamp for each line. The CLI output is hard to read, but every good IDE has the ability to show file blame information next to the actual code in a file. IDEs typically enhance the blame information to show you more details on the author, the commit message, and the commits before and after this one: Github offers a "blame" view as well, and makes it easy to jump back to view an earlier version of the repo. Github also lets you browse specific file versions and trees. For example, https://github.com/reduxjs/react-redux/tree/v7.1.2 shows the React-Redux codebase as of tag v7.1.2, and https://github.com/reduxjs/react-redux/blob/5f495b23bcf3f03e0bee85fa5f7b9c2afc193457/src/components/connectAdvanced.js shows that exact file version. (Press y while browsing a file on Github to change the URL to the exact file hash.) Bisecting Bugs Git has a really neat command called git bisect, which you can use to help find the exact commit where a bug was introduced. When you run git bisect, you can give it a commit range where you think the problem started. Git will then check out one commit, let you run whatever steps you need to with the app to determine if the bug is present or not, and then say git bisect good or git bisect bad. It then jumps to another commit and lets you repeat the process. It follows a splitting pattern that lets you narrow down the potential problem commit in just a few steps. Final Thoughts As software developers, we use lots of tools. Everyone has their own preferences for things like text editors and such, but everyone on a team is going to use the same version control system. In today's world, that's inevitably Git. Given how critical Git is to modern development, anything you can do to use it more effectively will pay dividends down the road, and anyone reading your commits will appreciate the effort you put into clearly describing what changes are happening and why. It might be a teammate reading your PR, an intern exploring the codebase next year, or even yourself revisiting code that you wrote many years ago. Ultimately, good Git practices are a key part of long-term codebase maintainability. Further Information Git Tutorials Atlassian Git tutorials Git for Humans slides Git Internals Git Internals ebook (PDF) Git from the Bottom Up A Visual Guide to Git Internals Commits are Snapshots, Not Diffs Git Internals exploration Commit Messages How to Write a Git Commit Message Git Commit Message Good Practices Operations Git Beyond the Basics Advanced Git Commands You Will Actually Use Cheat Sheets A Visual Git Guide Tower Git Cheat Sheet akras14 Git Cheat Sheet DevHints Git cheat sheets: revision syntax, log syntax, branching commands Other Resources Git Flight Rules: a guide for when things go wrong React/Redux Links: Git Resources

Blogged Answers: Why React Context is Not a "State Management" Tool (and Why It Doesn't Replace Redux)

Introduction "Context vs Redux" has been one of the most widely debated topics within the React community ever since the current React Context API was released. Sadly, most of this "debate" stems from confusion over the purpose and use cases for these two tools. I've answered various questions about Context and Redux hundreds of times across the internet (including my posts Redux - Not Dead Yet!, React, Redux, and Context Behavior, A (Mostly) Complete Guide to React Rendering Behavior, and When (and when not) to Reach for Redux), yet the confusion continues to get worse. Given the prevalence of questions on this topic, I'm putting together this post as a definitive answer to those questions. I'll try to clarify what Context and Redux actually are, how they're meant to be used, how they're different, and when you should use them. TL;DR Are Context and Redux the same thing? No. They are different tools that do different things, and you use them for different purposes. Is Context a "state management" tool? No. Context is a form of Dependency Injection. It is a transport mechanism - it doesn't "manage" anything. Any "state management" is done by you and your own code, typically via useState/useReducer. Are Context and useReducer a replacement for Redux? No. They have some similarities and overlap, but there are major differences in their capabilities. When should I use Context? Any time you have some value that you want to make accessible to a portion of your React component tree, without passing that value down as props through each level of components. When should I use Context and useReducer? When you have moderately complex React component state management needs within a specific section of your application. When should I use Redux instead? Redux is most useful in cases when: You have larger amounts of application state that are needed in many places in the app The app state is updated frequently over time The logic to update that state may be complex The app has a medium or large-sized codebase, and might be worked on by many people You want to be able to understand when, why, and how the state in your application has updated, and visualize the changes to your state over time You need more powerful capabilities for managing side effects, persistence, and data serialization Table of Contents Introduction TL;DR Understanding Context and Redux What is React Context? Using Context Purpose and Use Cases for Context What is Redux? Redux and React Purposes and Use Cases for (React-)Redux Why Context is Not "State Management" Comparing Context and Redux Context and useReducer Choosing the Right Tool Use Case Summary Recommendations Final Thoughts Further Information Understanding Context and Redux In order to use any tool correctly, it's critical to understand: What its purpose is What problems it's trying to solve When and why it was originally created It's also critical to understand what problems you are trying to solve in your own application right now, and pick the tools that solve your problem the best - not because someone else said you should use them, not because they’re popular, but because this is what works best for you in this particular situation. Most of the confusion over "Context vs Redux" stems from a lack of understanding about what these tools actually do, and what problems they solve. So, in order to actually know when to use them, we need to first clearly define what they do and what problems they solve. What is React Context? Let's start by looking at the actual description of Context from the React docs: Context provides a way to pass data through the component tree without having to pass props down manually at every level. In a typical React application, data is passed top-down (parent to child) via props, but this can be cumbersome for certain types of props (e.g. locale preference, UI theme) that are required by many components within an application. Context provides a way to share values like these between components without having to explicitly pass a prop through every level of the tree. Notice that it does not say anything about "managing" values - it only refers to "passing" and "sharing" values. The current React Context API (React.createContext()) was first released in React 16.3. It replaced the legacy context API, which had been available since early versions of React, but had major design flaws. The primary problem with legacy context was that updates to values passed down via context could be "blocked" if a component skipped rendering via shouldComponentUpdate. Since many components relied on shouldComponentUpdate for performance optimizations, that made legacy context useless for passing down plain data. createContext() was designed to solve that problem, so that any update to a value will be seen in child components even if a component in the middle skips rendering. Using Context Using React Context in an app requires a few steps: First, call const MyContext = React.createContext() to create a context object instance In a parent component, render <MyContext.Provider value={someValue}>. This puts some single piece of data into the context. That value could be anything - a string, a number, an object, an array, a class instance, an event emitter, and so on. Then, in any component nested inside that provider, call const theContextValue = useContext(MyContext). Whenever the parent component re-renders and passes in a new reference to the context provider as the value, any component that reads from that context will be forced to re-render. Most commonly, the value for a context is something that comes from React component state, along these lines: function ParentComponent() { const [counter, setCounter] = useState(0); // Create an object containing both the value and the setter const contextValue = {counter, setCounter}; return ( <MyContext.Provider value={contextValue}> <SomeChildComponent /> </MyContext.Provider> ) } A child component then can call useContext and read the value: function NestedChildComponent() { const { counter, setCounter } = useContext(MyContext); // do something with the counter value and setter } Purpose and Use Cases for Context Based on that, we can see that Context doesn't actually "manage" anything at all. Instead, it's like a pipe or a wormhole. You put something in the top end of the pipe using the <MyContext.Provider>, and that one thing (whatever it is) goes down through the pipe until it pops out the other end where another component asks for it with useContext(MyProvider). So, the primary purpose for using Context is to avoid "prop-drilling". Rather than pass this value down as a prop, explicitly, through every level of the component tree that needs it, any component that's nested inside the <MyContext.Provider> can just say useContext(MyContext) to grab the value as needed. This does simplify the code, because we don't have to write all the extra prop-passing logic. Conceptually, this is a form of "Dependency Injection". We know that the child component needs a value of a certain type, but it doesn't try to create or set up that value itself. Instead, it assumes that some parent component will pass down that value, at runtime. What is Redux? For comparison, let's look at the description from the "Redux Essentials" tutorial in the Redux docs: Redux is a pattern and library for managing and updating application state, using events called "actions". It serves as a centralized store for state that needs to be used across your entire application, with rules ensuring that the state can only be updated in a predictable fashion. Redux helps you manage "global" state - state that is needed across many parts of your application. The patterns and tools provided by Redux make it easier to understand when, where, why, and how the state in your application is being updated, and how your application logic will behave when those changes occur. Note that this description: specifically refers to "managing state" says that the purpose of Redux is to help you understand how state changes over time Historically, Redux was originally created as an implementation of the "Flux Architecture", which was a pattern first suggested by Facebook in 2014, a year after React came out. Following that announcement, the community created dozens of Flux-inspired libraries with varying approaches to the Flux concepts. Redux came out in 2015, and quickly won the "Flux Wars" because it had the best design, matched the problems people were trying to solve, and worked great with React. Architecturally, Redux emphasizes using functional programming principles to help you write as much of your code as possible as predictable "reducer" functions, and separating the idea of "what event happened" from the logic that determines "how the state updates when that event happens". Redux also uses middleware as a way to extend the capabilities of the Redux store, including handling side effects. Redux also has the Redux Devtools, which allow you to see the history of actions and state changes in your app over time. Redux and React Redux itself is UI-agnostic - you can use it with any UI layer (React, Vue, Angular, vanilla JS, etc), or without any UI at all. That said, Redux is most commonly used with React. The React-Redux library is the official UI binding layer that lets React components interact with a Redux store by reading values from Redux state and dispatching actions. So, when most people refer to "Redux", they actually mean "using a Redux store and the React-Redux library together". React-Redux allows any React component in the application to talk to the Redux store. This is only possible because React-Redux uses Context internally. However, it's critical to note that React-Redux only passes down the Redux store instance via context, not the current state value!. This is actually an example of using Context for dependency injection, as mentioned above. We know that our Redux-connected React components need to talk to a Redux store, but we don't know or care which Redux store that is when we define the component. The actual Redux store is injected into the tree at runtime using the React-Redux <Provider> component. Because of this, React-Redux can also be used to avoid prop-drilling, specifically because React-Redux uses Context internally. Instead of explicitly putting a new value into a <MyContext.Provider> yourself, you can put that data into the Redux store and then access it anywhere. Purposes and Use Cases for (React-)Redux The primary reason to use Redux is captured in the description from the Redux docs: The patterns and tools provided by Redux make it easier to understand when, where, why, and how the state in your application is being updated, and how your application logic will behave when those changes occur. There are additional reasons why you might want to use Redux. "Avoiding prop-drilling" is one of those other reasons. Many people chose Redux early on specifically to let them avoid prop-drilling, because React's legacy context was broken and React-Redux worked correctly. Other valid reasons to use Redux include: Wanting to write your state management logic completely separate from the UI layer Sharing state management logic between different UI layers (such as an application that is being migrated from AngularJS to React) Using the power of Redux middleware to add additional logic when actions are dispatched Being able to persist portions of the Redux state Enabling bug reports that can be replayed by developers Faster debugging of logic and UI while in development Dan Abramov listed a number of these use cases when he wrote his post You Might Not Need Redux, all the way back in 2016. Why Context is Not "State Management" "State" is any data that describes the behavior of an application. We could divide that into categories like "server state", "communications state", and "location state" if we want to, but the key point is that there is data being stored, read, updated, and used. David Khourshid, author of the XState library and an expert on state machines, said: "State management is how state changes over time." Based on that, we can say that "state management" means having ways to: store an initial value read the current value update a value There's also typically a way to be notified when the current value has changed. React's useState and useReducer hooks are good example of state management. With both of those hooks, you can: store an initial value by calling the hook read the current value, also by calling the hook update the value by calling the supplied setState or dispatch function Know that the value has been updated because the component re-rendered Similarly, Redux and MobX are clearly state management as well: Redux stores an initial value by calling the root reducer, lets you read the current value with store.getState(), updates the value with store.dispatch(action), and notifies listeners that the store updated via store.subscribe(listener) MobX stores an initial value by assigning field values in a store class, lets you read the current value by accessing the store's fields, updates values by assigning to those fields, and notifies that changes happened via autorun() and computed() We can even say that server caching tools like React-Query, SWR, Apollo, and Urql fit the definition of "state management" - they store initial values based on the fetched data, return the current value via their hooks, allow updates via "server mutations", and notify of changes via re-rendering the component. React Context does not meet those criteria. Therefore, Context is not a "state management" tool! As we established earlier, Context does not "store" anything itself. The parent component that renders a <MyContext.Provider> is responsible for deciding what value is passed into the context, and that value typically is based on React component state. The actual "state management" is happening with the useState/useReducer hook. As David Khourshid also said: Context is how state (that exists somewhere already) is shared with other components. Context has little to do with state management. Or, as a recent tweet put it: I guess Context is more like hidden props than abstracted state. Think of it this way. We could have written the exact same useState/useReducer code, but prop-drilled the data and the update function down through the component tree. The actual behavior of the app would have been the same overall. All Context does for us is let us skip the prop-drilling. Comparing Context and Redux Let's review what capabilities Context and React+Redux actually have: Context Does not store or "manage" anything Only works in React components Passes down a single value, which could be anything (primitive, objects, classes, etc) Allows reading that single value Can be used to avoid prop-drilling Does show the current context value for both Provider and Consumer components in the React DevTools, but does not show any history of how that value changed over time Updates consuming components when the context value changes, but with no way to skip updates Does not include any mechanism for side effects - it's purely for rendering components React+Redux Stores and manages a single value (which is typically an object) Works with any UI, including outside of React components Allows reading that single value Can be used to avoid prop-drilling Can update the value via dispatching an action and running reducers Has DevTools that show the history of all dispatched actions and state changes over time Uses middleware to allow app code to trigger side effects Allows components to subscribe to store updates, extract specific pieces of the store state, and only re-render when those values change So, clearly these are very different tools with different capabilities. The only overlap between them, really, is "can be used to avoid prop-drilling". Context and useReducer One problem with the "Context vs Redux" discussions is that people often actually mean "I'm using useReducer to manage my state, and Context to pass down that value". But, they never state that explicitly - they just say "I'm using Context". That's a common cause of the confusion I see, and it's really unfortunate because it helps perpetuate the idea that Context "manages state". So, let's talk about the Context + useReducer combination specifically. Yes, Context + useReducer does look an awful lot like Redux + React-Redux. They both have: A stored value A reducer function dispatching of actions a way to pass down that value and read it in nested components However, there's still a number of very significant differences in the capabilities and behaviors of Context + useReducer vs those of Redux + React-Redux. I covered the key points in my posts React, Redux, and Context Behavior and A (Mostly) Complete Guide to React Rendering Behavior. Summarizing here: Context + useReducer relies on passing the current state value via Context. React-Redux passes the current Redux store instance via Context. That means that when useReducer produces a new state value, all components that are subscribed to that context will be forced to re-render, even if they only care about part of the data. This may lead to performances issues, depending on the size of the state value, how many components are subscribed to that data, and how often they re-render. With React-Redux, components can subscribe to specific pieces of the store state, and only re-render when those values change. In addition, there's some other important differences as well: Context + useReducer are React features, and therefore cannot be used outside of React. A Redux store is independent of any UI, and so it can be used separate from React. The React DevTools allow viewing the current context value, but not any of the historical values or changes over time. The Redux DevTools allow seeing all actions that were dispatched, the contents of each action, the state as it existed after each action was processed, and the diffs between each state over time. useReducer does not have middleware. You can do some side-effect-y things with useEffect in combination with useReducer, and I've even seen some attempts to wrap useReducer with something that resembles a middleware, but both of those are severely limited in comparison to the functionality and capabilities of Redux middleware. It's worth repeating what Sebastian Markbage (React core team architect) said about the uses for Context: My personal summary is that new context is ready to be used for low frequency unlikely updates (like locale/theme). It's also good to use it in the same way as old context was used. I.e. for static values and then propagate updates through subscriptions. It's not ready to be used as a replacement for all Flux-like state propagation. There's a lot of posts out there that recommend setting up multiple separate contexts for different chunks of state, both to cut down on unnecessary re-renders and to scope concerns. Some of those also suggest adding your own "context selector components", which require a mixture of React.memo(), useMemo(), and carefully splitting things up so there's two separate contexts for each segment of state (one for the data, and one for the updater functions). Sure, it's possible to write code that way, but at that point you're just reinventing React-Redux, poorly. So, even though Context + useReducer sorta-resemble Redux + React-Redux at a quick glance... they are not fully equivalent and cannot truly replace Redux! Choosing the Right Tool As I said earlier, it's critical to understand what problems a tool solves, and know what problems you have, in order to correctly choose the right tool to solve your problems. Use Case Summary Let's recap the use cases for each of these: Context: Passing down a value to nested components without prop-drilling useReducer Moderately complex React component state management using a reducer function Context + useReducer: Moderately complex React component state management using a reducer function, and passing that state value down to nested components without prop-drilling Redux Moderate to highly complex state management using reducer functions Traceability for when, why, and how state changed over time Wanting to write your state management logic completely separate from the UI layer Sharing state management logic between different UI layers Using the power of Redux middleware to add additional logic when actions are dispatched Being able to persist portions of the Redux state Enabling bug reports that can be replayed by developers Faster debugging of logic and UI while in development Redux + React-Redux All of the use cases for Redux, plus interacting with the Redux store in your React components Again, these are different tools that solve different problems! Recommendations So, how do you decide whether to use Context, Context + useReducer, or Redux + React-Redux? You need to determine which of these tools best matches the set of problems that you're trying to solve! If the only thing you need to do is avoid prop-drilling, then use Context If you've got some moderately complex React component state, or just really don't want to use an external library, go with Context + useReducer If you want better traceability of the changes to your state over time, need to ensure that only specific components re-render when the state changes, need more powerful capabilities for managing side effects, or have other similar problems, use Redux + React-Redux My personal opinion is that if you get past 2-3 state-related contexts in an application, you're re-inventing a weaker version of React-Redux and should just switch to using Redux. Another common concern is that "using Redux means too much 'boilerplate'". Those complaints are very outdated, as "modern Redux" is significantly easier to learn and use than what you may have seen before. Our official Redux Toolkit package eliminates those "boilerplate" concerns, and the React-Redux hooks API simplifies using Redux in your React components. As one user recently told me: We just switched from context and hooks over to RTK on one of our production application's frontends. That thing processes a little over $1B/year. Fantastic stuff in the toolkit. The RTK is the polish that helped me convince the rest of the teams to buy into the refactor. I also did a boilerplate analysis for that refactor and it's actually LESS boilerplate to use the RTK than it is to use the recommended dispatch pattern in contexts. Not to mention how much easier it is to process data. Yes, adding RTK and React-Redux as dependencies does add additional byte size to your application bundle over just Context + useReducer, because those are built in to React. But, the tradeoffs are worth it - better state traceability, simpler and more predictable logic, and improved component rendering performance. It's also important to point out that these are not mutually exclusive options - you can use Redux, Context, and useReducer together at the same time! We specifically encourage putting "global state" in Redux and "local state" in React components, and carefully deciding whether each piece of state should live in Redux or component state. So, you can use Redux for some state that's global, and useReducer + Context for some state that's more local, and Context by itself for some semi-static values, all at the same time in the same application. To be clear, I'm not saying that all apps should use Redux, or that Redux is always a better choice! There's many nuances to this discussion. I am saying that Redux is a valid choice, there are many reasons to choose Redux, and the tradeoffs for choosing Redux are a net win more often than many people think. And finally, Context and Redux are not the only tools to think about. There's many other tools out there that solve other aspects of state management in different ways. MobX is another widely used option that uses OOP and observables to automatically update data dependencies. Jotai, Recoil, and Zustand offer lighter-weight state update approaches. Data fetching libraries like React Query, SWR, Apollo, and Urql all provide abstractions that simplify common patterns for working with cached server state (and the upcoming "RTK Query" library will do the same for Redux Toolkit). Again, these are different tools, with different purposes and use cases, and are worth evaluating based on your use case. Final Thoughts I realize that this post won't stop the seemingly never-ending debate over "Context vs Redux?!?!?!?!?". There's too many people out there, too many conflicting ideas, and too much miscommunication and misinformation. Having said that, I hope that this post has clarified what these tools actually do, how they're different, and when you should actually consider using them. (And maybe, just maybe, some folks will read this article and not feel the need to post the same question that's been asked a million times already...) Further Information Context Purpose and Design React docs: Context React blog: v16.3 - new Context API Sebastian Markbage: use cases for Context David Khourshid: Context is not "state management" React Contexts are Dynamic Scope Redux Purpose and Design The Tao of Redux, Part 1 - Implementation and Intent Redux docs: Understanding Redux - Motivation Redux Fundamentals tutorial: Modern Redux with Redux Toolkit Differences between Redux and Context React, Redux, and Context Behavior A (Mostly) Complete Guide to React Rendering Behavior When should I use Redux? Redux - Not Dead Yet! When (and when not) to reach for Redux You Might Not Need Redux Redux FAQ: When should I use Redux? Other Redux and Context Comparison Discussions Valentino Gagliardi: React Context API is not a state management tool Dave Ceddia: React Context API vs Redux Mike Green: You Might Not Need Redux (But You Can’t Replace It With Hooks) Sergey Ryzhov: From Redux to Hooks: A Case Study Eric Elliott: Do React Hooks Replace Redux? Chris Achard: Can You Replace Redux with React Hooks? Denny Scott: Redux vs Context vs State - an in-depth look at state management in React Maximilian Schwarzmüller: Redux vs React's Context API Jakob Lind: When to use Redux, Context, and props

Presentations: Intro to React, Redux, and TypeScript (2020)

I've done several previous iterations of an "Intro to React + Redux" presentation. The last one I did was back in 2018, and a lot has changed for both React and Redux since then. In particular, React Hooks have changed how we write React code, Redux Toolkit is now the standard approach for writing Redux logic, and the React-Redux hooks API is now the default. I've updated this presentation to teach React hooks instead of classes, filled out additional examples, and added material on Redux Toolkit and the React-Redux hooks. I've also included my intro to TS section as well, and expanded that to include advice on using TS with React and Redux. Slides: Intro to React, Redux, and TypeScript (2020) I've also published a copy of the slides as a PDF, per request: PDF: Intro to React, Redux, and TypeScript (2020)

Presentations: Podcast Appearances in 2020

Intro Due to the COVID pandemic canceling live conferences, I was unable to give any actual conference talks this year. (I had been scheduled to speak a conference in Japan in March, and one in NYC in July. Both were obviously canceled.) However, I was invited to be a guest on several different podcasts this year, so I want to round up those appearances for easy reference. I'll link them chronologically. July: Modern Web Podcast In mid July, Tracy Lee invited me to be a guest on the Modern Web Podcast, along with her co-host Ben Lesh. We covered everyone's favorite topic, "Is Redux Dead?", discussed the creation of Redux Toolkit, reviewed the history of React-Redux, and looked at the differences between NgRx and Redux: Modern Web Podcast S07E3: Is Redux Dead Yet? (No!) September: Newline Podcast Around the same time, I was a guest on the Newline Podcast with Amelia Wattenberger and Nate Murray. We hit some of the same topics, like the history of Redux, use cases for Redux, when and why to choose a particular tool, comparison with Context and Apollo, and the benefits of Redux Toolkit. The episode wasn't published until the end of September: Newline Podcast E8: Redux in 2020 October: JS Party In late September, Amal Hussein and Jerod Santo had me as a guest on the JS Party podcast. This discussion was a blast. Amal is a big fan of my work for the community and a constant cheerleader for my efforts, which I really appreciate. In fact, in this podcast, she dubbed me "Tech Support for the React Community" :) We chatted about how I keep up with answering questions, the history of Redux, why Redux Toolkit was invented and how some of its APIs help reduce boilerplate, and another comparison with Context, MobX, and Apollo. The episode was published a week later: JS Party #146: Redux is Not Dead The episode page has a full transcript, and the JS Party folks later turned an excerpt from that into an excellent standalone blog post: When (and when not) to reach for Redux October: Maintainable Software Podcast Finally, in mid October, Robby Russell invited me on the Maintainable Software Podcast. This discussion was a nice change of pace - while we did discuss a few things that touched on Redux, the discussion was more about codebase structures, maintainability, documentation, and techniques for migrations and interop instead of the Redux library itself. This was a nice change of pace from the usual discussions about Redux and RTK. Maintainable Software E78: Mark Erikson - Accidentally Becoming an Open Source Maintainer

Coding Career Advice: Searching and Evaluating Online Information Efficiently

Tip #3: Searching and Evaluating Online Information Efficiently A while back I saw this excellent quote on Twitter: When I learn something today, I usually do it by piecing together clues gleaned from out of date tutorials and random forum posts until a coherent picture forms, and that is a skill you should practice relentlessly because it's the most valuable skill you will ever acquire. I attribute much of my success as a developer to my ability to search for information online quickly and efficiently, absorb key pieces of info from the relevant search results, and then synthesize that information into a potential solution. To be honest, this is a really hard topic for me to write about, because it's something I've been doing my whole online life. It's kind of like trying to describe "How to learn and speak English" as a native speaker. It's not something I can exactly describe based on learning it, because it's a skill I naturally developed back during the dawn of the Web. It's also not something I think about, because by now it's entirely instinctive. Similarly, I can't go tell anyone "start using the internet back in 1995", or "spend most of your childhood learning to speed-read" - clearly that doesn't scale or actually help anyone else :) That said, I firmly believe that developing a strong ability to search for and evaluate information online is a key skill that every developer should aim for. I actually already described a form of this in my earlier post on evaluating software libraries and tools. Here's my suggestions for a general set of steps to follow. Suggested Search and Evaluation Process Know what it is you're trying to find. Obviously you don't know the exact details yet, otherwise you wouldn't be looking :) But, think of it as drawing an outline, and then trying to find a picture that matches that outline. The better idea you have up front of what you're looking for, the easier it is to look at a result and say "this does or does not match what I'm looking for" Know where to look for relevant information. Google is the typical starting point, but I'll use the search feature on other sites for specific things: NPM for JS libraries, Twitter for tweets, Github for code repos or usage snippets, and so on. Use specific phrases and keywords as part of your search. You might not even know what the relevant keywords are to begin with, but after opening up a few initial results, you might find some info that's in the same general category as what you're looking for. Start up a new search using specific relevant phrases you just found, and keep narrowing the results down based on those. (I'll throw an extra-special shout out to Shawn Swyx Wang's post on "How to Google Your Error Messages", which both has some excellent advice for a common problem and tweet-quotes me in the process.) Learn to skim search results quickly and do an initial filtering pass based on likely relevance. This includes looking at page titles or software library names, "last updated at" dates, URLs, and anything else in the actual search results list that can give you a sense of whether a given result is likely to be relevant and trustworthy. Learn to immediately disqualify search results as "too old", "likely spam or clickbait", or "just not relevant" as you move your eyes down the results list. Pop open potential candidates in a new tab for later review, and keep skimming results. Once you've got an initial set of potentially useful results opened up, skim the pages and get a sense of what the actual content is and whether it's worth looking at further. Consider the source (blog, docs site, tutorial), date, author, length, apparent level of detail, and overall content. If it doesn't look relevant, close the tab and move on to the next one. The emphasis here is on trying to figure out which of these results should be kept for the the real deeper reading and understanding portion of the process. Now that you have a good list of likely meaningful results, go back through them in more detail. You might want to take notes on your findings, bookmark some of the results and save them for later, or start up new searches based on some of the keywords you find. Finally, you should have accumulated a good set of info from across all these results. Maybe none of those pieces of info directly solves your problem by itself, but it's possible to take some of the info as a basis for putting together your own solution. Think of it as lego blocks - maybe you didn't find a complete pre-assembled kit, but you've got enough blocks that you can build something yourself. Note that these steps aren't limited to Google searches. They apply as you look at just about any other search site or tool, even if you're looking for something like a software library on NPM. Additionally, one of the main reasons why I still use Firefox as my primary browser is the Tree Style Tabs addon. It lets you switch from a row of tabs on top of the browser, to a sidebar that has a tree of tabs that can be nested and collapsed at any level. This makes it much easier to have a large number of tabs open at once, including grouping results from different searches based on the starting page. (My understanding is that Chrome addons can't do the same kind of UI changes that Firefox addons can, and last I checked there wasn't a good equivalent of Tree Style Tabs for Chrome. Could easily have changed since the last time I looked, though.) Examples I'll briefly describe some specific cases where I went through this process myself. Git History Rewriting In my post Rewriting Git History and JS Source for Fun and Profit, I described how I completely migrated our team's Git repository history, by stripping out unused files and rewriting the JS source to modernize it at all points in the history of the codebase. When I started this task, I had some prior experience with rewriting Git history. I frequently used commands like rebase in my daily work, and I had used git filter-branch a few times. I also generally understood how Git's data structures worked, and that tools like JS codemods existed. So, I understood the general shape of what I wanted to do ("create an alternate Git history but with changed content", "rewrite JS source automatically on a large scale") and key terms that I needed to look for ("git filter-branch", "rewrite Git history", "JS codemod"). I also assumed that what I wanted to do must be possible, somehow - I just didn't know the exact technical details of how. I probably spent at least a week doing initial research on this. I didn't find an exact solution, but I did find several critical resources that gave me much deeper understanding, key terms to look for, and ideas that shaped my potential solution: Understanding Git Filter-branch and the Git Storage Model: helped clarify my understanding of how filter-branch works and how Git stores its data Large scale Git history rewrites: gave me the idea for doing some of this work in Python A tale of three filter-branches: described a rewrite process, and linked to a Ruby script that did low-level Git repo manipulation to speed up the rewrite work sergelevin/pylter-branch: a Python lib that builds on pygit2 to provide an abstraction for iterating over Git history and rewriting it None of those pieces by itself was the entire solution, but together they gave me enough understanding to figure out how to build the actual solution I needed. By the time I was done building this, I had learned enough to write a 6000-word blog post on the topic, off the top of my head, in about 5 hours :) Searching for Info on How to Search I started this evening by searching online and evaluating info on how to search online and evaluate info efficiently. (How meta!) My first search was for how to search for information on the internet. In a sense I didn't follow my own advice here, because that's more of a question or a phrase than specific keywords, but I didn't quite know where to start looking for details on this topic. Most of these results were either blogspam about search engine optimization, or too narrowly focused on how to actually type things into a search engine rather than how to look at the results. I also avoided any results with dates older than the last couple years, on the grounds that they probably weren't as relevant. That initial search turned into effectively evaluating search results, which produced some better hits. I also saw the phrase "digital literacy", realized that was a relevant aspect of this topic, and searched for digital literacy searching for information. These queries gave me some better results, including a number of academic-related pages like university libraries. Ultimately, I didn't end up using those resources to write this post, but I have linked the best results in the "Further Information" section below. React-Redux, React Native, and Jest Last year I fixed a bug in React-Redux where it broke under React Native. As part of that, I needed to update our test suite to make it correctly test behavior with React Native, vs just ReactDOM. I remember having seen a tweet that day about "all the stuff I googled in a week as a software developer", so for fun I wrote down every Google search I did while trying to solve this particular React Native test setup problem. This included searches like react uselayouteffect ssr to find a particular issue thread I knew existed in the React repo, jest mockComponent.js Support for the experimental syntax 'classProperties' isn't currently enabled to find a solution to a particular error message I was seeing, and microsoft ipod naming as a sidebar based on a discussion with a friend, to find the classic Microsoft self-parody video on how they're so bad at naming things. Further Information Seven Ways to Find What You Want On the Internet Curtin University Library - Effective Internet Searching Girton College - Planning a search strategy and picking a search engine University of Exeter - Effective Search Techniques How to Evaluate Websites: A Guide for Teachers and Students Stark State Library - How to Evaluate Websites BYU Library - Step by Step Research Strategy Guide Webwise - Digital Literacy: Ten Steps to Better Web Research Media Literacy Starts with SEARCHing the Internet How to Google Your Error Messages How do you learn programming topics?

How Web Apps Work: AJAX, APIs, and Data Transfer

Web development is a huge field with a vast array of concepts, terms, tools, and technologies. For people just getting started in web dev, this landscape is often bewildering - it's unclear what most of these pieces are, much less how they fit together. This series provides an overview of fundamental web dev concepts and technologies, what these pieces are, why they're needed, and how they relate to each other. It's not a completely exhaustive reference to everything in web development, nor is it a "how to build apps" guide. Instead, it's a map of the territory, intended to give you a sense of what the landscape looks like, and enough information that you can go research these terms and topics in more depth if needed. Some of the descriptions will be more oriented towards modern client-side app development with JavaScript, but most of the topics in the series are fundamental enough that they apply to server-centric applications as well. Other posts in this series cover additional topics, such as: HTTP and Servers Client Development and Deployment Browsers, HTML, and CSS JavaScript and the DOM New terms will be marked in italics. I'll link references for some of them, but encourage you to search for definitions yourself. Also, some of the descriptions will be simplified to avoid taking up too much space or dealing with edge cases. This post in particular does not attempt to be a "how to program" tutorial or complete reference to JS, but will point out common gotchas and differences from other languages like Java, C++, and Python. The MDN JavaScript docs have a complete set of resources on everything related to JavaScript, AJAX, and browser functionality The Modern JavaScript Tutorial has information on built-in AJAX APIs and browser storage. My JavaScript for Java Developers slides also cover much of this post's content as well, showing examples of JS syntax and concepts in a cheatsheet-type format. I'll link relevant sections of these slides throughout this post rather than copy entire large code blocks. Table of Contents JSON JSON Overview Working with JSON AJAX AJAX APIs and Libraries XMLHttpRequest fetch Axios jQuery Other Request Libraries HTTP Request Variations Polling Long Polling HTTP Streaming Server Sent Events Websockets CORS Data Transfer Protocols REST RPC GraphQL Browser Storage Cookies localStorage sessionStorage indexedDB Client URL Routing Further Resources JSON JSON Overview JSON is the most widely used data transfer format for web apps. JSON is short for "JavaScript Object Notation", and is an adaptation of JS object/array syntax for use as a data format. All major programming languages have built-in support for reading and writing JSON data, but it naturally fits into JavaScript. JSON has become the de-facto standard data transfer format for HTTP-based APIs, and some databases even support storing and querying JSON document contents. (slides: JSON) JSON syntax consists of normal JS objects and array definitions, including primitives like numbers, strings, booleans, and null. However, JSON does have some additional rules and restrictions compared to normal JS syntax: No variables No comments All object keys must be quoted with double-quotes The final key in objects and the final item in arrays may not have a trailing comma The top level of JSON data may be either an object or an array (slides: (slides: JSON syntax examples)) A typical JSON document for a list of todo objects might look like: [ {"id": 0, "text": "Buy milk", "completed": false, "colors": ["red"]}, {"id": 1, "text": "Clean yard", "completed": true, "colors": []}, {"id": 2, "text": "Read book", "completed": false, "colors": ["red", "blue"]} ] Because JSON is a subset of JS syntax, it lacks several things that normal JS has. The inability to write comments is a frequently cited pain point. It also doesn't have native support for date values, so they have to be serialized as strings. You can write numbers in JSON of any size, but because JavaScript itself is limited to 64-bit floating point numbers, in practice that also limits the size of numbers you can pass via JSON. Working with JSON JSON content must be serialized by converting it into a single string for transfer, and then deserialized back into actual objects after it has been received. The JS language supports this with a pair of built-in functions: JSON.stringify(value) will convert any valid value (object, array, primitive) into a complete string. JSON.parse(string) will deserialize a string of JSON text back into its corresponding values. When sending JSON content over HTTP, the request or response should use the header content-type: application/json. Most server frameworks have shortcuts built in for returning a JSON-formatted response. These methods often allow returning an object or array directly, and the framework takes care of serializing the data into JSON format and setting the right HTTP headers. For example, the Express server framework for Node has res.json(obj). AJAX AJAX is a term used to describe making HTTP requests, via JavaScript, in the background of a page, without reloading it. It's an acronym that originally stood for "Asynchronous JavaScript And XML". At the time the phrase was created, XML was the standard data transfer format used for most requests. Since then, JSON has completely replaced XML as the standard data transfer format. Today, the term "AJAX request" normally refers to fetching data in JSON format from a web server over HTTP. Browsers have a couple built-in APIs that are used for making AJAX requests: XMLHttpRequest (XHR) and fetch. XHR is generally considered to be a clunky, low-level, and hard to use API design. fetch is intended to be a more modern replacement, but still has some noticeable annoyances in its design. Because of those, most apps use a third-party library that wraps one of those two built-in methods to provide an alternate API that is easier to use. Note: The term "API" always means "other code that you call to get something done", but depending on context, it can mean "an actual function you call in your code", or "a remote request to a server to fetch some data". In the "AJAX" section, "API" refers to actual functions you call in your JS code. In the Data Transfer Protocols section below, it refers to remote network requests to fetch data. Browsers can cache some HTTP response contents. This is most common with static files like HTML, CSS, and images, but some HTTP requests can be cached as well. This is most likely to happen if the HTTP request is using the HTTP GET method. Several HTTP headers also have an effect on caching as well. AJAX APIs and Libraries AJAX API usage typically falls into three categories: The built-in XMLHttpRequest and fetch APIs Third-party wrapper libraries like axios and jQuery that wrap the core APIs Higher-level data fetching abstractions that use the other APIs internally There are many third-party AJAX wrapper libraries, so we'll briefly look at a couple of the most popular options here. XMLHttpRequest XMLHttpRequest, typically referred to as "XHR", is the original browser API for making AJAX requests. It was introduced by Internet Explorer around 1999, and added by other browsers around over the next few years. Despite its name, it can be used to transfer any kind of content over HTTP, not just XML. XHR's API is based on creating an XHR object instance, and assigning callbacks to it for various request lifeycle events. That approach is notoriously hard to work with. (slides: XHR examples) XHR is supported in all browsers, and in Node.js. Because it's available everywhere and has been around longer, it's often used as the core implementation of other AJAX libraries. fetch The fetch API is a newer built-in browser API, meant to be a replacement for XHR. It has a nicer syntax, and uses Promises for handling the responses. (slides: fetch examples) A typical fetch request might look like: fetch("http://example.com/api/users") .then(response => response.json()) .then(users => { // do something with users data }); However, fetch has some behaviors that frequently surprise or annoy developers: fetch only returns a rejected promise if the actual network request failed completely. As long as the server returns a response, fetch will return a successfully resolved promise, even if the HTTP status code was a 4xx or 5xx HTTP error. The initial fetch('/some/url') call returns a promise containing a Response object. That object is not the actual serialized data from the server. Instead, a second call is required to ask the response to return the request body content in a specific format, such as JSON. fetch does not send cookies to the server by default - you must include a credentials: 'same-origin' option fetch does not automatically stringify objects or set headers when sending JSON - you must do those steps yourself Also, fetch is not supported in Internet Explorer or Node.js. Because of this, it's very common to use a library that wraps XHR or fetch. For cases where you want to use fetch but it's not available in your target environment, there are polyfills to add a reimplementation of fetch in environments that don't support it, such as isomorphic-fetch. Axios Axios is a widely popular AJAX wrapper library. Its API design is based on the "Resource" classes that were part of AngularJS 1.x, particularly the structure of its response objects. Axios makes common tasks fairly easy. The default axios object has .get() and .post() methods, which return promises. The .post() method accepts a data value as an argument, and if provided, automatically stringifies that into JSON and adds headers to the request. Axios response objects automatically deserialize JSON content, and the body content is always available in response.data. A typical request might look like: axios.get("http://www.example.com/api/users") .then(response => { // do something with response.data }) Because it uses XHR internally, it works across all browsers as well as Node. Axios also provides some powerful capabilities like "interceptors", which allow modification of requests and responses. This is often used for adding auth headers or transforming data. jQuery jQuery was historically used to enable powerful DOM manipulations, while smoothing over DOM API differences between browsers. jQuery also includes an AJAX wrapper, $.ajax(). Given the difficulties in working with the plain XHR API, jQuery's AJAX wrapper was another major reason for its popularity. (slides: jQuery AJAX) The jQuery AJAX API is nicer to work with than XHR, but definitely still shows that it was designed in an earlier era of JS usage. It primarily takes success and error callbacks, rather than working with promises. It does have the ability to handle responses by returning a precursor to Promises, called a "deferred", and can automatically deserialize JSON data. If you're working on a simple site and already are using jQuery for other things, it's reasonable to use jQuery to make AJAX calls. Otherwise, prefer using a modern option like fetch or axios. Definitely do not add jQuery to a site just to make AJAX calls. Other Request Libraries Many tools and frameworks come with their own HTTP request wrappers. For example, AngularJS 1.x had $http for general requests and a higher-level $resource class for working with REST APIs, while modern Angular has an HttpClient service. Since the fetch API has several annoyances in its design, many users opt to write their own fetch wrapper to customize the behavior, typically by auto-converting JSON data and rejecting promises on HTTP 4xx/5xx responses. Alternately, redaxios is a reimplementation of much of the axios API as a small wrapper around fetch. HTTP Request Variations HTTP is inherently a request/response protocol. That means that a server can never initiate sending a message to a client by itself. However, there are many cases when it's necessary for a server to send data to a client based on an event that occurred in the server. Over the years, the community has come up with several workarounds. The modern solution to this problem is Websockets, but it's useful to know about these other techniques as well. Polling Polling is the simplest option to implement. The client continually makes requests to the server every few seconds, on a timer. However, this is usually inefficient - most of the time the server won't have anything new to report, so the new request is a waste of time and bandwidth. Long Polling Long polling is a variation on the polling approach. Instead of continually making requests, the client opens one request to the server. If the server doesn't have any messages to report, it sleeps the request. The client still has the connection open, and continues to wait for a response from the server. Eventually, the server either has something new to report, or a timer expires. At that point, it wakes up the request, and finally sends back a response to the original open request connection. The client then immediately makes another request, and the server again either has something to send back immediately, or sleeps on the request. This approach requires a bit of coordination inside the server, but is more bandwidth-efficient. HTTP Streaming Once an HTTP connection is opened by the client, the server may continue to send data over the open connection without closing out the response, leaving it open indefinitely. Server Sent Events Server-Sent Events are a specific form of HTTP streaming that is actually part of the HTTP spec. The client creates an EventSource object and adds an event listener callback, and the server writes specifically formatted content to the response over time. Websockets All of those HTTP-based approaches to 2-way message and data transfer have distinct limitations. Websockets were created to address those limitations. Websockets allow opening a persistent connection between a client and a server, where either side may send messages over the connection at any time. (slides: websockets) A websocket starts out as a standard HTTP request, but the client adds a header telling the server it wants to "upgrade" the connection to a websocket. A websocket is similar conceptually to an actual OS-level networking socket, and like all HTTP requests, the data is sent over an actual socket. But, it's a higher-level abstraction, and is much more limited API-wise compared to a low-level OS socket. Once created, either side can add event listeners to process received messages, and send messages over the connection. Messages can contain text or binary content. Today, websockets are the standard approach used to provide live updates to a page after it's been updated, such as updates to sports scores. CORS HTTP requests often involve security and authorization concerns. As part of this, browsers specifically implement restrictions such as the Same-Origin Policy, so that scripts can only interact with resources loaded from the exact same server URL. Additionally, a server may only want to allow interactions with its own scripts and client code that it has served, instead of requests from any arbitrary site or client. Browsers enforce this via Cross Origin Resource Sharing (CORS). By default, an AJAX request to any URL other than the original URL for a host page will cause the browser to make a pre-flight request to ask the server "are you okay with other sites making data requests to you?". The server may then respond with a set of URL patterns that are allowed to successfully make requests, or a value indicating "all requests are okay". If the current client matches that pattern, the browser will actually make the real request. CORS is a frequent source of confusion for developers who are just trying to make a request and are surprised when the browser blocks it. Note that only browsers implement CORS - HTTP requests made outside a browser, such as from another server or another non-browser environments do not enforce CORS. This can also be confusing - "it worked in my CLI tool / API test client, why doesn't it work in the browser?". Data Transfer Protocols HTTP provides the standard mechanism for making some kind of request and response to a server. However, there are many ways that a server may structure its URLs and the expected format of requests and responses. In order to write code that makes requests to a server, you must know what structure and request formats that server expects your client to use. There are numerous variations on how server data APIs are structured, but there are a few common categories of API structures you'll see frequently. (slides: AJAX API design approaches) With all these approaches, the server defines specific endpoints - combinations of URLs, HTTP methods, and expected request/response formats. REST REST stands for "Representational State Transfer". A REST API is an HTTP-based API that uses URLs and HTTP methods as the primary approach for determining what kind of request the client is making, and how the server should handle that request. REST APIs are often used for CRUD (Create/Retrieve/Update/Delete) apps, where the app is conceptually doing operations to update data in a database on the server ("create this thing", "give me a list of these things", "update that thing", "delete that thing"). There are frequent arguments over what exactly constitutes a true REST API, but in general, a REST API: Offers different URLs for each kind of "resource" that it supports Expects different HTTP methods to be used for different kinds of operations on the same resource URL. Typically, HTTP POST/GET/PUT/DELETE map to relevant CRUD operations. Returns different HTTP status codes to indicate success or failure of the request Expects key pieces of data like item IDs to be included as part of the actual URL Example REST API usage might look like: Get list of users: GET /users Get one user: GET /users/42 Create one user: POST /users/42 (body: {name : "Mark"}) Update user: PUT /users/42 (body: {name : "Mark"}) Many server web frameworks have built-in support for defining a set of REST API endpoints based on metadata. RPC RPC stands for "Remote Procedure Call". This is a general style of network request that is intended to mimic a normal function call in your code, but with the actual logic running on some other server. There have been many RPC tools over the years for many languages. Here, the focus is on RPC-style HTTP requests. An RPC-style approach describes "methods" to be "called", instead of "resources" to be "operated on". Unlike a REST API, an RPC-style API will likely define URLs whose paths look like verbs or function names instead of nouns. For example, an HTTP-based RPC server approach might look like: Get a list of users: GET /getUsers Get one user: POST /getUser (body: {userId : 42}) Create one user: POST /createUser (body: {userId : 42, name : "Mark"}) Update one user: POST /updateUser (body: {userId : 42, name : "Mark"}) Alternately, there might be a single URL endpoint, and the request could have the name of the "method to call" in the body. The JSON-RPC spec uses this approach: POST /jsonrpc (body: {"method": "createUser", userId : 42, name : "Mark"}) Either way, the response is likely to have an HTTP 200 status code indicating that the server returned a response successfully, but the contents of the response might have some kind of a field indicating whether the actual operation succeeded, like {status: 'failed'}. GraphQL GraphQL is a relatively recent data transfer protocol created by Facebook. Conceptually, GraphQL is a defined format for API request and response content. In addition, there is an ecosystem of client and server tools that are typically used that abstract the details of making a GraphQL-formatted query, processing the request on the server and fetching requested data, formatting the response correctly, and caching the data on the client. Most discussions of "using GraphQL" assume use of the most popular libraries and tools for working with GraphQL. With GraphQL, the server offers a single URL endpoint, and defines schemas that describe what data types it supports and how they relate to each other. The client sends a GraphQL query request that asks for some subset of those data types, and defines the expected structure of the response format. The server then resolves the requested data types, extracts the requested fields for each type, formats the response as requested by the client, and sends the data back. The server may also support mutations, which allow the client to create/update/delete data, and subscriptions for changes to query results over time. GraphQL requests are almost always HTTP POSTs, and subscriptions are typically built on top of websockets. GraphQL query syntax looks somewhat like JSON, but without commas or values: { hero { name friends { name } } } which might result in this JSON response from the server: { "data": { "hero": { "name": "R2-D2", "friends": [ {"name": "Luke Skywalker"}, {"name": "Han Solo"}, {"name": "Leia Organa"} ] } } } GraphQL moves a lot of the query complexity to the server, and gives clients a lot of flexibility in what data they want to ask for. This can simplify cases where different portions of an app need to fetch different subsets of the same data and reduce the need to make numerous calls to fetch related values. On the other hand, it can be more difficult to deal with things like auth and business logic as part of the data resolution on the server, and the most popular GraphQL clients are very heavyweight. Browser Storage In addition to all these network requests for data fetching, browsers also offer several tools for storing data within the browser itself. These are typically used to manage data related to user sessions and preferences, but can be used for other scenarios as well. Cookies Browsers allow servers to set cookies - small pieces of text associated with a given URL. Any cookies set by a site will then be included on future requests to that site. Cookies are typically limited to a couple KB in length, and are most often used to store a unique session ID that can be read by the server to retrieve additional session data from a database or in memory. Cookies can have expiration timestamps set by the server. Cookies can be a source of security issues. A server can add an HttpOnly attribute when setting a cookie to ensure that JS code cannot read or modify that cookie. localStorage localStorage is a browser API that lets client code save key/value string data, then retrieve it later. localStorage persists indefinitely, so it's often used to save user settings so they can be reloaded the next time the user visits a page. An example of this might look like localStorage.setItem('userPrefs', JSON.stringify(userData)), and then reversing the process when a page is loaded. Data in localStorage can only be read by code from the same origin, and is limited to about 2MB in size. sessionStorage sessionStorage has the same basic API methods as localStorage, but is isolated on a per-tab basis. It will persist between tab reloads, but is cleared when the user closes the tab. indexedDB A more powerful database-style storage system that can hold larger amounts of data. Client URL Routing In HTTP and Servers, we talked about the idea of routing: determing application behavior based on the URL. For application servers, routing means looking at a URL and HTTP method, and choosing the right application handler logic to process that request. Client-side applications can do their own form of routing. Browsers offer access to the current URL via window.location, and the browser's URL history via a history object. Client applications can parse the URL and use that as the basis for dynamically showing and hiding portions of the UI, without needing to make an additional request to the server. Client-side routers typically offer special forms of a link tag that handle clicks by calling history.push('/new-route') to alter the URL, and then read the updated route to switch to showing a different piece of UI. However, client routing does add additional complications. If the browser tries to reload a client-routed URL by itself, the server now has to handle that URL somehow with a valid response. Typically, the server is configured so that if a URL like /some-route is not recognized, it returns the entire index page with the full JS bundle as the response, just as if the browser requested the / URL. Once the content is loaded, the client router kicks in, sees that /some-route is the active URL, and immediately shows the relevant content on the client side. Further Resources JSON MDN: Working with JSON What is JSON? AJAX Wikipedia: AJAX MDN: AJAX guide The Modern JS Tutorial: Network Requests Working with the fetch API How to connect to an API with JavaScript HTTP Request Variations Long Polling - Concepts and Considerations Long Polling vs Server Sent Events Using Server Sent Events to Power a Live UI Websockets Wikipedia: Websockets Websockets: A Conceptual Deep Dive Websockets vs Long Polling CORS Understanding CORS CORS Tutorial: A Guide to Cross-Origin Resource Sharing Data Transfer Protocols Understanding RPC vs REST for HTTP APIs Understanding RPC, REST, and GraphQL Comparing REST and RPC APIs Stack Overflow: What is the difference between REST and RPC web services? API Paradigms - which and when to use them API Developers Never REST REST API Tutorial GraphQL.org JSON-RPC.org Browser Storage A Primer on the Different Types of Browser Storage MDN: Cookies The Modern JS Tutorial: Storing data in the browser Client Routing MDN: History API Deep dive into client-side routing Understanding client side routing by implementing a router in vanilla JS

How Web Apps Work: JavaScript and the DOM

Web development is a huge field with a vast array of concepts, terms, tools, and technologies. For people just getting started in web dev, this landscape is often bewildering - it's unclear what most of these pieces are, much less how they fit together. This series provides an overview of fundamental web dev concepts and technologies, what these pieces are, why they're needed, and how they relate to each other. It's not a completely exhaustive reference to everything in web development, nor is it a "how to build apps" guide. Instead, it's a map of the territory, intended to give you a sense of what the landscape looks like, and enough information that you can go research these terms and topics in more depth if needed. Some of the descriptions will be more oriented towards modern client-side app development with JavaScript, but most of the topics in the series are fundamental enough that they apply to server-centric applications as well. Other posts in this series cover additional topics, such as: HTTP and Servers Client Development and Deployment Browsers, HTML, and CSS AJAX, APIs, and Data Transfer New terms will be marked in italics. I'll link references for some of them, but encourage you to search for definitions yourself. Also, some of the descriptions will be simplified to avoid taking up too much space or dealing with edge cases. This post in particular does not attempt to be a "how to program" tutorial or complete reference to JS, but will point out common gotchas and differences from other languages like Java, C++, and Python. The MDN JavaScript docs have a complete set of resources on JavaScript ranging from intro-level tutorials to detailed API references. FreeCodeCamp has multiple courses that cover JavaScript usage as well as other aspects of web dev. The Modern JavaScript Tutorial is an excellent and thorough set of explanations of JS syntax and DOM APIs. My JavaScript for Java Developers slides also cover much of this post's content as well, showing examples of JS syntax and concepts in a cheatsheet-type format. I'll link relevant sections of these slides throughout this post rather than copy entire large code blocks. Table of Contents JavaScript Overview Language Evolution Runtime Environments JS Development Constraints Core Language Basic Syntax Data Types Conditional Logic Comparisons and Boolean Conversions Functions JS in Depth Objects Arrays Scoping and Closures Functions and this Classes and Prototypes Async Timers and the Event Loop Promises async/await Syntax for Promises Other Topics JS Module Formats Regular Expxressions Immutability Lodash DOM DOM Query APIs DOM Manipulation Event Listeners jQuery Further Resources JavaScript Overview JavaScript is a dynamic interpreted language that is primarily used in web browsers, but can also be used outside a browser environment. It was created in the mid-1990s by Brendan Eich at Netscape, and takes inspiration from the Java, Scheme, and Self languages. Since then, it has grown from a tiny scripting language to the world's most popular language. It's definitely not a "toy" - it can be used to write anything from small scripts to HTTP server applications to complex client-side web apps with hundreds of thousands of lines of code. JS is multi-paradigm - it can be used for Object-Oriented Programming similar to Java or C#, but can also be used in a Functional Programming style as well. Note: Despite its name, JavaScript has no actual relation to the Java language! Some aspects of JS's syntax were indeed borrowed from Java, but they are completely different languages. (slides: Java vs JavaScript comparison table) Language Evolution To understand modern JS, it helps to understand how the language has evolved over time. (slides: JS language timeline) There's a neat timeline of milestones in JS history that helps visualize some of the key events and changes over time. JS was standardized early on by the ECMA standards organization. Since "JavaScript" was trademarked, the standardized name is "ECMAScript", and the specification is often referred to as the "ES Spec". Early revisions were numbered: "ES2", "ES3", etc. The ES spec is maintained by a committee known as TC39. The ES spec has had several new revisions published over time, which added new language syntax and features. However, the ES4 spec revision grew so large that it was abandoned in 2003 due to disagreements over potential features. As a result, when the ES5 spec came out in 2009, it only added some relatively minor changes to the language. The next major spec revision, ES6, didn't come out until 2015. Because of the long wait between revisions and the rapid growth in JS popularity and usage, the ES6 spec was huge and doubled the amount of syntax and features in the language. You can roughly divide JS history and syntax into "before ES6" and "after ES6". All the old syntax still works, but ES6 drastically changed how JS developers write code. After the ES6 spec was finished, the TC39 committee changed how they design changes to the JS language. Now, individual features are proposed and go through a series of stages over time (proposal, early implementation, completed implementation, finalized). New spec revisions are published yearly, and any new features finalized since the last revision are added to the new spec. Revisions are now known by year: ES2016, ES2017, and so on. (slides: TC39 stages and process) To give a sense of how these revisions have changed the language over time, here's an incomplete list of new features by revision: ES3 (1999): regular expressions; try/catch blocks and exceptions ES5 (2009): object and array methods; JSON parsing; function binding; syntax cleanup ES6 (2015): let/const variable declarations; arrow functions; classes; object literal shorthand; template strings; promises; generators; default function arguments; array spread syntax; module syntax; proxies ES2016: Array.includes(), ** exponent operator ES2017: async/await functions; Object.values() / Object.entries(); trailing commas in function args ES2018: async iteration; object rest/spread operators; Promise.finally() As you can see, ES6 was huge, while ES2016 was tiny. Since then, the rate of new features has been slower, but steady. (slides: ES spec revisions and features) Runtime Environments JavaScript was originally invented to give web browsers the ability to have interactive logic running inside a web page. By the mid 2000s, all browsers included a JS interpreter, and developers were starting to write larger client-side applications in JS. In 2009, Ryan Dahl announced Node.js - a JS runtime for executing JS outside of a browser environment. Node is built on top of Chrome's V8 JS engine, and adds a standard library of APIs for things like working with the local filesystem, network sockets, and more. Node also popularized the CommonJS module format and the NPM package management tool. Today, JS is widely used in multiple environments for many kinds of projects. The primary use cases are interactivity in web pages, full web app clients, web app servers, and build tools. So, some JS code is meant to run exclusively in a browser, some is meant to run exclusively under Node, and some code can work in both environments. JS Development Constraints The other key factor to understand in how JS has evolved and how it's used is the set of unique constraints web developers face. There's a great quote from one of the early browser developers that encapsulates this: The by-design purpose of JavaScript was to make the monkey dance when you moused over it. Scripts were often a single line. We considered ten line scripts to be pretty normal, hundred line scripts to be huge, and thousand line scripts were unheard of. The language was absolutely not designed for programming in the large, and our implementation decisions, performance targets, and so on, were based on that assumption. http://programmers.stackexchange.com/a/221658/214387 Compared to many other languages, JS is very different: No built-in module definition system (until ES6) No built-in private / public encapsulation Prototypal-based inheritance system unlike most languages No static type declarations or compilation Dynamically modified objects and data Minimal standard library Variations in browser capabilities Entire codebase has to be shipped to the browser every time a page is loaded, then parsed and executed Because of this, web client developers need to: Minimize bytes sent over the wire Handle browser compatibility issues Fill in gaps in the JS standard library and language spec Reuse and share code between apps Build increasingly complex full-blown applications that just happen to live inside a browser Core Language Basic Syntax JS uses much of the typical syntax seen in the C family of languages (C, C++, Java, and C#). Statements end with semicolons, variable names are case-sensitive, blocks are denoted with curly braces, comments can be written with // for a single line and /* */ for multiple lines, conditional logic uses if/else, and there are variations of for and while for looping. Semicolons are actually optional - the interpreter will automatically insert them in places where they're strictly needed. That said, there's an ongoing argument between people who prefer using semis and those who avoid them. (Personally, I'm in favor of always using semicolons.) The console.log() statement is the standard method for printing to the screen or the browser's debugging console. It accepts multiple arguments, including strings, objects, arrays, and other values, and will attempt to pretty-format any complex value. Other console methods exist, such as console.error() for error-formatted messages, as well as specialty formatting methods like console.table() and console.group(). (slides: comments and logging) The const and let keywords are used to declare variables, and are block-scoped - they scope the lifetime of variable to the nearest curly brace block. let variables can be assigned a new value, while const variables cannot be changed to point to something else. There's also an older var keyword, which has more confusing scoping behavior - it's function-scoped, which means that no matter where you use var to declare a variable, it's hoisted and acts as if it was declared at the first line of the function it's in. Modern JS usage avoids var, since const and let behave more consistently. (slides: basic variable declarations) As a personal recommendation, I suggest using const as the default, and let if you plan to reassign to that variable later, but there's others who suggest just using let all the time. JS only has a single number type: a 64-bit floating point number equivalent to a double in other languages. It can hold integer values as well. (slides: numbers and math methods) Strings can be written using three different quotes: single quotes and double quotes are equivalent, while the newer template literal strings use backticks and can have variable values interpolated in the middle. (slides: string syntax and methods) Booleans are written as true and false. Objects and arrays are normally written using object/array literal syntax, as {} for objects and [] for arrays. (slides: basic object syntax, basic array syntax) Object keys are always strings, but if the key name is a valid JS variable name, the quotes can be omitted (and usually are). const text1 = 'Test A'; let text2 = 'Test B'; const value = 42; // This is a template literal string const text3 = `The number is ${value}`; /* This log statement can take multiple arguments */ console.log('First variable: ', text1, 'second variable: ', text2); const object1 = { field1: 123, field2: true, }; const array1 = [text1, value, object1, 99]; Data Types JS has a variety of core data types. The basic data types are standalone values called primitives: undefined : a variable that has not been assigned a value null: a value that represents "no value" String: text Number : a 64-bit floating point number (aka IEEE double) Boolean : a true / false value Symbol: a unique reference value that can be used as a form of singleton identifier All other values in JS are objects. This includes actual plain objects, as well as other types that extend from the Object type: Object: a collection of string key / any value properties Function : a function that can be called Array : an expandable list of values Date : a date/time value RegExp : a regular expression The difference between null and undefined is subtle. undefined represents "there is no meaningful value here", while null means "there is a value, but it's empty / not available". Most of the other core data types have both a constructor function form and a literal syntax form, like new String('abcd') vs 'abcd', or new Array() vs []. Always prefer using the literal syntax to create new values instead of creating them via constructors. Functions, arrays, dates, and regexps are technically objects as well. Conditional Logic JS uses the standard C-like if/else if/else syntax for conditions, as well as the ternary operator (const value = condition ? trueValue : falseValue). There's also a typical switch/case/break statement. Comparisons include the usual < and > operators and &&, ||, and ! boolean logic operators. (slides: conditional statements) There are several forms of loops. You can do C-style counting loops with for(let i = 0; i < someValue; i++). The for (let key in obj) form iterates over keys, while for (let item of items) form will loop over values inside an iterable like an array. (slides: loops) try/catch/finally allows handling errors. There's an Error class, but technically any value can be thrown. Comparisons and Boolean Conversions Unlike strictly-typed languages, JS frequently does implicit conversions or coercions of values in various contexts. The most common example is treating values as truthy or falsy. In addition to the actual true and false boolean values, using other values in a comparison statement will implicitly convert them to their boolean equivalent. (slides: "truthy" and "falsy") For reference: "Truthy" values: all objects and arrays (even if empty); non-zero numbers; non-empty strings; dates; functions "Falsy" values: null, undefined, 0, NaN, and empty strings (Note that the "empty object/array" behavior differs from Python, where those are considered "falsy".) So, a comparison with an empty object like if ({}) { console.log('Truthy!')} will actually convert to true and print something, while a comparison with null like if (null) { console.log('Falsy!')} will convert to false and not print anything. The boolean/comparison operators will all implicitly convert values to their boolean equivalent as well. A doubled negation operator is often used to convert a value to its boolean equivalent. For example, an empty string like '' is falsy, so !!'' would implicitly convert '' to false, !false to true, and !true to a final result of false. JS also has two different comparison operators: == and ===. The double-equals == operator does loose comparisons that include implicit conversions of values. So, 0 == '' is true, because they will both end up being implicitly converted to false, and false is equal to false. The triple-equals === operator does strict reference comparisons that check to see if the two values are literally identical. For objects and arrays, it compares to see if they are the same reference in memory. So, {} === {} is false, because each of those is a separate new object reference. You should almost always prefer using === strict reference comparisons to avoid unexpected implicit conversions. (slides: comparisons and coercion) Functions JS has two different ways to define standalone functions: the function keyword, and the () => {} "arrow function" syntax. They have some differences in behavior. (slides: function syntax and behavior) function declarations are hoisted. Since arrow functions are assigned as variables, the declaration behavior is based on the keyword you use ( const, let, or var) function declarations create their own value for the this keyword used inside. Arrow functions inherit the value of the this keyword as it existed at the time and scope they were declared in. arrow functions can have optional parentheses if declared with only one parameter, and omitting curly braces adds an implicit return statement: const timesTwo = num => num * 2 JS functions are very flexible with arguments and parameters. No matter how many parameters were declared, you can always call a function with more or fewer arguments. If I have function myFunc(a, b, c) { }, but call it as myFunc(1, 2), then inside of the function a === 1, b === 2, and c === undefined because we didn't provide a value when we called it. If I call it as myFunc(1, 2, 3, 4), then the 4 argument is mostly ignored since we didn't declare a fourth parameter name. However, all function arguments can be accessed as an array-like value using the arguments keyword. Function parameters can have default values provided, which will be used if the incoming value is undefined. If we change the example to function myFunc(a, b, c=42){ }, and call it as myFunc(1, 2), then a === 1, b === 2, and c === 42. This is frequently used for initialization. Function arguments can also be destructured - extracting specific fields from object and array parameters. (slides: function declarations and arguments) Since functions are just variables, they can be passed around like any other variable. JS in Depth Objects Object fields can be accessed with dot notation (obj.a.b.c) or bracket notation (obj["a"]["b"["c"]). Always use dot notation as long as the field names are known ahead of time. Use bracket notation if using a variable to look up a field ( obj[key]), or if the field name is not a valid JS identifer ( obj["some-field"]). Reading a field that doesn't exist returns undefined, rather than throwing an error. Objects can be modified at any time. New fields can be added, existing fields can be reassigned, and fields can be deleted with delete obj.field. ES6 introduced a shorthand syntax for declaring objects when a key:value pair should be added based on the name and value of an existing variable - you can omit the : value portion: // Shorthand syntax for declaring key/value pairs: let x = 0, y = 1; let es5Obj = {x : x, y : y}; // Create a key 'x' whose value is the variable 'x' let es6Obj = {x, y}; There's also a computed property syntax that allows dynamically constructing keys based on the values of variables and expressions: let name = "abc"; let es5Obj = {}; es5[name] = 123; let es6Obj = { // Use the value of variable 'name' as the property key [name] : 123, [name + "2"] : 456, } // {abc : 123, abc2 : 456} Functions can be declared inside an object using function or arrow functions as the values, or directly inline as const obj = { someMethod() {} }. (slides: Object literal syntax) Objects can be destructured, which is a shorthand for creating local variables based on the names of object fields. Destructuring statements can provide default values if that field is undefined, or change the local variable name that's created (slides: object destructuring and spreads): let obj1 = {a : 1, b : 2, c : 3, d : 4}; // can create local variables the long way let a = obj1.a; let b = obj1.b; // or destructure to shorten it: let {a, b} = obj1; // Local variables can be given different names: let {a : differentA, c} = obj1; // And have default values in case of undefined: let {doesNotExist = 123} = obj1; New objects can be created by using the object spread operator to copy values onto a new object: let obj2 = {a : 1, b : 2, e : 5}; let obj3 = {...obj2, a : 99}; There are static methods on the Object built-in type that allow getting all of the keys or values as arrays (Object.keys(someObj)), or an array of key/value pairs as 2-element arrays (Object.entries(somObj)). Object.assign(target, src1, src1) can be used to mutate existing objects by copying properties, or create new objects by passing an empty object as the target. (slides: Object static methods) Arrays Arrays are 0-indexed, but can be sparse - any index can be assigned at any time (arr[97] = "stuff"). Arrays may hold any value and a mixture of different types of values. Like with objects, accessing a non-existing array index returns undefined. Like with objects, you can use destructuring to read values from arrays (const [a, , c] = ["a", "b", "c"]), and spread operators to create arrays (const arr3 = [...arr1, "b", "c", ...arr2, "d"]). (slides: array syntax, array destructuring) Arrays are actually objects as well, and have numerous methods built in for various purposes. Some of these methods mutate the existing array, others return new arrays or other values. (slides: array methods, array iteration methods, array searching) Name Description Arguments Returns Mutates slice() Copies an array subset Indices to copy A new array No concat() Creates array with more values New values/arrays to add A new array No splice() Inserts/deletes values Index to mutate/delete, values to insert Items removed Yes join() Stringifies array contents String used to separate items A new string No push() Inserts new items at end of array Items to insert Array size after Yes pop() Removes last item from array None Last array item Yes unshift() Inserts new items at start of array Items to insert Array size after Yes shift() Removes first item from array None First array item Yes sort() Sorts existing array Callback to compare two items Existing array reference Yes reverse() Reverses order of items in array None Existing array reference Yes map() New array by transforming values Callback to transform one item New array with callback return values No filter() New array with values matching comparison Callback to compare one item New array with original items where callback returned true No forEach() General iteration for side effects Callback to run logic for one item Nothing No reduce() Calculates one result value based on all items Callback to accumulate running result Final callback result value No indexOf() Finds index of exact value match Value to search for index or -1 No includes() Checks if exact value is in array Value to search fo boolean No find() Find item by comparison Callback to compare one item First match or undefined No findIndex() Find index of item by comparison Callback to compare one item index or -1 No some() Checks if any items match comparison Callback to compare one item boolean No every() Checks if all items match comparison Callback to compare one item boolean No To emphasize, array.sort() and array.reverse() mutate the existing array!. This can be very surprising if you're not expecting it. In many cases it's a good idea to make a copy of an array using [...arr] or arr.slice() first and then sort/reverse the copy, especially if you're using something like React or Redux. The core iteration methods each have a different semantic meaning and purpose: map(): creates a new array, with the same size as the original array, whose values are based on a transformation of the original values const doubledNumbers = [1, 2, 3].map(num => num * 2) filter(): creates a new array, with some of the values from the original, based on values where the comparison returned true: const oddNumbers = [1, 2, 3].filter(num => num % 2 === 0) forEach(): general iteration over the array, usually to produce side effects [1, 2, 3].forEach(num => console.log(num)) reduce(): calculates a single result value based on "accumulating" a running result at each step, plus a starting value const total = [1, 2, 3].reduce( (previousResult, currentValue) => previousResult + currentValue, 0) Scoping and Closures As mentioned above, the let, const, and var keywords differ in their scoping behavior. let and const are block-scoped - they only exist within the curly braces block where they're declared, and the same variable name can be reused and shadowed inside of nested blocks. var is function-scoped - no matter where it's used, the declaration is hoisted to act as if it was written on the first line of the function it's inside, and reusing the same name will override the earlier declaration. The function declaration keyword is also hoisted. (slides: variable scope and hoisting) There is a top level "global object" that can always be referenced, and any variables declared in the top-level global scope using var become fields on that object. In a browser, the object is window, and under Node, it's global. Code inside functions can reference variables outside functions, because those variables are in an outer scope. Functions can be declared inside of functions, and close over variables by referencing them. A nested function that references a variable in the parent scope is called a closure. This allows nested functions to continue to refer to and modify variables long after the outer function has finished running. This is frequently used when passing callback functions for use as event handlers. Understanding closures is a critical part of mastering JavaScript behavior. (slides: functions and closures) // Functions can create and return functions function createPrintName() { let name = "Mark"; function printName() { console.log(name); } return printName } // Functions can capture references to values. // These are called "closures". function makeCounter() { let timesCalled = 0; return function counter() { timesCalled++; console.log(`Times called: ${timesCalled}`); } } const actualCounter = makeCounter(); actualCounter(); // "Times called: 1" actualCounter(); // "Times called: 2" Functions and this The behavior of the this keyword in JS is one of the most confusing and difficult topics to grasp, especially if you are coming from a language like C# or Java. Unlike other languages, the value of this can point to different references depending on how a function is declared and what syntax was used to call it. Instead of always pointing to a specific instance of a class, what this points to is based on the execution context of a function. This is a long and complicated topic, so I'll summarize it briefly here and suggest reading up on it further. There are four ways to call a function in JS, which each have a different effect on the value of this inside of the function (slides: understanding this): "Function invocation": someFunction(1, 2) (slides: function invocation) this points to undefined or the global object, based on use of strict mode "Method invocation": someObject.someFunction(1, 2) (slides: method invocation) this points to someObject because of the dot operator "Constructor invocation": new SomeFunction(1, 2) (slides: constructor invocation) this points to the new function instance "Indirect invocation": someFunction.call(thisArg, 1, 2) or someFunction.apply(thisArg, [1, 2]) this points to the first argument of .call() or .apply() Functions can be bound to create a new function with pre-fixed values for some of their arguments, including forcing the value of this. (slides: indirect invocation and binding) Arrow functions inherit the value of this at the time they were defined (slides: arrow functions and this) Classes and Prototypes JS inheritance is unlike other most major languages - it's based on a system of prototypes for objects, and is known as prototypal inheritance. Loosely put, inheritance is dynamic and determined at runtime by tracing field lookups up a chain of references for a given object, instead of being defined strictly at definition time like in Java or C#. (slides: prototypal inheritance) // Define a function to act as a "class" function Animal(name, isAwake) { this.name = name; this.isAwake = isAwake; } // Create methods by adding functions to the "prototype" Animal.prototype.wakeUp = function() { this.isAwake = true; } Animal.prototype.sleep = function() { this.isAwake = false; } // Create an instance let jinx = new Animal("Jinx", true); // Call an instance method jinx.sleep(); ES6 added an actual class keyword to the language, but it is effectively syntax sugar over the existing prototypal inheritance behavior (slides: classes): class Animal { constructor(name, isAwake) { this.name = name; this.isAwake = isAwake; } wakeUp() { this.isAwake = true; } sleep() { this.isAwake = false; } } class Dog extends Animal { constructor(name, isAwake, isWaggingTail) { super(name, isAwake); this.isWaggingTail = isWaggingTail; } bark() { console.log("Woof! Woof!"); } } let spot = new Dog("Spot", true, false); spot.sleep(); Async Timers and the Event Loop JS execution is based on a single-threaded event loop + queue. Conceptually, the behavior is while(queue.waitForMessage()) queue.processNextMessage(). Events are added to the queue, and have JS code attached. This includes mouse events, timers, network requests, and much more. The event loop pops the next event off the queue, and executes the entire attached code to completion. The currently executing script cannot be interrupted. The script may start timers, which will add events with scripts to the queue for later execution. While there is only one JS execution thread, the browser itself may add events to the queue while code is executing. Rather than blocking for async requests, they will schedule events when they complete. Note that if you have an infinite loop in your code, the event loop is "blocked" and cannot move on to process the next event! JS logic relies heavily on callbacks - passing a function reference to be called and executed at some future point in time. Callback usage can be synchronous or asynchronous depending on the use case. (slides: callbacks and timers) JS provides two core functions for scheduling async logic: setTimeout(callback, ms), which runs the callback function once, and setInterval(callback, ms), which runs the callback repeatedly. Both functions return a unique ID value that can be used to cancel the queued timer via clearTimeout() or clearInterval(). Understanding the JS event loop is critical to understanding how to use async logic in JS! See the video What the heck is the event loop anyway? for an excellent explanation, as well as this accompanying interactive event loop visualization tool. Promises Nested async calls often result in a "callback pyramid of doom": fetchData("/endpoint1", (result1) => { fetchData("/endpoint2", (result2) => { fetchData("/endpoint3", (result3) => { // do something with results 1 through 3 }) }) }) This also makes error handling very difficult. The JS Promise data type provides a structured way to handle future async results (slides: promise basics. Promises are objects can be in one of three states: Pending: created, but there is no result yet Fulfilled: completed, with a positive/successful result Rejected: completed, with an error result A Promise has settled or resolved once it is either fulfilled or rejected. Promises can be chained using somePromise.then(callback) and somePromise.catch(callback). When somePromise resolves, any provided chained callbacks will be run - .then() callbacks if it fulfilled, or .catch() callbacks if it rejected. Callbacks can even be chained after the promise has resolved, in which case they will be executed almost immediately. Promise chains effectively form a pipeline. Each .then() or .catch() returns a new promise. (slides: creating and chaining promises, combining and resolving promises) let promise1 = new Promise( (resolve, reject) => { // This callback executes synchronously, immediately // Could "resolve" the promise with a value: resolve("a"); }) promise1 .then( (firstValue) => { // "a" return "b"; }) .then( (secondValue) => { // "b" // _Not_ returning a value returns `undefined` }) .then( (thirdValue) => { // undefined }) Inside a promise callback, you can run whatever calculations you want, but you can only do 3 things to complete the logic: Return a value: resolves the promise successfully, with that value. Note that returning nothing or undefined is the same as resolving the promise successfully with undefined Return another promise. The new promise for the callback will resolve or reject based on the promise you returned. Throw an error. This rejects the promise, with that error. async/await Syntax for Promises Chaining Promises can also be difficult. The newer async/await syntax lets you write Promise-handling logic with what appears to be synchronous-style syntax, including use of try/catch for handling errors. A function must be declared using the async keyword in order to use the await keyword inside. Every async function then automatically returns a Promise with whatever value is returned. Rejected promises in an async try/catch will jump to the catch block. (slides: async/await) // This function: function function1() { let resultPromise = Promise.resolve(42); return resultPromise; } // Is the same as this function: async function function2() { return 42; // converted to a promise! } Overall, async/await syntax is much nicer to read than promise chains, and should usually be preferred: // This promise chain: function fetchStuff() { return fetchData("/endpoint1") .then( (result) => { return firstProcessStep(result) }) .then( (processedResult) => { console.log(`Processed result: ${processedResult}`) return processedResult; }) .catch( (err) => { console.error("PANIC!", err); }) } // Can convert to: async function alsoFetchStuff() { try { let result = await fetchData("/endpoint1"); let processedResult = firstProcessStep(result); console.log(`Processed result: ${processedResult}`) return processedResult; } catch (err) { console.error("PANIC!", err); } } Other Topics JS Module Formats The ES Module format is now the standard for writing most new code. CommonJS modules are still widely used with Node and as a publishing format. For details and examples of the various module formats, see Client Development and Deployment: JS Module Formats Regular Expxressions JS has regular expressions available for complex text matching. Regexes can be declared using slashes, like /Have a good (day|afternoon|evening)/. (slides: regular expressions) Regex objects have methods like .test(str) and .match(str), and can be passed to some string methods as well ( "Hello world!".replace(/world/, 'dog')). Immutability Mutation means changing the contents of an existing object or array in memory. For example, obj1.a = 42 mutates the existing object pointed to by obj1, and arr1.push('abcd') mutates the existing array. Immutability is the concept of updating data by copying existing values and modifying the copies, rather than mutating the original values. JS is inherently a mutable language, so you have to explicitly write logic to make updates immutably. (slides: immutability) For objects, this usually is done via the object spread operator: let updatedObj2 = { ...obj2, // copy all fields from obj2, nested : { // provide a new `nested` value ...obj2.nested, // copy fields from `obj2.nested` d : 123, // but overwrite `obj2.nested.d` in this copy } } Arrays can be copied using the array spread operator or array.slice(). See the array methods table above for details on which array methods mutate the existing array, vs methods that return a new array. There are many utility libraries that can help with immutable updates, but by far the best is Immer. Immer provides a produce function that will wrap your original data in a Proxy object and let you "mutate" the value in a callback, but then converts all the mutations into safe immutable updates and returns the result: // hand-written immutable update: function toggleTodo(todos, index) { return todos.map( (todo, i) => { // Keep the items that aren't changing if(i !== index) return todo; // Return a new copied object for the item that is changing return { ...todo, completed : !todo.completed }; }); } // simpler with Immer because we can "mutate": import produce from "immer"; function toggleTodo(todos, index) { return produce(todos, (draftTodos) => { const todo = draftTodos[index]; todo.completed = !todo.completed // Safe "mutation"! }) } Lodash JS has a small and limited standard library built in. Lodash is a separate library of utility functions, and is one of the most widely used libraries in the JS ecosystem. It provides dozens of utility functions for everything from array and object operations, to working with functions, to checking the type of a value, to converting strings between different formats. (slides: Lodash) // Arrays _.difference(["a, b"], ["c", "b"]) // ["a"] _.head(["a", "b", "c"]) // "a" _.tail(["a", "b", "c"]) // "c" _.intersection(["a", "b"], ["b", "c"]) // ["b"] // "Collections" (objects and arrays) _.groupBy([6.1, 4.2, 6.3], Math.floor); // => { '4': [4.2], '6': [6.1, 6.3] } let items = [{n: "f", a: 48}, {n: "b", a : 36}, {n: "w", a: 38}]; let sortedItems = _.sortBy(items, "a"); // => [{n: "b", a:36}, {n: "w", a: 38}, {n: "f", a:48}] // Objects let obj1 = {a : 2, b : 3, c : 7}; _.mapValues(obj1, val => val * 2); // {a : 4, b : 6, c : 14} _.omit(obj1, ["b"]); // {a : 2, c : 7} _.pick(obj1, ["b", "c"]) // {b : 3, c: 7} // Strings _.camelCase("Foo Bar"); // "fooBar" _.kebabCase("Foo Bar"); // "foo-bar" DOM HTML is a text document format, organized into a hierarchy of nested tags like <h1>, <p>, and <img>, that represents the desired structure and content of a web page. Browsers download an HTML file, and parse the HTML tags. They then create a set of internal data structures that match the requested page content, and use that to lay out and draw the actual pixels on screen. Browsers also expose a live representation of the current page content to JS. This is known as the Document Object Model (DOM). While every browser has its own different internal data structures for a page, the DOM is a standardized set of APIs for interacting with the content of a page. The DOM APIs enable JS code to create, read, modify, and delete the actual contents of a page, dynamically, after the initial HTML has been parsed and the page was loaded. The DOM spec defines classes that correspond to each HTML element type. For example, a <div> tag, when parsed, will result in an HTMLDivElement class instance being created and added to the DOM. Each instance is called a DOM node, because the DOM contents form a tree. Different DOM classes support different methods - an HTMLInputElement instance has different methods available than an HTMLDivElement instance, but all DOM node instances share a common core set of base methods. DOM Query APIs The DOM API allows querying for nodes based on [selectors](HTML, such as the type of tag, ID attribute, classnames, node relational structure, and more. (slides: DOM query and manipulation) The root element for a page is globally accessible via the global document object. Most queries are run against document to find nodes anywhere inside a page. The primary query APIs are: getElementById(): finds a DOM node based on an ID attribute string. (Note that the ID string should be provided without a '#' prefix here, unlike other places where an actual selector is being used to query) querySelectorAll(): finds all DOM nodes that match the provided CSS selector query querySelector(): finds the first DOM node that matches the provided CSS selector query These same query APIs also exist on the DOM node subclasses, so you can run someNode.querySelectorAll() to scope a search to that particular portion of the page's DOM tree. Note that querySelectorAll() returns a NodeList object. This is an array-like object. It implements a couple array-type methods like .forEach(), but is not an actual array. It's common to convert a NodeList result into an actual array before doing further work with the query results. Other query methods like getElementsByName also return a NodeList, but those results are "live" views whose contents automatically change as the DOM is updated. This behavior can be confusing if you're not ready for it. DOM Manipulation DOM nodes support a wide variety of manipulation operations. New DOM nodes can be created with document.createElement('p'), which returns an instance of the requested node type, or copied with .cloneNode(). DOM nodes have .innerHTML and .innerText properties, which allow reading and writing the entire contents of a node at once. Input nodes typically have properties like .name and .value. Nodes have a .classlist property, which is an object that supports adding, removing, and toggling the existence of individual CSS classnames on that node (divNode.classlist.toggle('active')). Nodes have a .style property that allows reading and mutating any of the style-related properties, including values that were defined using CSS (divNode.style.backgroundColor = 'red'). Unlike CSS, the style property names are camelCased instead of kebab-cased, because that matches JS naming conventions. Nodes have properties that point to related nodes in the tree: .children, .firstChild/lastChild, .nextSibling/previousSibling, and .parentNode. Nodes can be inserted and moved around with with .appendChild(), .insertBefore()/insertAfter(). Event Listeners DOM nodes implement an event listener interface, which looks like node.addEventListener(eventName, callback). There's a wide variety of available events that will be triggered by the browser as the user interacts, such as 'click', 'mousemove', and 'keydown'. (slides: DOM event handling) Event handlers receive an event object with numerous fields describing the current event. Inside of an event handler callback, this will point to the actual DOM node as long as you declared the callback with function (arrow functions always use the same this value from the scope where they were defined). When a DOM event is triggered, it bubbles upwards from the original target node, through all of its parent nodes, to the root document, firing any matching event handlers along the way. After the event has reached the root, it then runs back down from the root to the original node, known as the capturing phase. Because events bubble up, it's possible to add a single event listener on some parent node that catches all events of a certain type on any children. This pattern is known as event delegation. Any event handler may cancel the rest of the handler processing by calling e.stopPropagation(). The browser also often has default behavior that will be run regardless of any user-provided listeners, which can be canceled with with e.preventDefault(). jQuery For many years, the DOM APIs were poorly defined, had widely varying support between browsers, and had limited query capabilities built in. The community created many libraries to fill in these gaps and provide common capabilities across browsers. jQuery became the most popular DOM manipulation library by far. Many of its capabilities have been added to the DOM APIs themselves, and the need for jQuery has lessened as the ecosystem has evolved, but it's still one of the most widely used JS libraries. (slides: jQuery examples) jQuery is an abstraction layer for querying nodes from the DOM and quickly manipulating them via chained operator functions. It also has utilities for interacting with inputs, creating DOM nodes from HTML strings, provides a nicer abstraction over the browser's built-in APIs for making AJAX requests, adds some animation capabilities, and has a plugin system that allows developers to extend its capabilities. jQuery is not a framework for developing entire applications - it's best suited for adding interactivity to an existing static or server-rendered web page. jQuery is not normally needed in today's modern client-rendered web app frameworks like React or Vue, because those frameworks take care of creating and updating the DOM for you, and using jQuery would interfere with their operations. Once loaded, the jQuery API is globally accessible as a variable named $. DOM queries are run by passing a selector, like $("#demo .content"), which returns an array-like object with numerous methods that can be chained. It's a common practice to prefix jQuery result object variable names with $, like $demo.css("backgroundColor", "green").slideUp(500). Further Resources JS Overviews and Tutorials MDN JavaScript docs: extensive tutorials and reference material on all aspects of JS FreeCodeCamp tutorials: beginner-focused project-based tutorials The Modern JS Tutorial: topic-based intros to specific aspects of JS and DOM syntax 2ality: JS books and articles: in-depth articles and books that cover JS syntax and behavior in fine detail, including articles that summarize the new features in each ES spec revision JS for Java Developers slides: a cheatsheet-style summary of JS syntax, concepts, and ecosystem tools Tania Rascia's JS tutorials: a broad set of tutorials that cover JS topics such as syntax, events, classes, functions, async, and more JS 25th Anniversary milestones timeline The this keyword A Gentle Explanation of this in JS Understanding JS Function Invocation and this this in JS Array Methods Which Array Method When? Does It Mutate? Array Functions and the Rule of Least Power JS Event Loop and Async Understanding the Event Loop, Callbacks, Promises, and Async/Await in JS What the heck is the event loop anyway? Loupe: Event loop interactive explorer JS Visualizer 9000: interactive event loop and async task queue explorer Other Topics A Visual Guide to References in JS Immutability in JS: The Complete Guide Understanding Modules, Import, and Export in JS DOM The Modern JS Tutorial: DOM Understanding the DOM (free ebook)

How Web Apps Work: Browsers, HTML, and CSS

Web development is a huge field with a vast array of concepts, terms, tools, and technologies. For people just getting started in web dev, this landscape is often bewildering - it's unclear what most of these pieces are, much less how they fit together. This series provides an overview of fundamental web dev concepts and technologies, what these pieces are, why they're needed, and how they relate to each other. It's not a completely exhaustive reference to everything in web development, nor is it a "how to build apps" guide. Instead, it's a map of the territory, intended to give you a sense of what the landscape looks like, and enough information that you can go research these terms and topics in more depth if needed. Some of the descriptions will be more oriented towards modern client-side app development with JavaScript, but most of the topics are fundamental enough that they apply to server-centric applications as well. Other posts in this series cover additional topics, such as: HTTP and Servers Client Development and Deployment JavaScript and the DOM AJAX, APIs, and Data Transfer New terms will be marked in italics. I'll link references for some of them, but encourage you to search for definitions yourself. Also, some of the descriptions will be simplified to avoid taking up too much space or dealing with edge cases. There's a comprehensive set of HTML and CSS tutorials at Interneting is Hard: HTML and CSS, with beautiful diagrams and great explanations of these terms and concepts. (I've borrowed a few of their excellent diagrams for this post.) CSS Tricks has some amazing guides to specific aspects of CSS, like their guides for Flexbox and CSS Grid, as well as numerous articles on other CSS and web dev topics. My JavaScript for Java Developers slides also cover much of this post's content as well. Table of Contents Browsers Browser Rendering Behavior Variations Browser Versions Major Browsers Browser Notes HTML HTML Tags HTML Page Structure Common HTML Tags Common Attributes Forms CSS CSS Syntax CSS Selectors CSS Styling Rules Page Layout The Box Model Display and Positioning Flexbox CSS Grid Responsive Screen Size UI Further Resources Browsers Any application can make requests using the HTTP protocol. However, web browsers are applications whose primary purpose is to request data using HTTP, format the content, and display it on screen. That content is normally defined using the HTML format, and the appearance of the content is defined using CSS. A browser's implementation can be divided into several areas: The networking logic that is used to request data The rendering engine that parses fetched content, determines how it should be laid out on screen, processes the styling rules, and actually draws the pixels on screen A JavaScript engine that parses and executes JS code The browser chrome that defines the visible UI around the page content (buttons, menus, address bar, etc) Different browser implementations may share and reuse implementation pieces, but use their own unique implementations for other sections. For example, several browsers are built on top of the "Blink" rendering engine, like Chrome and Edge, but have very different UI implementations and user-visible features. Browser Rendering Behavior Variations Browsers all attempt to implement the same language specifications, but each browser ends up implementing those specifications differently. Most of the time the behavior is the same, but browsers frequently have differences in behavior for the exact same features. This has become less of a problem in recent years, as browser support for major specifications has become more consistent, but browser "quirks" in layout, appearance, and JS support are still a major pain point for web developers. Browsers can also always try adding new features themselves, but in order for features to be considered "standardized", they typically need to be implemented by at least two different browsers and go through the appropriate web specification committees. Browser Versions Each time a browser ships a new version, that version has a fixed understanding of a certain subset of the HTML, CSS, and JS specifications. Since that browser version may stay in wide use for many years, developers need to determine what set of browsers and versions they plan to support when developing a site or application. Browsers originally had infrequent releases of new versions, and each new version had many significant changes. Today, most modern browsers release new versions frequently (every 6-8 weeks), and each new version only has a few meaningful changes. These browsers are referred to as evergreen browsers, because there's always a new version available and most users will automatically be upgraded to the latest version. The "Can I Use?" site is an incredibly useful resource that has a database of different HTML, CSS, and JS features, and lists which browser versions support each feature as well as how much worldwide market share those browsers have. This is a valuable tool for deciding whether you can safely use a given feature in your site, based on your target audience. For example, the CSS position: sticky layout feature currently has 95% support worldwide, but does not work in Internet Explorer. So, if your target audience still includes IE11 users, you might not want to use that feature. Major Browsers The majority of browser market share belongs to a few major browsers. By far the biggest one is Google Chrome. The market share of each browser has gone up and down over time. Internet Explorer had a near-monopoly in the early 2000s until Firefox became popular. Chrome took over from both of them, and Safari has become a major factor on mobile. Browser market share also varies significantly in different countries. The browser market share as of October 2020 looks roughly like this. Note that there are rather different usage percentages for desktop devices vs mobile devices: Name Company Version Evergreen Rendering Engine Desktop % Mobile % Chrome Google 86 Yes Blink 70% 63% Safari Apple 14 No WebKit 8% 24% Firefox Mozilla 82 Yes Gecko 7% 0.50% Edge Microsoft 86 Yes Blink 6% 0% Opera Opera 72 Yes Blink 2% 2% Samsung Internet Samsung 12 No Blink 0% 6% Edge (legacy) Microsoft 44 No EdgeHTML 1% 0% Internet Explorer Microsoft 11 No Trident 2% 0% Browser Notes Some notes on the major browsers: Chrome: Built on an open-source core called "Chromium", which is effectively Chrome minus all the actual Google account integration and branding The "Chrome" browser partly got its name because it tried to minimize the amount of "browser chrome" (the term for all the browser UI that surrounds the actual page content) Google has a tendency to announce new web features that only exist in Chrome and yet consider them "final" even if other browsers haven't added them yet Safari: The only actual browser rendering engine allowed by Apple on iOS devices. Any other browsers you see on iOS are actually the Safari core with a different UI skin and branding. Apple tends to be slow adding new features to Safari even after they've been added to other browsers. Apple has also refused to add certain features on the grounds that they might make act as privacy leaks in some way. Firefox: Open-source, and built by the Mozilla organization - it's the only major browser not built by one of the big tech companies Now that many browsers build on top of Chromium/Blink or Webkit, Firefox's "Gecko" engine is the main alternative rendering engine still in meaningful use. Edge: There's been two different variations of the MS Edge browser. The first was actually built entirely by Microsoft, using a cleaned-up and modernized version of the rendering engine previously used in Internet Explorer. Recently, Microsoft stopped developing their own rendering engine, labeled that version of Edge as "legacy", and rebuilt Edge on top of Chromium with their own UI. Internet Explorer: IE was the default browser on Windows up until recently. Many companies still have internal websites that only work correctly in IE, which keeps it as a small but persistent portion of the market IE has "stagnated" twice. IE6 came out in 2001 along with Windows XP, but Microsoft stopped developing it for several years until Firefox and Chrome began taking away market share. Today, IE11 is considered an obsolete browser without many modern capabilities, and many companies are finally dropping support for running their sites in IE11. HTML Browsers request data from servers using the HTTP protocol. The standard format that browsers read and use to display data is HyperText Markup Language (HTML). HTML is a plain-text format that uses a hierarchy of nested HTML tags to describe the content, layout, and formatting of a web page. HTML Tags An HTML tag is a unique name that is written surrounded by angle brackets, like <html>, <img>, or <p>. Most tags also need to have a corresponding closing tag element, and may contain content inside. For example, a paragraph tag may have text and images inside, and a list tag will have list item tags. Some tags are also self-closing, such as <img />, <hr />, and <input />. Tags are not case-sensitive, but are usually written lower-case: <img>, not <IMG>. Tags may also have attributes, written as attributeName="someValue". Some attributes don't need an actual value - the existence of the attribute name is all that's needed, such as <input disabled>. The HTML spec defines dozens of different tags, each with a different purpose. Browsers determine what should actually be displayed by parsing these tags. HTML Page Structure A standard HTML page structure consists of: an outer <html> tag that contains: an optional <head> tag that may have metadata-related tags inside, as well as links to additional asset files that the browser should download a <body> tag containing the actual content of the page A basic example would look like: <html> <head> <title>My Website</title> <link rel="stylesheet" type="text/css" href="styles.css"> <script type="text/javascript" src="logic.js"></script> </head> <body> <h1>Welcome to My Website</h1> <p> Here's some <b>bold</b> text, and an image: </p> <img src="party-parrot.gif" /> </body> </html> HTML tags are normally written with indented spacing for each level of nesting, but that's only needed for readability by people - browsers don't care about that. HTML pages may also start with a <doctype> tag that declares the HTML version being used by the file, although that isn't necessary. Common HTML Tags Core tags <html>: The outer container of an HTML page <head>: the header section with metadata <body>: the main content of an HTML page <h1> through <h6>: headers of varying sizes <p>: text paragraphs <div>: a generic block <span>: a generic inline piece of content <img src="some-image.png">: links to an image to be displayed <a href="http://www.example.com">: an anchor tag that may point to another page, or a specific location within this page. Clicking an anchor tells the browser to navigate to that URL. Headers: <title>: Page title, shown in the current tab / browser window <meta>: assorted metadata attributes describing a page <link>: link to assets such as stylesheets <script>: link to JavaScript files, or containing inline JS code Formatting <b>, <i>, <u>: bold, italic, underlined text <blockquote>: represents a formatted quotation Lists <ul>, <ol>: list parents. Unordered lists show bullet points for each list item, ordered lists show numbers. <li>: list items inside of lists Tables <table>: defines a table with rows columns <tr>: a single row in the table <thead>: the header area for the table <th>: header cells in a row in the header <tbody>: the body of the table <td>: a normal cell in a row in the body Forms <form>: a form area with inputs <input>: a user-editable input. Browsers have many types of inputs built in, most of which are used by passing a different type attribute to <input>. Common inputs are: "text": a one-row textbox "checkbox": a clickable true/false checkbox "radio": a clickable button where only one value of a group can be selected at a time "file": allows the user to select and upload a file from the local computer <select>: a dropdown for selecting a single value from a list <option>: a single selectable value in a dropdown <textarea> a multi-line textbox <label>: a descriptive label associated with an input Some tags do basically the same thing as other tags, but are considered to be more "semantic". Semantic markup is the idea that HTML tags should be meaningfully readable to both other developers and the browser itself. For example, a <div> is a generic block of content. The HTML spec now has tags like <header>, <footer>, <main>, and <section>, which behave the same way display-wise, but are intended to be more meaningful in terms of what content they will contain. Common Attributes There are several common attributes that can be applied to any tag: id="someValue": gives a globally unique "ID" name to this element. There can be many IDs in a page, but each unique ID should only exist once in a page at a time class="first second third": a space-separated list of "class names", used to help identify elements for styling and searches title="Some description": descriptive text, usually shown in a tooltip when hovering over the element style="color: red; background-color: blue;": a list of inline style rules to change the display of this element. Uses semi-colon-separated CSS rules alt="Some description": descriptive text, typically supplied for images for use when the image can't be displayed, or read by a screen reader Other attributes only work for certain tags: <img src="some-image.jpg">: the URL for the image to show <a href="http://www.example.com">: the URL for the link <input type="checkbox">: the type of input element to show Forms HTML forms allow users to enter values into the browser, such as typing in text or selecting items from a dropdown. The browser maintains the current value of each input internally as the user interacts with the inputs. Input elements can be put anywhere, but should normally be put inside a <form> tag, which adds some extra behavior. In particular, forms can be submitted - the browser will automatically gather up the values of each input and send them via HTTP request to the server (normally an HTTP POST request). A basic form might look like: <form action="/create-user" method="post"> <ul> <li> <label for="name">Name:</label> <input type="text" id="name" name="user_name" /> </li> <li> <label for="state">State</label> <select name="state"> <option value="ca">California</option> <option value="oh">Ohio</option> <option value="ny">New York</option> </select> </li> <li> <label for="msg">Message:</label> <textarea id="msg" name="user_message"></textarea> </li> </ul> <button type="submit">Create User</button> </form> Side note: a <button> in a form defaults to "submit this form" when clicked. If you want to do something else like handling logic in JavaScript (which is common in UI frameworks like React), you need to declare it as a <button type="button"> instead. CSS HTML describes a page's overall structure and layout, and browsers have a default basic appearance for each tag. However, every page needs additional styling information added: colors, fonts, layout, and more. This is done with Cascading Style Sheets (CSS), a declarative language that allows adding styling rules to ased on queries that match some set of HTML elements. CSS declarations can be added to an HTML page by: Linking a separate CSS file in the <head> of the document Adding a <style> tag to the <head> with CSS declarations inside Alternately, styling rules can be written directly on an element in the HTML with the style attribute, referred to as inline styles. CSS Syntax CSS syntax consists of two major pieces: selectors, which are queries that define what elements will be matched rules, which are the individual styling values that will be assigned to all matching elements The basic syntax consists of a selector followed by a curly-brace block. The block may contain one or more rules, where each rule looks like property-name: value; Multiple selectors may be declared in front of a rule block, and again any elements that match any of those selectors will be affected by the style rules. A basic CSS example might look like: div.active { font-weight: bold; font-size: 24px; color: red; border: 1px solid blue; } CSS only allows comments using the same multi-line comment syntax as many other languages: /* comment here */ CSS Selectors CSS selectors can contain many different pieces that uniquely identify some set of elements. Some of the most common matching pieces are: Specific HTML tag types: by name of the tag, without angle brackets ( h1 ) Tags with a given classname: by a . in front of the classname ( .myClassname ) Tags with a given ID: by a # in front of the ID value ( #myId ) Nesting: adding spaces between selector query pieces ( main form.signup input#firstName ) Immediate children: putting a > between parent and child ( ul.items > li ) The "universal selector" uses a * to match everything There are also pseudo-class selectors, which apply an additional modifier to the selector matching, and are usually based on some aspect of the element's behavior. For example, the :hover pseudo-class only matches an element where the mouse is currently hovering, and the :disabled pseudo-class only matches an element that is currently marked as disabled. The :nth-child pseudo-class is commonly used for striping the background color of alternating table rows based on even/odd row numbers, and there's dozens of others as well. An element's final style is made up of all of the combined rules that apply to it. Children inherit most properties from their parents. A rule html {font-family: Courier New;} would set that font as a default for the whole document, but a nested child could have its own font-family value that would override the value it inherited. There are rules for selector specificity, which define which values take effect when there are multiple competing rules+properties that apply to a given element. More specific rules will override less specific rules. Inline styles have much higher precedence than styles defined separately. Adding a !important at the end of a rule raises its specificity by a large amount, like color: red !important;. CSS Styling Rules There are hundreds of CSS properties that can be modified. They range from very simple (changing the text color) to highly complex (creating multi-point color gradients, defining polygon clip masks). Multi-word CSS property names are hyphenated (background-color, font-weight). Many CSS properties can be grouped by concept, such as: Element backgrounds: background-color, background-image, background-repeat Borders: border-left, border-color, border-width Fonts: font-family, font-weight, font-size Spacing: margin-top, padding-bottom Sizing: height, width Layout: display, position, overflow, z-index Different properties can accept different kinds of values, and values can often be given in different units. For example, font-size accepts values using "length units", which can be in pixels (px), size relative to parent ( em), size relative to the <html> element ( rem ), actual display sizes ( in, mm, cm ), and more. Layout widths and heights are often given in percentages ( % ) or pixels. CSS examples: div { /* will apply to all div tags */ color: red; /* text color */ } div span { /* only spans nested in divs */ color : blue; /* overrides red color */ font-weight: bold; } #some-id { /* only applies to tag with this ID */ font-size: 24px; } .someClassname { /* any tag with this classname */ border: 1px solid green; } p.first.second { /* only paragraphs with both classnames */ margin: 5px; } div.classA , span.classB { /* applies to either */ min-width: 50%; max-height: 300px; } div > p { /* only paras as direct children of divs */ display : none; /* hide entirely */ } div.myClass:hover { /* only applies on mouse hover */ border: 1px dotted orange; } Page Layout There are many factors that affect how elements are positioned in a page, and what their size and shape is compared to other elements. HTML has default layout before defined for each element type, but CSS properties can completely change an element's position and layout behavior. The Box Model The default layout behavior is called the box model. Every HTML element is treated as a rectangular box, and falls into one of two categories: block boxes stack downwards vertically inline boxes stack sideways horizontally For example, <div> tags are blocks. Adding <div /><div /><div /> will produce three elements in a column going down the page. On the other hand, <span> tags are inline. Adding <span /><span /><span /> will lay out all three spans side-by-side. An element's default behavior can be changed using the various options for the display and position properties. For example, adding display: inline; to a <div> tag will make it behave the same as a <span> normally would. The size of a given element is made up of several pieces: The actual content Padding: extra spacing that is still "inside" the element Border : extra lines around / "outside" the element Margin: extra spacing that is considered "outside" the element The "padding and border outside" sizing behavior can sometimes feel unexpected, The behavior can be altered using box-sizing: border-box;, which alters the sizing calculations so that the "total width" of an element includes all of its border and padding as well. Display and Positioning The display and position properties can be used to alter how an element's layout behavior works. The display: block; and display: inline; values force an element to one of the standard layout behaviors. There are also specialized layout behaviors, like Flexbox and CSS Grid (described below). Elements can be removed from the page completely using display: none;. The position property takes several values, some of which work in combination with the top/bottom/left/right numeric properties: position: static: the default position: relative: offset the location by some amount position: fixed: stays in a single location on screen, even as the document moves around position: absolute: stays in a single location within the document, without following normal layout flow position: sticky: a newer option that keeps items in place for a while until the page scrolls well past them Any value other than static creates a new stacking context. Loosely put, any children will be positioned with their top and left values relative to where this element is located. The z-index property affects how elements "stack" on top of each other when they overlap. (Frankly, the z-index property confuses just about everyone, so I'm not even going to explain more than that here because I'll probably get it wrong, but please research it further.) Flexbox Flexbox is a CSS layout algorithm that allows aligning items into rows or columns, with rules for aligning items and wrapping them as they overflow. A flex parent is defined by adding a display: flex rule to an element. All of the immediate children are then laid out automatically as flex items. Key properties: flex-direction: the primary layout direction for the flexbox parent (row, column) justify-content: how items are aligned in that direction (start, end, separated) align-items: how items are aligned in the cross direction Centering items both vertically and horizontally had been an exceptionally difficult task on the web for many years, but Flexbox allows centering things fairly easily, such as display: flex; justify-content: center; align-items: center;. CSS Grid CSS Grid is a newer CSS layout algorithm designed for 2D layouts with various numbers of rows and columns. A grid container is defined by adding a display: grid rule to an element. All of the immediate children are then laid out automatically as grid items. Grid syntax is extensive, and allows defining sets of columns and rows with varying repeated patterns. Responsive Screen Size UI Users may view a site using a wide variety of devices with varying screen sizes: desktops such as 1920x1080, tablets such as 768x1024, or phones such as 375x812. Designers and devs frequently need to create layouts that look good at many different screen sizes, referred to as responsive design. Rather than make completely different sites, they may add CSS rules that only apply in certain cases, such as when the viewing device screen is in a specific size range. This is normally done using media queries, which are CSS conditions that only apply the selectors and rules inside if the condition matches. The CSS rules typically choose certain screen sizes as breakpoints so that the layout changes if your device is larger or smaller than that size. An example media query and breakpoint setup might look like: /* Small devices (portrait tablets and large phones, 600px and up) */ @media only screen and (min-width: 600px) { .notice { color: red; } } /* Medium devices (landscape tablets, 768px and up) */ @media only screen and (min-width: 768px) { .notice { color: green; } } /* Large devices (laptops/desktops, 992px and up) */ @media only screen and (min-width: 992px) { .notice { color: blue; } } Further Resources Browsers How Browsers Work: Behind the scenes of modern web browsers Code Cartoons: The whole web at maximum FPS (rendering engine painting) Code Cartoons: Inside a super fast CSS engine (styling rules and layout) Roundup of Web Browser Internals Resources "Can I Use?" browser feature availability database HTML Interneting is Hard: HTML and CSS Tutorials MDN: Introduction to HTML MDN: HTML Cheatsheet CSS MDN: CSS introduction CSS Tricks: How CSS Selectors Work CSS Vocabulary: interactive CSS terms visual explainer CSS Positioning 101 CSS Tricks: Complete Guide to Flexbox CSS Tricks: Complete Guide to Grid

How Web Apps Work: Client Development and Deployment

Web development is a huge field with a vast array of concepts, terms, tools, and technologies. For people just getting started in web dev, this landscape is often bewildering - it's unclear what most of these pieces are, much less how they fit together. This series provides an overview of fundamental web dev concepts and technologies, what these pieces are, why they're needed, and how they relate to each other. It's not a completely exhaustive reference to everything in web development, nor is it a "how to build apps" guide. Instead, it's a map of the territory, intended to give you a sense of what the landscape looks like, and enough information that you can go research these terms and topics in more depth if needed. Some of the descriptions will be more oriented towards modern client-side app development with JavaScript, but most of the topics are fundamental enough that they apply to server-centric applications as well. Other posts in this series cover additional topics, such as: HTTP and Servers Browsers, HTML, and CSS JavaScript and the DOM AJAX, APIs, and Data Transfer New terms will be marked in italics. I'll link references for some of them, but encourage you to search for definitions yourself. Also, some of the descriptions will be simplified to avoid taking up too much space or dealing with edge cases. My JavaScript for Java Developers slides also cover much of this post's content as well. Table of Contents JavaScript Development and Build Processes JavaScript Module Formats Legacy Module Formats CommonJS Modules Universal Module Definition ES Modules Compiling Bundling Source Maps Development Environments and Tools Node.js NPM Node Build Tools Dev Servers Hot Module Reloading Deployment Serving Build Output Polyfills Code Splitting Further Resources JavaScript Development and Build Processes It's important to understand the evolution of JavaScript's design and usage over time to understand why and how it gets used today. This quote sums things up well: The by-design purpose of JavaScript was to make the monkey dance when you moused over it. Scripts were often a single line. We considered ten line scripts to be pretty normal, hundred line scripts to be huge, and thousand line scripts were unheard of. The language was absolutely not designed for programming in the large, and our implementation decisions, performance targets, and so on, were based on that assumption. http://programmers.stackexchange.com/a/221658/214387 Today, applications routinely consist of hundreds of thousands of lines of JavaScript. This introduces a completely different set of constraints on how developers write, build, and deliver client-side applications. Those constraints result in a very different development approach than the early days. Instead of writing a few lines of JS and inlining them into an HTML page, modern web clients use complex build toolchains that are directly equivalent to compilers for languages like C++ or Java. JavaScript Module Formats Almost all languages have built-in syntax for declaring encapsulated "modules" or "packages". For example, a Java file might declare it is part of package com.my.project, and then declare a dependency on another package with import some.other.project.SomeClass;. C#, Python, Go, and Swift all have their own package definition and import/export syntax. JavaScript is not one of those languages. Unlike all these other languages, JavaScript originally had no built-in module format syntax. For many years, JS code was written as inline <script> tags directly in HTML, or as small .js files with a few shared global variables. As developers began writing larger application codebases, the community eventually began inventing their own module formats to help provide structure and encapsulation. Each of these formats was invented to solve differing use cases. (Note: code snippets in this section are purely to illustrate the syntax differences between the various formats - the actual code is not intended to run or do anything useful.) Legacy Module Formats Adding numerous script tags to a page has several issues. It can be very hard to determine the dependencies between different script files and load them in the right order. Also, since all top-level variables occupy the same global namespace, it's really easy to accidentally have variables with the same name overriding each other: <script src="jquery.min.js"></script> <script src="jquery.someplugin.js"></script> <script src="./components/dropdown.js"></script> <script src="./components/modal.js"></script> <script src="./application.js"></script> // dropdown.js var delay = 2000; // in ms // modal.js var delay = 4000; // in ms // application.js // Oops - is it 2000 or 4000? console.log(delay) Immediately Invoked Function Expressions (IIFEs) are a pattern that rely on JS variables being scoped to the nearest function. An IIFE involves defining a new function and then immediately calling it to get a result. This provides encapsulation, and was used as the basis of the "revealing module" pattern, where an IIFE returns an object that defines its public API (equivalent to a factory function or a class constructor): // dropdown.js (function(){ var delay = 2000; // in ms APP.dropdown.delay = delay; }()); // modal.js const modalAPI = (function(){ // No name clash - encapsulated in the IIFE var delay = 4000; // in ms APP.modal.delay = delay; $("#myModal").show(); function hideModal() { $("#myModal").hide() } // return a "public API" for the modal by exposing methods return { hideModal : hideModal } }()); The Asynchronous Module Definition format (AMD) was designed specifically to be used by browsers. A specialized AMD loader library first creates a global define function. AMD modules then call define() and pass in an array of module names they depend on, and a function that acts as the body of the module. The module body function receives all its requested dependencies as arguments, and may return any single value to act as its "export". The loader library then checks to see if all of the requested dependencies have been registered and loaded. If not, it recursively downloads other dependencies waterfall-style, and works its way back up the dependency chain to initialize each module function with its dependencies. // moduleA.js // Loader library adds a global `define()` function define(["jquery", "myOtherModule"], function($, myOtherModule) { // Body of the function is the module definition const a = 42; const b = 123; function someFunction() { } // Return value is the "exports" of the module // Can do "named exports" by returning object with many values return {a : a, publicName : b, someFunction : someFunction} }); // moduleB.js define(["backbone"], function(Backbone) { const MyModel = Backbone.Model.extend({}); // Can do a "default" export by just returning one thing // instead of an object with multiple things inside return MyModel; }); IIFEs and AMD modules are no longer actively being used for new development, but code using those patterns is still out there. CommonJS Modules The CommonJS module format was developed specifically for use with the Node.js runtime (a JS interpreter running outside the browser). Since Node has access to the filesystem, the CommonJS format was designed to load modules from disk synchronously as soon as they are imported. The Node.js interpreter defines a global require function that accepts either relative paths, absolute paths, or library names. Node then follows a complex lookup formula to find a file matching the requested path/name, and if found, immediately reads and loads the requested file. CommonJS modules do not have any outer wrapping function. The interpreter also defines a global module.exports variable, and the module defines its exported values by assigning to that variable. // moduleA.js // Node runtime system adds `require()` function and infrastructure const $ = require("jquery"); const myOtherModule = require("myOtherModule"); // The entire file is the module definition const a = 42; const b = 123; function someFunction() { } // Node runtime adds a `module.exports` keyword to define exports // Can do "named exports" by assigning an object with many values module.exports = { a : a, publicName : b, someFunction : someFunction } // moduleB.js const Backbone = require("backbone"); const MyModel = Backbone.Model.extend({}); // Can do a "default" export by just assigning // one value to `module.exports` module.exports = MyModel; CommonJS modules allow dynamically importing other modules at any time, and imports can be done conditionally. The flip side is that CommonJS modules cannot be used as-is in a browser - some kind of adapter or repackaging is needed. Universal Module Definition Some libraries need to be able to be used in multiple environments with the same build artifact: a plain global <script> tag in a browser, an AMD module in a browser, or a CommonJS file under Node. The community invented a bizarre-looking hack that allowed a module to work correctly in all three environments by feature-detecting capabilities, which was dubbed the Universal Module Definition (UMD) format. Today this is still semi-commonly used as a build output target for some libraries. // File log.js (function (global, factory) { if (typeof define === "function" && define.amd) { define(["exports"], factory); } else if (typeof exports !== "undefined") { factory(exports); } else { var mod = { exports: {} }; factory(mod.exports); global.log = mod.exports; } })(this, function (exports) { "use strict"; function log() { console.log("Example of UMD module system"); } // expose log to other modules exports.log = log; }); ES Modules The ES2015 language spec finally added an official module syntax to the JS language, which is now referred to as ES Modules or "ESM". It provides syntax for defining both named and default imports and exports. However, due to the differences between browsers and Node.js, the specification did not define how modules would actually be loaded by an interpreter, or what the import strings would refer to, and instead left it up to the differing environments to figure out how to load modules appropriately. Modern browsers have all implemented loading ES Modules based on URLs as the import strings. Node has had significantly more trouble determining a path forward, due to its reliance on CommonJS modules as the default format. As of Node 15, Node has some support for loading ES Modules, but there are still difficulties determining how CommonJS and ES Module files should interop with each other. ES Modules were designed to be statically analyzable. The downside is that you cannot do dynamic or conditional imports - all imports and exports must be at the top level of the file. // moduleA.js // ES6 language spec defines import/export keywords import $ from "jquery"; // Can do "default" imports - no curly braces around the variable name import myOtherModule from "myOtherModule"; // Define "named exports" by adding `export` in front of a variable export const a = 42; export const b = 123; export {b as publicName}; export function someFunction() { } // moduleB.js // Can do "named imports" from other modules import {Model} from "backbone"; const MyModel = Model.extend({}); // Can do a "default" export with the `export default` keyword export default MyModel; Compiling The JS language spec has added lots of additional syntax over the years. The ES2015 spec in particular effectively doubled the amount of syntax in the language. Each time a browser ships a new version, that version has a fixed understanding of a certain subset of the JS language. Since that browser version may stay in wide use for many years, developers need to ship code using only syntax that is supported by the set of browser versions they intend to support. However, developers also want to be able to write and develop code using the latest and greatest syntax. This means that developers need to compile the original "current" JS source code they've written into an equivalent version that only uses older syntax. The Babel JS compiler is the standard tool used to cross-compile JS code into a different variation of JS. It has a wide array of plugins that support compiling specific pieces of newer JS language syntax into their older equivalent forms. As an example, this ES2015-compatible snippet uses ES Module syntax, arrow functions, the const/let variable declaration keywords, and the shorthand object declaration syntax: export const myFunc = () => { let longVariableName = 1; return {longVariableName}; } When compiled by Babel with the ES2015 plugin enabled and targeting the CommonJS module format, it becomes: "use strict"; Object.defineProperty(exports, "__esModule", { value: true }); exports.myFunc = void 0; var myFunc = function myFunc() { var longVariableName = 1; return { longVariableName: longVariableName }; }; exports.myFunc = myFunc; In addition, there are many "compile-to-JS" languages in use in the industry. Some are obscure niche languages, some were popular for a few years and have since died out (CoffeeScript). The most commonly used compile-to-JS language at this point is TypeScript, which is a statically-typed superset of JS created by Microsoft. The TypeScript compiler itself strips out type annotations at compile time and outputs plain JS. Similar to Babel, it can also compile newer syntax into varying older language versions. // Input: TypeScript syntax is JS with type annotations const add2 = (x: number, y: number) => { return x + y; }; // Output: plain JS, no type annotations const add2 = (x, y) => { return x + y; }; Bundling There are multiple reasons why the original JS source cannot be delivered as-is to the browser: Code written in CommonJS format cannot be loaded by browsers Code written in ES Module format can be, but requires careful work to have all the files and path URLs line up correctly Target browsers used by consumers likely don't support all modern syntax Codebases may consist of thousands of separate JS files, and downloading each file separately would take too long to load Original source contains comments, whitespace, and longer variable names, and developers need to minimize the number of bytes sent to the browser to let pages load faster Languages like TypeScript are not supported by JS interpreters - the original source has to be compiled to plain JS syntax Because of this, JS code is also bundled to prepare it for use in a browser. This has to happen in both a development environment, and production. The bundling process traces the tree of imports and dependencies starting from a set of entry point files (such as src/index.js). Any imported file is then added to the list of files to be processed. The bundler resolves all requested imports, determines the necessary loading order, and outputs the module sources wrapped in some scaffolding that initializes the application when the bundle file is loaded. Bundling tools also typically support multiple additional processing steps during the bundling process. In particular, bundlers will usually be configured to: run a compiler like Babel or TypeScript on all JS/TS source files If using TS, do typechecks with the TS compiler to verify the code actually compiles Enable importing and processing additional assets like CSS and images Optimize the size of the output to make it as small as possibly, by minifying it (also known as uglifying). Minifying JS source involves shrinking the code as much as possible, by stripping out whitespace and comments, replacing long variable names with shorter names, and using the shortest possible versions of syntax. Finally, minifiers can detect dead code and remove it, and JS code is often written with flags like if (process.env.NODE_ENV !== 'production') to add development-only checks that will be removed in a production build. The same Babel-compiled output above looks like this when minified: "use strict";Object.defineProperty(exports,"__esModule",{value:!0}),exports.myFunc=void 0;var myFunc=function(){return{longVariableName:1}};exports.myFunc=myFunc; Webpack is the most widely used JS bundler. Other tools such as Parcel, Snowpack, and ESBuild fulfill the same roles, but with different goals and constraints. Source Maps Because of these transformations, the code loaded by a browser has been mangled into a completely unrecognizable form that makes it impossible to actually debug as-is. To solve this, development tools also write out source maps, which map segments of the output file back to their original lines of source. This allows browser debuggers to show the original source code, even if it's a language not actually supported by the JS interpreter in the browser. The browser will show the "original source" in its individual files, and allows developers to debug that "original source" by setting breakpoints and viewing variable contents. Development Environments and Tools Node.js Node.js is a runtime for executing JS outside of a browser environment. It's equivalent to the JRE for Java, the .NET Framework SDK, or the Python runtime. It consists of the V8 JS engine from Chrome repackaged for use as a standalone executable, along with a standard library of APIs for interacting with the file system, creating sockets and servers, and much more. NPM "NPM" has three meanings: NPM is the publicly-hosted JS package registry that hosts third-party JS libraries and packages published by the community npm is an open-source CLI client used for installing packages from that registry NPM is a company that runs the registry and develops the CLI client (recently bought by Microsoft) Libraries and packages installed off of NPM are put into a ./node_modules folder. So, npm install redux downloads a published archive from the NPM registry servers, and extracts the contents into a new ./node_modules/redux folder. Yarn is an alternative package manager that installs the same packages from the same public NPM package registry. It has the same core capabilities as the npm tool, but provides some different configuration options. Node Build Tools Since most JS build tools are written by JS developers for JS developers, the build tools themselves are typically written in JS. This includes widely used tools like Babel, Webpack, ESLint, and many others. So, to run them, you must have Node.js installed in your development environment. (As described below, you do not need Node.js installed on a server machine just to run your client code in a browser, unless you also have written your server application itself in JS.) There's been a recent trend of alternative JS build tools being written in Rust or Go, with a goal of making compilation and bundling much faster through use of native code and parallelism. Most of these tools haven't hit mainstream usage yet, but the potential speedups are big enough that those tools are likely to pick up usage. Dev Servers Because the original source needs to be repeatedly re-compiled and re-bundled as a developer makes changes locally, the typical development process involves launching a dev server, a separate process that detects edits to the original source files and rebuilds the client code with the changes. The dev server process typically acts as an HTTP proxy, and forwards requests for data and assets onwards to an actual application server. A typical example might look like: App server process listening on port 8080 Dev server process listening on port 3000 The developer would then browse to http://localhost:3000 to load the page. The dev server on port 3000 receives the request, loads the HTML host page and bundled JS source from its memory, and returns it. When the browser requests http://localhost:3000/images/avatar.png, the dev server forwards that to the app server at http://localhost:8080/images/avatar.png instead. Similarly, a request for data by the browser to GET http://localhost:3000/items would be forwarded to the app server at http://localhost:8080/items, and the response passed back through the dev server to the browser. Webpack has a prebuilt dev server package available, and other tools such as Create-React-App often wrap around the Webpack dev server to provide additional configuration and capabilities. Hot Module Reloading Normally, recompiling a web app requires completely reloading the page to see the changed code running. This wipes out any state loaded into the app when the page is refreshed. Tools like Webpack offer a "hot module reloading" ability. When a file is edited, the dev server recompiles with the changes, then pushes a notification to the client code in the browser. The app code can then subscribe to "some file changed" notifications, re-import the new version of the code, and swap out the old code for the new code as the app is still running. Other tools like React's "Fast Refresh" mode can then leverage that reloading capability to swap specific parts of the app, such as replacing individual React components in real time. Deployment Serving Build Output The output of a normal bundler build process is a folder full of static JS, HTML, CSS, and image files. Here's the output structure of a typical React app: /my-project/build - index.html /static /css - main.34928ada.chunk.css - 2.7110e618.chunk.css /js - 2.e5df1c81.chunk.js - 2.e5df1c81.chunk.js.map - main.caa84d88.chunk.js - main.caa84d88.chunk.js.map - runtime-main.d653cc00.js - runtime-main.d653cc00.js.map /media - image1.png - image2.jpg - fancy-font.woff2 These are simple static files that can be served up by any web server. To deploy these files, they just need to be uploaded to an appropriate location on the machine that hosts the server application. This is often done using a file transfer protocol such as SFTP. Polyfills There are many browser APIs that do not exist in older browsers, but cannot be handled by backwards-compiling syntax. This includes built-in functions, classes, and data types. A couple examples of this are the String.padStart() method and the Map data structure. However, some of these can still be polyfilled with developer-provided implementations. Polyfills are extra code that is executed when an app is loaded, detects if a given feature exists at runtime in the current environment, and adds an artificial-but-equivalent implementation dynamically. As an example, a polyfill for String.padStart() might look something like: if (!String.prototype.padStart) { String.prototype.padStart = function padStart(targetLength,padString) { // actual logic here } } Code Splitting Even with minification, JS bundles can get much too large (250K, 1MB, or worse). This is often due to use of many third-party libraries for additional functionality. Code splitting allows bundlers to break very large bundles into smaller chunks. The extra chunks are either added as additional <script> tags into the host HTML page, or dynamically downloaded as the app is running. Some chunks might contain only third-party code, so that the "vendor chunks" can be cached by the browser and only downloaded the first time a user visits a site. Or, chunks might be common logic shared between multiple parts of an app, such as common utilities used by the main user-facing page and an admin page. Some chunks might be "lazy loaded" only when the user activates a certain feature. For example, a rich text editor implementation might add an extra 500K to the bundle, but if it's only used in a specific modal dialog, the dialog code can dynamically import the text editor library. The bundler then detects that dynamic import, splits the editor code into a separate chunk, and the chunk is only downloaded when the user opens that modal. Further Resources JS Module Formats History of JavaScript: Evolution of JS Modularity JavaScript Modules: From IIFEs to CommonJS to ES6 Modules Understanding ES6 Modules via Their History ES Modules: A cartoon deep-dive Build Tools The Many Jobs of JS Build Tools Unpacking Webpack Babel and Webpack Fundamentals How does the Development Mode Work? Deployment Create-React-App docs: Deployment Fullstack React: Deployment introduction

How Web Apps Work: HTTP and Servers

Web development is a huge field with a vast array of concepts, terms, tools, and technologies. For people just getting started in web dev, this landscape is often bewildering - it's unclear what most of these pieces are, much less how they fit together. This series of posts provides an overview of fundamental web dev concepts and technologies, what these pieces are, why they're needed, and how they relate to each other. It's not a completely exhaustive reference to everything in web development, nor is it a "how to build apps" guide. Instead, it's a map of the territory, intended to give you a sense of what the landscape looks like, and enough information that you can go research these terms and topics in more depth if needed. Some of the descriptions will be more oriented towards modern client-side app development with JavaScript, but most of the topics are fundamental enough that they apply to server-centric applications as well. Other posts in this series cover additional topics, such as: Client Development and Deployment Browsers, HTML, and CSS JavaScript and the DOM AJAX, APIs, and Data Transfer New terms will be marked in italics. I'll link references for some of them, but encourage you to search for definitions yourself. Also, some of the descriptions will be simplified to avoid taking up too much space or dealing with edge cases. The MDN web technologies documentation is an excellent resource for reviewing many of the specific terms and concepts listed here in more detail. Table of Contents HTTP HTTP Networking HTTP Requests and Responses HTTP Headers HTTP Methods GET Requests POST Requests PUT Requests PATCH Requests DELETE Requests HTTP Status Codes Cookies Servers Basic HTTP Request Processing Routing Static Files Dynamic Server Application Logic Response Formats Further Resources HTTP Browsers effectively only speak one language to request information: the HyperText Transfer Protocol (HTTP). This means that a browser must talk to a server that also speaks HTTP and can respond to the browser's requests for information. HTTP Networking All computer networking at the software level is based on an abstraction called a socket. Sockets represent a specific open connection to another computer. A server starts listening for incoming connections, and a client asks to open a new connection. Connections are defined using numeric dot-separated IP addresses like 192.168.0.255, and an additional numeric port numbers like 8080. An individual IP address segment can be from 0-255, and port numbers range from 0-65535. Think of an IP address as a street address for an apartment building, and a port number as a specific room in that building. Any data can be sent over a socket - binary or text - it's all just bytes. Plain HTTP is an unencrypted protocol. HTTP Secure (HTTPS) is the same content, but encrypted before being sent. Standardized networking protocols use specific well-known port numbers. Plain HTTP defaults to port 80, and HTTPS defaults to port 443. So, an address of http://192.168.0.255 implies port 80, and https://192.168.0.255 implies port 443. Other ports are frequently used depending on project setup. For example, many web server applications listen on port 8080 or 3000 in a development environment, so in that case you would use an address like http://192.168.0.255:3000 as the destination for an HTTP connection. Since numeric URLs are hard to remember, and a site may be made up of many systems working together, the Domain Name System (DNS) maps text name URLs like www.google.com to a specific set of numeric IP addresses. A URL like http://www.example.com:8080/some-page?id=42#intro can be split into multiple pieces: protocol: http:// subdomain: www domain name: example.com port: 8080 path: some-page query parameters: ?id=42 fragment: #intro HTTP Requests and Responses HTTP is a plain-text, "stateless", request/response-based protocol. In other words: A client application has to open a connection to the server and send a request using a text-based format The server then parses the request, runs code to process and handle the request appropriately, and sends back a text-based response. Once the response has been sent, it closes the open socket connection. Each request/response is an individual transaction between a client and a server, and by default there is no correlation between separate requests Both requests and responses may also contain binary data, but the HTTP-related contents of the request and response are readable text instructions. A typical HTTP request looks like: GET https://www.example.com/description.html HTTP/1.1 Host: www.example.com User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) Chrome/16.0.912.75 Safari/535.7 Accept: text/html,application/xhtml+xml Referer: http://www.google.com/url?&q=example Accept-Language: en-US,en The first line of the request is broken down into three pieces: The HTTP method or verb, such as GET or POST The path of the request The HTTP protocol version After that, the request may contain additional headers. Each header is a piece of metadata that the client can include so that the server better understands what kind of information is being requested. The request may also include a body after the headers, with additional data. Responses follow the same structure: HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 Server: Apache/2.4 Date: Sun, 01 Nov 2020 16:38:23 GMT Content-Length: 17151 <html> <head> <title>Page Title Here</title> </head> <body> Content here </body> </html> The first line contains: The HTTP protocol version The HTTP status code, in a numeric form The HTTP status reason, in a text form That is followed by additional header lines containing metadata about the response, including the total byte size of the body. Finally, the response includes the actual body contents. HTTP Headers There are dozens of header fields that can be sent over HTTP, and both clients and servers can send additional arbitrary headers they've defined. Some of the most common headers are: Requests User-Agent: a string describing the specific type and version of the client Cookie: a small piece of data previously attached by the server to help track this client Referer: URL of the previous page where a link was clicked Responses Content-Length: size of the response in bytes Location: redirect request to a new URL Set-Cookie: creates a new cookie value Both Content-Type: name of the format used in the request / response body HTTP Methods The HTTP protocol specifies several possible methods or verbs. Each HTTP method represents a different type of request intent: GET: the client wants to retrieve information POST: the client wants to create or update data on the server PUT: the client wants to update or replace data on the server DELETE: the client wants to delete some data on the server Each HTTP method has different variations in how the request is formatted, and when/why it should be used by a client. In addition, some types of requests can be treated as "idempotent" (can be done many times without causing additional changes). GET Requests GET requests are used to retrieve information from the server, based on a specific URL. GET requests do not contain a request body. However, clients may include additional data as query parameters options attached to the main URL. Query params start with a ?, and are formatted as key=value pairs separated by ampersands: /endpoint?a=1&b=stuff. Spaces and special characters in URLs may need to be URL-encoded, where the original value is replaced by a % and a number: ?a=Some Value might become ?a=Some%20Value. Since GET requests are only used for retrieving data, servers should not update data in response to a GET. This means it should be safe to make the same GET request multiple times without causing side effects. POST Requests POST requests are used to tell the server to update some data or process some information. POSTs typically include all relevant information in the body of the request, and rarely include query params. POST request bodies typically use a few common formats "Form-encoded": the same key=value structure as query parameters, but in the body of a POST "Multi-part form data": a delimited format that splits the body into sections "JSON": a string representation of JavaScript data structures like objects and arrays PUT Requests PUT requests are very similar to POST requests. Both involve sending data to the server with an intent to update. The intended difference is that a PUT is intended to create or replace a value, while a POST is intended to create or update a value. Conceptually, a PUT should be safe to do multiple times in a row, while a POST is likely to cause something to happen separately for each request. PATCH Requests PATCH requests are also similar to PUT requests, but the intent is to send a partial representation of an item, while PUT is meant to send the complete representation of an item. DELETE Requests DELETE requests are used to ask a server to delete some data. Conceptually, it should be safe to make a DELETE request multiple times - if a value is already gone, the server ought to ignore the request. HTTP Status Codes All HTTP responses include a numeric status code, along with a text "reason" string that describes the status code value. Status codes are 3-digit numbers, grouped into these ranges: 1xx: Informational 2xx: Successful 3xx: Redirection 4xx: Client error 5xx: Server error Common HTTP status codes include: 200 ("OK"): successful response 301 ("Moved permanently"): the resource at this URL has been moved to a new URL 304 ("Not Modified"): The value at this URL has not changed since last asked 400 ("Bad Request"): the request was not formatted properly 401 ("Not Authorized"): not authenticated to view this 403 ("Forbidden"): not allowed to view this 404 ("Not Found"): server does not recognize this URL 500 ("Internal Server Error"): something went wrong on the server while handling this 503 ("Service Unavailable"): a portion of the server is down and cannot respond Cookies HTTP is inherently "stateless" by default, but servers normally want some way to correlate a specific client's identity between multiple requests. For example, if I log into a forum site, the server should remember that I'm logged in as I browse through different pages. The HTTP spec allows servers to include Set-Cookie headers in responses with specific keys and values: HTTP/2.0 200 OK Content-Type: text/html Set-Cookie: yummy_cookie=choco Set-Cookie: tasty_cookie=strawberry Whenever a client makes future requests to that server, the cookie values will be automatically included in the request: GET /sample_page.html HTTP/2.0 Host: www.example.org Cookie: yummy_cookie=choco; tasty_cookie=strawberry Typically, a server will set a "session cookie" that contains some unique ID value, and map that unique ID internally to additional data (such as "session ID 12345 is actually user mark"). That way, each time a request is made, the server can look up the additional data needed to handle that specific user ("Request is for /messages, and the session data says the user is mark - query the database for all messages sent to user mark"). Servers The word "server" has multiple related meanings depending on the context: The application that opened up a socket to listen for incoming requests An application that is specifically able to handle HTTP requests and send back responses The physical or virtual machine that is running that server application Here, we're going to focus on the "HTTP request handling application" meaning. Basic HTTP Request Processing Every HTTP server application starts handling a request by accepting an incoming socket connection, and parsing the request contents into some internal data structure. The server then inspects the HTTP request path and the method to determine the intent of the request. Depending on how the server has been written and configured, it will then handle the request by doing some combination of: Reading files from disk Connecting to a database and loading / updating data Updating session tracking information internally Running developer-specified logic Ultimately, the server application will fill out an HTTP response message based on all of the processing, write it to the socket, and close the connection. Routing Routing refers to determining what code to run based on a request URL. Servers normally use a combination of the HTTP request method and the request URL path to determine what code should handle a request. For example, GET /users might be handled by one specific function, POST /users by a second function, and GET /images/header.png by another function. Servers typically use a convention that if /some-folder/index.html exists, it will be used as the response contents for a request to GET /some-folder/. This pattern has carried over into many other areas as well. For example, a server written using the PHP language might use /some-folder/index.php to handle requests, and the Node.js CommonJS module format will handle an attempt to load require('some-folder') by opening some-folder/index.js if it exists. Static Files Many HTTP requests are asking for the server to return a copy of a specific file, such as GET /images/header.png or GET /app/index.html. A server will usually handle these by mapping URLs to a specific set of folders on disk, and checking to see if a file by that name exists in the specific folder path. If it does, the server will immediately handle the request by reading the entire file from disk and returning its contents as the body of the response. Since these files on disk typically do not change often, they are known as static files. Serving static files is a baseline capability that all HTTP server applications can do, and most web server frameworks provide a built-in mechanism for automatically serving files from a designated folder. There are many sites and tools which only host and serve static files - a user just needs to upload their files, and the site is already configured to serve them automatically. Dynamic Server Application Logic However, servers also need to respond to incoming requests by running additional logic written by a developer. The exact syntax and techniques used to define server logic vary widely across languages and server application frameworks, but they all share some common aspects. A typical web application server framework will allow the developer to: Specify which HTTP method is being handled Specify what URL routes are being handled Read an object representing the parsed HTTP request, which will contain fields with all the headers, other metadata, and body contents Interact with an object representing the in-progress HTTP response, which will contain fields and methods to help generate the final response contents Many web app frameworks also implement some form of middleware, which are individual chunks of logic that are combined together. Each individual middleware typically has a specific purpose, such as adding session info to the request based on a cookie. The combined middleware then form a pipeline of preprocessing steps that are executed on every incoming request before the request is handled by the specific application logic. The request handling code may then use whatever logic it wants to dynamically generate a response. The handler logic might: Check the URL and HTTP method Check cookies for a session ID and look up a user's details internally Read query parameters Extract data from a request body Connect to a database to retrieve information Run some calculations Update the database based on the calculations or the request contents Construct an HTML document or a JSON data structure Send that content back as the response body Here's a (fictional!) typical example using the Express HTTP library for the Node.js runtime: const express = require('express'); const session = require('express-session'); const db = require('./dbConnection'); const app = express(); // Add middleware to populate req.session app.use(session({ secret: 'keyboard cat' })); // Add middleware for processing incoming request contents app.use(bodyParser.json()) app.use(bodyParser.urlencoded({extended: true})) // Add middleware to serve static files // URLs starting with '/static' will be mapped to files in the './public' folder app.use('/static', express.static('public')) // Define a request handler function that receives objects for request and response function countViews(req, res){ // Build up an HTML body string as we go through let body = ''; // Session middleware added data to the request object if (req.session.views) { ++req.session.views; } else { req.session.views = 1; body += '<p>First time visiting? view this page in several browsers :)</p>'; } // Send back HTML content in the response res.send(body + '<p>viewed <strong>' + req.session.views + '</strong> times.</p>'); } async function addBook(req, res) { // JSON body contents were processed into plain JS objects const {author, title} = request.body;; // Send update request to DB and wait for it to complete await db.query( 'INSERT INTO books (author, title) VALUES ($1, $2)', [author, title] ); // Send a specific HTTP status message and JSON-formatted response res.status(201).json({status: 'success', message: 'Book added.'}) } // Define a handler for GET requests to the '/' URL path. app.get('/', countViews); // Define a handler for POST requests to the '/books' URL path app.post('/books', addBook); if (!module.parent) { // Use port 3000 to listen for incoming request socket connections app.listen(3000); console.log('Express started on port 3000'); } The exact syntax for each of these steps varies based on programming language and web framework, but all frameworks share these same basic concepts and capabilities in some form. Response Formats Servers can send back any data they want in a response, but most responses are sent back with one of a few common formats: HTML: the standard markup language that describes the structure and content of a web page CSS: the standard styling language used to define the appearance of a web page JSON: a text-based data format based on the syntax for JavaScript objects, arrays, and primitive values XML: a customizable text-based data format based on nested tags, similar to HTML Actual static files (images, JavaScript files, etc) For data, JSON has become the de-facto standard data transfer format. Further Resources HTTP MDN: An Overview of HTTP Anatomy of a URL HTTP Made Really Easy Introductory HTTP Book A Software Developer's Guide to HTTP History of the browser user-agent string Stack Overflow: How are parameters sent in an HTTP POST request What is DNS? Other Overviews MDN web technologies documentation What happens when you type 'google.com' and press Enter? Web Applications 101 Build a Node, Express, and PostgreSQL REST API

Global React Meetup: The State of Redux 2020

Intro I've previously given "State of Redux" talks at React Boston 2018 and Reactathon 2019. Today I got to give an updated version of the talk for the Global React Meetup vidcast, hosted by ThisDot. I covered the same general topics as before, but with the latest updates on what's happening with Redux: Is Redux dead? (nope!) React-Redux status and future plans Redux Toolkit background, APIs, and future enhancements Redux docs rewrites, including the new "Redux Essentials" and "Redux Fundamentals" tutorials As usual, here's the video link: Video: The State of Redux, October 2020 And the slides: Slides: The State of Redux, October 2020

Coding Career Advice: Evaluating Software Libraries and Tools

Tip #2: How to Correctly Evaluate Third-Party Software Libraries and Tools As a senior developer, I spend a lot of my time looking at tools and libraries to decide if they're something we can or should use as part of our projects. I've developed a strong ability to quickly filter through lists of possible tools to narrow them down, and then do further research on the remaining candidates. This is definitely not the kind of skill that they teach you in school, which is sad given the way modern software development is strongly dependent on reusing libraries. To help with that, here's some of the things I take into consideration when evaluating tools. Starting Point: Search Locations For software libraries, you should typically start with the appropriate package manager site for the language you're using, such as NPM for JavaScript, PyPi for Python, or Maven Central for Java). If you're looking for non-library tools such as IDEs, Google is your best bet. Searching for keywords on those sites should turn up a list of things that are potentially relevant. Potential Evaluation Criteria When choosing any tool, you have to evaluate it on multiple criteria. Here's an (incomplete) list of things you may want to consider when looking at any software library or tool that you're considering using: Zeroth Pass: Relevance The first step is to select a list of tools that actually look relevant to your topic. For example, searching NPM for react number input right now turns up 219 hits. There's probably only a much smaller number that actually match what you're looking for. Skim through the hits and pop open the most likely looking options based on the initial descriptions. First Pass: Basic Suitability Now that you've got the initial candidate list, it's time to determine which of these may be a good possibility: Does this even solve the problems I have right now? Does it solve those problems well? What's the license of this tool? Is it compatible with my company's policies? The goal of using any tool is to solve a problem you have. If a tool doesn't solve your problem, you shouldn't even consider using it in the first place. (Note that this implies that you should understand what problems you're trying to solve before you start investigating tools to help with them :) ) Along with that, there's the question of whether a tool is something you can use based on legal or policy concerns. For example, many companies disallow use of GPL-licensed software libraries to avoid any possible issues mixing their proprietary code with GPL-licensed code and being forced to distribute their own code publicly. Other companies might restrict libraries and applications based on the nationality of the authors, purchased licensing terms, or license costs. One-line summary of the most common open-source (OSS) licenses: MIT, BSD: use the code for whatever you want, just credit the original author LGPL: Use it as long as you don't modify the original library GPL: If you use it, your own software must be licensed under the GPL as well, and your source must be made available on request If you know ahead of time that you can't use a particular tool, there's no point in digging into it further. Second Pass: Detailed Evaluations Once you've narrowed down the list down to the top 3-4 choices, you can do a more detailed evaluation of each of them. Here's some of the criteria that I take into consideration. I don't necessarily have these in a formal checklist, but they're usually in the back of my head in some form. (This list is biased towards OSS software libraries, particularly for JavaScript, but most apply more broadly as well). How easy is it to use? What's the size of the community? How many projects appear to use this tool? How many downloads does this have weekly/monthly? How much info is there on using it outside the core docs? What's the size of the ecosystem? How many related packages exist? If we have an additional use case not solved by the core tool, do addons exist that handle that use case? How easy is it to extend or modify this tool? How easy is it to understand what the tool does internally and conceptually? How much documentation does this have? Is there an actual meaningful docs site, or is it just a Github readme? How well tested is this tool? How much prior experience does my team have with this tool or something similar? Is it well maintained? How many bugs does it have? Has this ever had any critical vulnerabilities / CVEs? If so, how many, how often, and how long did they take to get fixed? How often do new releases occur? When was the last release? How many open issues / PRs are there? How fast will things get fixed if there's a bug? How many maintainers are there? Are they active? What's the long-term roadmap for this project? How many other dependencies does this tool have? How many of them are runtime dependencies vs development-only dependencies? Do we need to be concerned about any of them? How much does this affect the architecture of our app? Can we rip it out if we choose to switch to something else? How hard is it to upgrade this tool when a new version comes out? Where does this fall in the "hype cycle"? Brand new tool that is experimental? It works, but still only used by early adopters? Mainstream? Dying? Are people actively starting new projects with this tool, or are they moving away? What technical requirements and system constraints does this have? What's the minimum runtime version? Are there size concerns, such as min+gzip size for a JS library used in the browser? This isn't an all-inclusive list - there's likely other factors you might want to consider as well. These all affect a decision on whether using a given tool is a good idea or not for your team. It's also likely that some of these may be more relevant than others depending on your situation. Note that you could easily spend an infinite amount of time recursing through the transitive dependencies of a given tool, especially if this is a JavaScript package of some kind. For example, a brand new Create-React-App project only has a handful of dependencies: react, react-dom, @testing-library/react, and react-scripts. However, react-scripts itself depends on Jest, ESLint, Webpack, Babel, PostCSS, and a couple dozen other packages, and Webpack in turn ends up pulling in around 700 packages, for a final total of just over 1500 individual packages installed into your node_modules folder. Clearly, inspecting every one of those manually is not feasible, so you have to draw a line somewhere. Given the rise of "supply chain" attacks, where malicious packages are deliberately created to steal information or cause harm, it's critical to make these decisions wisely. It can sometimes be worth doing a "trade study" to evaluate alternatives, especially if multiple people are involved in this decision. Define several categories, decide on the possible range of numeric scores for each category, and rate each tooling option in each category. You may want to add together the scores for each tool, or average them out. Also, you might want to weight some categories as being more important than others. I suggest defining the weights and scoring approach ahead of time, so that the final decision is more fair. When searching for critical security vulnerabilities like CVEs, you can use sites like https://cve.mitre.org/cve/search_cve_list.html and https://nvd.nist.gov/vuln/search as references. For open-source projects on Github, it's worth looking at the number of stars, open issues, and PRs. Those aren't final decision criteria, but they can give you a sense of the popularity and health of a project. Also keep an eye on the recent commits to get an idea how active it is. NPM shows a weekly download stats graph on the package description page, but I prefer using npm-stat.com, which lets you compare the download stats for multiple packages over a specific period of time, with values shown for daily/weekly/monthly/yearly downloads totals. For JS client bundle size questions specifically, I strongly recommend using Bundlephobia to determine the true cost of adding a given package to your app. Michel Weststrate's import-size util can also be helpful here. Further Information TLDR Legal: shortened plain English explanations of common software licenses Bundlephobia mweststrate/import-size Trade Study guidelines npm-stat.com 17 criteria for evaluating third-party libraries The 12 Things You Need to Consider when Evaluating Any New JavaScript Library Evaluating 3rd party libraries

Coding Career Advice: Keeping a Daily Work Journal

Introduction As I've progressed through my career as a developer, I've learned a bunch of useful things based on my experiences. I've frequently shared some of these thoughts with other developers on my team and various folks I've chatted with online, so it's worth trying to write these down so I can share them more widely. None of these thoughts are particularly new, and in fact I'll try to link to other resources that say similar things. But, writing them down means I've got somewhere to point people to in the future, and maybe they'll be new to someone who hasn't seen these thoughts before. Note: For a much more extensive set of career advice on many topics, I strongly recommend Shawn Swyx Wang's book The Coding Career Handbook: Guides, Principles, Strategies, and Tactics from Code Newbie to Senior Dev. There's a ton of valuable info in there, and it's totally worth the purchase. Tip #1: Keeping a Daily Work Journal At the start of 2013, I started keeping a daily work journal. At the end of every workday, I take 10-15 minutes to write down a few paragraphs covering what I did that day, such as: Tasks I worked on and what specific things I did for those Conversations with other developers on my team and what topics we covered, or key takeways from discussions outside the team Problems I ran into, and (hopefully) how I solved them Where I left off at the end of the day What tasks I need to prioritize the next day Useful code snippets I don't hit all of those points every day, but they're all common things I cover. Why Keep a Work Journal? Writing journal entries like this has a lot of potential benefits: When I pick up work after a long weekend, it reminds me what I'm even supposed to be working on now I sometimes run into a bug that I know I solved months or even years ago. There's been several times I dug back into my notes and found the solution I'd written down the last time. It's often interesting to look back at how I approached problems years ago, and see how I've grown since then But most of all, it's extremely valuable come performance review season. I can go back through every single workday of the past year, see exactly what tasks I worked on, and put together an extensive list of accomplishments for my self-evaluation writeup. Journaling Tools I strongly prefer to write notes in Markdown format, with actual syntax highlighting in the Markdown text and the rendered output. I currently use the "legacy" Boostnote desktop app (v0.16), although I may end up switching to the "modern" Boostnote app in the near future once it looks mature enough. Boostnote does a pretty good job handling Markdown and syntax highlighting. It also has decent search capabilities. I've also been writing a lot of Markdown documentation in VS Code lately, and there's some notebook plugins for VS Code that look like they might be viable options as well. You should feel free to use whatever works best for you, electronic or analog, although I would suggest something electronic because it's probably easier to search for text keywords. Journal Organization I have a separate folder for each month, typically named like 2020-09 (September), and create a separate Markdown note for each day, named like 2020-09-21 (Monday). Legacy Boostnote sadly doesn't have nested folders, so all folders for all years are flattened out, but that makes it pretty easy to jump straight to a period of time that I'm interested in. (Looks like new Boostnote does support nested folders, so that may be a reason for me to switch.) I also have a separate parent category for other notes organized by topics. Example Journal Entry Here's a very slightly anonymized version of a real journal entry I wrote recently: 2020-09-03 (Thursday) Talked to $DEV_1 and split the remaining "UI polish" tasks up. Fixed an issue with input cells not focusing in FF. Turns out FF doesn't trigger click events on disabled inputs. Had to apply input[disabled] { pointer-events: none; }, per Stack Overflow. I tried to get sticky table columns to work, and bounced off that. Too tough for now, so we'll put that for later. Had a good talk with $UX_DESIGNER. He's been busy the last few weeks with other projects, so we caught up on the state of our work. He's going to see if he can get a couple other UX folks to look at what we've got. He'll also think about the best way to display things like the couple dozen columns for $BUSINESS_FEATURE. I tossed out the idea of showing $FEATURE instruction help as a separate new popup window. He agreed that was viable. Found react-new-window, pulled that in. Was able to copy and paste field definitions from Excel and convert to HTML via Tableizer. Threw together help pages for the three $FEATURE tabs so far. Reviewed $DEV_2's PR that had removal of dead $FEATURE code, backend models converted to TS, a unit test for deleting $ITEM, and a bunch of other cleanup. Looks pretty good overall. I'll try to knock off the remaining UI polish tasks next. Further Information Keep journals to become a better developer Getting started with a code journal Tools for Brainstorming and Tracking Accomplishments The More Senior Your Job Title, the More You Need to Keep a Journal HN discussion thread

Greatest Hits: The Most Popular and Most Useful Posts I've Written

Intro I've accumulated a decent number of posts on this blog over the last few years. I've received a lot of compliments on the quality and usefulness of many of them, with comments like: This is by far the best, most comprehensive blog post I've seen on this sub; by an author who clearly has a deep and precise understanding of how React and Redux work. Well done! /u/minty901 on Reddit I imagine it's a pretty thankless task being the redux maintainer these days, especially since useReducer appeared. Despite this Mark is always a positive, calm and insightful voice in all the React discussions I read. And this is a post that embodies all of those things. @RichieAHB on Twitter Just wanted to say that ever since I started using redux in a professional setting a little while back, I've become a huge proponent. And that's definitely thanks in part to your engagement with the community and all your writings. So, thanks for taking the time to answer so many people's questions. /u/Yodiddlyyo on Reddit I frequently find myself copying and pasting links to several specific posts over and over to answer people's questions. I also recently asked folks which of my posts they found to be the most useful, and other folks have said they wished there was a list of some of the best posts in one place so they don't have to browse through the whole blog. So, here you go: the posts that I'd classify as my "greatest hits", based on how often I link them and how much people seem to appreciate them. Greatest Hits, By Category Redux: Status, Use Cases, and Comparisons Redux - Not Dead Yet!: a clarification of Redux's status ("IT AIN'T DEAD!"), with comparisons of use cases and usefulness vs other tools like React Context Global React Meetup: The State of Redux 2020: a vidcast talk that covers Redux's usage and popularity, the evolution of React-Redux, and how Redux Toolkit simplifies Redux logic JS Party #146: When (and when not) to reach for Redux: a transcript excerpt from my JS Party podcast appearance, where I addressed the importance of choosing tools based on the problems you need to solve, and compare the use cases for Redux, Context, Apollo, and React-Query React, Redux, and Context Behavior: a short comparison of how Context and React-Redux pass down values and handle updates React and Redux Internals and Design A (Mostly) Complete Guide to React Rendering Behavior: my exhaustively detailed walkthrough of how React handles rendering, and how that behavior changes with regards to optimizations, Context, and React-Redux The Tao of Redux, Part 1: Implementation and Intent: looks at how the Redux core works internally, how different parts of the Redux ecosystem rely on aspects like immutability, and why Redux was designed to work in certain ways The History and Implementation of React-Redux: covers how Redux works with a UI layer, the history of how React-Redux has changed over time, and key implementation details of each version ReactNext 2019: A Deep Dive into React-Redux: a conference talk based on "The History and Implementation of React-Redux" Redux Toolkit 1.0: a recap of how why we created Redux Toolkit, the problems it's meant to solve, and the evolution of its API React Hooks and Architecture Thoughts on React Hooks, Redux, and Separation of Concerns: compares the tradeoffs of differing approaches to organizing React logic and components, particularly in relation to React-Redux ReactBoston 2019: Hooks, HOCs, and Tradeoffs: a follow-up talk that looks at how approaches for reusing logic in React have changed over time, with thoughts on the tradeoffs involved between hooks and HOCs Redux Architecture and Practices The Tao of Redux, Part 2: Practice and Philosophy: looks at many of the common usage patterns that exist around Redux, and explains why they even exist in the first place and why certain patterns are considered "idiomatic Redux" Redux Fundamentals Workshop Slides: the slides from my 2018 "Redux Fundamentals" workshop, a full 2-day set of materials that cover everything about how Redux works from the ground up and how to use it. Thoughts on Thunks, Sagas, Abstraction, and Reusability: responds to several concerns about using thunks and sagas with Redux, and discusses why those tools are useful Using Reselect Selectors for Encapsulation and Performance: explains the concept of "selector functions", when and how to use them, and how to use the Reselect library to create memoized selectors Why Redux Toolkit Uses Thunks for Async Logic: a quick look at reasons why I selected thunks instead of sagas as the default async middleware for Redux Toolkit Practical Redux, Part 6: Connected Lists and Performance: teaches why connecting multiple components can be useful and how it can improve performance in React-Redux apps Practical Redux, Part 7: Form Change Handling and Feature Reducers: shows alternate patterns for organizing "per-feature" logic besides the typical combineReducers, as well as techniques for debouncing form inputs when updating Redux data Practical Redux, Part 10: Managing Modals and Context Menus: demonstrates techniques for controlling pieces of UI like modals and context menus while staying within the restrictions of normal Redux usage How Web Apps Work I wrote a blog post series on "How Web Apps Work", intended to be an introduction to key concepts, terms, and technologies related to web development, aimed at new developers in general as well as devs with Java/C++ experience who are getting into web dev for the first time: How Web Apps Work: HTTP and Servers How Web Apps Work: Client Development and Deployment How Web Apps Work: Browsers, HTML, and CSS How Web Apps Work: JavaScript and the DOM How Web Apps Work: AJAX, APIs, and Data Transfer Other Technical Content Learning and Using TypeScript as an App Dev and a Library Maintainer: a recap of my own sequence of learning various static and dynamically-typed languages, how I eventually learned TypeScript, and my opinions on the pros and cons of using TS and how to use it effectively. Debugging Tips: My tips on learning how to effectively debug code, with some war stories of interesting problems I've debugged and fixed Git Under the Hood: Internals, Techniques, and Rewriting History: a slideset that covers how Git works internally and techniques you can use to modify your Git history Rewriting Your Git History and JS Source for Fun and Profit: a highly detailed look at how I rewrote an entire Git repository's history while rewriting the JS source to modernize it Building a MEAN.js AngularJS Project with Create-React-App: details on how I used CRA's build tooling to compile a legacy AngularJS project Learning Resources JavaScript for Java Developers: an extensive slideshow that covers everything you need to know about modern JS syntax, concepts, and ecosystem usage. Great for experienced devs coming to JS from another language. Resources for Learning React: my top recommended tools and tutorials for learning React Resources for Learning Redux: my top recommended tools and tutorials for learning Redux External Writing Besides this blog, I've written several major sections of the Redux docs: Redux core docs: Redux Essentials tutorial: a brand-new comprehensive tutorial sequence that covers everything you need to use Redux, "the right way", using our latest recommended tools and practices Redux Style Guide: a list of recommended patterns and practices for writing Redux apps Redux FAQ: my first contribution, a large FAQ section answering many common questions about Redux with links to additional resources Redux core docs: Structuring Reducers: a usage guide section covering techniques for organizing and writing reducer logic, including topics like normalizing data and immutable updates Redux Toolkit docs: RTK tutorials: the RTK "Basic/Intermediate/Advanced" tutorial sequence Usage Guide: instructions on how to use Redux Toolkit effectively and my links list repos: markerikson/react-redux-links: React/Redux Links: a curated list of React and Redux-related articles and resources markerikson/redux-ecosystem-links: Redux Addons Catalog: a curated list of Redux-related addons and tools

Blogged Answers: React Components, Reusability, and Abstraction

I just replied to a question by a developer who was concerned that they were writing components that aren't "reusable", and wanted to know how important that is, and figured it's worth pasting that here for reference. Reusability is great, but it's also not necessarily a primary goal. As I've grown as a software developer, I've learned that it's always easier to make something hardcoded and specific, and harder to make it generic / abstract / reusable. Genericizing code means that you have to think through all the possible ways that this might get used in the future, and handle all those possible use cases. As a specific example here, see the new createAsyncThunk API we recently added to Redux Toolkit. Lenz Weber, Matt Sutkowski, and I spent weeks thinking through ways to deal with error handling, API parameters, and return values. I think we came up with something that works pretty well, but it took a lot of effort. As programmers, we do get focused on "oh, these two pieces of code look almost the same, we can turn this into one piece of code and use it in both places!" I used to be really bad about this in code reviews - if I saw the same 3-line snippet in a couple places, I'd ask the PR author to extract it into a function or something. What finally snapped me out of that was a post by Sandi Metz entitled The Wrong Abstraction, where she talks about how trying to abstract or genericize code too early can often lead to making bad abstractions, and that it's often better to let code be duplicated for now rather than try to abstract it right away. This applies to any code in general, as well as React components. Specifically for React, I've found that my components generally fall into one of three categories: Truly generic components that could theoretically be used across all projects, like a <Button>. This is especially true if you're in a larger team that has some kind of a design system. Components that may be more specific to an app, but still have some potential for reuse App-specific components that are very tied to the structure and logic of this one app Overall, the smaller and more "primitive" a component is, the better a candidate it is for reuse. Don't get me wrong, reusability is a good thing. But, I do think that trying to focus on reusability as the primary goal is misleading. The primary goal is to have code that runs, accomplishes the desired functionality for the feature and the app, and does so without bugs. I've always been a huge fan of the mantra "make it work, make it right, make it fast". Focus on getting something working, even if the code is ugly and/or duplicated. Then you can improve it. I wouldn't worry about trying to make components "reusable outside this project" for now. Make something that works. If that means copy-pasting some code, do so. Once you've got this working, then take some time to evaluate what you've got, look for duplication and patterns, and extract something reusable, if it even makes sense to do so.