I can’t believe it’s already the end of June. ESR is only a few days away, and things are moving faster than ever.
Preparing For ESR
This is going to be a slightly shorter update since the majority of our effort revolved around testing and polishing 128 beta, which will turn into ESR on July 10th.
We fixed a total of 127 bugs and a few more things are getting tackled.
Account Colors In Compose
You can now see the custom colors you chose for your accounts in the compose windows. This was an 18 year-old request that we were finally able to fulfill thanks to the incredible work that many core developers put in place during the past 2 years.
By implementing a much more reliable and modular code base, with a clearer separation between data and UI, we’re finally able to ship these long standing requested features much faster. There’s still a lot to do, but working on our code base is getting better and better.
Mozilla Sync
The client code is finished, everything is in place and we’re testing syncing server data against a temporary staging server.
We’re still working on spawning our own production server, which turned out more challenging than expected. This means that potentially we won’t enable Sync by default for the first ESR release and instead keep it hidden temporarily, with the objective of enabling it in a future point release (maybe 128.1 or 128.2) depending on when the production server will be ready.
We will keep you posted every step of the way.
Thunderbird Beta 128
If you haven’t downloaded 128 beta, please do so and help us test and report bugs if you spot them. You can download Thunderbird 128 Beta here, and if you find any issue, please open a bug report against this Meta Bug we’re using to track any potential regression specific to 128. Thank you!
See ya next month.
Alessandro Castellani(he, him) Director, Desktop and Mobile Apps
If you’re interested in joining the technical discussion around Thunderbird development, consider joining one or several of our mailing list groups here.
Imagine for a moment if we treated our laundry the same way we treat our email. It might look something like this: At least ten times an hour, we’d look in the dryer, sigh at the mix of wet and dry clothes, wonder where the shirt we needed was, and then close the dryer door again without emptying a thing. Laura Mae Martin, author of Uptime: A Practical Guide to Personal Productivity and Wellbeing, has a better approach. Treat your email like you would ideally treat your laundry.
How do we put this metaphor to work in our inboxes? Martin has some steps for getting the most out of this analogy, and the first is to set aside a specific time in your day to tackle your inbox. This is the email equivalent of emptying your dryer, not just looking in it, and sorting the clothes into baskets. You’re already setting future you up for a better day with this first step!
The Process
At this set time, you’ll have a first pass at everything in your inbox, or as much as you can, sorting your messages into one of four ‘baskets’ – Respond, To Read, Revisit, and Relax (aka, the archive where the email lives once you’ve acted on it from a basket, and the trash for deleted emails). Acting on those messages comes after the sorting is done. So instead of ‘touching’ your email a dozen times with your attention, you only touch it twice: sorting it, and acting on it.
Let’s discuss those first three baskets in a little more detail.
First, the ‘Respond’ basket is for emails that require a response from you, which need you and your time to complete. Next, the ‘To Read’ basket is for emails that you’d like to read for informative purposes, but don’t require a response. Finally, the ‘Revisit’ basket is for emails where you need to respond but can’t right now because you’re waiting for the appropriate time, a response from someone, etc.
Here’s more info on how treating your email like laundry looks in your inbox. You don’t have separate dryers for work clothes and personal clothes, so ideally you want your multiple inboxes in one place, like Thunderbird’s Unified Folders view. The baskets (Respond, To Read, Revisit) are labels, tags, or folders. Unread messages should not be in the same place with sorted email; that’s like putting in wet clothes with your nice, dry laundry!
Baskets and Batch Tasking
You might be wondering “why not just use this time to sort AND respond to messages?” The answer is that this kind of multitasking saps your focus, thanks to something called attention residue. Hopping between sorting and replying – and increasing the chance of falling down attention rabbit holes doing the latter – makes attention residue thicker, stickier, and ultimately harder to shake. Batch tasking, or putting related tasks together for longer stretches of time, keeps potentially distracting tasks like email in check. So, sorting is one batch, responding is another, etc. No matter how much you’re tempted, don’t mix the tasks!
Putting It Into Practice
You know why you should treat your email like laundry, and you know the process. Here’s some steps for day one and beyond to make this efficient approach a habit.
One-time Setup:
Put active emails in your inbox in one of the first three baskets (Respond, To Read, Revisit)
If email doesn’t need one of these baskets, archive or delete them
Daily Tasks
Remember the 4 Baskets are tasks to be done separately
Pick a time to sort your email each day – at least once, and hopefully no more than two or three more times. Remember, this is time ONLY to sort emails into your baskets.
Give future you the gift of a sorted inbox
Find and schedule time during the day to deal with the baskets – but only one basket at a time! Have slots just for responding, reading, or checking on the progress of your Revisit emails. Think of your energy flow during the day, and assign your most mentally strenuous boxes for your peak energy times.
One Last Fold
Thanks for joining us in our continuing journey to turn our inboxes, calendars, and tasks lists into inspiring productivity tools instead of burdens. We know opening our inboxes can sometimes feel overwhelming, which makes it easier for them to steal our focus and our time. But if you treat your email like laundry, this chore can help make your inbox manageable and put you in control of it, instead of the other way around.
We’re excited to try this method, and we hope you are too. We’re also eager to try this advice with our actual laundry. Watch out, inboxes and floor wardrobes. We’re coming for you!
Our Community Office Hours session for May 2024 has concluded, and it was quite informative (especially for non-developers like me)! Wayne and Daniel shed light on Thunderbird’s build and release process, ran through a detailed presentation, answered questions, and treated us to live demos showing how a new Thunderbird build gets pushed and promoted to release.
Below you’ll find a lightly edited recording of the session, and the presentation slides in PDF format.
We’ll be announcing the topic of our June Office Hours session soon, so keep an eye on the Thunderbird blog.
Have you ever wondered what the release process of Thunderbird is like? Wanted to know if a particular bug would be fixed in the next release? Or how long release support lasts? Or just how many point releases are there?
In the May Office Hours, we’ll demystify the current Thunderbird release process as we get closer to the next Extended Security Release on July 10, 2024.
May Office Hours: The Thunderbird Release Process
One of our guests you may know already: Wayne Mery, our release and community manager. Daniel Darnell, a key release engineer, will also join us. They’ll answer questions about what roles they play, how we stage releases, and when they know if releases are ready. Additionally, they’ll tell us about the future of Thunderbird releases, including working with add-on developers and exploring a monthly release cadence.
Join us as our guests answer these questions and more in the next edition of our Community Office Hours! You can also submit your own questions about this topic beforehand and we’ll be sure to answer them: officehours@thunderbird.net
Catch Up On Last Month’s Thunderbird Community Office Hours
While you’re thinking of questions to ask, watch last month’s office hours where we chatted with three key developers bringing Rust and native Microsoft Exchange support into Thunderbird. You can find the video on our TILvids page.
Join The Video Chat
We’ll be back in our Big Blue Button room, provided by KDE and the Linux Application Summit. We’re grateful for their support and to have an open source web conferencing solution for our community office hours.
Thunderbird wouldn’t be here today without its incredible and dedicated contributors. The people developing Thunderbird and all of its add-ons, testing new releases, and supporting fellow users, for example, are the wind beneath our wings. It’s time to give them the spotlight in our new Contributor Highlight series.
We kick things off with Arthur, who contributes to Thunderbird by triaging and filing bug reports at Bugzilla, as well as assisting others.
Arthur, Chicago USA
Why do you like using Thunderbird?
Thunderbird helps me organize my life and I could not function in this world without its Calendar feature. It syncs well with things I do on my Android device and I can even run a portable version of it on my USB drive when I don’t have physical access to my home or office PC. Try doing that with that “other” email client.
What do you do in the Thunderbird community and why do you enjoy it? What motivates you to contribute?
Being a user myself, I can help other users because I know where they’re coming from. Also, having a forum like Bugzilla allows regular users to bring bugs to the attention of the Devs and for me to interface with those users to see if I can reproduce bugs or help them resolve issues. Having a direct line to Mozilla is an amazing resource. If you don’t have skin in the game, you can’t complain about the direction in which a product goes.
How do you relate your professional background and volunteerism to your involvement in Thunderbird?
As an IT veteran of 33+ years, I am very comfortable in user facing support and working with app vendors to resolve app problems but volunteering takes on many forms and is good for personal growth. Some choose to volunteer at their local Food Panty or Homeless shelter. I’ve found my comfort zone in leveraging my decades of IT experience to make something I know millions of users use and help make it better.
Share Your Contributor Highlight (or Get Involved!)
A big thanks to Arthur and all our Thunderbird contributors who have kept us alive and are helping us thrive! We’ll be back soon with more contributor highlights to spotlight more of our community.
If you’re a contributor who would like to share your story, get in touch with us at community@thunderbird.net. If you’re reading this and want to know more about getting involved with Thunderbird, check out our new and improved guide to learn about all the ways to contribute your skills to Thunderbird.
Welcome reader! This is the place where we, the Thunderbird for Android team, inform you about what we worked on in May 2024.
We’ve been publishing monthly progress reports for quite a while now. If you haven’t subscribed to the RSS feed yet, now would be a good time to start. You can even use your favorite desktop app to do so – see Thunderbird + RSS: How To Bring Your Favorite Content To The Inbox.
And if you need a reminder on where we left off last month, head over to April’s progress report.
Material 3
The most noticeable development effort going on right now is the conversion of the user interface to the design system Material 3. You can follow our progress by becoming a beta tester and installing the K-9 Mail 6.9xx beta versions.
The first step consisted of changing the theme to Material 3. That changes things like the style of buttons and dialogs.
Next, we replaced the many icons used throughout the app. But when using the beta version we — and some of you — noticed that not all of the new icons are a good fit. So we’ll update those icons in the next design iteration.
One of the main reasons for switching to Material 3 is the ability to support dynamic colors. It will allow the app to (optionally) use the system color scheme e.g. derived from the wallpaper. But in order for this to work properly, we need to update many places in the app that currently use fixed theme colors. This is an ongoing effort.
Targeting Android 14
As mentioned in April’s progress report, we’ve included the changes necessary to target Android 14 in the latest beta versions. So far we haven’t seen any crashes or bug reports related to these changes. So we plan to include them in the next maintenance release – K-9 Mail 6.804.
F-Droid metadata (part 3)
Unfortunately, this topic was part of the last two progress reports. So we’re very happy to report that the app description is now finally available again on our F-Droid app listing.
Other things we’ve worked on
Developer documentation
We’ve done some work on making our developer documentation more accessible. There’s now a table of contents and we have the capability to render it to HTML using mdbook. However, we haven’t set up automatic publishing yet. Until that happens, the documentation can be browsed on GitHub: K-9 Mail developer documentation.
Small IMAP improvements
We took some time to have a closer look at the communication between the app and the server when using the IMAP protocol and noticed a few places where the app could be more efficient. We’ve started addressing some of these inefficiencies. The result is that K-9 Mail can now perform some action with fewer network packets going back and forth between the app and the server.
Support for predictive back
Google is working on improving the user experience of the back gesture in Android. This effort is called predictive back. The idea is to reveal (part of) the screen to which a successful back gesture will navigate while the swipe gesture is still in progress.
In order for this to work properly, apps that currently intercept the back button/gesture will have to make some changes. We’ve started making the necessary modifications. But it’s still a work in progress.
Community Contributions
GitHub user Silas217209 added support for mailto: URIs on NFC tags (#7804). This was a feature a user requested in April.
Thank you for the contribution!
Releases
In May 2024 we published the following stable release:
We’re tossing May behind our shoulders, which means we’re in the final sprint before the next ESR (Extended Support Release). During the next couple of weeks you can expect some official communication on all the things that are going in the next major release of Thunderbird. Until then, here are some appetizers on our most recent efforts.
Rust-enabled builds
Our build and release team is working hard to ship Rust enabled builds by default. The first beta version of 128 will ship with Rust enabled by default, which will allow all of you to test experimental features without needing to compile the code locally.
Microsoft Exchange support
We’re very very very close!
So far we have the main flow completed, and we’re able to set up an account, fetch folders, fetch messages, and display messages. We’re finalizing the outgoing flow in order to send messages, and after that we will start an audit to ensure that all the usual features you expect from interacting with your email are working.
Expect some future call to actions to test things and invites to switch the experimental pref ON.
Native Linux system tray support
Enabling Rust builds in Thunderbird also gives us the ability to implement some long awaited features much faster. We’re still testing and cleaning things up, but if you’re adventurous you can check out our GitHub repositories for Linux System Tray and DBus hooks and run them locally.
Folder multi-selection
Folder pane multi-selection is almost completed and it should land soon. There are still some rough edges we need to tackle, mostly due to some C++ code not liking multiple folders copy/move and undo actions, but we’re confident that we will have this done before the end of June.
You can check the code and follow the progress here.
Account color customization
Another requested feature we’re aiming to ship in 128 is the customization of account colors. This is the first patch of an upcoming stack that will add some nice visual cues in the message list and the compose window for users with multiple accounts.
Folder compaction
We shared this in our Daily mailing list, but in case you missed it, we rebuilt the Folder Compaction code from scratch. This should potentially solve all the issues of profiles bubbling up in size, or compact operations silently failing and piling up on each other.
These changes should be uplifted to Beta soon. Please test it as much as possible and report any bugs as soon as you encounter them.
Native Windows notifications
Another important achievement was the ability to completely support native Windows 10/11 notifications and make them fully functional.
You can already consume this feature on Daily, and moving forward Thunderbird will be using native OS notifications by default.
We plan to add some nice quick actions and improve the usefulness of native notifications in the future, so stay tuned!
As usual, if you want to see things as they land you can always check the pushlog and try running daily, which would be immensely helpful for catching bugs early.
See ya next month.
Alessandro Castellani(he, him) Director, Desktop and Mobile Apps
If you’re interested in joining the technical discussion around Thunderbird development, consider joining one or several of our mailing list groups here.
This might be unexpected coming from an email app developer, but hear us out: we want you to spend the least amount of time possible in your inbox.
The Thunderbird Team wants to help you manage your most precious, nonrenewable resource: your time. This post kicks off a series of Thunderbird tips and tricks focusing on our favorite time management and productivity advice.
When we asked Director of Product, Ryan Sipes, what time management strategy he wanted to share first, he said time blocking. Time blocking is a favorite of author and productivity guru Cal Newport. This technique schedules your entire day to minimize focus-stealing activities and maximize deep work that requires your full attention.
This sounds daunting, but we’re on this productivity journey with you! All you need to start is your calendar or planner, whether it’s in Thunderbird, another app, or using pen and paper. Personally, we’re fans of having our notebook and laptop on hand when we schedule our day ahead.
Don’t worry that an all-day schedule won’t leave time for fun or impromptu plans. You’ll be more present when you’re off work, whether it’s cherished time with loved ones or working on that novel you always wanted to write. And since you’re adjusting your schedule as you go, you’ll be able to add plans without overwhelming yourself.
With that, let’s get started time blocking with Thunderbird!
Get Your Calendars in One Place
First, have all your calendars (work, personal, school, etc.) in one place. Thunderbird can combine online calendars from different accounts for you (and let you customize their colors)! This SUMO article explains — with screenshots — how to add your calendars and create new ones.
Suggestions for Getting Started
<figcaption class="wp-element-caption">Example of a time-blocked Thunderbird Calendar. For other examples, see Todoist.</figcaption>
It’s hard to change some fixed blocks of time, like team meetings or scheduled personal obligations. But the spaces in between are a blank canvas waiting to be filled with everything you need and want to do. How you fill them is up to you, but here are some suggestions for time blocking with Thunderbird:
Know when you do your best work. If you can focus more in the morning, or after a 30-minute walk, schedule blocks of deep work around that. If you don’t know when you do your best work, observe how you work for a week or two! Take notes on your energy and focus levels during the day.
Use your professional and personal priorities to fill out your time blocks. Whether your planning exists in project management software or handwritten notes, identify your urgent and important tasks that need your focus and time.
Use breaks between longer blocks for less urgent and potentially distracting tasks like checking your email or catching up on chats. If you limit the amount of time you spend on these tasks (a technique known as time boxing), and minimize or turn off their notifications, your day becomes a lot more productive.
Adjust your schedule whenever you need it, not just at the end of the day or week. Move blocks, shorten or lengthen them, etc. As you learn to be more aware of how you use your time, you’ll become better at estimating how long you need for tasks, and the best time of day to do them.
Time Blocking and Beyond
Thanks for joining us for this first productivity newsletter. We hope this post and the ones to come help you reclaim the time for things you need and want to do. Next month we’ll share more advice, and techniques you can use in Thunderbird to maximize your valuable time.
We’re on this journey with you, learning new skills and working them into our lives until they become habits. Making changes, even for the better, is hard. If a day or two or seven go by and you’re losing track of your time again, it’s okay. Make this the cue to start your productivity training montage, and let us be the awesome 80s rock soundtrack to support you.
Thunderbird.net has a new look, but the improvements go beyond that. We wanted a website where you could quickly find the information you need, from support to contribution, in clear and easy to understand text. While staying grateful to the many amazing contributors who have helped build and maintain our website over the past 20 years, we wanted to refresh our information along with our look. Finally, we wanted to partner with Freehive’s Ryan Gorley for their sleek, cohesive design vision and commitment to open source.
We wanted a website that’s ready for the next 20 years of Thunderbird, including the upcoming arrival of Thunderbird on mobile devices. But you don’t have to wait for that future to experience the new website now.
The New Thunderbird.net
The new, more organized framework starts with the refreshed Home page. All the great content you’ve relied on is still here, just easier to find! The expanded navigation menu makes it almost effortless to find the information and resources you need.
Resources provide a quick link to all the news and updates in the Thunderbird Blog and the unmatched community assistance in Mozilla Support, aka SUMO. Release notes are linked from the download and other options page. That page has also been simplified while still maintaining all the usual options. It’s now the main way to get links to download Beta and Daily, and in the future any other apps or versions we produce.
The About section introduces the values and the people behind the Thunderbird project, which includes our growing MZLA team. Our contact page connects you with the right community resources or team member, no matter your question or concern. And if you’d like to join us, or just see what positions are open, you’ll find a link to our career page here.
Whether it’s giving your time and skill or making a financial donation, it’s easy to discover all the ways to contribute to the project. Our new and improved Participate page shows how to get involved, from coding and testing to everyday advocacy. No matter your talents and experience, everyone can contribute!
If you want to download the latest stable release, or to donate and help bring Thunderbird everywhere, those options are still an easy click from the navigation menu.
Your Feedback
We’d love to have your thoughts and feedback on the new website. Is there a new and improved section you love? Is there something we missed? Let us know in the comments below. Want to see all the changes we made? Check the repository for the detailed commit log.
Welcome to our monthly report on turning K-9 Mail into Thunderbird for Android! Last month you could read about how we found and fixed bugs after publishing a new stable release. This month we start with… telling you that we fixed even more bugs.
Fixing bugs
After the release of K-9 Mail 6.800 we dedicated some time to fixing bugs. We published the first bugfix release in March and continued that work in April.
K-9 Mail 6.802
The second bugfix release contained these changes:
Push: Notify user if permission to schedule exact alarms is missing
Renamed “Send client ID” setting to “Send client information”
IMAP: Added support for the \NonExistent LIST response attribute
IMAP: Issue EXPUNGE command after moving without MOVE extension
Updated translations; added Hebrew translation
I’m especially happy that we were able to add back the Hebrew translation. We removed it prior to the K-9 Mail 6.800 release due to the translation being less than 70% complete (it was at 49%). Since then volunteers translated the missing bits of the app and in April the translation was almost complete.
Unfortunately, the same isn’t true for the Korean translation that was also removed. It was 69% complete, right below the threshold. Since then there has been no significant change. If you are a K-9 Mail user and a native Korean speaker, please consider helping out.
F-Droid metadata (again?)
In the previous progress report we described what change had led to the app description disappearing on F-Droid and how we intended to fix it. Unfortunately we found out that our approach to fixing the issue didn’t work due to the way F-Droid builds their app index. So we changed our approach once again and hope that the app description will be restored with the next app release.
Push & the permission to schedule alarms
K-9 Mail 6.802 notifies the user when Push is enabled in settings, but the permission to schedule exact alarms is missing. However, what we really want to do is ask the user for this permission before we allow them to enable Push.
This change was completed in April and will be included in the next bugfix release, K-9 Mail 6.803.
Material 3
As briefly mentioned in March’s progress report, we’ve started work on switching the app to Google’s latest version of Material Design – Material 3. In April we completed the technical conversion. The app is now using Material 3 components instead of the Material Design 2 ones.
The next step is to clean up the different screens in the app. This means adjusting spacings, text sizes, colors, and sometimes more extensive changes.
We didn’t release any beta versions while the development version was still a mix of Material Design 2 and Material 3. Now that the first step is complete, we’ll resume publishing beta versions.
If you are a beta tester, please be aware that the app still looks quite rough in a couple of places. While the app should be fully functional, you might want to leave the beta program for a while if the look of the app is important to you.
Targeting Android 14
Part of the necessary app maintenance is to update the app to target the latest Android version. This is required for the app to use the latest security features and to cope with added restrictions the system puts in place. It’s also required by Google in order to be able to publish updates on Google Play.
The work to target Android 14 is now mostly complete. This involved some behind the scenes changes that users hopefully won’t notice at all. We’ll be testing these changes in a future beta version before including them in a K-9 Mail 6.8xx release.
Building two apps
If you’re reading this, it’s probably because you’re excited for Thunderbird for Android to be finally released. However, we’ve also heard numerous times that people love K-9 Mail and wished the app would stay around. That’s why we’ve announced in December to do just that.
We’ve started work on this and are now able to build two apps from the same source code. Thunderbird for Android already includes the fancy new Thunderbird logo and a first version of a blue theme.
But as you can see in the screenshots above, we’re not quite done yet. We still have to change parts of the app where the app name is displayed to use a placeholder instead of a hard-coded string. Then there’s the About screen and a couple of other places that require app-specific behavior.
We’ll keep you posted.
Releases
In April 2024 we published the following stable release:
Hello Thunderbird Community, and welcome back to the monthly Thunderbird development digest. April just ended and we’re running at full speed into May. We’re only a couple of months away from the next ESR, so things are landing faster and we’re seeing the finalization of a lot of parallel efforts.
20-Year-Old bugs
Something that has been requested for almost 20 years finally landed on Daily. The ability to control the display of recipients in the message list and better distinguish unknown addresses from those saved in the Address Book was finally implemented in Bug 243258 – Show email address in message list.
This is one of the many examples of features that in the past were very complicated and tricky to implement, but that we were finally able to address thanks to the improvements of our architecture and being able to work with a more flexible and modular code.
We’re aiming at going through those very very old requests and slowly addressing them when possible.
Exchange alpha
More Exchange support improvements and features are landing on Daily almost…daily (pun intended). If you want to test things with a local build, you can follow this overview from Ikey.
We will soon look at the possibility of enabling Rust builds by default, making sure that all users will be able to consume our Rust code from next beta, and only needing to switch a pref in order to test Exchange.
Folder compaction
If you’ve been tracking our most recent struggles, you’re probably aware of one of the lingering annoying issues which sees the bubbling up of the size of the user profile caused by local folder corruption.
Ben dive bombed into the code and found a spaghetti mess that was hard to untangle. You can read more about his exploration and discoveries in his recent post on TB-Planning.
We’re aiming to land this code hopefully before the end of the week and start calling for some testing and feedback from the community to ensure that all the various issues have been addressed correctly.
If you’re running Beta or Daily, you might have noticed some very fancy new UI for the Cards View. This has been a culmination of many weeks of UX analysis to ensure a flexible and consistent hover, selection, and focus state.
Micah and Sol identified a total of 27 different interaction states on that list, and implementing visual consistency while guaranteeing optimal accessibility levels for all operating systems and potential custom themes was not easy.
We’re very curious to hear your feedback.
Context menu
A more refined and updated context menu for the message list also landed on Daily.
This update is only the first step of many more to come, so we apologize in advance if some things are not super polished or things seem temporarily off.
ESR Preview
If you’re curious about what the next ESR will look like or checking new features, please consider downloading and installing Beta (preferably in another directory to not override your current profile.) Help us test this new upcoming release and find bugs early.
As usual, if you want to see things as they land you can always check the pushlog and try running daily, which would be immensely helpful for catching bugs early.
See ya next month.
Alessandro Castellani(he, him) Director, Desktop and Mobile Apps
If you’re interested in joining the technical discussion around Thunderbird development, consider joining one or several of our mailing list groups here.
Microsoft Exchange is a popular choice of email service for corporations and educational institutions, and so it’s no surprise that there’s demand among Thunderbird users to support Exchange. Until recently, this functionality was only available through an add-on. But, in the next ESR (Extended Support) release of Thunderbird in July 2024, we expect to provide this support natively within Thunderbird. Because of the size of this undertaking, the first roll-out of the Exchange support will initially cover only email, with calendar and address book support coming at a later date.
This article will go into technical detail on how we are implementing support for the Microsoft Exchange Web Services mail protocol, and some idea of where we’re going next with the knowledge gained from this adventure.
Historical context
Thunderbird is a long-lived project, which means there’s lots of old code. The current architecture for supporting mail protocols predates Thunderbird itself, having been developed more than 20 years ago as part of Netscape Communicator. There was also no paid maintainership from about 2012 — when Mozilla divested and transferred ownership of Thunderbird to its community — until 2017, when Thunderbird rejoined the Mozilla Foundation. That means years of ad hoc changes without a larger architectural vision and a lot of decaying C++ code that was not using modern standards.
Furthermore, in the entire 20 year lifetime of the Thunderbird project, no one has added support for a new mail protocol before. As such, no one has updated the architecture as mail protocols change and adapt to modern usage patterns, and a great deal of institutional knowledge has been lost. Implementing this much-needed feature is the first organization-led effort to actually understand and address limitations of Thunderbird’s architecture in an incremental fashion.
Why we chose Rust
Thunderbird is a large project maintained by a small team, so choosing a language for new work cannot be taken lightly. We need powerful tools to develop complex features relatively quickly, but we absolutely must balance this with long-term maintainability. Selecting Rust as the language for our new protocol support brings some important benefits:
Memory safety. Thunderbird takes input from anyone who sends an email, so we need to be diligent about keeping security bugs out.
Performance. Rust runs as native code with all of the associated performance benefits.
Modularity and Ecosystem. The built-in modularity of Rust gives us access to a large ecosystem where there are already a lot of people doing things related to email which we can benefit from.
The above are all on the standard list of benefits when discussing Rust. However, there are some additional considerations for Thunderbird:
Firefox. Thunderbird is built on top of Firefox code and we use a shared CI infrastructure with Firefox which already enables Rust. Additionally, Firefox provides a language interop layer called XPCOM (Cross-Platform Component Object Model), which has Rust support and allows us to call between Rust, C++, and JavaScript.
Powerful tools. Rust gives us a large toolbox for building APIs which are difficult to misuse by pushing logical errors into the domain of the compiler. We can easily avoid circular references or provide functions which simply cannot be called with values which don’t make sense, letting us have a high degree of confidence in features with a large scope. Rust also provides first-class tooling for documentation, which is critically important on a small team.
Addressing architectural technical debt. Introducing a new language gives us a chance to reconsider some aging architectures while benefiting from a growing language community.
Platform support and portability. Rust supports a broad set of host platforms. By building modular crates, we can reuse our work in other projects, such as Thunderbird for Android/K-9 Mail.
Some mishaps along the way
Of course, the endeavor to introduce our first Rust component in Thunderbird is not without its challenges, mostly related to the size of the Thunderbird codebase. For example, there is a lot of existing code with idiosyncratic asynchronous patterns that don’t integrate nicely with idiomatic Rust. There are also lots of features and capabilities in the Firefox and Thunderbird codebase that don’t have any existing Rust bindings.
The first roadblock: the build system
Our first hurdle came with getting any Rust code to run in Thunderbird at all. There are two things you need to know to understand why:
First, since the Firefox code is a dependency of Thunderbird, you might expect that we pull in their code as a subtree of our own, or some similar mechanism. However, for historical reasons, it’s the other way around: building Thunderbird requires fetching Firefox’s code, fetching Thunderbird’s code as a subtree of Firefox’s, and using a build configuration file to point into that subtree.
Second, because Firefox’s entrypoint is written in C++ and Rust calls happen via an interoperability layer, there is no single point of entry for Rust. In order to create a tree-wide dependency graph for Cargo and avoid duplicate builds or version/feature conflicts, Firefox introduced a hack to generate a single Cargo workspace which aggregates all the individual crates in the tree.
In isolation, neither of these is a problem in itself. However, in order to build Rust into Thunderbird, we needed to define our own Cargo workspace which lives in our tree, and Cargo does not allow nesting workspaces. To solve this issue, we had to define our own workspace and add configuration to the upstream build tool, mach, to build from this workspace instead of Firefox’s. We then use a newly-added mach subcommand to sync our dependencies and lockfile with upstream and to vendor the resulting superset.
XPCOM
While the availability of language interop through XPCOM is important for integrating our frontend and backend, the developer experience has presented some challenges. Because XPCOM was originally designed with C++ in mind, implementing or consuming an XPCOM interface requires a lot of boilerplate and prevents us from taking full advantage of tools like rust-analyzer. Over time, Firefox has significantly reduced its reliance on XPCOM, making a clunky Rust+XPCOM experience a relatively minor consideration. However, as part of the previously-discussed maintenance gap, Thunderbird never undertook a similar project, and supporting a new mail protocol requires implementing hundreds of functions defined in XPCOM.
Existing protocol implementations ease this burden by inheriting C++ classes which provide the basis for most of the shared behavior. Since we can’t do this directly, we are instead implementing our protocol-specific logic in Rust and communicating with a bridge class in C++ which combines our Rust implementations (an internal crate called ews_xpcom) with the existing code for shared behavior, with as small an interface between the two as we can manage.
Please visit our documentation to learn more about how to create Rust components in Thunderbird.
Implementing Exchange support with Rust
Despite the technical hiccups experienced along the way, we were able to clear the hurdles, use, and build Rust within Thunderbird. Now we can talk about how we’re using it and the tools we’re building. Remember all the way back to the beginning of this blog post, where we stated that our goal is to support Microsoft’s Exchange Web Services (EWS) API. EWS communicates over HTTP with request and response bodies in XML.
Sending HTTP requests
Firefox already includes a full-featured HTTP stack via its necko networking component. However, necko is written in C++ and exposed over XPCOM, which as previously stated does not make for nice, idiomatic Rust. Simply sending a GET request requires a great deal of boilerplate, including nasty-looking unsafe blocks where we call into XPCOM. (XPCOM manages the lifetime of pointers and their referents, ensuring memory safety, but the Rust compiler doesn’t know this.) Additionally, the interfaces we need are callback-based. For making HTTP requests to be simple for developers, we need to do two things:
Support native Rust async/await syntax. For this, we added a new Thunderbird-internal crate, xpcom_async. This is a low-level crate which translates asynchronous operations in XPCOM into Rust’s native async syntax by defining callbacks to buffer incoming data and expose it by implementing Rust’s Future trait so that it can be awaited by consumers. (If you’re not familiar with the Future concept in Rust, it is similar to a JS Promise or a Python coroutine.)
Provide an idiomatic HTTP API. Now that we had native async/await support, we created another internal crate (moz_http) which provides an HTTP client inspired by reqwest. This crate handles creating all of the necessary XPCOM objects and providing Rustic error handling (much nicer than the standard XPCOM error handling).
Handling XML requests and responses
The hardest task in working with EWS is translating between our code’s own data structures and the XML expected/provided by EWS. Existing crates for serializing/deserializing XML didn’t meet our needs. serde’s data model doesn’t align well with XML, making distinguishing XML attributes and elements difficult. EWS is also sensitive to XML namespaces, which are completely foreign to serde. Various serde-inspired crates designed for XML exist, but these require explicit annotation of how to serialize every field. EWS defines hundreds of types which can have dozens of fields, making that amount of boilerplate untenable.
Ultimately, we found that existing serde-based implementations worked fine for deserializing XML into Rust, but we were unable to find a satisfactory tool for serialization. To that end, we introduced another new crate, xml_struct. This crate defines traits governing serialization behavior and uses Rust’s procedural derive macros to automatically generate implementations of these traits for Rust data structures. It is built on top of the existing quick_xml crate and designed to create a low-boilerplate, intuitive mapping between XML and Rust. While it is in the early stages of development, it does not make use of any Thunderbird/Firefox internals and is available on GitHub.
We have also introduced one more new crate, ews, which defines types for working with EWS and an API for XML serialization/deserialization, based on xml_struct and serde. Like xml_struct, it is in the early stages of development, but is available on GitHub.
Overall flow chart
Below, you can find a handy flow chart to help understand the logical flow for making an Exchange request and handling the response.
Fig 1. A bird’s eye view of the flow
What’s next?
Testing all the things
Before landing our next major features, we are taking some time to build out our automated tests. In addition to unit tests, we just landed a mock EWS server for integration testing. The current focus on testing is already paying dividends, having exposed a couple of crashes and some double-sync issues which have since been rectified. Going forward, new features can now be easily tested and verified.
Improving error handling
While we are working on testing, we are also busy improving the story around error handling. EWS’s error behavior is often poorly documented, and errors can occur at multiple levels (e.g., a request may fail as a whole due to throttling or incorrect structure, or parts of a request may succeed while other parts fail due to incorrect IDs). Some errors we can handle at the protocol level, while others may require user intervention or may be intractable. In taking the time now to improve error handling, we can provide a more polished implementation and set ourselves up for easier long-term maintenance.
Expanding support
We are working on expanding protocol support for EWS (via ews and the internal ews_xpcom crate) and hooking it into the Thunderbird UI. Earlier this month, we landed a series of patches which allow adding an EWS account to Thunderbird, syncing the account’s folder hierarchy from the remote server, and displaying those folders in the UI. (At present, this alpha-state functionality is gated behind a build flag and a preference.) Next up, we’ll work on fetching message lists from the remote server as well as generalizing outgoing mail support in Thunderbird.
Documentation
Of course, all of our work on maintainability is for naught if no one understands what the code does. To that end, we’re producing documentation on how all of the bits we have talked about here come together, as well as describing the existing architecture of mail protocols in Thunderbird and thoughts on future improvements, so that once the work of supporting EWS is done, we can continue building and improving on the Thunderbird you know and love.
QUESTIONS FROM YOU
EWS is deprecated for removal in 2026. Are there plans to add support for Microsoft Graph into Thunderbird?
This is a common enough question that we probably should have addressed it in the post! EWS will no longer be available for Exchange Online in October 2026, but our research in the lead-up to this project showed that there’s a significant number of users who are still using on-premise installs of Exchange Server. That is, many companies and educational institutions are running Exchange Server on their own hardware.
These on-premise installs largely support EWS, but they cannot support the Azure-based Graph API. We expect that this will continue to be the case for some time to come, and EWS provides a means of supporting those users for the foreseeable future. Additionally, we found a few outstanding issues with the Graph API (which is built with web-based services in mind, not desktop applications), and adding EWS support allows us to take some extra time to find solutions to those problems before building Graph API support.
Diving into the past has enabled a sound engineering-led strategy for dealing with the future: Thanks to the deep dive into the existing Thunderbird architecture we can begin to leverage more efficient and productive patterns and technologies when implementing protocols.
In time this will have far reaching consequences for the Thunderbird code base which will not only run faster and more reliably, but significantly reduce maintenance burden when landing bug fixes and new features.
Rust and EWS are elements of a larger effort in Thunderbird to reduce turnarounds and build resilience into the very core of the software.
We admit it. Thunderbird is getting a bit Rusty, but in a good way! In our monthly Development Digests, we’ve been updating the community about enabling Rust in Thunderbirdto implementnative support for Exchange. Now, we’d like to invite you for a chat with Team Thunderbird and the developers making this change possible. As always, send your questions in advance to officehours@thunderbird.net! This is a great way to get answers even if you can’t join live.
Be sure to note the change in day of the week and the UTC time. (At least the time changes are done for now!) We had to shift our calendar a bit to fit everyone’s schedules and time zones!
<figcaption class="wp-element-caption">UPDATE: Watch the entire conversation here. </figcaption>
April Office Hours: Rust and Exchange
This month’s topic is a new and exciting change to the core functionality: using Rust to natively support Microsoft Exchange. Join us and talk with the three key Thunderbird developers responsible for this shiny (rusty) new addition: Sean Burke, Ikey Doherty, and Brendan Abolivier! You’ll find out why we chose Rust, challenges we encountered, how we used Rust to interface with XPCOM and Necko to provide Exchange support. We’ll also give you a peek into some future plans around Rust.
Catch Up On Last Month’s Thunderbird Community Office Hours
While you’re thinking of questions to ask, watch last month’s office hours where we answered some of your frequently asked recent questions. You can watch clips of specific questions and answers on our TILvids channel. If you’d prefer a written summary, this blog post has you covered.
Join The Video Chat
We’ve also got a shiny new Big Blue Button room, thanks to KDE! We encourage everyone to check out their Get Involved page. We’re grateful for their support and to have an open source web conferencing solution for our community office hours.
We know the Thunderbird community has LOTS of questions! We get them on Mozilla Support, Mastodon, and X.com (formerly Twitter). They pop up everywhere, from the Thunderbird subreddit to the teeming halls of conferences like FOSDEM and SCaLE. During our March Community Office Hours, we took your most frequently asked questions to Team Thunderbird and got some answers. If you couldn’t watch the full session, or would rather have the answers in abbreviated text clips, this post is for you!
Thunderbird for Android / K-9 Mail
The upcoming release on Android is definitely on everyone’s mind! We received lots of questions about this at our conference booths, so let’s answer them!
Will there be Exchange support for Thunderbird for Android?
Yes! Implementing Exchange in Rust in the Thunderbird Desktop client will enable us to reuse those Rust crates as shared libraries with the Mobile client. Stay up to date on Exchange support progress via our monthly Developer Digests.
Will Thunderbird Add-ons be available on Android?
Right now, no, they will not be available. K-9 Mail uses a different code base than Thunderbird Desktop. Thunderbird add-ons are designed for a desktop experience, not a mobile one. We want to have add-ons in the future, but this will likely not happen within the next two years.
When Thunderbird for Android launches, will it be available on F-Droid?
It absolutely will.
When Thunderbird for Android is ready to be released, what will the upgrade path look like?
We know some in the K-9 Mail community love their adorable robot dog and don’t want to give him up yet. So we will support K-9 Mail (same code, different brand) in parallel for a year or two, until the product is more mature, and we see that more K-9 Mail users are organically switching.
Because of Android security, users will need to manually migrate from K-9 Mail to Thunderbird for Android, versus an automatic migration. We want to make that effortless and unobtrusive, and the Sync feature using Mozilla accounts will be a large part of that. We are exploring one-tap migration tools that will prompt you to switch easily and keep all your data and settings – and your peace of mind.
Will CalDAV and CardDAV be available on Thunderbird for Android?
Probably! We’re still determining this, but we know our users like having their contacts and calendars inside one app for convenience, as well as out of privacy concerns. While it would be a lot of engineering effort, we understand the reasoning behind these requests. As we consider how to go forward, we’ll release all these explorations and ideas in our monthly updates, where people can give us feedback.
Will the K-9 Mail API provide the ability to download the save preferences that Sync stores locally to plug into automation like Ansible?
Yes! Sync is open source, so users can self-host their own instead of using Mozilla services. This question touches on the differences between data structure for desktop and mobile, and how they handle settings. So this will take a while, but once we have something stable in a beta release, we’ll have articles on how to hook up your own sync server and do your own automation.
Thunderbird for Desktop
When will we have native Exchange support for desktop Thunderbird?
We hope to land this in the next ESR (Extended Support Release), version 128, in limited capacity. Users will still need to use the OWL Add-on for all situations where the standard exchange web service is not available. We don’t yet know if native calendar and address book support will be included in the ESR. We want to support every aspect of Exchange, but there is a lot of code complexity and a history of changes from Microsoft. So our primary goal is good, stable support for email by default, and calendar and address book if possible, for the next ESR.
When will conversations and a true threaded view be added to Thunderbird?
Viewing your own sent emails is an important component of a true conversation view. This is a top priority and we’re actively working towards it. Unfortunately, this requires overhauling the backend database that underlies Thunderbird, which is 20 years old. Our legacy database is not built to handle conversation views with received and sent messages listed in the same thread. Restructuring a two decades old database is not easy. Our goal is to have a new global message database in place by May 31. If nothing has exploded, it should be much easier to enable conversation view in the front end.
When will we get a full sender name columnwith the raw email address of the sender?This will help further avoid phishing and spam.
We plan to make this available in the next ESR — Thunderbird 128 — which is due July 2024.
Will there ever be a browser-based view of Thunderbird?
Despite our foundations in Firefox, this is a huge effort that would have to be built from scratch. This isn’t on our roadmap and not in our plans for now. If there was a high demand, we might examine how feasible this could be. Alex explains this in more detail during the short video below:
If you’ve been wondering how the work to turn K-9 Mail into Thunderbird for Android is coming along, you’ve found the right place. This blog post contains a report of our development activities in March 2024.
We’ve published monthly progress reports for a while now. If you’re interested in what happened previously, check out February’s progress report. The report for the preceding month is usually linked in the first section of a post. But you can also browse the Android section of our blog to find progress reports and release announcements.
Fixing bugs
For K-9 Mail, new stable releases typically include a lot of changes. K-9 Mail 6.800 was no exception. That means a lot of opportunities to accidentally introduce new bugs. And while we test the app in several ways – manual tests, automated tests, and via beta releases – there’s always some bugs that aren’t caught and make it into a stable version. So we typically spend a couple of weeks after a new major release fixing the bugs reported by our users.
K-9 Mail 6.801
Stop capitalizing email addresses
One of the known bugs was that some software keyboards automatically capitalized words when entering the email address in the first account setup screen. A user opened a bug and provided enough information () for us to reproduce the issue and come up with a fix.
Line breaks in single line text inputs
At the end of the beta phase a user noticed that K-9 Mail wasn’t able to connect to their email account even though they copy-pasted the correct password to the app. It turned out that the text in the clipboard ended with a line break. The single line text input we use for the password field didn’t automatically strip that line break and didn’t give any visual indication that there was one.
While we knew about this issue, we decided it wasn’t important enough to delay the release of K-9 Mail 6.800. After the release we took some time to fix the problem.
DNSSEC? Is anyone using that?
When setting up an account, the app attempts to automatically find the server settings for the given email address. One part of this mechanism is looking up the email domain’s MX record. We intended for this lookup to support DNSSEC and specifically looked for a library supporting this.
A user in our support forum reported a strange error message (“Cannot serialize abstract class com.fsck.k9.mail.oauth.XOAuth2Response”) when using OAuth 2.0 while adding their email account. Our intention was to display the error message returned by the OAuth server. Instead an internal error occurred.
We tracked this down to the tool optimizing the app by stripping unused code and resources when building the final APK. The optimizer was removing a bit too much. But once the issue was identified, the fix was simple enough.
Crash when downloading an attachment
Shortly after K-9 Mail 6.800 was made available on Google Play, I checked the list of reported app crashes in the developer console. Not a lot of users had gotten the update yet. So there were only very few reports. One was about a crash that occurred when the progress dialog was displayed while downloading an attachment.
The crash had been reported before. But the number of crashes never crossed the threshold where we consider a crash important enough to actually look at.
It turned out that the code contained the bug since it was first added in 2017. It was a race condition that was very timing sensitive. And so it worked fine much more often than it did not.
The fix was simple enough. So now this bug is history.
Don’t write novels in the subject line
The app was crashing when trying to send a message with a very long subject line (around 1000 characters). This, too, wasn’t a new bug. But the crash occurred rarely enough that we didn’t notice it before.
The bug is fixed now. But it’s still best practice to keep the subject short!
Work on K-9 Mail 6.802
Even though we fixed quite a few bugs in K-9 Mail 6.801, there’s still more work to do. Besides fixing a couple of minor issues, K-9 Mail 6.802 will include the following changes.
F-Droid metadata
In preparation of building two apps (Thunderbird for Android and K-9 Mail), we moved the app description and screenshots that are used for F-Droid’s app listing to a new location inside our source code repository. We later found out that this new location is not supported by F-Droid, leading to an empty app description on the F-Droid website and inside their app.
We switched to a different approach and hope this will fix the app description once K-9 Mail 6.802 is released.
Push not working due to missing permission
Fresh installs of the app on Android 14 no longer automatically get the permission to schedule exact alarms. But this permission is necessary for Push to work. This was a known issue. But since it only affects new installs and users can manually grant this permission via Android settings, we decided not to delay the stable release until we added a user interface to guide the user through the permission flow.
K-9 Mail 6.802 will include a first step to improve the user experience. If Push is enabled but the permission to schedule exact alarms hasn’t been granted, the app will change the ongoing Push notification to ask the user to grant this permission.
In a future update we’ll expand on that and ask the user to grant the permission before allowing them to enable Push.
What about new features?
Of course we haven’t forgotten about our roadmap. As mentioned in February’s progress report we’ve started work on switching the user interface to use Material 3 and adding/improving Android 14 compatibility.
There’s not much to show yet. Some Material 3 changes have been merged already. But the user interface in our development version is currently very much in a transitional phase.
The Android 14 compatibility changes will be tested in beta versions first, and then back-ported to K-9 Mail 6.8xx.
Releases
In March 2024 we published the following stable release:
Since the release of Thunderbird 115, a big focus has been on improving the state of our automated testing. Automated testing increases the software quality by minimizing the number of bugs accidentally introduced by changes to the code. For each change made to Thunderbird, our testing machines run a set of tests across Windows, macOS, and Linux to detect mistakes and unintended consequences. For a single change (or a group of changes that land at the same time), 60 to 80 hours of machine time is used running tests.
Our code is going to be under more pressure than ever before – with a bigger team making more changes, and monthly releases reducing the time code spends on testing channels before being released.
We want to find the bugs before our users do.
Why We’re Testing
We’re not writing tests merely to make ourselves feel better. Tests improve Thunderbird by:
Preventing mistakes If we test that some code behaves in an expected way, we’ll find out immediately if it no longer behaves that way. This means a shorter feedback loop, and we can fix the problem before it annoys the users.
Finding out when somebody upstream breaks us Thunderbird is built from the Firefox code. The Firefox code, which we are not responsible for, is 30 to 40 times the size of the code we are responsible for. When something inevitably changes in Firefox that affects us, we want to know about it immediately so that we can respond.
Freeing up human testers If we use computers to prove that the program does what it’s supposed to do, particularly if we avoid tedious repetition and difficult-to-set-up tasks, then the limited human resources we have can do more things that humans are better at. For example, I’ve recently added tests that check 22 ways to trigger fetching mail, and 10 circumstances fetching mail might not work. There’s no way our human testers (great though they are) are testing all of them, but our automated tests can and do, several times a day.
Thinking through what the code should be doing Testing forces an engineer to look at the code from a different point-of-view, and this is helpful to think about what the code is supposed to do in more circumstances. It also makes it easier to prove that the code does work in obscure circumstances.
Finding existing bugs In software terms we’re working with some very old code, and much of it is untested. Testing it puts a fresh set of eyes on the code and reveals some of the mistakes of the past, and where the ravages of time have broken things. It also helps the person writing the tests to understand what the code does, a lot better than just reading the code does.
We’re not trying to completely cover a feature or every edge case in tests. We are trying to create a testing framework around the feature so that when we find a bug, as well as fixing it, we can easily write a test preventing the bug from happening again without being noticed. For too much of the code, this has been impossible without a weeks-long detour into tests.
Breaking New Ground
In the past few months we’ve figured out how to make automated tests for things that were previously impossible:
Communication with mail servers using encrypted channels.
OAuth2 authentication with mail servers.
Communication with web servers where a specific address must be used and an unencrypted channel must not be used.
Servers at any given host name or port. Previously, if we wanted to start a server for automated testing, it had to be on the local machine at a non-standard location. Now we can pretend that the server is anywhere, and using standard ports, which is needed for proper testing of account configuration features. (Actually, this was possible before, but now it’s much easier.)
These new abilities are being used to wrap better testing around account set-up features, ahead of the new Account Hub development, so that we can be sure nothing breaks without being noticed. They’re also helping test that collecting mail works when it should, or gives the error prompts we expect when it doesn’t.
Code coverage
We record every line of code that runs during our tests. Collecting all that data tells what code doesn’t run during our tests. If a block of code doesn’t run during any of our tests, nothing will tell us when it breaks until somebody uses the code and complains.
Looking at the data, you might notice that our overall number is now lower than it was when we started measuring. This doesn’t mean that our testing got worse, it actually shows where we added a lot of code (that isn’t maintained by us) in the third_party directory. For a better reflection of the progress we’ve made, check out the individual directories, especially mail/base which contains the most important user interface code.
Just setting up the code coverage tools and looking at the results uncovered several memory leaks. (A memory leak is where memory is allocated for a task and not released when it is no longer needed.) We fixed these leaks and some more that existed in our test code. We now have very low levels of memory leaking in our test runs, so if we make a mistake it is easy to spot.
Code coverage data can also point to code that is no longer used. We’ve removed some big chunks of this dead code, which means we’re not wasting time maintaining it.
Mozmill no more
Towards the end of last year we finally retired an old test suite known as Mozmill. Those tests were partially migrated to a different test suite (Mochitest) about four years ago, and things were mostly working fine so it wasn’t a priority to finish. These tests now do things in a more conventional way instead of relying on a bunch of clever but weird tricks.
How much of the code is test code?
About 27%. This is a very rough estimate based on the files in our code repository (minus some third-party directories) and whether they are inside a directory with “test” in the name or not. That’s risen from about 19% in the last five years.
There is no particular goal in mind, but I can imagine a future where there is as much test code as non-test code. If we achieve that, Thunderbird will be in a very healthy place.
Looking ahead, we’ll be asking contributors to add tests to their patches more often. This obviously depends on the circumstance. But if you’re adding or fixing something, that is the best time to ensure it continues to work in the future. As always, feel free to reach out if you need help writing or running tests, either via Matrix or Topicbox mailing lists:
Let’s step back into the Thunderbird Time Machine and teleport ourselves back to December 2009. If you were on the bleeding edge, maybe you were upgrading your computer to the newly released Windows 7 (or checking out Ubuntu 9.10 “Karmic Koala”.) Perhaps you were pouring all your free time into Valve’s ridiculously fun team-based survival shooter Left 4 Dead 2. And maybe, just maybe, you were eagerly anticipating installing Thunderbird 3.0 — especially since it had been a lengthy two years since Thunderbird 2.0 had launched.
What happened during those two years? The Thunderbird developer community — and Mozilla Messaging — clearly stayed busy and productive. Thunderbird 3.0 introduced several new feature milestones!
1) The Email Account Wizard
We take it for granted now, but in the 2000s, adding an account to an email client wasn’t remotely simple. Traditionally you needed to know your IMAP/POP3 and SMTP server URLs, port numbers, and authentication settings. When Thunderbird 3.0 launched, all that was required was your username and password for most mainstream email service providers like Yahoo, Hotmail, or Gmail. Thunderbird went out and detected the rest of the settings for you. Neat!
2) A New Tabbed Interface
With Firefox at its core, Thunderbird followed in the footsteps of most web browsers by offering a tabbed interface. Imagine! Being able to quickly tab between various searches and emails without navigating a chaotic mess of separate windows!
3) A New Add-on Manager
<figcaption class="wp-element-caption">Screenshot from HowToGeek’s Thunderbird 3.0 review.</figcaption>
Speaking of Firefox, Thunderbird quickly adopted the same kind of Add-on Manager that Firefox had recently integrated. No need to fire up a browser to search for useful extensions to Thunderbird — now you could search and install new functionality from right inside Thunderbird itself.
4) Advanced Search Options
Searching your emails got a massive boost in Thunderbird 3.0. Advanced filtering tools means you could filter your results by sender, attachments, people, folders, and more. A shiny new timeline view was also introduced, letting you jump directly to a certain date’s results.
5) The Migration Assistant
Tying this all together was a simple but wonderful migration assistant. It served as a way to introduce users to certain new features (like per-account IMAP synchronization), and visually toggle them on or off (useful for displaying the revised Message Toolbar and giving users a choice of where to enjoy it). To me, this particular addition felt ahead of its time. We’ve been discussing the idea of re-introducing it in a future Thunderbird release, but one of the steep hurdles to doing so now is localization. If it’s something you’d like to see, let us know in the comments.
Try It Out For Yourself
If you want to personally step into the Thunderbird Time Machine, every version ever released for Windows, Linux, and macOS is available in this archive. I ran mine inside of a Windows 7 virtual machine, since my native Linux install complained about missing libraries when trying to get Thunderbird 3.0 running.
Regardless if you’re a new Thunderbird user or a veteran who’s been with us since 2003, thanks for being on the journey with us!
Last year we took ownership of the Thunderbird Flatpak, and it has been our officially recommended package for Linux users. However, we are expanding our horizons to make sure the Thunderbird Snap experience is officially supported too. We at Thunderbird are team “free software”, independent of the packaging technology. This will mostly affect our Ubuntu users but there are plenty of other Snap users out there as well.
Why support both the Snap and Flatpak?
In the spirit of free software, we want to support as many of our users as possible without discriminating on their package preferences. We are not a large company with infinite resources, so we can’t support everything under the sun. But we can make informed decisions that reach the majority of our Linux users.
The Thunderbird Snap has been well maintained by the Ubuntu desktop team for years, and we felt it was time to step up and help out.
What does this mean for me?
If you are an Ubuntu user, then you may already be using the Thunderbird Snap. The next release of Ubuntu is 24.04 (available April 25) and will be the first Ubuntu release that seeds the Thunderbird Snap on the ISO. So if you do a fresh full install of Ubuntu, you will be using the Thunderbird Snap that you know is directly supported by the Thunderbird team.
If you are not an Ubuntu user but Snaps are still a part of your life, then you will still benefit from the same rolling updates provided by the Snap experience.
What changes are expected?
From a user perspective, you should see no changes. Just keep using whichever Thunderbird Snap channel you are comfortable with.
From a developer perspective, we have added the Snap build to our build infrastructure on treeherder. This means whenever a full build is triggered automatically from commits, the Snap is built as well for testing. Whenever the build is one we want to release to the public, this will trigger a general flow:
The existing launchpad mirror will pick up this change and automatically build the Snap for x86 and arm64.
If the launchpad Snap build succeeds, the Snap will be uploaded to the designated Snap store channel.
So all we are changing is adding the snap build into the Thunderbird build infrastructure and plugging it into the existing automation that feeds the snap store.
Hello Thunderbird Community! March is over, which means it’s time for another Development Digest to share the current progress and product direction of Thunderbird development.
Is this your first time reading the Development Digest? Find them all using the Dev Digest tag!
Rust and Exchange
It seems that this section is part of every Development Digest! But that’s the reality of these large efforts, spanning across multiple months with slow but steady progress.
This month we completed initial Exchange Autodiscovery and compatibility with OAuth in our account setup flow, as well as fetching and rendering of all folders. Some areas still need polish and clean up. But work continues towards having things behind a pref in the next beta release. You can follow the progress in this bug.
Meanwhile, here are some goodies to try if you need to parse the Microsoft Exchange Web Services data set and the current crates for serializing and deserializing XML don’t serve you well. https://github.com/thunderbird/xml_struct
List management
Shout out to Magnus for implementing the first step towards a more manageable mailing list subscription flow. An initial implementation of the List Management feature just landed on daily and beta, and it was recently announced in the tb-beta mailing list with a screenshot to show it in action.
It’s currently accessible via a context menu on the List ID. But we’re planning to do some UX and UI explorations to find the best way to expose it without making it annoying.
Another big shout out to Magnus for finishing the ESMification effort! As users, you won’t see or notice any difference, but for developers this substantial architectural change saw the removal of all .jsm files in favor of standard JavaScript modules.
A huge win for a more standardized code base! This allows us to leverage all the nice features of modern JavaScript in Thunderbird development.
Tiny changes and improvements in Thunderbird development
A lot of nice quality of life improvements tend to happen in small chunks that are not easy to see or spot right away.
Here’s a list of the projects we’re actively working on and will be focusing on for the next month:
Cards view UI completion.
Fixed missing FindBar in multimessage and browser view.
Implementation of a new visual selection paradigm.
Improvements to usability and accessibility of the Quick Filter bar.
Completion of the email setup in the new Account Hub.
Many add-ons API improvements and additions (big shout out to John).
Support for viewing nested signed messages and other OpenPGP improvements.
Stay tuned and make sure to sign up to our mailing lists to get detailed updates on all the items in this list, and a lot more.
As usual, if you want to see things as they land you can always check the pushlog and try running daily, which would be immensely helpful for catching bugs early.
See ya next time in our April Development Digest.
Alessandro Castellani(he, him) Director of Product Engineering
If you’re interested in joining the technical discussion around Thunderbird development, consider joining one or several of our mailing list groups here.
This month’s topics for our Thunderbird Community Office Hours will be decided by you! We’d like to invite the community to bring their questions, comments, and general conversation to Team Thunderbird for an informal and informational chat. As always, send your questions in advance to officehours@thunderbird.net!
Be sure to note the change in day of the week and time, especially if you’re in Europe and not on summer time yet!
March Office Hours: Open Forum and FAQ
While we love having community office hours with specific topics, from our design process to Add-ons, we want to make time for an open forum, where you bring the topics of discussion. Do you have a great idea for a feature request, or need help filing a bug? Or do you want to know how to use SUMO better, or get some Thunderbird tips? Maybe you want to know more about Team Thunderbird, whether it’s how we got started in open source to how we like our coffee. This is the time to ask these questions and more!
We also just got back from SCaLE21x, and we had so many great questions from people who stopped by the booth. So in addition to answering your questions, whether emailed or live, we’d like to tackle some the things people asked most during our first SCaLE appearance.
Catch Up On Last Month’s Thunderbird Community Office Hours
While you’re thinking of questions to ask, watch last month’s office hours with John Bieling all about Add-on development. We had a fantastic chat about the history, present state, and future of Add-ons, with advice on getting involved in development and support. Watch the video below and read more about our guest at last month’s blog post.
Welcome to a new report on the progress of transforming K-9 Mail into Thunderbird for Android. I hope you’ve enjoyed the extra day in February. We certainly did and used this opportunity to release a new stable version on February 29.
If you’re new to this series or the unusually long February made you forget what happened the previous month, you might want to check out January’s progress report.
New stable release
We spent most of our time in February getting ready for a new stable release – K-9 Mail 6.800. That mostly meant fixing bugs and usability issues reported by beta testers. Thanks to everyone who tested the app and reported bugs
With the new account setup being mostly done, we’ll concentrate on the following two areas.
Material 3
The question of whether to update the user interface to match the design used by the latest Android version seems to have always split the K-9 Mail user base. One group prefers that we work on adding new features instead. The other group wants their email app of choice to look similar to the apps that ship with Android.
Never updating the user interface to the latest design is not really an option. At some point all third-party libraries we’re using will only support the latest platform design. Not updating those libraries is also not an option because Android itself is constantly changing and requires app/library updates just to keep existing functionality working.
I think we found a good balance by not being the first ones to update to Material 3. By now a lot of other app developers have done so and countless bugs related to Material 3 have been found and fixed. So it’s a good time for us to start switching to Android’s latest design system now.
We’re currently still in a research phase to figure out what parts of the app need changing. Once that’s done, we’ll change the base theme and fix up the app screen by screen. You will be able to follow along by becoming a beta tester and installing K-9 Mail 6.9xx beta versions once those become available.
Android 14 compatibility
K-9 Mail is affected by a couple of changes that were introduced with Android 14. We’ve started to look into which parts of the app need to be updated to be able to target Android 14.
Our current plan is to include the necessary changes in updates to the K-9 Mail 6.8xx line.
Community Contributions
S Tanveer Hussain submitted a pull request to update the information about third-party libraries in K-9 Mail’s About screen (#7601)
GitHub user LorenzHo provided a patch to not focus the recipient input field when the Compose screen was opened using a mailto: URI (#7623). Unfortunately, this change had to be backed out later because of unintended side effects. But we’re hopeful a modified version of this change will make it into the app soon.
Thank you for your contributions!
Releases
In February 2024 we published a new stable release:
We’re happy to announce the release of K-9 Mail 6.800. The main goal of this version is to make it easier for you to add your email accounts to the app.
Setting up an email account in K-9 Mail is something many new users have struggled with in the past. That’s mainly because automatic setup was only supported for a handful of large email providers. If you had an email account with another email provider, you had to manually enter the incoming and outgoing server settings. But finding the correct server settings can be challenging.
So we set out to improve the setup experience. Since this part of the app was quite old and had a couple of other problems, we used this opportunity to rewrite the whole account setup component. This turned out to be more work than originally anticipated. But we’re quite happy with the result.
Let’s have a brief look at the steps involved in setting up a new account.
1. Enter email address
To get the process started, all you have to do is enter the email address of the account you want to set up in K-9 Mail.
2. Provide login credentials
After tapping the Next button, the app will use Thunderbird’s Autoconfig mechanism to try to find the appropriate incoming and outgoing server settings. Then you’ll be asked to provide a password or use the web login flow, depending on the email provider.
The app will then try to log in to the incoming and outgoing server using the provided credentials.
3. Provide some basic information about the account
If your login credentials check out, you’ll be asked to provide your name for outgoing messages. For all the other inputs you can go with the defaults. All settings can be changed later, once an account has been set up.
If everything goes well, that’s all it takes to set up an account.
Of course there’s still cases where the app won’t be able to automatically find a configuration and the user will be asked to manually provide the incoming and outgoing server settings. But we’ll be working with email providers to hopefully reduce the number of times this happens.
What else is new?
While the account setup rewrite was our main development focus, we’ve also made a couple of smaller changes and bug fixes. You can find a list of the most notable ones below.
Improvements and behavior changes
Made it harder to accidentally trigger swipe actions in the message list screen
IMAP: Added support for sending the ID command (that is required by some email providers)
Improved screen reader experience in various places
Improved display of some HTML messages
Changed background color in message view and compose screens when using dark theme
Adding to contacts should now allow you again to add the email address to an existing contact
Added image handling within the context menu for hyperlinks
A URI pasted when composing a message will now be surrounded by angle brackets
Don’t use nickname as display name when auto-completing recipient using the nickname
Changed compose icon in the message list widget to match the icon inside the app
Don’t attempt to open file: URIs in an email; tapping such a link will now copy the URL to the clipboard instead
Added option to return to the message list after marking a message as unread in the message view
Combined settings “Return to list after delete” and “Show next message after delete” into “After deleting or moving a message”
Moved “Show only subscribed folders” setting to “Folders” section
Added copy action to recipient dropdown in compose screen (to work around missing drag & drop functionality)
Simplified the app icon so it can be a vector drawable
Added support for the IMAP MOVE extension
Bug fixes
Fixed bug where account name wasn’t displayed in the message view when it should
Fixed bugs with importing and exporting identities
The app will no longer ask to save a draft when no changes have been made to an existing draft message
Fixed bug where “Cannot connect to crypto provider” was displayed when the problem wasn’t the crypto provider
Fixed a crash caused by an interaction with OpenKeychain 6.0.0
Fixed inconsistent behavior when replying to messages
Fixed display issue with recipients in message view screen
Fixed display issues when rendering a message/rfc822 inline part
Fixed display issue when removing an account
Fixed notification sounds on WearOS devices
Fixed the app so it runs on devices that don’t support home screen widgets
A fresh app install on Android 14 will be missing the “alarms & reminders” permission required for Push to work. Please allow setting alarms and reminders in Android’s app settings under Alarms & reminders.
Some software keyboards automatically capitalize words when entering the email address in the first account setup screen.
When a password containing line breaks is pasted during account setup, these line breaks are neither ignored nor flagged as an error. This will most likely lead to an authentication error when checking server settings.
Where To Get K-9 Mail Version 6.800
Version 6.800 has started gradually rolling out. As always, you can get it on the following platforms:
(Note that the release will gradually roll out on the Google Play Store, and should appear shortly on F-Droid, so please be patient if it doesn’t automatically update.)
Hello Thunderbird Community! I can’t believe it’s already the end of February. Time goes by very fast and it seems that there’s never enough time to do all the things that you set your mind to. Nonetheless, it’s that time of the month again for a juicy and hopefully interesting Thunderbird Development Digest.
If this is your first time reading our monthly Dev Digest, these are short posts to give our community visibility into features and updates being planned for Thunderbird, as well as progress reports on work that’s in the early stages of development.
Let’s jump right into it, because there’s a lot to get excited about!
Rust and Exchange
Things are moving steadily on this front. Maybe not as fast as we would like, but we’re handling a complicated implementation and we’re adding a new protocol for the first time in more than a decade, so some friction is to be expected.
Nonetheless, you can start following the progress in our Thundercell repository. We’re using this repo to temporarily “park” crates and other libraries we’re aiming to vendor inside Thunderbird.
We’re aiming at reaching an alpha state where we can land in Thunderbird later next month and start asking for user feedback on Daily.
Mozilla Account + Thunderbird Sync
<figcaption class="wp-element-caption">Illustration by Alessandro Castellani</figcaption>
Things are moving forward on this front as well. We’re currently in the process of setting up our own SyncServer and TokenStorage in order to allow users to log in with their Mozilla Account but sync the Thunderbird data in an independent location from the Firefox data. This gives us an extra layer of security as it will prevent an app from accessing the other app’s data and vice versa.
In case you didn’t know, you can already use a Mozilla account and Sync on Daily, but this only works with a staging server and you’ll need an alternate Mozilla account for testing. There are a couple of known bugs but overall things seem to be working properly. Once we switch to our storage server, we will expose this feature more and enable it on Beta for everyone to test.
Oh, Snap!
Our continuous efforts to own our packages and distribution methods is moving forward with the internal creation of a Snap package. (For background, last year we took ownership of the Thunderbird Flatpak.)
We’re currently internally testing the Beta and things seem to work accordingly. We will announce it publicly when it’s available from the Snap Store, with the objective of offering both Stable and Beta channels.
We’re exploring the possibility of also offering a Daily channel, but that’s a bit more complicated and we will need more time to make sure it’s doable and automated, so stay tuned.
As usual, if you want to see things as they land you can always check the pushlog and try running daily, which would be immensely helpful for catching bugs early.
See ya next month,
Alessandro Castellani(he, him) Director of Product Engineering
If you’re interested in joining the technical discussion around Thunderbird development, consider joining one or several of our mailing list groups here.
A new year, a new progress report! Learn what we did in January on our journey to transform K-9 Mail into Thunderbird for Android. If you’re new here or you forgot where we left off last year, check out the previous progress report.
Account setup
In January most of our work went into polishing the user interface and user experience of the new and improved account setup. However, there was still one feature missing that we really wanted to get in there: the ability to configure special folders.
Special folders
K-9 Mail supports the following special folders:
Archive: When configured, an Archive action will be available that moves a message to the designated archive folder.
Drafts: When configured, the Save as draft action will be available in the compose screen.
Sent: Messages that have been successfully submitted to the outgoing server will be uploaded to this folder. If this special folder is set to None, the app won’t save a copy of sent messages. Note: There’s also the setting Upload sent messages that can be disabled to prevent sent messages from being uploaded, e.g. if your email provider automatically saves a copy of outgoing messages.
Spam: When configured, a Spam action will be available that moves a message to the designated spam folder. (Please note that K-9 Mail currently does not include spam detection. So besides moving the message, this doesn’t do anything on its own. However, moving a message to and from the spam folder often trains the server-side spam filter available at many email providers.)
Trash: When configured, deleting a message in the app will move it to the designated trash folder. If the special folder is set to None, emails are deleted permanently right away.
In the distant past, K-9 Mail was simply using common names for these folders and created them on the server if they didn’t exist yet. But some email clients were using different names. And so a user could end up with e.g. multiple folders for sent messages. Of course there was an option to manually change the special folder assignment. But usually people only noticed when it was too late and the new folder already contained a couple of messages. Manually cleaning this up and making sure all email clients are configured to use the same folders is not fun.
To solve this problem, RFC 6154 introduced the SPECIAL-USE IMAP extension. That’s a mechanism to save this special folder mapping on an IMAP server. Having this information on the server means all email clients can simply fetch that mapping and then there should be no disagreement on e.g. which folder is used for sent messages.
Unfortunately, there’s still some email providers that don’t support this extension. There’s also cases where the server supports the feature, but none of the special roles are assigned to any folder. When K-9 Mail added support for the SPECIAL-USE extension, it simply used the data from the server, even if it meant not using any special folders. Unfortunately, that could be even worse than creating new folders, because you might end up e.g. not having a copy of sent messages.
So now the app is displaying a screen to ask the user to assign special folders when setting up an account.
This screen is skipped if the app receives a full mapping from the server, i.e. all special roles are assigned to a folder. Of course you’ll still be able to change the special folder assignment after the account has been created.
Splitting account options
We split what used to be the account options screen into two different screens: display options and sync options.
With the special folders screen done, we’re now feature complete. So we took a step back to look at the whole experience of setting up an account. And we’ve found several areas where we could improve the app.
Here’s an (incomplete) list of things we’ve changed:
We reduced the font weight of the header text to be less distracting.
In some parts of the flow there’s enough content on the screen that a user has to scroll. The area between the header and the navigation buttons at the bottom can be very small depending on the device size. So we included the header in the scrollable area to improve the experience on devices with a small screen.
There are a couple of transient screens, e.g. when checking server settings. Previously the app first displayed a progress indicator when checking server settings, then a success message for 2 seconds, but allowed the user to skip this screen by pressing the Next button. This turned out to be annoying and confusing. Annoying because the user has to wait longer than necessary; and confusing because it looked like user input was required, but by the time the user realizes that, the app will have most likely switched to the next screen automatically. We updated these transient screens to always show a progress indicator and hide the Next button, so users know something is happening and there’s currently nothing for them to do.
We also fixed a couple of smaller issues, like the inbox not being synchronized during setup when an account was configured for manual synchronization.
Fixing bugs
Some of the more interesting bugs we fixed in January:
When rotating the screen while selecting a notification sound in settings, some of the notification settings were accidentally disabled (#7468).
When importing settings a preview lines value of 0 was ignored and the default of 2 was used instead (#7493).
When viewing a message and long-pressing an image that is also a link, only menu items relevant for images were displayed, but not ones relevant for links (#7457).
Opening an attachment from K-9 Mail’s message view in an external app and then sharing the content to K-9 Mail opened the compose screen for a new message but didn’t add an attachment (#7557).
Community Contributions
new-sashok724 fixed a bug that prevented the use of IP addresses for incoming or outgoing servers (#7483).
The topic for this month’s Thunderbird Community Office Hours takes a short break from the core of Thunderbird and takes us into the world of extensions we call Add-ons. These allow our users to add features and options beyond the customization already available in Thunderbird by default.
We want it to be easy to make Thunderbird yours, and so does our community. The Thunderbird Add-on page shows the power of community-driven extensions. There are Add-ons for everything, from themes to integrations, that add even more customization to Thunderbird.
Our guest for this month’s Thunderbird Community Office Hours is John Bieling, who is the person responsible for Thunderbird’s add-on component. This includes the WebExtension APIs, add-on documentation, as well as community support. He hosts a frequent open call about Add-on development and is welcoming to any developers seeking help. Come join us to learn about Add-on development and meet a key developer in the space.
Catch Up On Last Month’s Thunderbird Community Office Hours
Before you join us on February 22 at 18:00 UTC, watch last month’s office hours with UX Engineer Elizabeth Mitchell. We had some great discussion around the Message Context Menu and testing beta and daily images. Watch the video and read more about our guest at last month’s blog post.
<figcaption class="wp-element-caption">Watch January’s Office Hours session, all about the message context menu</figcaption>
Join Us On Zoom
(Yes, we’re still on Zoom for now, but a Jitsi server for future office hours is in the works!)
The Thunderbird Project enjoyed a fantastic 2023. From my point of view – as someone who regularly engages with both the community and our team on a daily basis – the past year brought a renewed sense of purpose, sustainability, and excitement to Thunderbird. Let’s talk about a few of the awesome milestones Thunderbird achieved, but let’s also discuss where we stumbled and what lessons we learned along the way.
Our 2023 Milestones
The biggest milestone of 2023 was Thunderbird 115 “Supernova.” This release marked the first step towards a more flexible, reliable, and customizable Thunderbird that will accommodate different needs and workflows. Work has been long underway to modernize huge amounts of old code, with the aim of modernizing Thunderbird to deliver new features even faster. The “Supernova” release represented the first fruits of those efforts, and there’s a lot more in the pipeline!
Alongside Supernova came a brand new Thunderbird logo to signal the revitalization of the project. We finally (even a bit reluctantly) said goodbye to our beloved “wig on an envelope” and ushered in a new era of Thunderbird with a refreshed, redesigned logo. But it was important to honor our roots, which is why we hired Jon Hicks – the designer of the original Firefox and Thunderbird logos – to help us bring it to life. (Now that you’ve all been living with it for the last several months, has it grown on you? Let us know in the comments of this post!)
One 2023 milestone that deserves more attention is that we hired a dedicated User Support Specialist! Roland Tanglao has been working enthusiastically towards removing “documentation debt” and updating the 100s of Thunderbird support articles at support.mozilla.org (which you’ll see us refer to internally as “SUMO”). Beyond that, he keeps a watchful eye on our Matrix community support channel for emerging issues, and is in the forums answering as many help questions as humanly possible, alongside our amazing support volunteers. In a nutshell, Roland is doing everything he can to improve the experience of asking for and receiving support, modernize existing documentation, and create new guides and articles that make using Thunderbird easier.
These are some – not all – of our accomplishments from last year. But it’s time to shift focus to where we stumbled, and how we’ll do better.
The Lessons We Learned In 2023
In 2023, we failed to finish some of the great features we wanted to bring to Thunderbird, including Sync and Account Hub (both of which, however, are still in development). We also missed our target release window for Thunderbird on Android, after deciding it was worth the extra development time to add the kind of functionality and flexibility you expect from Thunderbird software.
Speaking of functionality you expect, we hear you loud and clear: you want Exchange support in Thunderbird. We’ve already done some exploratory work, and have enabled the usage of Rust in Thunderbird. This is a complex topic, but the short version is that this opens the doors for us to start implementing native support for the Exchange protocol. It’s officially on our roadmap!
We also believe our communication with you has fallen short of where it needs to be. There are times when we get so excited about things we’re working on that it seems like marketing hype. In other situations, we have over-promised and under-delivered because these projects haven’t been extensively scoped out.
We’re beginning to solve the latter issue with the recent hiring of Kelly McSweeney, Senior Technical PM. She joined our team late last year and brings 20 years of valuable experience to Thunderbird. In a nutshell, Kelly is building processes and tools to accurately gauge how long development time will realistically take, from extensive projects to the tiniest tasks. Basically, she’s getting us very organized and making things run much more efficiently! This not only means smoother operations across the organization, but also clearer communication with you going forward.
And communication is our biggest area of opportunity right now, specifically with our global Thunderbird community. We haven’t been as transparent as an open source project should be, nor have we discussed our future plans frequently enough. We’ve had several meetings about this over the past few weeks, and we’re taking immediate steps to do better.
To begin with, you’ll start seeing monthly Developer Digests like this one from Alex, aimed at giving you a closer look at the work currently being planned. We’re also increasing our activity on the Thunderbird mailing lists, where you can give us direct feedback about future improvements and features.
In 2024 you can also look forward to monthly community Office Hours sessions. This is where you can get some face time (or just voice time) with our team, and watch presentations about upcoming features and improvements by the developer(s) working on them.
One last thing: In 2023, Thunderbird’s Marketing & Communications team consisted of myself and Wayne Mery. This year Wayne and I are fortunate to be working alongside new team members Heather Ellsworth, Monica Ayhens-Madon, and Natalia Ivanova. Together, we’re going to work diligently to create more tutorials on the blog, more video guides, and more content to help you get the most out of Thunderbird – with a focus on productivity.
How To Stay Updated
Thank you for being on this journey with us! If you want to get more involved and stay in touch, here are the best places to keep up with what’s happening at Thunderbird:
We will be more active right here on this blog, so come back once or twice per month to see what’s new.
If you enjoy the technical bits, want to help test Thunderbird, or you’re part of our contributor community, these mailing lists at Topicbox are ideal.
Follow us on Mastodon or X/Twitter for more frequent – and fun – updates!
Hello Thunderbird Community! I’m very happy to kick off a new monthly Thunderbird development recap in order to bring a deeper look and understanding of what we’re working on, and the status of these efforts. (We also publish monthly progress reports on Thunderbird for Android.)
These monthly digests will be in a very short format, focusing primarily on the work that is currently being planned or initiated that is not yet fully captured in BugZilla. Nonetheless, we’re putting it out there to cherish and fully embrace the open nature of Thunderbird.
Without further ado, let’s get into it!
2024 Thunderbird Development RoadmapsPublished
Over at DTN, we’ve published initial 2024 roadmaps for the work we have planned on Thunderbird for desktop, and Thunderbird for Android. These will be updated periodically as we continue to scope out each project.
Global Message Database
Our database is currently based on Mork, which is a very old paradigm that creates a lot of limitations, blocking us from doing anything remotely modern or expected (a real threaded conversation view is a classic example). Removing and reworking this implementation, which is at the very core of every message and folder interaction, is not an easy lift and requires a lot of careful planning and exploration, but the work is underway.
The first clean up effort is targeted at removing the old and bad paradigm of the “non-unique unique ID” (kudos to our very own Ben Campbell for coining this term), which causes all sorts of problems. You can follow the work in Bug 1806770.
Cards view final sprint
If you’re using Daily or Beta you might have already seen a lot of drastic differences from 115 for Cards View.
Currently, we’re shaping up the final sprint to polish what we’ve implemented and add extra needed features. We’re in the process of opening all the needed bugs and assigning resources for this final sprint. You can follow the progress by tracking this meta bug and all its child bugs.
As usual, we will continue sharing plans and mock-ups in the UX mailing list, so make sure to follow that if you’re interested in seeing early visual prototypes before any code is touched.
Rust Implementation and ExchangeSupport
This is a very large topic and exploration that requires dedicated posts (which are coming) and extensive recaps. The short story is that we were able to enable the usage of Rust in Thunderbird, therefore opening the doors for us to start implementing native support for the Exchange protocol by building and vendoring a Rust crate.
Once we have a stable and safe implementation, we will share that crate publicly on a GitHub repo so everyone will be able to vendor it and improve it.
Make sure to follow tb-planning and tb-developers mailing lists to soon get more detailed and very in depth info on Rust and Exchange in Thunderbird.
As usual, if you want to see things as they land you can always check the pushlog and try running daily, which would be immensely helpful for catching bugs early.
Alessandro Castellani(he, him) Director of Product Engineering
If you’re interested in joining the discussion around Thunderbird development, consider joining one or several of our mailing list groups here.
UPDATE: Our January Office Hours was fantastic! Here’s the full video replay.
A New Year of New Office Hours
We’re back from our end of year break, breaking in our new calendars, and ready to start 2024 with our renewed, refreshed, and refocused community office hours. Thank you to everyone who joined us for our November session! If you missed out on our chat about the new Cards View and the Thunderbird design process, you can find the video (which also describes the new format) in this blog post.
We’re excited for another year of bringing you expert insights from the Thunderbird Team and our broader community. To kick off 2024, and to build on November’s excellent discussion, we’ll be continuing our dive into another important aspect of the Thunderbird design process.
January Office Hours Topic: Message Context Menu
<figcaption class="wp-element-caption">Mock-up: designs shown are not final and subject to change. </figcaption>
We’ve been working on some significant (and what we think are pretty fantastic) UI changes to Thunderbird. Besides the new Cards View, we have some exciting overhauls to the Message Context Menu (aka the right-click menu) planned. UX Engineer Elizabeth Mitchell will discuss these changes, and most importantly, why we’re making them. Additionally, Elizabeth is one of the leaders on making Thunderbird accessible for all! We’re excited to hear how the new Message Context Menu will make your email experience easier and more effective.
If you’d like a sneak peak of the Context Menu plans, you can find them here.
And as always, if you have any questions you’d like to ask during the January office hours, you can e-mail them to officehours@thunderbird.net.
Join Us On Zoom
(Yes, we’re still on Zoom for now, but a Jitsi server for future office hours is in the works!)
When will Thunderbird for Android be released? This is a question that comes up quite a lot, and we appreciate that you’re all excited to finally put Thunderbird in your pocket. It’s not a simple answer, but we’ll do our best to explain why things are taking longer than expected.
We have always been a bit vague on when we were going to release Thunderbird for Android. At first this was because we still had to figure out what features we wanted to add to K-9 Mail before we were comfortable calling it Thunderbird. Once we had a list, we estimated how long it would take to add those features to the app. Then something happened that always happens in software projects – things took longer than expected. So we cut down on features and aimed for a release at the end of 2023. As we got closer to the end of the year, it became clear that even with the reduced set of features, the release date would have almost certainly slipped into early 2024.
We then sat together and reevaluated the situation. In the end we decided that there’s no rush. We’ll work on the features we wanted in the app in the first place, because you deserve the best mobile experience we can give you. Once those features have been added, we’ll release the app as Thunderbird for Android.
Why Wait? Try K-9 Mail Now
But of course you don’t have to wait until then. All our development happens out in the open. The stable version of K-9 Mail contains all of the features we have already completed. The beta version of K-9 Mail contains the feature(s) we’re currently working on.
Both stable and beta versions can be installed via F-Droid or Google Play.
Side note: Quite a few people seem to love K-9 Mail and have asked us to keep the robot dog around. We believe it should be relatively little effort to build two apps from one code base. The apps would be virtually identical and only differ in app name, app icon, and the color scheme. So our current plan is to keep K-9 Mail around.
Whether you prefer metal dogs or mythical birds, we’ve got you covered.
In February 2023 we started publishing monthly reports on the progress of transforming K-9 Mail into Thunderbird for Android. Somewhat to my surprise, we managed to keep this up throughout the entire year.
But since the end of the year company shutdown is coming up and both Wolf and I have some vacation days left, this will be the last progress report of the year, covering both November and December. If you need a refresher on where we left off previously, know that the progress report for October is only one click away.
New Home On Google Play
If you’ve recently visited K-9 Mail’s page on Google Play you might have noticed that the developer name changed from “K-9 Dog Walkers” to “Mozilla Thunderbird”. That’s because we finally got around to moving the app to a developer account owned by Thunderbird.
I’d like to use this opportunity to thank Jesse Vincent, who not only founded the K-9 Mail project, but also managed the Google Play developer account for all these years. Thank you
Previously, the app asked the user to grant the permission to access contacts when the message list or compose screens were displayed.
<figcaption class="wp-element-caption">Permission prompt in message list screen</figcaption><figcaption class="wp-element-caption">Permission prompt in compose screen</figcaption>
The app asked for the contacts permission every time one of these screens was opened. That’s not as bad as it sounds. Android automatically ignores such a request after the user has selected the “deny” option twice. Unfortunately, dismissing the dialog e.g. by using the back button, doesn’t count as denying the permission request. So users who chose that option to get rid of the dialog were asked again and again. Clearly not a great experience.
So we changed it. Now, the app no longer asks for the contacts permission in those screens. Instead, asking the user to grant permissions is now part of the onboarding flow. After adding the first account, users will see the following screen:
The keen observer will have noticed that the app is now also asking for the permission to create notifications. Since the introduction of notification categories in Android 8, users have always had the option to disable some or all notifications created by an app. But starting with Android 13, users now have to explicitly grant the permission to create notifications.
While the app will work without the notification permission, you should still grant it to the app, at least for now. Currently, some errors (e.g. when sending an email has failed) are only communicated via a notification.
And don’t worry, granting the permission doesn’t mean you’ll be bombarded with notifications. You can still configure whether you want to get notifications for new messages on a per account basis.
Improved Account Setup
This section has been a fixture in the last couple of progress reports. The new account setup code has been a lot of work. And we’re still not quite done yet. However, it already is in a state where it’s a vast improvement over what we had previously.
Bug fixes
Thanks to feedback from beta testers, we identified and fixed a couple of bugs.
The app was crashing when trying to display an error message after the user had entered an invalid or unsupported email address.
While fixing the bug above, we also noticed that some placeholder code to validate email addresses was still used. We replaced that code and improved error messages, e.g. when encountering a syntactically valid, but deliberately unsupported email address like test@[127.0.0.1].
A user reported a crash when trying to set up an account with a particular email domain. We tracked this down to an MX DNS record containing an underscore. That’s not a valid character for a hostname. The app already checked for that, but the error wasn’t caught and so crashed the app.
User experience improvements
Thanks to feedback from people who went through the manual setup flow multiple times, we identified a couple of usability issues. We made some changes like disabling auto-correct in the server name text field and copying the password entered in the incoming server settings screen to the outgoing server settings screen.
Hopefully, automatic account setup will just work for you. But if you have to use the manual setup route, at least now it should be a tiny bit less annoying.
Edit server settings
Editing incoming or outgoing server settings is not strictly part of setting up an account. However, the same screens used in the manual account setup flow are also used when editing server settings of an existing account (e.g. by going to Settings → [Account] → Fetching mail → Incoming server).
<figcaption class="wp-element-caption">Incoming server settings screen during manual account setup</figcaption><figcaption class="wp-element-caption">Incoming server settings screen when editing an existing account</figcaption>
The screens don’t behave exactly the same in both instances, so some changes were necessary. In November we finally got around to adapting the screens. And now the new UI is also used when editing server settings.
Targeting Android 13
Every year Google requires Android developers to change their apps to support the new (security) features and restrictions of the Android version that was released the prior year. This is automatically enforced by only allowing developers to publish app updates on Google Play when they “target” the required Android version. This year’s deadline was August 31.
There was only one change in Android 13 that affected K-9 Mail. Once an app targets this Android version, it has to ask the user for permission before being able to create notifications. Since our plans already included adding a new screen to ask for permissions during onboarding, we didn’t spend too much time worrying about the deadline.
But due to us being busy working on other features, we only got around to adding the permission screen in November. We requested an extension to the deadline, which (to my surprise) seems to have been granted automatically. Still, there was a brief period of time where we weren’t able to publish new beta versions because we missed the extended deadline by a couple of days.
We’ll prioritize updating the app to target the latest Android version in the future.
Push Not Working On Android 14
When Push is enabled, K-9 Mail uses what the developer documentation calls “exact alarms” to periodically refresh its Push connection to the server. Starting with Android 12, apps need to request a separate permission to use exact alarms. But the permission itself was granted automatically.
In Android 14 (released in October 2023) Google changed the behavior and Android no longer pre-grants this permission to newly installed apps. However, instead of limiting this to apps targeting Android 14, for some reason they decided to extend this behavior change to apps targeting Android 13.
This unfortunate choice by the creator of Android means that Push is currently not working for users who perform a fresh install of K-9 Mail 6.712 or newer on Android 14. Upgrading from a previous version of K-9 Mail should be fine since the permission was then granted automatically in the past.
At the beginning of next year we’ll be working on adding a screen to guide the user to grant the necessary permission when enabling Push on Android 14. Until then, you can manually grant the permission by opening Android’s App info screen for the app, then enable Allow setting alarms and reminders under Alarms & reminders.
Community Contributions
In November and December the following contributions by community members were merged into K-9 Mail:
Welcome to the progress report for October! If you missed the one for September, you’re in luck, because we always try to include a link to the progress report of the previous month in the introduction.
Translations
All the translations of the app are provided by volunteers from the community. This work is mostly independent from software development and design work. So sometimes it’s easy for us to forget that a lot more people regularly spend time working on the app.
Dear translators, thank you for all the work you’re putting into making the app more accessible by translating it to so many languages
Move to Weblate
In October we switched to Weblate to manage translations of the app. This has a couple of reasons:
Some of the volunteers providing translations have expressed a strong preference for Weblate. It’s also not uncommon for volunteers to also translate other open source apps. Weblate has become a popular option for that.
Weblate has good tools to allow us to better integrate it into our development process.
Unlike the previous translation service we used, Weblate itself is open source software. And you all know we love open source.
Call for translators
Currently K-9 Mail ships with support for the following 49 languages:
There’s an additional 7 languages where partial translations exist, but are currently not included in the app:
Armenian (36% translated), Azerbaijani (6%), Georgian (51%), Hindi (23%), Norwegian Nynorsk (43%), Tamil (16%), Vietnamese (63%).
However, some of the included translations haven’t been updated in a while. And so in addition to the 7 languages above, there’s quite a few languages that could use the help of new translators. Here’s the list of languages where less than 97% of strings have been translated:
Arabic (88%), Belarusian (89%), Breton (74%), Bulgarian (87%), Croatian (75%), Danish (92%), English (GB) (1%), Esperanto (82%), Gaelic (73%), Galician (82%), Greek (89%), Hebrew (49%), Indonesian (79%), Korean (69%), Latvian (89%), Lithuanian (87%), Malayalam (81%), Norwegian Bokmål (93%), Persian (89%), Portuguese (Portugal) (88%), Russian (88%), Serbian (79%), Slovak (71%), Slovenian (90%), Ukrainian (89%), Welsh (86%).
If you want to help out with translations for any of these, please head over to our Weblate project.
GitHub organization renamed
On October 31 our GitHub organization was renamed from thundernest to thunderbird. We also used this opportunity to rename the repository from k-9 to thunderbird-android.
Old links should redirect to the new location. But if you’re maintaining a website that links to our GitHub organization or repository, please update those links.
Improved account setup
The new account setup is still our main focus. In October most of our work went on behind the scenes. The only user-visible changes were fixing a crash when entering an unsupported email address and respecting the app theme and app language settings in the account setup UI.
We also worked on replacing the screens to edit incoming and outgoing server settings. However, that work is still ongoing and not available in beta releases yet.
Architecture decision records
We’re trying to get better at documenting decisions we made about the architecture of the app. Wolf started us off by creating the following architecture decision records (ADR):
At the end of October cketti attended Make Better Email, a meeting to bring together email client authors, email server authors, and email service providers to talk about how to improve the ecosystem through the better use of open standards.
Some of the topics we talked about:
Modern Authentication – What mechanism do we need to be able to support interoperable modern authentication for clients of open protocols, such as IMAP, JMAP, SMTP, CalDAV, and CardDAV.
Scheduled Send – How can this feature be implemented in an interoperable way.
IMAP Extensions – We came up with a recommendation for which existing standards should be targeted by living IMAP code.
In the upcoming months this should result in new RFC drafts and a best current practice (BCP) document at the IETF.
Community contributions
fishchimp made a change that hopefully makes it harder to accidentally trigger swipe actions in the message list (see #7280).
Marcel M. fixed a bug that lead to notifications always being silent on Android Wear devices (see #7284).
Many thanks to both of you!
Releases
In October 2023 we published the following stable releases:
Please join us for our upcoming Thunderbird community office hours all about Cards View on November 29, 2023 at 18:00 UTC! (Keep reading for joining information.)
A New Era
We are trying out a new format in this edition of the office hours. Previously, we had several sessions with no agenda. This was intended to provide an open forum for anyone to come and ask any questions, during a few different times. Since we have seen low engagement, we’re trying something new.
The new office hours format will feature a key Thunderbird guest to talk about the area they specialize in. Going forward, we’ll hold just one monthly session at a time we hope is convenient for the majority of our users around the globe. The session will be recorded and shared with the community afterwards. On air, we will do our best to answer any questions you have. Everyone is welcome to attend and ask questions live. If you cannot make it in person, we encourage you to submit any questions you have in advance by sending them to
With the substantial UI changes that the new cards view in Thunderbird 115 “Supernova” offers, we thought we would bring an expert on cards view, Micah Ilbery to discuss the improvements with you. Micah has been a designer on the Thunderbird team and played a major role in the beautiful cards view we have today. So if you have questions or helpful feedback you would like to share face-to-face with Micah, this is your opportunity! If you can’t make it, please submit any questions you have to the above email address and we will ask them on air for you.
(You can see where Cards View is headed next by looking here.)
<figcaption class="wp-element-caption">Ryan Sipes, Product and Business Development Manager</figcaption>
Hi, my name is Ryan Sipes and I run product and business development at MZLA Technologies Corporation, the subsidiary of the Mozilla Foundation that develops Thunderbird. I have been working on Thunderbird for my day job since November of 2017. It doesn’t seem like that long ago, but looking at the calendar I see that it has been six years this month. A lot has happened in that time and Thunderbird is in a much different place than it was when I started. I’ve seen multiple people online share accounts of “the Thunderbird story,” and each time I’ve thought “that’s really great, but they missed some important parts.” It’s not their fault, we’ve simply never shared the whole story. So today, I thought I’d sit down and write that.
To tell the story correctly, we must go back to 2012. That’s when Thunderbird began to transition from a project that was funded and developed by the Mozilla Corporation, to a community run project. The reasons behind that move were sound and made sense given the state of the project at the time.
One of the biggest issues for Thunderbird throughout its life is that, while it was a well-loved product with over 20 million users, it never had any substantial revenue that could adequately cover its development. So, whomever managed the incredibly large project had to simply eat the cost of developing, maintaining, fixing, and distributing the software. A few attempts were made to make the project sustainable (Mozilla Messaging, for example) but they ultimately didn’t work.
Surviving With A Skeleton Crew
By the time I joined in 2017, Thunderbird lived in the nonprofit Mozilla Foundation and was governed by the Thunderbird Council, an elected group of contributors to the project. At some point, a Thunderbird donation form was set up and donations were high enough to hire 3 people. A developer was hired, someone else to keep the infrastructure running, and a Community Manager (me). The team was way too small for the task at hand (by contrast, we now have 29 people working on Thunderbird). Shortly after being hired, I joined the Thunderbird Council and acted as its Treasurer for many years.
To say that this period was challenging would be an understatement. Thunderbird was being maintained by a skeleton crew. There were long stretches where we couldn’t even get the software to build. On top of that, there was no real organization or guiding vision for where we were trying to go. In fact, there were competing ideas within the community, and a consensus looked difficult to reach.
Throughout the first couple of years that I was around, I was acting as the Treasurer for the project and kept thinking, “we’ve got 20,000,000 users who rely on Thunderbird every day – is an application with that many users really going to just die? We really need to tell our users that we need their support.”
The project simply wasn’t on a sustainable path, and a few of us suspected it was only a matter of a few years until Thunderbird would be unmaintainable.
Asking For Help
So we defined a vision, made a roadmap, and focused our work. It was still kind of chaotic, but it was more organized chaos. At the same time, I worked with the team on how best to convey the message that we needed the support of our users. It started with updates to the familiar Start Page that shows inside Thunderbird. Then we tried letting folks know when they downloaded the software that “Thunderbird is supported by donations from our users. Please consider giving to keep Thunderbird alive” (not the exact wording, but that was the spirit of the message).
Eventually, we showed an in-app donation appeal to all our users explaining that we truly need their support if Thunderbird is going to stay around. Each time we tried to be honest about our need, and tasteful about how and when we asked.
And… It actually worked! Our users donated to help Thunderbird and we continued to get more organized. Eventually our team grew large enough that it made sense to move to our own organization, the aforementioned MZLA Technologies Corporation. The team working on Thunderbird got way more organized, and we put out many of the fires that made developing Thunderbird so hard.
I could probably write a book on this part, but trust me: it was really, really tough to solve all these problems. Fortunately, it has all paid off in allowing us to do more in a faster and more efficient way.
Thunderbird Adopts A Puppy
Then we were able to do bigger things. As noted in another blog post, I’d been talking to cketti who ran the K-9 Mail open source email client project. K-9 Mail was exactly what I thought Thunderbird on Android would be if it existed. Talking to Cketti for a couple of years showed me that the K-9 Mail project also faced a sustainability challenge.
After many Thunderbird Council discussions, we concluded that K-9 Mail and Thunderbird shared common values and that we could better serve our users by adopting the project. K-9 Mail is amazing software, with a long legacy like Thunderbird and I’m excited that soon we will ship Thunderbird for Android based on the awesome foundation the K-9 Mail team has built and continues to improve.
At the same time, there was a dreaded task to take on. Parts of the Thunderbird codebase are 20 years old. In software time, that’s ancient, like the pyramids are ancient. That old code made fixing long standing bugs or adding a feature extraordinarily difficult. Often a small change would break something random in a totally different part of the application and it would take a lot of time to figure out why. We had to fix this in the areas that were most affecting our work.
Enter the “Supernova” project.
But WHY The Name “Supernova?”
I named the 115 release Thunderbird “Supernova” after a cosmic explosion because I knew we were going to have to blow some things up and rebuild them. But I also knew the result was going to be beautiful and more powerful for our users. (Quick aside: a supernova often results in a beautiful nebula – search “Crab Nebula” to see what I mean). We rebuilt some really core parts of Thunderbird (a ton of the front-end, the message list, and message display areas to name a few). And in the process, we began a journey of future-proofing Thunderbird so that it can live for another 20 years.
<figcaption class="wp-element-caption">This view of the Crab Nebula in visible light comes from the Hubble Space Telescope and spans 12 light-years. The supernova remnant, located 6,500 light-years away in the constellation Taurus, is among the best-studied objects in the sky.Credits: NASA/ESA/ASU/J. Hester</figcaption>
Now, we look forward to the future, but email as a standard is under threat. It is becoming less of an open ecosystem and one of large fiefdoms with high walls.
The biggest providers are making features and tools that only work within their walled garden. They espouse “privacy” while maintaining access to all your messages in their privacy policies. They make it increasingly harder to use clients like Thunderbird because they want you to live in their apps in order to monetize you and your data.
Next: Thunderbird for Android and iOS
But Thunderbird has an important story to tell. A story of decentralized, privacy-respecting email based on open standards. We’ve always created open source software that anyone can use, contribute to, and extend to their specific needs.
By the time Q1 2024 rolls around, we’ll have opened the aperture a bit and given folks a new choice on Android. Next year we’re also going to start working on Thunderbird for iOS. In addition, we’re going to develop the tools that give people choices other than the big three. Thunderbird has come a long way these past few years, but we’re not done yet – come and join us as we get ready to do so much more!
If you have recently upgraded to Thunderbird 115 “Supernova” and noticed a smaller font size or a presentation that feels too compact, there might be an easy solution. You should be able to simply change the font from the app menu to achieve your ideal font setting by clicking the +/- to change the font size globally (see below).
As the GIF above illustrates, here’s how to do it:
Change the density to suit your taste: Click ≡ > Density. Then click “Relaxed” to increase the size of UI elements in Thunderbird or click “Compact” to decrease the size.
Change the font size to suit your needs: Click ≡ > Font Size. Click + to increase the size of the fonts or click – to decrease the size of the fonts.
If for some reason that does not fix your font resizing problem, then you may have hit a known issue about the font size going back to a smaller size due to some unsupported properties being adjusted either directly or via an add-on.
You can find more detailed info on fixing these technical issues in our support article.
As part of our continuing efforts to strengthen the security of Thunderbird, a change was made in version 115.0 that rejects the use of the SHA-1 algorithm in digital signatures of S/MIME emails.
The SHA-1 algorithm is nowadays considered insecure in most contexts, which includes digital signatures, as explained in the related Wikipedia article.
Because of the change in Thunderbird 115, when an affected message is displayed, an invalid signature will be reported.
You can spot such messages by looking at the message source, and search for the text micalg= in the message headers. If it is followed by the text sha-1 or sha1, you should contact your correspondent and ask them to upgrade.
Most modern email software that supports S/MIME should already be able to use another hash algorithm, for example SHA-256 is a more modern alternative. It might be necessary to change a setting to enable its use.
The Thunderbird team was recently made aware that the use of SHA-1 is still required in some environments, as some government agencies continue to send out messages based on SHA-1. Recipients of such messages asked for a way to confirm the validity of such signatures, despite the risk that the signature could have been forged.
To accommodate those Thunderbird users, starting with version 115.4.1 a new configuration mechanism will be made available. It can be used to accept S/MIME signatures based on SHA-1. To enable it, use Thunderbird’s settings, access the advanced config editor, search for the setting mail.smime.accept_insecure_sha1_message_signatures and set it to the value true.
Note that changing this setting is not recommended, and if you decide to set it, you should work with your correspondents to get them to change to SHA-256 or newer as soon as possible. Once your correspondents have upgraded, you should revert the setting to false.
Changing the setting will have no effect on the messages that Thunderbird sends. Thunderbird uses SHA-256 when sending digitally signed S/MIME email messages, and has been doing so for several years already.
The Thunderbird team understands that it might seem early to demand the deprecation of insecure algorithms while other software is still using it, given the incompatibilities that some users experience. However, aligned with our mission to increase the security of users, we hope that our actions can raise awareness and motivate deployments to upgrade to more secure settings, which otherwise they might not have done.
Welcome back to your monthly K-9 Mail update! The previous month ended with Mozilla’s All-Hands event in Montreal, Canada. While I used this opportunity for a three weeks long vacation in Canada (it was great, see picture below), Wolf went back to work on your favorite mobile email client as it transforms into Thunderbird for Android.
<figcaption class="wp-element-caption">Algonquin Provincial Park, Ontario, Canada | Photo credit: cketti</figcaption>
Improved account setup
Wolf continued to work on the new and improved account setup code. This mostly involved fixing bugs and improving the internal architecture, so the code will be easier to maintain in the future.
With the switch to the new account setup code, we were able to remove (some of) the old setup code. If you’re software developer, you know that being able to delete a significant amount of old code is one of the best feelings on the job. If you’re not, just take my word for it.
Wolf also started work on using the new server settings screens when editing the incoming and outgoing server of an existing account. Once that work is complete we’ll be able to delete even more old code.
Unfortunately, none of this work resulted in new screens that we could show off in this progress report. But maybe the following stats can give an idea of how busy Wolf was.
App maintenance
These are some of the more notable bugs we fixed in September.
Vector image as app icon
Some users reported that the splash screen newer Android versions automatically display, shows a blurry app icon. The reason was that we used a bitmap that looked fine when used as regular-sized icon, but that looked blurry when being scaled up, e.g. for the splash screen.
We fixed this by converting the icon into the vector image format supported by Android. To be able to do that we had to remove some details from the icon. But the result is a sharp app icon on the splash screen.
In our new setup code we accidentally broke OAuth 2.0 support for Yahoo and AOL accounts. Apparently some people still use those email providers. So we fixed the bug.
Cleaned up “Return to list after delete” setting
K-9 Mail allows the user to specify what is displayed next after a message has been deleted from the message view screen. Available options are:
return to message list
show next message
show previous message
However, those are not the options a user could select in app settings. There were two preferences: Return to list after delete and Show next message after delete.
During one of our design meetings we quickly decided this is not a great user experience and changed it to one setting with three options.
Since the same behavior is also used after moving a message, we also used this opportunity to change the name of the setting.
Community contributions
In September we merged the following pull requests by external contributors:
We didn’t release any beta or stable versions in September. However, that’s an exception. Usually we publish a couple of beta releases a month. If you want to help shape future versions of the app, become a beta tester and provide feedback on new features while they are still in development.
Recently, the Thunderbird team became aware of some ransomware masquerading as Thunderbird. There are several ransomware groups that run ads in an attempt to trick people into downloading and installing fake software, which pretends to be other software, e.g. Thunderbird. Remember that the Thunderbird project doesn’t require payments for downloading the Thunderbird software (although you may see a donation request when downloading from thunderbird.net), so if you are being demanded for payment for a properly working Thunderbird, something is surely wrong with the package you are downloading.
There are ongoing Mozilla efforts to take down these sites but since they are hosted in Russia, takedowns are difficult and often not effective. What you can do in the meantime is to make sure you are getting Thunderbird from a trusted location like thunderbird.net directly, or your Linux distribution’s software store.
For your convenience, we have also included a list of trusted sources below:
Trusted Thunderbird Download Sources
There are a handful of trusted places you can get Thunderbird from, aside from our thunderbird.net website. Additionally, you may choose to download Thunderbird from sources related to your distro.
Windows
There are two ways to download Thunderbird onto your Windows system:
Click Download on thunderbird.net to download the latest release for your platform.
When downloading from the thunderbird.net website, it automatically allows you to download the correct Thunderbird package for your language and your operating system. Advanced users can find all available localized versions for the current release at https://www.thunderbird.net/thunderbird/all/.
The Thunderbird team is a remote-first, globally distributed group, so it made perfect sense to devote an episode to Remote Work! Join Heather, Chris, and Jason for some useful tips and tricks to make your daily remote work more enjoyable and more productive. We also include tips from ThunderCast listeners Pedro and Mike, who emailed us at podcast@thunderbird.net. (You can do the same if something’s on your mind.)
Plus: An inside look at the upcoming Thunderbird Send service, some fascinating origin stories, and geeky Raspberry Pi solutions for weather and BBQ.
<figcaption class="wp-element-caption">Listen to this episode on PeerTube</figcaption>
August was a relatively calm month for the K-9 Mail team, with many taking well-deserved summer vacations and attending our first Mozilla All-Hands event. Despite the quieter pace, we managed to hit a significant milestone on our journey to Thunderbird for Android: the beta release of our new account setup interface.
Beta Release with New Account Setup: We Want Your Feedback!
We’re thrilled to announce that we rolled out a beta version featuring the new account setup UI. This has been a long-awaited feature, and even though the team was partially on vacation, we managed to get it out for user testing. The initial feedback has been encouraging, and we’re eager to hear your thoughts.
<figcaption class="wp-element-caption">Add your email address</figcaption><figcaption class="wp-element-caption">Configuration found through autoconfig</figcaption><figcaption class="wp-element-caption">Authenticate with your OAuth provider</figcaption><figcaption class="wp-element-caption">Addjust your account options</figcaption>
If you’ve tried the beta, we’d love to get your feedback. What did you like? What could be improved? Your insights will help us refine the feature for its official release.
How to Provide Feedback
You can provide feedback through the following channels:
The Thunderbird team is back from Mozilla’s All-Hands event, and we’re overwhelmed in the most positive way. In addition to the happy and positive vibes we’re feeling from meeting our colleagues in person for the first time, we have a lot of thoughts and impressions to share. Ryan, Jason, and Alex talk about how Mozilla is building AI tools for the good of humanity, and how our perception of AI has changed dramatically. Plus, the problem with the “hey Mozilla, just build a browser” argument.
Today’s episode is light on actual Thunderbird talk, and more focused on Mozilla as an organization and the future of AI. We hope you enjoy it as much as we enjoyed discussing it!
Hello Thunderbird community! We’re bringing back monthly Office Hours, now with a Saturday option to make attendance more convenient. Please see the details below to learn how and when you can meet with us to share your feedback and ask questions.
Now that Thunderbird 115 Supernova has been released, we have a lot to discuss, plan, and do! And we’re rolling out monthly Office Hours sessions so that you can:
Share your Thunderbird experiences with us
Share your ideas for the future of Thunderbird
Discuss ways to get involved with Thunderbird
Ask us questions about Thunderbird and the Thunderbird Project
Meet new team members, and meet your fellow Thunderbird users
This month we’re hosting these sessions using Zoom (in the future we plan to stand up a dedicated Jitsi instance). You can easily participate using video, dialing in by phone, or asking questions in our community Matrix channel at #thunderbird:mozilla.org.
In the table above please click a session for meeting time converted to your local time. If you encounter difficulty joining please post in Matrix back channel #thunderbird:mozilla.org or email us.
Hosts Wayne (Thunderbird Community Manager) and Jason (Marketing & Communications Manager), plus special guests from the Thunderbird team look forward to meeting you!
PLEASE NOTE: We’ll be recording this call for internal use and distribution only. In the future, we may explore publishing these on video platforms.
How To Join Community Office Hours By Phone
Meeting ID: 981 2417 3850 Password: 647025
Dial by your location +1 646 518 9805 US (New York) +1 669 219 2599 US (San Jose) +1 647 558 0588 Canada +33 1 7095 0103 France +49 69 7104 9922 Germany +44 330 088 5830 United Kingdom Find your local number: https://mozilla.zoom.us/u/adPpRGsVms
The day I write this, it’s very hot outside. Too hot to think of a good introduction to this blog post that also includes a link to the previous month’s progress report… Well, I guess this will have to do. I’m off to get some ice cream
Please enjoy this brief report of our development activities in July 2023.
Improved account setup
Since Wolf joined in February of this year, he has spent a considerable amount of time on many of the individual pieces that make up the new and improved account setup user interface. July was the month when things started coming together. For the first time we were able to test the whole flow and not just individual parts.
Things were looking good. But a few small issues kept us busy and prevented us from releasing a beta version containing the new account setup.
Material 3 experiments
We’ve done some experiments to get a better idea of how much work it will be to switch the app to Material 3, the latest version of Google’s open-source design system. We’re now cautiously optimistic. And so the current plan is switch to Material 3 before renaming the app from K-9 Mail to Thunderbird.
Community contributions
In July we merged the following pull requests by external contributors:
After a few busy days surrounding the Thunderbird Supernova release, we finally managed to publish the report of the security audit organized by OSTIF and performed by 7ASecurity. We’re happy to report that no high-risk vulnerabilities were found. The security audit did uncover a handful of low-to-medium risk vulnerabilities.
Thunderbird 115 “Supernova” ships with brand new layout options to give you a more beautiful and more productive email experience. But those new options aren’t on by default (for now), out of respect to those who have grown comfortable with Thunderbird’s Classic View throughout the years. Fortunately, getting that shiny new “Supernova” look is accomplished in just a few seconds. In this short guide, we’ll show you how to do it!
Step 1: Turn On Vertical View
First, click on the App Menu (≡) and choose “View”, followed by “Layout,” and then select “Vertical View.” This will rearrange the Folder pane, Message List pane, and Message pane to be displayed side-by-side.
Step 2: Turn On Cards View
Next, let’s switch on “Cards” view. This new way of displaying your message list is simpler and more compact, to help reduce cognitive burden when you view the list. And it’s easy to activate.
Look immediately to the right of the “Quick Filter” area for the “Message List Display Options” icon. Click it, and select “Cards View” as seen in the GIF above.
Cards View is still in active development. More features such as a message preview line and sender avatars will be added in the future.
TIP #1: Are you having trouble finding that menu? The area of Thunderbird where you activate Cards View is called the Message List Header. Some people choose to hide this section to reclaim a bit of vertical space. It’s easy to get it back: Just return to the App menu (≡), then select View Layout, and make sure that “Message List Header” is checked ✓
TIP #2: Cards View is also available when using Classic or Wide layouts.
Step 3: Turn On “Tags” Folder Mode
If Tags are an important part of your workflow, now it’s easier than ever to access them.
In the Folder Pane Options menu (⋯) next to the “New Message” button, click “Folder Modes” and then choose “Tags.” That’s it! But if you want to continue customizing Thunderbird 115, you can also use this menu to hide Local Folders.
Tip: Want to move your Tags up or down in the Folder Pane? Click on the 3 vertical dots menu (⋮) next to Tags, and simply choose “Move Up” or “Move Down” as seen in the above GIF.
Step 4: Customize The Message Header
And finally, we arrive at the Message Header Settings. That’s the section at the top of your email showing all the information such as sender’s name, contact photo (which pulls from the Address Book), subject, any associated tags, and more. Configuring this to fit your preferences is easy. Just click the “More” button with the downward facing arrow, then select “Customize” and make it yours!
We hope this helps you enjoy an even better Thunderbird experience. Thanks for being part of the Thunderbird family, and make sure to check back later for more customization guides and usage tips.
The roadmap item we’re currently working on is Improve Account Setup. Most of our time went into working on this. However, for June there’s no exciting news to share. We mostly worked on the internal plumbing; that is important to get right, but not necessarily great material for a blog post. Hopefully there will be new screenshots to share in July’s progress report.
App maintenance
Having an app with a large user base means we can’t spend all of our time working on new features. Fixing bugs is a large and important part of the job. Here’s a writeup of just three of the bugs we fixed in June.
Folder appears to be empty
A user reported that some of their folders appear to be empty in K-9 Mail. Using the provided debug log () we were able to track this down to a message containing an invalid email address, specifically one whose local part (the text before the @ symbol) exceeds the limit of 64 characters.
The error was thrown by a newly added email address parser that is stricter than what we used before. At first it was a bit surprising that this would lead to messages in a folder not being shown. We deliberately kept this new implementation out of the code responsible for parsing emails after download and the code for displaying messages.
However, it turned out the new email address parser was used when getting the contact name belonging to an email address. This lookup is performed when loading the message list of a folder from the local database. When an error occurs during this step, an empty message list is shown to the user.
To fix this bug and limit the impact of similar problems in the future, we made the following changes:
Ignore most errors when parsing email addresses from messages that the user has received. The world is full of email addresses that violate the specification but work mostly fine in practice. However, we still want to be strict when it comes to the email addresses we accept, e.g. when setting up a new account.
Ignore errors with the email address when trying to fetch the system contact belonging to that email address. This may lead to the app not being able to fetch a contact name for an spec-violating email address. But this will no longer lead to failing to load the entire message list.
We added a message with an email address whose local part exceeds the length limit to our test account. That way we are likely to catch bugs related to such email addresses before they make it into a beta release.
We’re very grateful to our beta testers for finding and reporting bugs like this one. That way we can fix them before they make it into a stable release.
Adding an email address to an existing contact
With the introduction of the message details screen we added a button to add a message sender or recipient to the contacts database using an installed contact app. If the email address can already be found in the contacts database, this button is hidden and tapping the participant name, email address, or contact picture opens the contacts app.
Previously the app didn’t make that distinction and tapping an email address or participant name would open the contacts app using the “show or create” action. Apparently this reliably allowed to add an email address to an existing contact. However, the “insert” action used by the details screen only allows adding the email address to an existing contact with some contacts apps, but not others
<figcaption class="wp-element-caption">“Insert” action with contacts app that supports adding to an existing contact</figcaption><figcaption class="wp-element-caption">“Insert” action with contacts app that doesn’t support adding to an existing contact</figcaption>
We changed the action to start the contacts app from “insert” to “insert or edit”, and this seems to reliably offer the option to add the email address to an existing contact.
<figcaption class="wp-element-caption">“Insert or edit” action with Google’s Contacts app</figcaption><figcaption class="wp-element-caption">“Insert or edit” action with AOSP Contacts app</figcaption>
Reply behavior depends on message size
A user reported that the behavior when replying to a message retrieved via a mailing list was different depending on whether the message had been downloaded completely or only partially.
K-9 Mail supports only downloading parts of a message when the email exceeds a configured size. In that case also only selected parts of the message header are downloaded. Unfortunately, we forgot to include the List-Post header field that is used to determine the email address to which to reply.
The fix was simply adding List-Post to the list of header fields to fetch from the IMAP server.
Community contributions
In June we merged the following pull requests by external contributors:
Hello Thunderbird family! First and foremost, we want to express our deepest appreciation for your patience. The road to Thunderbird 115 “Supernova” has been a long one, and we’re confident you’ll love the final result. If you’re already using Thunderbird 115, you may have noticed a feature that is conspicuously absent: Thunderbird Sync.
When we started creating our roadmap for Supernova, our feature targets were ambitious. As it turns out, a little too ambitious. We did our very best to finish up Thunderbird Sync for the initial release of version 115, but some technical blockers prevented us from moving forward fast enough to deliver it to you. Besides, this is a feature that absolutely must be secure and reliable. And it needs months of user testing to ensure that stability.
We do have the basic user interface for Thunderbird Sync designed already. However, what slowed us down is the need to hire a Site Reliability Engineer (SRE) to help us spin up our own back-end infrastructure that is independent of Firefox Sync. While our Sync functionality does use Firefox Sync code, they will end up being completely different products with different use-cases.
When Will Thunderbird Sync Be Finished?
We don’t have a solid release date, but our objective is to have Thunderbird Sync finished in time for the next ESR release, or shortly after we switch to a monthly release schedule.
Once we have a server and a proper back-end infrastructure, we’ll enable it on beta for you all to test.
What Data Will Thunderbird Sync Support?
We plan to support syncing of your email account definitions, credentials, signatures, saved searches, tags, tasks, filters, and most major preferences across multiple installations of Thunderbird on PC, (Yes, this is cross-compatible with Windows, macOS, and Linux.) You’ll also be able to sync your Thunderbird accounts with the forthcoming Thunderbird for Android.
Thank you again for being patient with us as we continue to build the best possible software for managing your email and personal communications. In the future, we’re hopeful that a switch to monthly releases will allow us to put new features in your hands faster than what’s previously been possible.
Hello Thunderbird family, and welcome back to a long-overdue episode of the ThunderCast! Ryan, Alex, and Jason get together to talk about the new features and improvements in Thunderbird 115 “Supernova.” But they also share WHY those features were developed, and what’s being worked on right now.
Plus, Ryan shares some breaking news about the future of the Thunderbird Project! It’s a casual, informative, behind-the-scenes chat.
Our journey to transform K-9 Mail to Thunderbird for Android involves more than just improving the user interface and adding new features. K-9 Mail is already an important part of the open source ecosystem on Android, and because it lays the foundation for the future of Thunderbird on Android, we believe it’s important to invest in the security health of the software.
To that end, we recently enjoyed a collaboration with the Open Source Technology Improvement Fund (OSTIF) and 7ASecurity on an extensive security audit of K-9 Mail.
A team of six auditors at 7ASecurity worked diligently to identify and address any potential security or stability issues found in K-9 Mail. The audit focused specifically on threat modeling, fuzzing (a technique that simulates real-world scenarios where software might encounter unexpected or malicious inputs), and our software supply chain.
We are happy to report that zero high-risk vulnerabilities were found. The security audit did uncover a handful of low-to-medium risk vulnerabilities, the majority of which the K-9 Mail team has already resolved or is in the process of addressing.
Additionally, we’re very pleased to share this promising conclusion from OSTIF:
“[Mozilla] has an incredible foundation to begin this new chapter with, as the report notes seven wide-ranging points of secure and healthy practices and conditions of K-9 Mail that the 7ASecurity team evidenced during the engagement.”
Amir Montazery, OSTIF
The entire process was educational and productive, and we sincerely appreciated working with such professional and knowledgeable teams. We’d like to extend our deepest thanks to everyone at OSTIF, including Amir Montazery, Ashley Leszkiewicz, and Derek Zimmer, who were instrumental in orchestrating a smooth experience.
And our sincerest gratitude goes out to Abraham Aranguren, Dariusz Jastrzębski, Daniel Ortiz, Dr. Miroslav Štampar, Óscar Martínez, and Patrick Ventuzelo at 7ASecurity for their hard work and attention to detail.
“OSTIF has a strong understanding of how open source projects operate, and we really appreciated that they were able to jump in and help us coordinate this security audit of our K-9 Mail software. OSTIF and 7ASecurity were amazing partners that provided a helpful guiding hand, and made the process of doing the audit a breeze. We really appreciated their professionalism and expertise. I can confidently say that we plan on working with them again.”
Ryan Sipes, Thunderbird Product and Business Development Manager.
Thunderbird 115 “Supernova” has been released! During the next several weeks, you should begin to see your installation of Thunderbird 102 automatically update to Thunderbird 115. Whether you’re upgrading or coming in as a brand new user, now is the perfect time to introduce you to the various elements of the software. This guide will introduce you to the main window of Thunderbird (in its Classic layout and new “Vertical Cards” layout), get you familiar with the terminology, and help you navigate Thunderbird like a pro!
(If you enjoy keyboard navigation, we encourage you to learn the many keyboard shortcuts available in the main window to help you quickly perform functions, such as message actions and navigation, searching, and calendar actions.)
So, let’s take a tour around the various sections of your main Thunderbird window.
Classic View Layout with “Table” View
The following screenshot shows the Classic View Layout with Table View. This is still the default layout for new Thunderbird installs, and it’s what you’re probably used to if you’ve used Thunderbird 102 or older versions.
Here are the 11 main components of the user interface when using the main window:
1. Spaces Toolbar
Since version 102, Thunderbird has had a central Spaces Toolbar for fast and easy access to your most important activities. Click on the buttons to move between the various “Spaces”: Mail, Address Book, Calendar, Tasks, Chat, and your add-ons. At the bottom of the Spaces Toolbar, there is a button for hiding the Spaces Toolbar. Above this is a button to change the application Settings.
2. Unified Toolbar
There is a new “Unified Toolbar” on top of the Tab Bar, which is fully screenreader and keyboard accessible. This toolbar is used for all “Spaces” of the Spaces Toolbar, which are Mail, Address Book, Calendar, Tasks, Chat, and Settings.
3. Tab Bar
In Thunderbird 115, the Tab Bar is only displayed if you have two or more tabs. For example, if you have a Mail tab and a Calendar Tab, the Tab Bar is visible. If you just have one tab like the Mail tab, the Tab Bar automatically disappears.
4. Message List Header
The Message List Header is new in Thunderbird 115. It shows the current folder name, number of messages, “Quick Filter”, and “Message list display options” icon .
5. Folder Pane
The Folder Pane is where your accounts and folders will appear. It will show the account name(s) specified in the account settings window, with the default account first, and any additional accounts listed in the order you added them. (Want to rearrange them? Read this!)
Within each account, special folders (Inbox, Sent, Trash) will be listed first with unique icons, while other folders will appear in alphabetical order.
Sub-folders will appear underneath the parent folder. To expand or collapse an account or parent folder, click on the arrow beside the account or parent folder, or double click on the name.
The Quick Filter Bar helps you sort your message list based on specific criteria. See Quick Filter Bar for more details and help.
7. Message List Pane
The Message List Pane contains the list of messages in the selected folder, including details like the date/time, subject, sender, and more. To pick which details are displayed, and how wide each column is, see this Message List Columns article.
8. Message Header Pane
The Message Header Pane will show you information about the selected message, such as the From, To, and Subject headers, as well as the date/time of the message.
Tip: Clicking “More” at the top of the Message Header Pane, then “Customize” will let you change this area’s appearance and options.
9. Message Pane
The Message Pane is where you see the content of the selected message. To show/hide the Message Pane, press F8.
10. Today Pane
The Today Pane will display information from your calendar, such as any events scheduled for today, tomorrow, and upcoming events within the next 5 days. Click on the arrows at the top of the Today Pane to switch to the Events and Tasks view, which will include your calendar tasks as well as events, or the Tasks view which will only include your calendar tasks, not events. To hide the Today Pane, click on the X beside the arrows. To bring it back, click on Today Pane in the Status Bar. You can also show/hide the Today Pane, by pressing F11.
11. Status Bar
Here’s a breakdown of the Status Bar from left to right:
A button to switch back and forth from online to offline and vice versa.
An area to display status messages about operations like downloading email, compacting folders etc.
A button to switch back and forth between displaying the “Today Pane” and not displaying it.
New Vertical View Layout with “Cards” View
In this example we’ll show you the main Thunderbird window with “the Supernova look.” That’s when the Vertical layout is activated in Thunderbird 115, along with the new “Cards” view.
Card view (one of the cards is highlighted in red, see #11), is a multi-line, non-column alternative view of the message list. Currently there is a two line “card” for each message in the Message List Pane (our plan is to eventually expand this to 3 or 4 customizable lines). The first line has the sender’s display name and time, and the second line displays the subject.
On behalf of the Thunderbird team, Thunderbird Council, our global community of contributors, and our extended Mozilla family, I am incredibly excited to announce the initial launch of Thunderbird 115 “Supernova” for Linux, macOS, and Windows! With this year’s version, we’re delivering much more than just another yearly release. Supernova represents a modernized overhaul of the software – both visually and technically – while retaining the familiarity and flexibility you expect from Thunderbird.
“Supernova is the beginning of a new era for Thunderbird,” says Ryan Sipes, Thunderbird’s Product and Business Development Manager. “It lays a beautiful, solid groundwork for future releases that will make Thunderbird a significantly better email client – and we won’t stop until it’s the best option out there.”
“During the past year we reworked some of the oldest parts of our code base with the goal of modernizing them to make maintenance and extensibility easier,” says Alex Castellani, Thunderbird’s Director of Product Engineering. “We did our best to strike a comfortable balance between new features and interface changes to appeal to new users, while maintaining the current layout and interface that millions of our users know and love. This is the first step towards a more flexible, reliable, and customizable Thunderbird that will accommodate different needs, preferences, and workflows.”
Here’s a brief look at some of the best new features and improvements in Thunderbird 115. For the complete list, please visit this support article.
Modernized Cards View
<figcaption class="wp-element-caption">(Mockup, subject to change)</figcaption>
Supernova’s new Card View is our version of the well-known vertical layout for the Message List, perfect for people used to modern webmail. Card View emulates a mobile interface list with multi-line support, offering a more comfortable appearance to reduce cognitive burden. (And because we want Supernova to feel familiar for veteran users, the legacy “Table” view is still available.)
Brand New Thunderbird Logo
Supernova ships with a brand new Thunderbird logo honoring our history and vital connection to Mozilla, while representing the revitalization of our brand and a promising, sustainable future.
Supernova features a single dynamic toolbar, presenting frequently used and contextual options based on the tab or Space that is currently active. Take full control by customizing the toolbar and window layout to perfectly fit your workflow.
Intuitive AppMenu
The new Application Menu (≡) in Supernova is now keyboard-accessible, and redesigned for faster, easier navigation. Fewer submenus, persistent global options, and consistent iconography combine to create a much more intuitive experience.
Improved Calendar Design
<figcaption class="wp-element-caption">(Mockup, subject to change)</figcaption>
As part of an ongoing effort to modernize and upgrade Thunderbird’s Calendar, Supernova introduces an improved “mini-month” layout, improvements to the day/week/month grid, a pleasing color palette, and several more minor changes.
Elegant Density Control
<figcaption class="wp-element-caption">Change your density setting and font size easily from the new AppMenu</figcaption>
Working with multiple monitors and display resolutions? Supernova lets you dial in the perfect density settings and font sizes for the entire application, with just a single click from the AppMenu.
New + Sortable Folder Modes
Supernova gives you more control by introducing sortable Folder Modes. Display all of your Tags in the Folder Pane, turn on and off Local Folders, or move your favorite Folder Mode sections up and down with one click. Less scrolling, more productivity.
Eye-Catching Tags View
Tags are a familiar and powerful part of your workflow, so Supernova adds an eye-catching “Tags” view to the Folder Pane. Quickly jump to your custom, color-coded tag categories to find and filter your messages even faster.
Improved Address Book
Supernova continues to iterate on the modernized Address Book introduced in Thunderbird 102. You’ll enjoy a new tabular view, an improved Edit view, delete buttons, and better accessibility.
Better Accessibility
Speaking of accessibility, Supernova substantially improves Thunderbird’s keyboard navigation and screen reader accessibility across the entire application. We’ve also greatly expanded the ability to navigate Mail content and buttons using the TAB and arrow keys.
More To Come…
Supernova is constantly evolving. Throughout the next year, we’ll deliver many improvements to existing Supernova features and introduce brand new ones, including Thunderbird Sync. Upgrade to version 115 and experience the future of Thunderbird!
How To Get Thunderbird 115 “Supernova”
As with any major release, we sometimes become aware of corner cases after significant public testing. The Thunderbird team will wait to enable automatic updates until we’re confident no such issues exist. We’ll keep you posted right here, and on social media platforms. (Existing Thunderbird 102 users will continue to see security updates during this time).
On Linux, the timing of this release and subsequent automatic updates may be determined by your distribution’s maintainers. If it’s convenient for you, we recommend installing the Thunderbird Flatpak from Flathub, which is now packaged and maintained by the Thunderbird team. (Please note that 115 is not live on Flathub at time of publication, but as always we’ll keep you updated.)
If you don’t want to wait for an automatic update, Thunderbird 115 is also available to download directly at thunderbird.net for Windows, Linux, and macOS.
Want to visualize the differences between 102 and 115 at a glance? Check out this awesome comparison slider.)
In the last few months, I have been building up a new business called Trade Post 47. While we envision it as a little space station in orbit of a nice planet, to most people it will be a science fiction merchandise trading company, with an online shop and booths at events like local Comic-Cons in Central Europe, especially in Austria. If you want to learn more, we have put up a complete page about us on our shop website.
To manage our products, which we get from different vendors (sometimes the same product via different vendors) as well as plan and manage our orders, I built an internal, custom merchandise management in my own PHP framework or CMS CBSM (which is also used for this blog, for example). I did this mostly out of convenience as I have and maintain this system anyhow and I needed some database tables with fitting UI for managing our merchandise, vendors, and more (even conventions we may want to run booths at).
OTOH, the public shop is (as you may notice when looking at the website) an installation of Magento 2 (i.e. the open-source version of what is nowadays called "Adobe Commerce"). We decided to run that system because we are partnering closely with MCO Shop, which is a local ham radio and electronics shop, and they already had this software running previously on the servers we share and know how to work with it, run the upgrades, and maintain it. After all, when building a new business, as in so many areas of life, it always helps if you can share some resources and knowledge with others. First, I adapted the Magento theme to make it look more "space-like", mostly importantly, having a dark instead of light background. Once that worked well enough, I still had to get those products that we actually ordered from my custom management system into this Magento shop. Initially, I did this via creating a big CSV file and importing that into the shop, but it was clear that we needed a more fine-grained solution in the long run that can add and update entries individually.
Additionally, when we run booths on our "away missions" to events/conventions (or whenever we otherwise sell anything in person), Austrian law requires us to use a cash register system that follows strict rules and passes a certification so that the government can be sure we pay taxes for everything we sell. For that we decided to use a solution integrated into our bookkeeping system, which runs online as a web service as well, a specialized Austrian solution called FreeFinance. And of course, the cash register needs a full list of products and prices as well, which we also initially solved with a CSV creation and import in anticipation of a more fine-grained solution after our first big appearance at Austria Comic-Con in early June.
As icing on the cake, we also wanted to generate nicely styled invoice documents in FreeFinance for all online shop orders that weren't paid via the cash register, and in the future, we'll want to make the online shop automatically aware of merchandise sold at events so they are removed from available stock for online purchases.
To achieve that, I looked into the APIs that both Magento and FreeFinance provide, accessing them from the custom internal system that I have full access to and that is required for providing the merchandise data anyhow. I found that the FreeFinance API is relatively simple, well-documented and does authentication via OAuth2, which I already had some knowledge of (and code to access it) from other projects, including some code already in the CBSM system for facilitating its own logins. That said, Magento is a different beast: its product catalog feature set is way more complex, and so is its API. Also, there is no well-structured collective documentation that would explain what various things mean or what is preferably done in what way (often there are multiple paths to the same result), it's a lot of turning on developer mode so that a Swagger/OpenAPI UI is available on your installation and then trying around there and searching the web for what could work how and what value could mean what. In addition, authentication is done via OAuth1, which is more complicated than its successor, and which I didn't have any pre-existing code for, though I could build on some code from their tutorial. Also, as we're running Magento ourselves on the server side, I could more easily try around things than with FreeFinance, which is a hosted service and I needed to request access credentials from their team. But FreeFinance gave us access to a testing system, whereas for Magento, for various reasons, we only have a live system and no staging/testing environment, so we can't "play around" very much when testing.
I wrote quite a bit of code for all those cases, the simplest part was and is surely updating the cash register with our products, the only slight complication there is adding categories if needed. For adding products to the shop, I needed to respect all kinds of things, like creating and managing configurable products, adding values to some attributes, uploading images, managing categories, and more. And the curious structure of the Magento API, which requires way more detailed action than the CSV import route, did at times make this even more complicated - but it works now and I can just add or change a product in the merch management and at the latest on the next day, both the shop and the cash register have updated to those changes (I can trigger the sync jobs earlier if required). For creating the invoice documents, I could base some things on a make.com "blueprint" provided by FreeFinance, but for one thing, we don't want to use a paid third-party service if we can automate this ourselves, and for the other, we have some restrictions and specialties of our own there (like only generating invoices for orders actually paid via the web shop payment integration and not in person via the cash register). I did run into some curiosities there, like the Magento order API result containing several pieces of data multiple times, or us initially using a document layout template that didn't allow for different products having different VAT rates (which we require) - but that's working now as well. The reverse part about getting cash register purchases into the online shop is still on my plate, but I now have a good plan for how to do that, and some time until our next big "away mission" where this will be important to have.
All in all, this has been a quite interesting experience, and I'm sure now that I am comfortable with working with those systems and APIs, I will do more with them in the long run - and our Trade Post 47 hopefully will still grow as well and therefore have additional requirements in the future. If you are a developer and have questions about some details, feel free to contact me - and if you are running such systems yourself and need a developer who can adapt them in a similar fashion, I'm happy to offer those services as a contractor!
Thunderbird doesn’t come bundled with a great suite of office productivity software, but there are many excellent free and open source applications that work well alongside it. One fantastic example is LibreOffice from The Document Foundation. And just like Thunderbird, it’s open source and multiplatform, available on Linux, macOS, and Windows.
Thanks a lot to everyone who responded! There were dozens of great ideas, and here are our top five:
“A consistent UI/UX could help familiarize newcomers to both tools more quickly… possibly a unified Settings area if you were to combine the functionality of both apps. Include a Thunderbird app launcher in the LibreOffice dashboard.”~Jay Robbie
“Link documents to calendar events or tasks, enabling quick access to relevant files during meetings or when working on projects and create calendar events or tasks directly from within LibreOffice.” ~Briani Davide
“What about a better integration of the viewing, editing and exporting capabilities from LibreOffice into Thunderbird for files send via mail? For example, an ODT file can be viewed and edited directly in Thunderbird (without leaving the window) and exported as PDF file.”~this.ven
“For me it would help if both would have the same shortcuts for styling the text. For example Ctrl+1 for Header 1, etc.”~Johannes
“Being able to insert from Thunderbird contacts into LibreOffice while writing, like when you type “@” on Twitter.”~Fabián Valverde
Let’s make some of these happen! LibreOffice and Thunderbird are community-driven open source projects, so the more help we get, the better the software becomes for everyone.
We’ve set up this meta bug for tracking interoperability and integration, and this page shows how to get involved on the LibreOffice side. If you want to help out from the Thunderbird side, see this page. (And if you’re a business running LibreOffice and Thunderbird, please consider funding developers to work on the features you need.)
Thanks in advance for all help! Let’s make these apps awesome together
As announced in the progress report for the previous month, in May we mostly worked on a new account setup experience. To learn more about that and what else has happened in the world of K-9 Mail as it evolves to Thunderbird For Android, read on.
[This month’s progress report was co-written by cketti and Wolf]
The Revamped Account Setup UI
In our continued efforts to enhance the user experience, we have redesigned the account setup UI. This is the first point of contact for our users and we wanted to ensure it is as intuitive and straightforward as possible. We integrate Thunderbird’s Autoconfiguration for seamless email account setup, start the transition of our app’s UI from XML Android layouts to Jetpack Compose, and adopt Atomic Design principles for a cohesive, intuitive design.
At the heart of this transformation is the integration of Thunderbird Autoconfig. This system enables automatic configuration of most email accounts, simplifying the user experience by making the connection to email servers effortless. Users only need to enter their email address and the server settings are obtained via multiple methods. They can be sourced from a central database (ISPDB), retrieved directly from ISPs through a configuration server, drawn from provided configuration files. In cases where these methods are unsuccessful, the configuration is derived from common server names. If all else fails, manual configuration is always available. With this powerful tool, setting up an email account becomes a breeze.
Our redesigned account setup UI has been written using Jetpack Compose. Although XML layouts have long been the standard for building Android UIs, the Android ecosystem is transitioning towards Jetpack Compose as the new norm. This declarative UI toolkit has enabled us to build UIs more efficiently, with less boilerplate code and reduced likelihood of bugs. Jetpack Compose defines UI elements as composable functions written in Kotlin code. This results in more intuitive, expressive, and readable components and promotes the reuse of UI components, streamlining the UI design process. This aligns well with the implementation of our new design system.
In combination with our new design system, which follows Atomic Design principles, the account setup UI presents a cohesive and intuitive user interface. The fundamental concept of Atomic Design involves breaking down design into smaller, reusable components (atoms), which can be combined to form larger, more complex structures (molecules and organisms). This approach has allowed us to systematically deconstruct our UI interfaces into their basic components and combine them to form more complex, reusable components for the account setup UI. The result is a more structured, scalable, and efficient design system, improving the consistency and usability of our app.
We’re confident that these changes have resulted in a smoother, more user-friendly account setup process. As we prepare to release a new beta version that includes this feature, we are eager to receive your thoughts and feedback on these improvements.
To Reply, or To Reply All?
When we released the overhauled message view in K‑9 Mail 6.600, the button to reply to a message changed from a menu offering Reply, Reply All, and Forward to a single button with a default action and the additional actions moving to the overflow menu (⋮). The default action in K‑9 Mail 6.600 was Reply All. Due to many complaints we changed it to Reply in K‑9 Mail 6.602.
However, it became clear that neither default works well for everyone, or even for one person in every situation. We’ll continue to work on this and hope we’ll find a solution where everyone can have their favorite behavior without forcing a significant portion of users having to dig through settings to get it.
Community Contributions
Ilya Bizyaev made a small change to the app so it also runs on Android devices that don’t support home screen widgets (see #6889).
Georg Sieber fixed a bug where we got the initialization order wrong (see #6909). This lead to the error message “Cannot connect to crypto provider” (erroneously) being displayed in certain situations (see #6068). If you were affected by this and want to say thank you, we’ll point out that there’s a way to sponsor Georg’s open source work.
Siva Subramaniam noticed that the “Sign in with Google” button was unreadable when using the dark theme (see #6924) and worked on a fix for the problem (see #6928).
Simon Arlott fixed a couple of issues with importing and exporting identities (see #6936).
In May we spent a couple of days fixing most of the security issues that were identified during the audit. We’ve created entries in our issue tracker for findings that still need to be addressed.
The final report should be ready soon. We’ll announce its availability in this blog.
Releases
In May 2023 we published the following stable releases:
Searchfox (source, config source) is Mozilla’s primary code searching tool for Firefox introduced by Bill McCloskey in 2016 which built upon prior work on DXR. This roadmap post is the second of two posts attempting to lay out where my personal efforts to enhance searchfox are headed and the decision making framework that guides them. The first post was a more abstract product vision document and can be found here.
Discoverable, Extensible, Powerful Queries
Searchfox has a new “query” endpoint introduced in bug 1762817 which is intended to enable more powerful queries. Queries are parsed using :kats‘ query-parser crate which allows us to support our existing (secret) key:value syntax in a more rigorous way (and with automatic parse correction for the inevitable typos). In order to have a sane execution model, these key/value pairs are mapped through an extensible configuration file into a pipeline / graph execution model whose clap-based commands form the basis of our testing mechanism and can also be manually built and run from the command-line via searchfox-tool.
Above you will find a diagram rendering the execution pipeline of searching for foo manually created from the JSON insta crate check output for the query. Bug 1763005 will add automatically generated diagrams as well as further expanding on the existing capability to produce markdown explanations of what is happening in each stage of the pipeline and the values at each stage.
While a new query syntax isn’t exciting on its own, what is exciting is that this infrastructure makes it easier to add functionality with confidence (and tests!). Some particular details worth discussing:
Customizable, Shareable Queries
Bug 1799796: Do you really wish that you could issue a query like webidl:CacheStorage to search just our WebIDL files for “CacheStorage”? Does your team have terminology that’s specific to your team and it would be great to have special search terms/aliases but it would feel wrong to use up all the cool short prefixes for your team? The new query mechanism has plans for these situations!
The new searchfox query endpoint looks like /mozilla-central/query/default. You’ll note that default looks like something that implies there are non-default options. And indeed, the plan is to allow files like this example “preset” dom.toml file to layer additional “terms” and “aliases” onto the base query_core.toml file as well as any other presets you want to build off of. You will need to add your preset to the mozsearch-mozilla repository for the tree in question, but the upside is that any query links you share will work for other people as well!
Faceting in Search Results with Shareable URLs
Bug 1799802: The basic idea of faceted search/filtering is:
You start with a basic search query.
Your results come back, potentially quite a lot of them. Too many, even!
The faceting logic looks at the various attributes of the results and classifies or “facets” them. Does that sound too circular? We just throw things in bins. If a bin ends up having a lot of things in it and there’s some hierarchy to its contents, we recursively bin those contents.
The UI presents these facets (bins), giving you a high level overview of the shape of your results, and letting you limit your results to only include certain attribute values, or to exclude based on others.
The UI is able to react quickly because it already knows about the result set
The cool screenshot above is of a SIMILE Exhibit-based faceting UI I created for bugzilla a while back which may help provide a more immediate concept of how faceting works. See my exhibit blog tag for more in the space.
Here are some example facets that search can soon support:
Individual result paths: Categorize results by the path in which they happen. Do you not want to look at any results under devtools/? Push a button and filter out all those devtool results in an instant! Do you only care about layout/? Push a button and only see layout results!
Subsystem facets: moz.build files labels every file in mozilla-central so that it has an associated Bugzilla Component. As of Bug 1783761 searchfox now also derives a subsystem mapping from the bugzilla components, which really just means that if you have a component that looks like “Core :: Storage: IndexedDB”, searchfox transforms that first colon into a slash so we get “Core/Storage/IndexedDB”. This would let you restrict your results to “Core/Storage” without having to manually select every Storage bugzilla component or path by hand.
Symbol relationships: Did you search for a base class or virtual method which has a number of subclasses/overrides? Do you only care about some subset of the class hierarchy? Then restrict your results to whatever the portion of the set you care about.
Recency of changes: Do you only care about seeing results whose blame history indicates it happened recently? Can do! Or maybe you only want to see code that hasn’t been touched in a long time? Uh, that might work less well until we improve the blame situation in Bug 1517978, but it’s always nice to have something to dream about.
Code coverage: Only want to see results that runs a lot under our tests? Sure thing! Only want to see results that seem like we don’t have test coverage for? By looking at the result you’re now morally obligated to add test coverage!
Key to this enhancement is that the faceting state will be reflected in the URL (likely the hash) so that you can share it or navigate forward and back and the state will be the same. It’s all too common on the web for state like this to be local to the page, but key to my searchfox vision is that URLs are key. If you do a lot of faceting, the URL may become large an unwieldy, but continuing in the style of :arai‘s fantastic work on Bug 1769936 and follow-ups to make it easy to get usable markdown out of searchfox, we can help wrap your URL in markdown link syntax so that when you paste it somewhere markdown-aware, it looks nice.
Additional Query Constraints
A bunch of those facets mentioned above sound like things that it would be neat to query on, right? Maybe even put them in a preset that you can share with others? Yes, we would add explicit query constraints for those as well, as well as to provide a way to convert faceted query results into a specific query that does not need to be faceted in Bug 1799805.
A variety of other additional queries become possible as well:
Searching for lines of text that are near each other, or not near each other, or maybe both inside the same argument list.
Locating member fields by type (Bug 1733217), like if you wanted to find all member fields that are smart or raw pointer references to nsILoadInfo.
Result Context Lines For All Result Types, Including Automatic Context
<figcaption class="wp-element-caption">Current query results for C:4 AddOrPut</figcaption>
A major limitation for searchfox searches has been a lack of support for context lines. (Disclaimer: in Bug 1414954 I added secret support for fulltext-only queries by prefixing a search with context:4 or similar, but you would then want to force a fulltext query like context:4 text:my actual search or context:4 re:my.*regexp[y]?.*search.) The query mechanism already supports full context, as the above screenshot is taken from the query for C:4 AddOrPut but note that the UX needs more passes and the gathering mechanism currently needs optimization which I have a WIP for in Bug 1794177
The next steps in diagramming will happen in Bug 1773165 with a focus on making the graphs interactive and applying heuristics related to graph clustering based on work on the “fancy branch” prototype and my recent work to derive the sub-component mapping for files that can in turn be propagated to classes/methods so that we can automatically collapse edges that cross sub-component boundaries (but which can be interactively expanded). This has involved a bit of yak-shaving on Bug 1776522 and Bug 1783761 and others.
Note that we also support calls-to:'Identifier' in the query endpoint as well, but the graphs look a lot messier without the clustering heuristics, so I’m not including any in this post.
Most of my work on searchfox is motivated by my desire to use diagrams in system understanding, with much of the other work being necessary because to make useful diagrams, you need to have useful and deep models of the underlying data. I’ll try and write more about this in the future, but this is definitely a case where:
A picture is worth a thousand words and iterations on the diagrams are more useful than the relevant prose.
Providing screen-reader accessible versions of the underlying data is fundamental. I have not yet ported the tree-dual version of the diagram logic from the “fancy” branch and I think this is a precondition to an initial release that’s more than just a proof-of-sorta-works.
Documentation Integration
Our in-tree docs rendered at https://firefox-source-docs.mozilla.org/ are fantastic. Searchfox cannot replace human-authored documentation, but it can help you find them! Have you spent hours understanding code only to find that there was documentation that would help clarify what was going on only after the fact? Bug 1763532 will teach searchfox to index markdown so that documentation definitions and references show up in search and that we can potentially expose those in context menus. Subsequent steps could also index comment contents.
Bug 1458882 will teach searchfox how to link to the rendered documentation.
Improved Language Support
New Language Support via SCIP
With the advent of LSIF and SCIP and in particular the work by the team at sourcegraph to add language indexing built on existing analysis tools, there is now a tremendous amount of low hanging fruit in terms of off-the-shelf language indexing that searchfox can potentially ingest. Thanks to Emilio‘s initial work in Bug 1761287 we know that it’s reasonably straightforward to ingest SCIP data from these indexers.
For each additional language we want to index, we expect the primary effort required will be to make the indexer available in a taskcluster task and appropriately configure it to index the potentially many component roots within the mozilla-central mono-repo. There will also be some searchfox-specific effort required to map the symbols into searchfox’s symbol namespace.
Specific languages we can support (better):
Javascript / Typescript via scip-typescript (Bug 1740290): scip-typescript potentially allows us to expose the same enhanced understanding of JS code, especially module-based JS code, that you experience in VS code, including type inference/extraction from JSDoc. Additionally, in Bug 1775130 we can leverage the amazing eslint work already done to bring enhanced analysis to more confusing situations like our mochitests which deal with more complex global situations. Overall, this can allow us to move away from searchfox’s current “soupy” understanding of JS code where it assumes that all the JS it ever sees is running in a single global.
Searchfox’s strongest support is for C++ (and its interactions with XPIDL and IPDL), but there is still more to do here. Thankfully Botond is working to improve C++ template handling in Bug 1781178 and related bugs.
Other enhancements:
Bug 1419018: Allow distinguishing LHS/RHS usages of fields, with a prototype of relevance in Bug 1733217.
Improved Mozilla-Specific Language Support
mozilla-central contains a number of Mozilla-specific Interface Definition Languages (IDLs) and Domain Specific Languages (DSLs). Searchfox has existing support for:
XPIDL.idl files: Our C++ support here is pretty good because XPIDL files are not preprocessed (beyond in-language support of #include and the ability to put pass-through C++ code including preprocessor directives inside %{C++ and %}demarcated blocks. Bug 1761689 tracks adding support for constants/enums which is not currently supported, and I have WIPs for this. Bug 1800008 tracks adding awareness of the rust bindings.
IPDL.ipdl and .ipdlh files: Our C++ support here is good as long as the file is not pre-processed and the rust IPDL parser hasn’t fallen behind the Python parser. Unfortunately a lot of critical files like PContent.ipdl are pre-processed so this currently creates massive blind-spots in searchfox’s understanding of the system. Bug 1661067 will move us to having the Python parser/code generator emit data searchfox can ingest
Searchfox has planned support for:
WebIDL.webidl files: Bug 1416899 tracks indexing WebIDL files, and Bug 1525189 tracks the JS interaction side of this.
StaticPrefList.yaml and related preference bindings: Bug 1699048 with discussion in Bug 1727789 on which it certainly depends.
Searchfox’s language indexing is inherently a form of static analysis. Consistent with the searchfox vision saying that “searchfox is not the only tool”, it makes sense to attempt to integrate with and build upon the tools that Firefox engineers are already using. Mozilla’s code-coverage data is already integrated with searchfox, and the next logical step is to integrate with pernosco, why not. I created pernosco-bridge as an experimental means of extracting data from pernosco and allowing for interactive visualizations.
The screenshot above is an example of a timeline graph automatically extracted from a config file to show data relevant to IndexedDB. IndexedDB transactions were hierarchically related to their corresponding database and the origin that opened those databases. Within each transaction, ObjectStoreAddOrPutRequestOp and CommitOp operations are graphed. Clicking on the timeline would direct the pernosco tab to jump to those instants in time.
The above is a different visualization based on a config file for DocumentChannel to help group what’s going on in a pernosco trace and surface the relevant information. If you check out the config file, you will probably find it inscrutable, but with searchfox’s structured understanding of C++ classes landed last year in Bug 1641372 we can imagine leveraging searchfox’s understanding of the codebase to make this process friendly. More importantly, there is the potential to collaboratively build a shared knowledge base of what’s most relevant for classes, so everyone doesn’t need to re-do the same work.
Using the same information pernosco-bridge used to build the hierarchically organized timelines above with extracted values like URIs, it can also build graphs of the live objects at any moment in time in the trace. Above we can see the relationship between windowGlobalParent instances and their corresponding canonicalBrowsingContexts, plus the URIs of the canonicalBrowsingContexts. We can imagine using this to help visualize representative object graphs in searchfox.
We can also imagine doing something like the above screenshot from my prior experiment pecobro where we interleave graphs of function activity into source listings.
Token-Centric Blame / “hyperannotate” Support via Microannotate
<figcaption class="wp-element-caption">A demonstration of microannotate’s output</figcaption>
Quoting my dev-platform post about the unfortunate removal of searchfox’s first attempt at blame-skipping: “Revision history and the “annotate” / “blame” UIs for revision control are tricky because they’re built on a sequential, line-centric data-model where moving a function above another function in a file results in a destructive representational decision to treat one function as continuing through history and the other function as removed and then re-added as new code. Reformatting that maintains the overall sequence of tokens but changes how they are distributed across multiple lines also looks like removal of all of the old code and the addition of new code. Tools frequently perform heuristic-based post-passes to help identify intra-line changes which are reflected in diff UIs, as well as (entire) lines of code that are copied/moved in a revision (ex: Phabricator does this).”
The plan to move forward is to move to a token-centric approach using :marco‘s microannotate project as tracked in Bug 1517978. We would also likely want to combine this with heuristics that skip over backout pairs. The screenshot at the top of this section is of example output for nsWebBrowserPersist.cpp where colors distinguish between different blame revision origins. Note that the addition of specific arguments can be seen as well as changes to comments.
Source Listings
Bug 1781179: Improved syntax and semantic highlighting in C++ for the tip/head indexed revision.
Bug 1583635: Show expansion of C++ macros. Do you ever look at our XPCOM macrology and wish you weren’t about to spend several minutes clicking through those macros to understand what’s happening? This bug, my friend, this bug.
Bug 1796870: Adopt use of tree-sitter as a tokenizer which can improve syntax highlighting for other languages as well as position: sticky context for both the tip/head indexed revision but also for historical revisions!
Bug 1799557: Improved handling of links to source files that no longer exist by offering to show the last version of the file that existed or try and direct the user to the successor code.
Bug 1697671: Link resource:// and chrome:// URLs in source listings to the underlying source files
Test Info Boxes
Bug 1785129: Add an info box mechanism to indicate the need for data collection review (“data review”) in info boxes on searchfox source listing pages
Bug 1797855: Joel Maher and friends have been adding all kinds of great test metadata for searchfox to expose, and soon, this bug shall expose that information. Unfortunately there’s some yak shaving related to logging that remains underway.
Bug 1797857: Extend the “Go to header file”/”Go to source file” mechanism to support WPT `.headers` files and xpcshell/mochitest `^headers^` files.
Alternate Views of Data
Searchfox is able to provide more than source listings. The above screenshot shows searchfox’s understanding of the field layouts of C++ classes across all the platforms searchfox indexes on as rendered by the “fancy” branch prototype. Bug 1468445 tracks implementing a production quality version of this, noting that the data is already available, so this is straightforward. Bug 1799517 is a variation on this which would help us explicitly capture the destructor order of C++ fields.
Bug 1672307 tracks showing the binary size impact of a given file, class, etc.
Source Directory Listings
In the recently landed Bug 1783761 I moved our directory listings into rust after shaving a whole bunch of yaks. Now we can do a bunch of queries on data about files. Would you like to see all the tests that are disabled in your components? We could do this! Would you like to see all the files in your components that have been modified in the last month but have bad coverage? We could also do that! There are many possibilities here but I haven’t filed bugs for them.
Bug 1732585: Provide a way to search related (phabricator revision) review/bugzilla comments related to the current file
Bug 1657786: Create searchfox taskcluster mode/variant that can run the C++ indexer only against changed files for try builds / phabricator requests
Bug 1778802: Consider storing m-c analysis data in a git repo artifact with a bounded history via `git checkout –orphan` to enable try branch/code review features and recent semantic history support
Easier Contributions
The largest hurdle new contributors have faced is standing up a virtual machine. In Bug 1612525 we’ve added core support for docker, and we have additional work to do in that bug to document using docker and add additional support for using docker under WSL2 on Windows. Please feel free to drop by https://chat.mozilla.org/#/room/#searchfox:mozilla.org if you need help getting started.
Deeper Integration with Mozilla Infrastructure
Currently much of searchfox runs as EC2 jobs that exists outside of taskcluster, although C++ and rust indexing artifacts as well as all coverage data and test info data comes from taskcluster. Bug 1598502 tracks moving more of searchfox into taskcluster, although presumably the web-servers will still need to exist outside of taskcluster.
Searchfox (source, config source) is Mozilla’s primary code searching tool for Firefox introduced by Bill McCloskey in 2016 which built upon prior work on DXR. This product vision post describes my personal vision for searchfox and the rationale that underpins it. I’m also writing an accompanying road map that describes specific potential enhancements in support of this vision which I will publish soon and goes into the concrete potential features that would be implemented in the spirit of this vision.
Note that the process of developing searchfox is iterative and done in consultation with its users and other contributors, primarily in the searchfox channel on chat.mozilla.org and in its bugzilla component. Accordingly, these documents should be viewed as a basis for discussion rather than a strict project plan.
The Whys Of Searchfox
Searchfox is a Tool For System Understanding
Searchfox enables exploration and understanding of the Firefox codebase as it exists now, as it existed in the past, and to support understanding of the ramifications of potential changes.
Searchfox Is A Tool For Shared System Understanding
Firefox is a complex piece of software which has more going on than any one person can understand at a time. Searchfox enables Firefox’s contributors to leverage the documentation artifacts of other teams and contributors when exploring in isolation, and to communicate more effectively when interacting.
Searchfox Is Not The Only Tool
Searchfox integrates relevant data from automation and other tools in the Firefox development ecosystem where they make sense and provides deep links into those tools or first steps to help you get started without having to start from nothing.
The Hows Of Searchfox
Searchfox is Immediate: Low Latency and Complete
Searchfox’s results should be available in their entirety when the page load completes, ideally in much less than a second. There should be no asynchronous lazy loading or spinners. Practically speaking, if you could potentially see something on a page, you should be able to ctrl-f for it.
In situations where results are too voluminous to be practically useful, Searchfox should offer targeted follow-on searches that can relax limits and optionally provide for additional constraints so that iterative progress is made.
Searchfox is Accessible
Searchfox should always present a usable accessibility tree. This includes ensuring that any dynamically generated graphical representations such as graphviz-style diagrams have a directly usable accessibility tree or an alternate representation that maximally captures any hierarchy or clustering present in the visual presentation.
Searchfox Favors Iterative Exploration and Low Activation Energy
Searchfox seeks to avoid UX patterns where you have to metaphorically start from a blank sheet of paper or face decision paralysis choosing among a long list of options. Instead, start from whatever needle you have (an identifier name, a source file location, a string you saw in the UI) and searchfox will help you iterate and refine from there.
Searchfox Provides Stable, Useful URLs When Possible and Markdown For More Complex Situations
If you’re looking at something in searchfox, you should be able to share it as a URL, although there may be a few URLs to choose from such as whether to use a permalink which includes a specific revision identifier. More complicated situations may merit searchfox providing you with markdown that you can paste in tools that understand markdown.
After some behind-the-scenes discussions with Michael Kohler on what I could contribute at this year's FOSDEM, I ended up doing a presentation about my personal Suggestions for a Stronger Mozilla Community (video is available on the linked page). While figuring out the points I wanted to talk about and assembling my slides for that talk, I realized that one of the largest issues I'm seeing is that the Mozilla community nowadays feels very disconnected to me, like several islands, within each there is good stuff being done, but most people not knowing much about what's happening elsewhere. That has been helped a lot by a lot of interesting projects being split off Mozilla into separate projects in recent years (see e.g. Coqui, WebThings, and others) - which is often taking them off the radar of many people even though I still consider them as being part of this wider community around the Mozilla Manifesto and the Open Web.
Following the talk, I brought that topic to the Reps Weekly Call this last week (see linked video), esp. focusing on one slide from my FOSDEM talk that talks about finding some kind of communication channel to cross-connect the community. As Reps are already a somewhat cross-function community group, my hope is that a push from that direction can help getting such a channel in place - and figuring out what exactly is a good idea and doable with the resources we have available (I for example like the idea of a podcast as I like how those can be listened to while traveling, cooking, doing house work, and others things - but it would be a ton of work to organize and produce that).
Some ideas that came up in the Reps Call were for example a regular newsletter on Mozilla Discourse in the style of the MoCo-internal "tl;dr" (which Reps have access to via NDA), but as something that is public, as well as from and for the community - or maybe morphing some Reps Calls regularly into some sort of "Community News" calls that would highlight activities around the wider community, even bringing in people from those various projects/efforts there. But there may be more, maybe even better ideas out there.
To get this effort to the next level, we agreed that we'll first get the discussion rolling on a Discourse thread that I started after the meeting and then probably do a brainstorming video call. Then we'll take all that input and actually start experimenting with the formats that sound good and are practically achievable, to find what works for us the best way.
If you have ideas or other input on this, please join the conversation on Discourse - and also let us know if you can help in some form!
tl;dr: Happy 23rd birthday, Mozilla. And for the question: yes.
Here's a bit more rambling on this topic...
First of all, the Mozilla project was officially started on March 31, 1998, which is 23 years ago today. Happy birthday to my favorite "dino" out there! For more background, take a look at my Mozilla History talk from this year's FOSDEM, and/or watch the "Code Rush" documentary that conserved that moment in time so well and also gives nice insight into late-90's Silicon Valley culture.
Now, while Mozilla initially was there to "act as the virtual meeting place for the Mozilla code" as Netscape was still there with the target to win back the browser market that was slipping over to Micosoft. The revolutionary stance to develop a large consumer application in the open along with the marketing of "hack - this technology could fall into the right hands" as well as the general novenly of the open-source movement back then - and last not least a very friendly community (as I could find out myself) made this young project grow fast to be more than a development vehicle for AOL/Netscape, though. And in 2003, a mission to "preserve choice and innovation on the Internet" was set up for the project, shortly after backed by a non-profit Mozilla Foundation, and then with an independently developed Firefox browser, implementing "the idea [...] to design the best web browser for most people" - and starting to take back the web from the stagnation and lack of choice represented by >95% of the landscape being dominated by Microsoft Internet Explorer.
The exact phrasing of Mozilla's mission has been massages a few times, but from the view of the core contributors, it always meant the same thing, it currently reads:
Quote:
Our mission is to ensure the Internet is a global public resource, open and accessible to all. An Internet that truly puts people first, where individuals can shape their own experience and are empowered, safe and independent.
On the Foundation site, there's the sentence "It is Mozilla’s duty to ensure the internet remains a force for good." - also pretty much meaning the same thing with that, just in less specific terms. Of course, the spirit of the project was also put into 10 pretty concrete technical principles, prefaced by 4 social pledges, in the Mozilla Manifesto, which make it even more clear and concrete what the project sees as its core purpose.
So, if we think about the question whether we still need Mozilla nowadays, we should take a look if moving in that direction is still required and helpful, and if Mozilla is still able and willing to push those principles forward.
When quite a few communities I'm part of - or would like to be part of - are moving to Discord or are adding it as an additional option to Facebook groups, and I read the Terms of Services of those two tightly closed and privacy-unfriendly services, I have to conclude that the current Internet is not open, not putting people first, and I don't feel neither empowered, safe or independent in that space. When YouTube selects recommendations so I live in a weird bubble that pulls me into conspiracies and negativity pretty fast, I don't feel like individuals can shape their own experience. When watching videos stored on certain sites is cheaper or less throttled than other sources with any new data plan I can get for my phone, or when geoblocking hinders me from watching even a trailer of my favorite series, I don't feel like the Internet is equally accessible to all. Neither do I when political misinformation is targeted at certain groups of users in election ads on social networks without any transparency to the public. But I would long for that all to be different, and to follow the principles I talked of above. So, I'd say those are still required, and would be helpful to push for.
I definitely think we badly need a Mozilla that works on all those issues, and we need a whole lot of other projects and people help in the space as well. Be it in advocacy, in communication, in technology (links are just examples), or in other topics.
Can all that actually succeed in improving the Internet? Well, it definitely needs all of us to help, starting with using products like Firefox, supporting organizations like Mozilla, spreading the word, maybe helping to build a community, or even to contribute where we can.
We definitely need Mozilla today, even 23 years after its inception. Maybe we need it more than ever, actually. Are you in?
As mentioned in a previous post, I've been working with the Capacity Blockchain Solutions team on the Crypto stamp project, the first physical postage stamp with a unique digital twin, issued by the Austrian Postal Service (Österreichische Post AG). After a successful release of Crypto stamp 1, one of our core ideas for a second edition was to represent stamp albums (or stamp collections) in the digital world as well - and not just the stamps themselves.
We set off to find existing standards on Ethereum contracts for grouping NFTs (ERC-721 and potentially ERC-1155 tokens) together and we found that there are a few possibilities (like EIP-998) but those ares getting complicated very fast. We wanted a collection (a stamp album) to actually be the owner of those NFTs or "assets" but at the same time being owned by an Ethereum account and able to be transferred (or traded) as an NFT by itself. So, for the former (being the owner of assets), it needs to be an Ethereum account (in this case, a contract) and for the latter (being owned and traded) be a single ERC-721 NFT as well. The Ethereum account should not be shared with other collections so ownership of an asset is as transparent as people and (distributed) apps expect. Also, we wanted to be able to give names to collections (via ENS) so it would be easier to work with them for normal users - and that also requires every collection to have a distinct Ethereum account address (which the before-mentioned EIP-998 is unable to do, for example). That said, to be NFTs themselves, the collections need to be "indexed" by what we could call a "registry of Collections".
To achieve all that, we came up with a system that we think could be a model for future similar project as well and would ideally form the basis of a future standard itself.
At its core, a common "Collections" ERC-721 contract acts as the "registry" for all Crypto stamp collections, every individual collection is represented as an NFT in this "registry". Additionally, every time a new NFT for a collection is created, this core contract acts a "factory" and creates a new separate contract for the collection itself, connecting this new "Collection" contract with the newly created NFT. On that new contract, we set the requested ENS name for easier addressing of the Collection.
Now this Collection contract is the account that receives ERC-721 and ERC-1155 assets, and becomes their owner. It also does some bookkeeping so it can actually be queried for assets and has functionality so the owner of the Collection's own NFT (the actual owner of the Collection itself) and full control over those assets, including functions to safely transfer those away again or even call functions on other contracts in the name of the Collection (similar to what you would see on e.g. multisig wallets).
As the owner of the Collection's NFT in the "registry" contract ("Collections") is the one that has power over all functionality of this Collection contract (and therefore the assets it owns), just transferring ownership of that NFT via a normal ERC-721 transfer can give a different person control, and therefore a single trade can move a whole collection of assets to a new owner, just like handing a full album of stamps physically to a different person.
To go into more details, you can look up the code of our Collections contract on Etherscan. You'll find that it exposes an ERC-721 name of "Crypto stamp Collections" with a symbol of "CSC" for the NFTs. The collections are registered as NFTs in this contract, so there's also an Etherscan Token Tracker for it, and as of writing this post, over 1600 collections are listed there. The contract lets anyone create new collections, and optionally hand over a "notification contract" address and data for registering an ENS name. When doing that, a new Collection contract is deployed and an NFT minted - but the contract deployment is done with a twist: As deploying a lot of full contracts with a larger set of code is costly, an EIP-1167 minimal proxy contract is deployed instead, which is able to hold all the data for the specific collection while calling all its code via proxying to - in this case - our Collection Prototype contract. This makes creating a new Collection contract as cheap as possible in terms of gas cost while still giving the user a good amount of functionality. Thankfully, Etherscan for example has knowledge of those minimal proxy contracts and even can show you their "read/write contract" UI with the actually available functionality - and additionally they know ENS names as well, so you can go to e.g. etherscan.io/address/kairo.c.cryptostamp.eth and read the code and data of my own collection contract. For connecting that Collection contract with its own NFT, the Collections (CSC) contract could have a translation table between token IDs and contract addresses, but we even went a step further and just set the token ID to the integer value of the address itself - as an Ethereum address is a 40-byte hexadecimal value, this results in large integer numbers (like 675946817706768216998960957837194760936536071597 for mine) but as Ethereum uses 256-bit values by default anyhow, it works perfectly well and no translation table between IDs and addresses is needed. We still do have explicit functions on the main Collections (CSC) contract to get from token IDs to addresses and vice versa, though, even if in our case, it can be calculated directly in both ways.
Both the proxy contract pattern and the address-to-token-ID conversion scheme are optimizations we are using but if we were to standardize collections, those would not be in the core standard but instead to be recommended implementation practices instead.
Of course, users do not need to care about those details at all - they just go to crypto.post.at, click "Collections" and create their own collection for there (when logged in via MetaMask or a similar Ethereum browser module), and they also go the the website to look at its contents (e.g. crypto.post.at/collection/kairo). Ideally, they'll also be able to view and trade them on platforms like OpenSea - but the viewing needs specific support (which probably would need standardization to at least be in good progress), and the trading only works well if the platform can deal with NFTs that can change value while they are on auction or the trade market (and then any bids made before need to be invalidated or re-confirmed in some fashion). Because the latter needs a way to detect those value changes and OpenSea doesn't have that, they had to suspend trade for collections for now after someone exploited that missing support by transferring assets out of the collection while it was on auction. That said, there are ideas on how to get this back again the right way but it will need work on both the NFT creator side (us in the specific case of collections) and platforms that support trade, like OpenSea. Most importantly, the meta data of the NFT needs to contain some kind of "fingerprint" value that changes when any property changes that influences the value, and the trading platform needs to check for that and react properly to changes of that "fingerprint" so bids are only automatically processed as long as it doesn't change.
For showing contents or calculating such a "fingerprint", there needs to be a way to find out, which assets the collection actually owns. There are three ways to do that in theory: 1) Have a list of all assets you care about, and look up if the collection address is listed as their owner, 2) look at the complete event log on the blockchain since creation of the collection and filter all NFT Transfer events for ones going to the collection address or away from it, or 3) have some way of so the collection itself can record what assets it owns and allow enumeration of that. Option 1 is working well as long as your use case only covers a small amount of different NFT contracts, as e.g. the Crypto stamp website is doing right now. Option 2 gives general results and is actually pretty feasible with the functionality existing in the Ethereum tool set, but it requires a full node and is somewhat slow.
So, for allowing general usage with decent performance, we actually implemented everything needed for option 3 in the collections contract. Any "safe transfer" of ERC-721 or ERC-1155 tokens (e.g. via a call to the safeTransferFrom() function) - which is the normal way that those are transferred between owners - does actually test if the new owner is a simple account or a contract, and if it actually is a contract, it "asks" if that contract can receive tokens via a contract function call. The collection contract does use that function call to register any such transfer into the collection and puts such received assets into a list. As for transferring away an asset, you need to make a function call on the collection contract anyhow, removing from that list can be done there. So, this list can be made available for querying and will always be accurate - as long as "safe" transfers are used. Unfortunately, ERC-721 allows "unsafe" transfers via transferFrom() even though it warns that NFTs "MAY BE PERMANENTLY LOST" when that function is used. This was probably added into the standard mostly for compatibility with CryptoKitties, which predate this standard and only supported "unsafe" transfers. To deal with that, the collections contract has a function to "sync" ownership, which is given a contract address and token ID, and it adjusts it assets list accordingly by either adding or removing it from there. Note that there is a theoretical possibility to also lose an assets without being able to track it there, that's why both directions are supported there. (Note: OpenSea has used "unsafe" transfers in their "gift" functionality at least in the past, but that hopefully has been fixed by now.)
So, when using "safe" transfers or - when "unsafe" ones are used - "syncing" afterwards, we can query the collection for its owned assets and list those in a generic way, no matter which ERC-721 or ERC-1155 assets are sent to it. As usual, any additional data and meta data of those assets can then be retrieved via their NFT contracts and their meta data URLs.
I mentioned a "notification contract" before which can be specified at creation of a collection. When adding or removing an assets from the internal list in the collection, it also calls to that notification contract (if one is set) as a notification of this asset list change. Using that feature, it was possible to award achievements directly on the blockchain for e.g. collecting a certain number of NFTs of a specific type or one of each motif of Crypto stamps. Unfortunately, this additional contract call costs even more gas on Ethereum, as does tracking and awarding of achievements themselves, so rising gas costs forced us to remove that functionality and not set a notification contract for new collections as well as offer an "optimization" feature that would remove it from collections already created with one. This removal made transaction costs for using collections more bearable again for users, though I still believe that on-chain achievements were a great idea and probably a feature that was ahead of its time. We may come back to that idea when it can be done with an acceptably small impact on transaction cost.
One thing I also mentioned before is that the owner of a Collection can actually call functions in other contracts in the name of the Collection, similar to functionality that multisig wallets provide. This is done via an externalCall() function, to which the caller needs to hand over a contract address to call and an encoded payload (which can relatively easily be generated e.g. via the web3.js library). The result is that the Collection can e.g. call the function for Crypto stamps sold via the OnChain shop to have their physical versions sent to a postage address, which is a function that only the owner of a Crypto stamp can call - as the Collection is that owner and its own owner can call this "external" function, things like this can still be achieved.
To conclude, with Crypto stamp Collections we have created a simple but feature-rich solution to bring the experience of physical stamp albums to the digital world, and we see a good possibility to use the same concept generally for collecting NFTs and enabling a whole such collection of NFTs to be transferred or traded easily as one unit. And after all, NFT collectors would probably expect a collection of NFTs or a "stamp album" to have its own NFT, right? I hope we can push this concept to larger adoption in the future!
The FOSDEM conference in Brussels has become a bit of a ritual for me. Ever since 2002, there has only been a single year of the conference that I missed, and any time I was there, I did take part in the Mozilla devroom - most years also with a talk, as you can see on my slides page.
This year, things were a bit different as for obvious reasons the conference couldn't bring together thousands of developers in Brussels but almost a month ago, in its usual spot, the conference took place in a virtual setting instead. The team did an incredibly good job of hosting this huge conference in a setting completely run on Free and Open Source Software, backed by Matrix (as explained in a great talk by Matthew Hodgson) and Jitsi (see talk by Saúl Ibarra Corretgé).
On short notice, I also added my bit to the conference - this time not talking about all the shiny new software, but diving into the past with "Mozilla History: 20+ Years And Counting". After that long a time that the project exists, I figured many people may not realize its origins and especially early history, so I tried to bring that to the audience, together with important milestones and projects on the way up to today.
The video of the talk has been available for a short time now, and if you are interested yourself in Mozilla's history, then it's surely worth a watch. Of course, my slides are online as well.
If you want to watch more videos to dig deeper into Mozilla history, I heavily recommend the Code Rush documentary from when Netscape initially open-sourced Mozilla (also an awesome time capsule of late-90s Silicon Valley) and a talk on early Mozilla history from Mitchell Baker that she gave at an all-hands in 2012.
The Firefox part of the history is also where my song "Rock Me Firefox" (demo recording on YouTube) starts off, for anyone who wants some music to go along with all this!
We’re working on a thing to make Firefox start faster! It appears to work! Here’s a video showing off a before (left) and after (right):
Improving Firefox Startup Time With The about:home Startup Cache
For the past year or so, the Firefox Desktop Front-End Performance team has been concentrating on making improvements to browser startup performance.
The launching of an application like Firefox is quite complex. Meticulous profiling of Firefox startup in various conditions has, thankfully, helped reveal a number of opportunities where we can make improvements. We’ve been evaluating and addressing these opportunities, and several have made it into the past few Firefox releases.
This blog post is about one of those improvements that is currently in the later stages of development. I’m going to describe the improvement, and how we went about integrating it.
In a default installation of Firefox, the first (and only) tab that loads is about:home1.
The about:home page is actually the same thing that appears when you open a new tab (about:newtab). The fact that they have different addresses allows us to treat their loading differently.
Your about:home might look slightly different from the above — depending on your locale, it may or may not include the Pocket stories.
Do not be fooled by what appears to be a very simple page of images and text. This page is actually quite sophisticated under the hood. It is designed to be customized by the user in the following ways:
Users can
Collapse or expand sections
Remove sections entirely
Reorganize the order of their Top Sites by dragging and dropping
Pin and unpin Top Sites to their positions
Add their own custom Top Sites with custom thumbnails
Add or remove search engines from their Top Sites
Change the number of rows in the Top Sites and Recommended by Pocket sections
Choose to have the Highlights composed of any of the following:
Visited pages
Recent bookmarks
Recent downloads
Pages recently saved to Pocket
The user can customize these things at any time, and any open copies of the page are expected to reflect those customizations immediately.
There are further complexities beyond user customization. The page is also designed to be easy for our design and engineering teams to experiment with reorganizing the layout and composition of the page so that they can test variations on its layout in the wild.
The about:home page also has special privileges not afforded to normal websites. It can
Save and remove bookmarks
Add pages to Pocket
Cause the URL bar to be focused and selected
Show thumbnails for pages that the user has visited
Access both high and normal resolution favicons
Render information about the user’s recent activity (recent page visits, downloads, saves to Pocket, etc.)
So while at first glance, this appears to be a static page of just images and text, rest assured that the page can do much more.
Like the Firefox Developer Tools UI, about:home is written with the help of the React and Redux libraries. This has allowed the about:home development team to create sophisticated, reusable, and composable components that could be easily tested using modern JavaScript testing methods.
Unsurprisingly, this complexity and customizability comes at a cost. The page needs to request a state object from the parent process in order to have the Redux store populated and to have React render it. Essentially, the page is dynamically rendering itself after the markup of the page loads.
Startup is a critical time for an application. The user has expressed a need for their browser, and we have an obligation to serve the user as quickly and efficiently as possible. The user’s time is a resource that we should not squander. Similarly, because so much needs to occur during startup,2 disk reads, disk writes, and CPU time are also considered precious resources. They should only be used if there’s no other choice.
In this case, we believed that the CPU time and disk accesses spent constructing the state object and dynamically rendering the about:home page was competing with all of the other CPU and disk access happening during startup, and this was slowing us down from presenting about:home to the user in a timely way.
Generally speaking, in my mind there are four broad approaches to performance problems once a bottleneck has been identified.
You can widen the bottleneck (make the operations more efficient)
You can divide the bottleneck (split the work into smaller slices that can be done over a longer period of time with rests in between)
You can move the bottleneck (defer work until later when it seems that there is less competition for resources, or move it to a different thread)
You can remove the bottleneck (don’t do the work)
We started by trying to apply the last two approaches, wondering what startup performance would be like if the page did not render itself dynamically, but was instead a static page generated periodically and pulled off of the disk at startup.
Prototype when possible
The first step to improving something is finding a way to measure it. Thankfully, we already have a number of logged measurements for startup. One of those measurements gives us the time from process start to rendering the Top Sites section of about:home. This is not a perfect measurement—ideally, we’d measure to the point that the page finally “settles” and stops changing3—but for this project, this measurement served our purposes.
Before investing a bunch of time into a potential improvement, it’s usually a good idea to try to see if what you’re gaining is worth the development time. It’s not always possible to build a prototype for performance improvements, but in this case it was.
So, with that information, we thought we had a real improvement opportunity here. We decided to proceed with the idea, and began a long arduous search for “the right way to do it.”
Pre-production
As I mentioned earlier, about:home is complex. The infrastructure that powers it is complex. Coupled with the fact that no one on the Firefox Front-End Performance team had spent much time studying React and Redux meant that we had a lot of learning to do.
The first step was to get some React and Redux fundamentals under our belt. This meant building some small toy applications and getting familiar with the framework idioms and how things are organized.
With that grounding, the next step was to start reading the code — starting from the entrypoint into the code that powers about:home when the browser starts. This was an intense period of study that branched into many different directions. Part of the complexity was because much of the code is asynchronous and launched work on different threads, which introduced some non-determinism. While it is generally good for responsiveness to move work off of the main thread, it can lead to some complex reading and interpretation of the code when more than two threads are involved.
A tool we used during this analysis was the Firefox Profiler, to get a realistic sense of the order of executions during startup. These profiles helped to inform much of our reading of the code.
This analysis helped us solidify our mental model of how about:home loads. With that model in place, it was much easier to propose practical approaches for introducing a static about:home document into the ecosystem of pre-existing code. The Firefox Front-End Performance team documented our findings and recommendations and then presented them to the team that originally built the about:home system to ensure that we were all on the same page and that we hadn’t missed anything critical. They were already aware that we were investigating potential performance improvements, and had very useful feedback for us, as well as historical product decision context that clarified our understanding.
Critically, we presented our recommendation for loading a static about:home page at startup and ensured that there were no upcoming plans for about:home that would break our mental model or render the recommendation no longer valid. Thankfully, it sounded like we were aligned and fine to proceed with our plan.
So what was the plan? We knew that since about:home is quite dynamic and can change over time4 we needed a startup cache for about:home that could be periodically updated during the course of a browsing session. We would then load from that cache at startup. Clearly, I’m glossing over some details here, but that was the general plan.
As usual, no plan survives breakfast, and as we started to architect our solution, we identified things we would need to change along the way.
Development
We knew that the process that loads about:home would need to be able to read from the about:home startup cache. We also knew that about:home can potentially contain information about what pages the user has visited, and that about:home can do privileged things that normal web pages cannot. It seemed that this project would be a good opportunity to finish a project that was started (and mothballed) a year or so earlier: creating a special privileged content process for about:home. We would load about:home in that process, and add assertions to ensure that privileged actions from about:home could only happen from that content process type5
So getting the “privileged about content process”6 fixed up and ready for shipping was the first step.
This also paved the way for solving the next step, which was to enable the moz-page-thumb:// protocol for the “privileged about content process.” The moz-page-thumb:// protocol is used to show the screenshot thumbnails for pages that the user has visited in the past. The previous implementation was using Blob URLs to send those thumbnails down to the page, and those Blob URLs exist only during runtime and would not work properly after a restart.
The next step was figuring out how to build the document that would be stored in the cache. Thankfully, ReactDOMServer has the ability to render a React application to a string. This is normally used for server-side rendering of React-powered applications. This feature also allows the React library to passively attach to the server-side page without causing the DOM to be modified. With some small modifications, we were able to build a simple mechanism in a Web Worker to produce this cached document string off of the main thread. Keeping this work off of the main thread would help maintain responsiveness.
With those pieces of foundational work out of the way, it was time to figure out the cache storage mechanism. Firefox already has a startupcache module that it uses for static resources like markup and JavaScript, but that cache is not designed to be written to periodically at runtime. We would need something different.
We had originally supposed that we would need to give the privileged about content process special access to a file on the filesystem to read from and to write to (since our sandbox prevents content processes from accessing disks directly). Initial experiments along this line worried us — we didn’t like the idea of poking holes in the sandbox if we didn’t need to. Also, adding yet another read from the filesystem during startup seemed counter to our purposes.
We evaluated IndexedDB as a storage mechanism, but the DOM team talked us out of it. The performance characteristics of IndexedDB, especially during startup, were unlikely to work for us.
Finally, after some consulting, we were directed to the HTTP cache. The HTTP cache’s job is to cache pages that the user visits (when appropriate) and to offer those caches to the user instead of hitting the network when retrieving the resource within the expiration time7. Functionally speaking, this seemed like a storage mechanism perfectly suited to our purposes.
After consulting with the Necko team and building a few proof-of-concepts, we figured out how to tie the whole system together. Importantly, we figured out how to get the initial about:home load to pull a document out from the HTTP cache rather than reading it from the application resource package.
We also figured out the cache writing mechanism. The cached document that would periodically get built inside of the privileged about content process inside of a Worker off of the main thread, would then send that data back up to the parent to stream into the cache.
At this point, we felt we had all of the pieces that we needed. Construction on each component began.
One of the more gratifying parts of implementation was when we modified one of our startup tests to use the new caching mechanism.
In this graph, the Y axis is the geometric mean time to render the about:home Top Sites over 20 restarts of the browser, in milliseconds. Lower is better. The dots along the top are without the cache. The dots along the bottom are with the cache enabled. According to our measurements, we improved the rendering time from process start to Top Sites by just over 20%! We beat our prototype!
Noticeable differences
But the real proof will be if there’s actually a noticeable visual change. Here’s that screen recording again from one of our reference devices8.
The screen on the left is with the cache disabled, and on the right with the cache enabled. Looks to me like we made a noticeable dent!
Try it out!
We haven’t yet enabled the about:home startup cache in Nightly by default, but we hope to do so soon. In the meantime, Nightly users can try it out right now by going to about:preferences#experimental and toggling it on. If you find problems and have a Bugzilla account, here’s a form for submitting bugs to the right place.
You can tell if the about:home you’re looking at is from the cache by opening up the DevTools Inspector and looking for a <!-- Cached: <some date> --> comment just above the <body> tag.
Caveat emptor
There are a few cases where the cache isn’t used or is invalidated.
The first case is if you’ve configured something other than about:home as your home page (where the cache isn’t used). In this case, the cache won’t be read from, and the code to create the cache won’t ever run. If the user ever resets about:home to be their home page, then the caching code will start working for them.
The second case is if you’ve configured Firefox to restore your previous session by default. In this case, it’s unlikely that the first tab you’ll see is about:home, so the cache won’t be read from, and the code to create the cache won’t ever run. As before, if the user switches to not loading their previous session by default, then the cache will start working for them.
Another case is when the Firefox build identifier doesn’t match the build identifier from when the cache was created. This is also how the other startupcache module for static resources works. This ensures that when an update is applied, we don’t accidentally load old assets from the cache. So the first time you launch Firefox after you apply an update will not pull the about:home document from the cache, even if one exists (and will throw the cache out if it does). For Nightly users that generally receive updated builds twice a day, this makes the cache somewhat useless. Beta and Release users update much less frequently, so we expect to see a greater impact there.
The last case is in the event that your disk was in a situation such that reading the dynamic code from the asset bundle was faster than reading from the cache. If by the time the about:home document attempts to load and the cache isn’t ready, we fall back to loading it the old way. We don’t expect this to happen too often, but it’s theoretically possible, so we handle the case.
Future work
The next major step is to get the about:home startup cache turned on by default on Nightly and get it tested by a broader audience. At that point, hopefully we’ll get a better sense of its behaviour out in the wild via bug reports and Telemetry. Then our improvement will either ride the release train, or we might turn it on for subsets of the Beta or Release populations to measure its impact on more realistic restart scenarios. Once we’re confident that it’s helping more than hindering, we’ll turn it on by default for everyone.
After that, I think it would be worth seeing if we can load from the cache more often. Perhaps we could load about:newtab from there as well, for example.
One step at a time!
Thanks to
Florian Quèze and Doug Thayer, both of whom independently approached me with the idea of creating a static about:home
Jay Lim and Ursula Sarracini, both of whom wrote some of the groundwork code that was needed for this feature (namely, the infrastructure for the privileged about content process, and the first version of the moz-page-thumbs support)
Gijs Kruitbosch, Kate Hudson, Ed Lee, Scott Downe, and Gavin Suntop for all of the consulting and reviews
Honza Bambas and Andrew Sutherland for storage consultations
Markus Stange, Andrew Creskey, Eric Smyth, Asif Youssuff, Dan Mosedale, Emily Derr for their generous feedback on this post
This is only true if the user hasn’t just restarted after applying an update, and if they haven’t set a custom home page or configured Firefox to restore their previous session on start. ↩
You can think of startup like a traveling circus coming to town. You have to get the trucks and trailers parked, get the tents set up, hook up power, then lighting and sound … it’s a big, complex operation, and we haven’t even shot a clown out of a cannon yet. ↩
As the user browses, bookmarks and downloads things, their Highlights and Top Sites sections might change. If Pocket is enabled, new stories will also be downloaded periodically. ↩
It’s vitally important that content processes have limited abilities. That way, if they’re ever compromised by a bad actor, there are limits to what damage they can do. The assertions mentioned in this case mean that if a compromised content process tries to “pretend” to be the privileged about content process by sending one of its messages, that the parent process will terminate that content process immediately. ↩
This has changed slightly in the past few years with a feature called Race Cache With Network, which races the disk cache with the network instead of relying on the disk entirely. ↩
Thunderbird Conversations is an add-on for Thunderbird that provides a conversation view for messages. It groups message threads together, including those stored in different folders, and allows easier reading and control for a more efficient workflow.
Over the last couple of years, Conversations has been largely rewritten to adapt to changes in Thunderbird’s architecture for add-ons. Conversations 3.1 is the result of that effort so far.
<figcaption>Message Controls Menu</figcaption>
The new version will work with Thunderbird 68, and Thunderbird 78 that will be released soon.
<figcaption>Attachment preview area with gallery view available for images.</figcaption>
The one feature that is currently missing after the rewrite is inline quick reply. This has been of lower priority, as we have focussed on being able to keep the main part of the add-on running with the newer versions of Thunderbird. However, now that 3.1 is stable, I hope to be able to start work on a new version of quick reply soon.
More rewriting will also be continuing for the foreseeable future to further support Thunderbird’s new architecture. I’m planning a more technical blog post about this in future.
If you find an issue, or would like to help contribute to Conversations’ code, please head over to our GitHub repository.
Last year, I worked with the Capacity team on the Crypto stamp project, the first physical postage stamp with a unique digital twin, issued by the Austrian Postal Service (Österreichische Post AG). Those stamps are mainly intended as collectibles, but their physical "half" can be used as valid postage on packages or letters, and a QR code on that physical stamp links to a website presenting the digital collectible. Our job (at Capacity Blockchain Solutions) was to build that digital collectible, the website at crypto.post.at, and the back-end service delivering both public meta data and the back end for the website. I specifically did most of the work on the EthereumSmart Contract for the digital collectible, a "non-fungible token" (NFT) using the ERC-721 standard (publicly visible), as well as the back-end REST service, which I implemented in Python (based on Flask and Web3.py). The coding for the website was done by colleagues, of course using JavaScript for the dynamic elements.
One feature we have in this project is that people can purchase Crypto stamps directly from the blockchain, with the website guiding those with an Ethreum-enabled browser (e.g. with the MetaMask add-on) through that. By sending Ether cryptocurrency to the right address (the OnChainShop contract), they will directly receive the digital NFT - but then, every Crypto stamp consists of both a digital and physical item, so what about the physical part?
Of course, we cannot send a physical item to an Ethereum address (which is just a mostly-random number) so we needed a way for the owner of the NFT to give us (or actually Post AG) a postal address to send the physical stamp to. For this, we added a form to allow them to enter the postal address for stamps that were bought via the OnChain shop - but then the issue arose of how would we would verify that the sender was the actual owner of the NFT. Additionally, we had to figure out how do we do this without requiring a separate database or authentication in the back end, as we also did not need those features for anything else, since authentication for purchases are already done via signed transactions on the blockchain, and any data that needs to be stored is either static or on the blockchain.
We can easily verify the ownership if we send the information to a Smart Contract function on the blockchain, given that the owner has proven to be able to do such calls by purchasing via the OnChain shop already, and anyone sending transactions there has to sign those. To not need to store the whole postage address in the blockchain state database, which is expensive, we just emit an event and therefore put it in the event log, which is much cheaper and can still be read by our back end service and forwarded to Post AG. But then, anything sent to the public Ethereum blockchain (no matter if we put it into state or logs afterwards) is also visible to everyone else, and postal address are private data, so we need to ensure others reading the message cannot actually read that data.
So, our basic idea sounded simple: We generate a public/private key pair, use the public key to encrypt the postage address on the website, call a Smart Contract function with that data, signed by the user, emit an event with the data, and decrypt the information on the back-end service when receiving the event, before forwarding it to the actual shipping department in a nice format. As someone who has heard a lot about encryption but not actually coded encryption usage, I was surprised how many issues we ran into when actually writing the code.
So, first thing I did was seeing what techniques there are for sending encrypted messages, and pretty soon I found ECIES and was enthusiastic that sending encrypted messages was standardized, there are libraries for this in many languages and we just need to use implementations of that standard on both sides and it's all solved! Yay!
So I looked for ECIES libraries, both for JavaScript to be used in the browser and for Python, making sure they are still maintained. After some looking, I settled for eccrypto (JS) and eciespy, which both sounded pretty decent in usage and being kept up to date. I created a private/public key pair, trying to encrypt back and forth via eccrypto worked, so I went for trying to decrypt via eciespy, with great hope - only to see that eccrypto.encrypt() results in an object with 4 member strings while eciespy expects a string as input. Hmm.
With some digging, I found out that ECIES is not the same as ECIES. Sigh. It's a standard in terms of providing a standard framework for encrypting messages but there are multiple variants for the steps in the standardized mechanism, and both sides (encryption and decryption) need to agree on using the same to make it work correctly. Now, both eccrypto and eciespy implement exactly one variant, and of course two different ones, of course. Things would have been too easy if the implementations would be compatible, right?
So, I had to unpack what ECIES does to understand better what happens there. For one thing, ECIES basically does an ECDH exchange with the receiver's public key and a random "ephemeral" private key to derive a shared secret, which is then used as the key for AES-encrypting the message. The message is sent over to the recipient along with the AES parameters (IV, MAC) and the "ephemeral" public key. The recipient can use that public key along with their private key in ECDH, get the same shared secret, and do another round of AES with the given parameters to decrypt (as AES is symmetric, i.e. encryption and decryption are the same operation).
While both libraries use the secp256k1 curve (which incidentally is also used by Ethereum and Bitcoin) for ECDH, and both use AES-256, the main difference there, as I figured, is the AES cipher block mode - eccrypto uses CBC while eciespy uses GCM. Both modes are fine for what we are doing here, but we need to make sure we use the same on both sides. And additional difference is that eccrypto gives us the IV, MAC, ciphertext, and ephemeral public key as separate values while eciespy expects them packed into a single string - but that would be easier to cope with.
In any case, I would need to change one of the two sides and not use the simple-to-use libraries. Given that I was writing the Python code while my collegues working on the website were already busy enough with other feature work needed there, I decided that the JavaScript-side code would stay with eccrypto and I'd figure out the decoding part on the Python side, taking apart and adapting the steps that ecies would have done.
We'd convert the 4 values returned from eccrypto.encrypt() to hex strings, stick them into a JSON and stringify that to hand it over to the blockchain function - using code very similar to this:
var data = JSON.stringify(addressfields);
var eccrypto = require("eccrypto");
eccrypto.encrypt(pubkey, Buffer(data))
.then((encrypted) => {
var sendData = {
iv: encrypted.iv.toString("hex"),
ephemPublicKey: encrypted.ephemPublicKey.toString("hex"),
ciphertext: encrypted.ciphertext.toString("hex"),
mac: encrypted.mac.toString("hex"),
};
var finalString = JSON.stringify(sendData);
// Call the token shipping function with that final string.
OnChainShopContract.methods.shipToMe(finalString, tokenId)
.send({from: web3.eth.defaultAccount}).then(...)...
};
So, on the Python side, I went and took the ECDH bits from eciespy, and by looking at eccrypto code as an example and the relevant Python libraries, implemented code to make AES-CBC work with the data we get from our blockchain event listener. And then I found out that it still did not work, as I got garbage out instead of the expected result. Ouch. Adding more debug messages, I realized that the key used for AES was already wrong, so ECDH resulted in the wrong shared secret. Now I was really confused: Same elliptic curve, right public and private keys used, but the much-proven ECDH algorithm gives me a wrong result? How can that be? I was fully of disbelief and despair, wondering if this could be solved at all.
But I went for web searches trying to find out why in the world ECDH could give different results on different libraries that all use the secp256k1 curve. And I found documents ofthat same issue. And it comes down to this: While standard ECDH returns the x coordinate of the resulting point, the libsecp256k1 developers (I believe that's a part of the Bitcoin community) found it would be more secure to instead return the SHA256 hash of both coordinates of that point. This may be a good idea when everyone uses the same library, but eccrypto uses a standard library while eciespy uses libsecp256k1 - and so they disagree on the shared secret, which is pretty unhelpful in our case.
In the end, I also replaced the ECDH pieces from eciespy with equivalent code using a standard library - and suddenly things worked! \o/
I was fully of joy, and we had code we could use for Crypto stamp - and since the release in June 2019, this mechanism has been used successfully for over a hundred shipments of stamps to postal addresses (note that we had a limited amount available in the OnChainShop).
So, here's the Python code used for decrypting (we pip install eciespy cryptography in our virtualenv - not sure if eciespy is still needed but it may for dependencies we end up using):
from Crypto.Cipher import AES
import hashlib
import hmac
from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.hazmat.backends import default_backend
def ecies_decrypt(privkey, message_parts):
# Do ECDH via the cryptography module to get the non-libsecp256k1 version.
sender_public_key_obj = ec.EllipticCurvePublicNumbers.from_encoded_point(ec.SECP256K1(), message_parts["ephemPublicKey"]).public_key(default_backend())
private_key_obj = ec.derive_private_key(Web3.toInt(hexstr=privkey),ec.SECP256K1(), default_backend())
aes_shared_key = private_key_obj.exchange(ec.ECDH(), sender_public_key_obj)
# Now let's do AES-CBC with this, including the hmac matching (modeled after eccrypto code).
aes_keyhash = hashlib.sha512(aes_shared_key).digest()
hmac_key = aes_keyhash[32:]
test_hmac = hmac.new(hmac_key, message_parts["iv"] + message_parts["ephemPublicKey"] + message_parts["ciphertext"], hashlib.sha256).digest()
if test_hmac != message_parts["mac"]:
logger.error("Mac doesn't match: %s vs. %s", test_hmac, message_parts["mac"])
return False
aes_key = aes_keyhash[:32]
# Actual decrypt is modeled after ecies.utils.aes_decrypt() - but with CBC mode to match eccrypto.
aes_cipher = AES.new(aes_key, AES.MODE_CBC, iv=message_parts["iv"])
try:
decrypted_bytes = aes_cipher.decrypt(message_parts["ciphertext"])
# Padding characters (unprintable) may be at the end to fit AES block size, so strip them.
unprintable_chars = bytes(''.join(map(chr, range(0,32))).join(map(chr, range(127,160))), 'utf-8')
decrypted_string = decrypted_bytes.rstrip(unprintable_chars).decode("utf-8")
return decrypted_string
except:
logger.error("Could not decode ciphertext: %s", sys.exc_info()[0])
return False
So, this mechanism has caused me quite a bit of work and you probably don't want to know the word I shouted at my computer at times while trying to figure this all out, but the results works great, and if you are ever in need of something like this, I hope I could shed some light on how to achieve it!
For further illustration, here's a flow graph of how the data gets from the user to Post AG in the end - the ECIES code samples are highlighted with light blue, all encryption-related things are blue in general, red is unencrypted data, while green is encrypted data:
Thanks to Post AG and Capacity for letting me work on interesting projects like that - and keep checking crypto.post.at for news about the next iteration of Crypto stamp!
Ever since I was on a tour to Star Trek filming sites in 2016 with Geek Nation Tours and Larry Nemecek, I've become ever more interested in finding out to which actual real-world places TV/film crews have gone "on location" and shot scenes for our favorite on-screen stories. While the background of production of TV and film is of interest to me in general, I focus mostly on everything Star Trek and I love visiting locations they used and try to catch pictures that recreate the base setting of the shots in the production - but just the way the place looks "in the real world" and right now.
This has gone as far as me doing several presentations about the topic - two of which (one in German, one in English language) I will give at this year's FedCon as well, and creating an experimental website at filmingsites.com where I note all locations used in Star Trek productions as soon as I become aware of them.
In the last few years, around the Star Trek Las Vegas Conventions, I did get the chance to have a few days traveling around Los Angeles and vicinity, visit a few locations and take pictures there. And after Discovery being filmed up in the Toronto area (and generally using quite few locations outside the studios), Picard is back producing in Southern California and using plenty of interesting places! And now with the first half of season 1 in the books (or at least ready to watch for us via streaming), here are a few filming sites I found in those episodes:
And we actually get started with our first location (picture is a still from the series) in "Remembrance" right after Picard wakes up from the "cold open" dream sequence: Château Picard was filmed at Sunstone Winery's Villa this time (after different places were used in its TNG appearances). The Winery's general manager even said "We encourage all the Trekkies and Trekkers to come visit us." - so I guess I'll need to put it in my travels plans soon.
Another one I haven't seen yet but will need to put in my plans to see is One Culver, previously known as Sony Pictures Plaza. That's where the scenes in the Daystrom Institute were shot - interestingly, in walking distance to the location of the former Desilu Culver soundstages (now "The Culver Studios") and its backlot (now a residential area), where the original Star Trek series shot its first episodes and several outdoor scenes of later ones as well. One Culver's big glass front structure and the huge screen on its inside are clearly visible multiple times in Picard's Daystrom Institute scenes, as is the rainbow arch behind it on the Sony Studios parking lot. Not having been there, I could only include a promotional picture from their website here.
Now a third filming site that appears in "Remembrance" is actually one I do have my own pictures of: After seeing the first trailer for Picard and getting a hint where that building depicted that clip is, I made my way last summer to a place close to Disneyland and took a few pictures of Anaheim Convention Center. Walking by to the main entrance, I found the attached Arena to just look good, so I also got one shot of that one in - and then I see that in this episode, they used it as the Starfleet Archive Museum!
Of course, in the second episode, "Maps and Legends", we then see the main entrance, where Picard goes to meet the C-in-C, so presumably Starfleet headquarters. It looks like the roof scenes with Dahj would actually be on the same building, on satellite pictures, there seems to be an area with those stairs South of the main entrance. I'm still a bit sad though that Starfleet seems to have moved their headquarters and it's not the Tillman administration building any more that was used in previous series (actually, for both headquarters and the Academy - so maybe it comes back in some series as the Academy, with its beautiful Japanese garden).
Of course, at the end of this episode we get to Raffi's home, and we stay there for a bit and see more of it in "The End is the Beginning". The description in the episode tells us it's located at a place called "Vasquez Rocks" - and this time, that's actually the real filming site! Now, Trekkies know this of course, as a whole lot of Trek has been filmed there - most famously the fight between Kirk and the Gorn captain in "Arena". Vasquez Rocks has surely been of the most-used Star Trek filming sites over the years, though - at least before Picard - I'd say that it ranked second behind Bronson Canyon. How what's nowadays a Natural Area park becomes a place to live in by 2399 is up to anyone's speculation.
I guess in the 3 introductory episodes we had more different filming sites than in any of the two whole seasons of Discovery seen so far, but right in the next episode, Absolute Candor, we got yet another interesting place! A lot of that episode plays on the planet Vashti, with three sets of scenes on their main place with the bar setting: In the "cold open" / flashback, when Picard beams down to the planet again in the show's present, and before he leaves, including the fight scene. Given that there were multiple hints of shooting taking place at Universal Studios Hollywood, and the sets having a somewhat familiar look, more Mexican than totally alien, it did not take long to identify where those scenes were filmed: It's the standing "Mexican Street" / "Old Mexico Place" set on Universal's backlot - which you usually can visit with the Studio Tour as an attraction of their Theme Park. The pictures, of the bar area, and basically from there in the direction of Picard's beam-in point, are from a one of those tours I took in 2013.
In the following two episodes, I could not make out any filming sites, so I guess they pretty much filmed those at Santa Clarita Studios where the production of the series is based. I know we will have some location(s) to talk about in the second half of the season though - not sure if there's as many as in the first few episodes, but I hope we'll have a few good ones!
I've been meaning to blog again for some time, and just looked in disbelief at the date of my last post. Yes, I'm still around. I hope I get to write more often in the future.
Ludo just posted his thoughts on FOSDEM, which I also attended last weekend as a volunteer for Mozilla. I have been attending this conference since 2002, when it first went by that exact name, and since then AFAIK only missed the 2010 edition, giving talks in the Mozilla dev room almost every year - though funnily enough, in two of the three years where I've been a member of the Mozilla Tech Speakers program, my talks were not accepted into that room, while I made it all the years before. In fact, that's more telling a story of how interested speakers are in getting into this room nowadays, while in the past there were probably fewer submissions in total. So, this year I helped out Sunday's Mozilla developer room by managing the crowd entering/leaving at the door(s), similar to what I did in the last few years, and given that we had fewer volunteers this year, I also helped out at the Mozilla booth on Saturday. Unfortunately, being busy volunteering on both days meant that I did not catch any talks at all at the conference (I hear there were some good ones esp. in our dev room), but I had a number of good hallway and booth conversations with various people, esp. within the Mozilla community - be it with friends I had not seen for a while, new interesting people within and outside of Mozilla, or conversations clearing up lingering questions.
(pictures by Rabimba & Bob Chao)
Now, this was the 20th conference by the FOSDEM team (their first one went by "OSDEM", before they added the "F" in 2002), and the number 20 is coming up for me all over the place - not just that it works double duty in the current year's number 2020, but even in the months before, I started my row of 20-year anniversaries in terms of my Mozilla contributions: first bug reported in May, first contribution contact in December, first German-language Mozilla suite release on January 1, and will will continue with the 20th anniversaries of my first patches to shared code this summer - see 'My Web Story' post from 2013 for more details. So, being part of an Open-Source project with more than 20 years of history, celebrating a number of 20th anniversaries in that community, I see that number popping up quite a bit nowadays. Around the turn of the century/millennium, a lot of change happened, for me personally but all around as well. Since then, it has been a whirlwind, and change is the one constant that really stayed with me and has become almost a good friend. A lot of changes are going on in the Mozilla community right now as well, and after a bit of a slump and trying to find my new place in this community (since I switched back from staff to volunteer in 2016), I'm definitely excited again to try and help building this next chapter of the future with my fellow Mozillians.
There's so much more going around in my mind, but for now I'll leave it at that: In past times, when I was invited as volunteer or staff, the Mozilla Summits and All-hands were points that energized me and gave me motivation to push forward on making Mozilla better. This year, FOSDEM, with my volunteering and the conversations I had, did the same job. Let's build a better Internet and a better Mozilla community!
I’m writing this in lieu of a traditional Firefox Front-end Performance Update, as I think this will be more useful in the long run than just a snapshot of what my team is doing.
I want to talk about main thread disk access (sometimes referred to more generally as “main thread IO”). Specifically, I’m going to argue that main thread disk access is lethal to program responsiveness. For some folks reading this, that might be an obvious argument not worth making, or one already made ad nauseam — if that’s you, this blog post is probably not for you. You can go ahead and skip most or all of it, if you’d like. Or just skim it. You never know — there might be something in here you didn’t know or hadn’t thought about!
For everybody else, scoot your chairs forward, grab a snack, and read on.
Disclaimer: I wouldn’t call myself a disk specialist. I don’t work for Western Digital or Seagate. I don’t design file systems. I have, however, been using and writing software for computers for a significant chunk of my life, and I seem to have accumulated a bunch of information about disks. Some of that information might be incorrect or imprecise. Please send me mail at mike dot d dot conley at gmail dot com if any of this strikes you as wildly inaccurate (though forgive me if I politely disregard pedantry), and then I can update the post.
The mechanical parts of a computer
If you grab a screwdriver and (carefully) open up a laptop or desktop computer, what do you see? Circuit boards, chips, wires and plugs. Lots of electrons flowing around in there, moving quickly and invisibly.
Notably, there aren’t many mechanical moving parts of a modern computer. Nothing to grease up, nowhere to pour lubricant. Opening up my desktop at home, the only moving parts I can really see are the cooling fans near the CPU and power supply (and if you’re like me, you’ll also notice that your cooling fans are caked with dust and in need of a cleaning).
There’s another moving part that’s harder to see — the hard drive. This might not be obvious, because most mechanical drives (I’ve heard them sometimes referred to as magnetic drives, spinny drives, physical drives, platter drives and HDDs. There are probably more terms.) hide their moving parts inside of the disk enclosure.1
If you ever get the opportunity to open one of these enclosures (perhaps the disk has failed or is otherwise being replaced, and you’re just about to get rid of it) I encourage you to.
As you disassemble the drive, what you’ll probably notice are circular parts, layered on top of one another on a motor that spins them. In between those circles are little arms that can move back and forth. This next image shows one of those circles, and one of those little arms.
<figcaption>There are several of those circles stacked on top of one another, and several of those arms in between them. We’re only seeing the top one in this photo.</figcaption>
Does this remind you of anything? The circular parts remind me of CDs and DVDs, but the arms reaching across them remind me of vinyl players.
<figcaption>Vinyl’s back, baby!</figcaption>
The comparison isn’t that outlandish. If you ignore some of the lower-level details, CDs, DVDs, vinyl players and hard drives all operate under the same basic principles:
The circular part has information encoded on it.
An arm of some kind is able to reach across the radius of the circular part.
Because the circular part is spinning, the arm is able to reach all parts of it.
The end of the arm is used to read the information encoded on it.
There’s some extra complexity for hard drives. Normally there’s more than one spinning platter and one arm, all stacked up, so it’s more like several vinyl players piled on top of one another.
Hard drives are also typically written to as well as read from, whereas CDs, DVDs and vinyls tend to be written to once, and then used as “read-only memory.” (Though, yes, there are exceptions there.)
Lastly, for hard drives, there’s a bit I’m skipping over involving caches, where parts of the information encoded on the spinning platters are temporarily held elsewhere for easier and faster access, but we’ll ignore that for now for simplicity, and because it wrecks my simile.2
So, in general, when you’re asking a computer to read a file off of your hard drive, it’s a bit like asking it to play a tune on a vinyl. It needs to find the right starting place to put the needle, then it needs to put the needle there and only then will the song play.
For hard drives, the act of moving the “arm” to find the right spot is called seeking.
Contiguous blocks of information and fragmentation
Have you ever had to defragment your hard drive? What does that even mean? I’m going to spend a few moments trying to explain that at a high-level. Again, if this is something you already understand, go ahead and skip this part.
Most functional hard drives allow you to do the following useful operations:
Write data to the drive
Read data from the drive
Remove data from the drive
That last one is interesting, because usually when you delete a file from your computer, the information isn’t actually erased from the disk. This is true even after emptying your Trash / Recycling Bin — perhaps surprisingly, the files that you asked to be removed are still there encoded on the circular platters as 1’s and 0’s. This is why it’s sometimes possible to recover deleted files even when it seems that all is lost.
Allow me to explain.
Just like there are different ways of organizing a sock drawer (at random, by colour, by type, by age, by amount of damage), there are ways of organizing a hard drive. These “ways” are called file systems. There are lots of different file systems. If you’re using a modern version of Windows, you’re probably using a file system called NTFS. One of the things that a file system is responsible for is knowing where your files are on the spinning platters. This file system is also responsible for knowing where there’s free space on the spinning platters to write new data to.
When you delete a file, what tends to happen is that your file system marks those sectors of the platter as places where new information can be written to, but doesn’t immediately overwrite those sectors. That’s one reason why sometimes deleted files can be recovered.
Depending on your file system, there’s a natural consequence as you delete and write files of different sizes to the hard drive: fragmentation. This kinda sounds like the actual physical disk is falling apart, but that’s not what it means. Data fragmentation is probably a more precise way of thinking about it.
Imagine you have a sheet of white paper broken up into a grid of 5 boxes by 5 boxes (25 boxes in total), and a box of paints and paintbrushes.
Each square on the paper is white to start. Now, starting from the top-left, and going from left-to-right, top-to-bottom, use your paint to fill in 10 of those boxes with the colour red. Now use your paint to fill in the next 5 boxes with blue. Now do 3 more boxes with yellow.
So we’ve got our colour-filled boxes in neat, organized rows (red, then blue, then yellow), and we’ve got 18 of them filled, and 7 of them still white.
Now let’s say we don’t care about the colour blue. We’re okay to paint over those now with a new colour. We also want to fill in 10 boxes with the colour purple. Hm… there aren’t enough free white boxes to put in that many purple ones, but we have these 5 blue ones we can paint over. Let’s paint over them with purple, and then put the next 5 at the end in the white boxes.
So now 23 of the boxes are filled, we’ve got 2 left at the end that are white, but also, notice that the purple boxes aren’t all together — they’ve been broken apart into two sections. They’ve been fragmented.
This is an incredibly simplified model, but (I hope) it demonstrates what happens when you delete and write files to a hard drive. Gaps open up that can be written to, and bits and pieces of files end up being distributed across the platters as fragments.
This also occurs as files grow. If, for example, we decided to paint two more white boxes red, we’d need to paint the ones at the very end, breaking up the red boxes so that they’re fragmented.
So going back to our vinyl player example for a second — the ideal scenario is that you start a song at the beginning and it plays straight through until the end, right? The more common case with disk drives, however, is you read bits and pieces of a song from different parts of the vinyl: you have to lift and move the arm each time until eventually you have heard the song from start to finish. That seeking of the arm adds overhead to the time it takes to listen to the song from beginning to end.
When your hard drive undergoes defragmentation, what your computer does is try to re-organize your disk so that files are in contiguous sectors on the platters. That’s a fancy way of saying that they’re all in a row on the platter, so they can be read in without the overhead of seeking around to assemble it as fragments.
Skipping that overhead can have huge benefits to your computer’s performance, because the disk is usually the slowest part of your computer.
On the relative input / output speeds of modern computing components
I mentioned in the disclaimer at the start of this post that I’m not a disk specialist or expert. Scott Davis is probably a better bet as one of those. His bio lists an impressive wealth of experience, and mentions that he’s “a recognized expert in virtualization, clustering, operating systems, cloud computing, file systems, storage, end user computing and cloud native applications.”
I don’t know Scott at all (if you’re reading this, Hi, Scott!), but let’s just agree for now that he probably knows more about disks than I do.
I’m picking Scott as an expert because of a particularly illustrative analogy that was posted to a blog for a company he used to work for. The analogy compares the speeds of different media that can be used to store information on a computer. Specifically, it compares the following:
RAM
The network with a decent connection
Flash drives
Magnetic hard drives — what we’ve been discussing up until now.
For these media, the post claims that input / output speed can be measured using the following units:
RAM is in nanoseconds
10GbE Network speed is in microseconds (~50 microseconds)
Flash speed is in microseconds (between 20-500+ microseconds)
Disk speed is in milliseconds
That all seems pretty fast. What’s the big deal? Well, it helps if we zoom in a little bit. The post does this by supposing that we pretend that RAM speed happens in minutes.
If that’s the case, then we’d have to measure network speed in weeks.
And if that’s the case, then we’d want to measure the speed of a Flash drive in months.
And if that’s the case, then we’d have to measure the speed of a magnetic spinny disk in decades.
Update (May 23, 2019): My Uncle Mark, who also works in computing, sent me links that show similar visualizations of computing latency: this one has a really excellent infographic, and this one has more discussion. These articles highlight network latency as the worst offender, which is true especially when the quality of service is low, but I’m mostly writing this post for folks who hack on Firefox where the vast majority of networking occurs off of the main thread.
I wish I had some ACM paper, or something written by a computer science professor that I could point to you to bolster the following claim. I don’t, not because one doesn’t exist, but because I’m too lazy to look for one. I hope you’ll forgive me for that, but I don’t think I’m saying anything super controversial when I say:
In the common case, for a personal computer, it’s best to assume that reading and writing to the disk is the slowest operation you can perform.
Sure, there are edge cases where other things in the system might be slower. And there is that disk cache that I breezed over earlier that might make reading or writing cheaper. And sometimes the operating system tries to do smart things to help you. For now, just let it go. I’m making a broad generalization that I think covers the common cases, and I’m talking about what’s best to assume.
Single and multi-threaded restaurants
When I try to describe threading and concurrency to someone, I inevitably fall back to the metaphor of cooks in a kitchen in a restaurant. This is a special restaurant where there’s only one seat, for a single customer — you, the user.
Single-threaded programs
Let’s imagine a restaurant that’s very, very small and simple. In this restaurant, the cook is also acting as the waiter / waitress / server. That means when you place your order, the server / cook goes into the kitchen and makes it for you. While they’re gone, you can’t really ask for anything else — the server / cook is busy making the thing you asked for last.
This is how most simple, single-threaded programs work—the user feeds in requests, maybe by clicking a button, or typing something in, maybe something else entirely—and then the program goes off and does it and returns some kind of result. Maybe at that point, the program just exits (“The restaurant is closed! Come back tomorrow!”), or maybe you can ask for something else. It’s really up to how the restaurant / program is designed that dictates this.
Suppose you’re very, very hungry, and you’ve just ordered a complex five-course meal for yourself at this restaurant. Blanching, your server / cook goes off to the kitchen. While they’re gone, nobody is refilling your water glass or giving you breadsticks. You’re pretty sure there’s activity going in the kitchen and that the server / cook hasn’t had a heart attack back there, but you’re going to be waiting a looooong time since there’s only one person working in this place.
Maybe in some restaurants, the server / cook will dash out periodically to refill your water glass, give you some breadsticks, and update you on how things are going, but it sure would be nice if we gave this person some help back there, wouldn’t it?
Multi-threaded programs
Let’s imagine a slightly different restaurant. There are more cooks in the kitchen. The server is available to take your order (but is also able to cook in the kitchen if need be), and you make your request from the menu.
Now suppose again that you order a five-course meal. The server goes to the kitchen and tells the cooks what you just ordered. In this restaurant, suppose the kitchen staff are a really great team and don’t get in each other’s way3, so they divide up the order in a way that makes sense and get to work.
The server can come back and refill your water glass, feed you breadsticks, perhaps they can tell you an entertaining joke, perhaps they can take additional orders that won’t take as long. At any rate, in this restaurant, the interaction between the user and the server is frequent and rarely interrupted.
The waiter / waitress / server is the main thread
In these two examples, the waiter / waitress / server is what is usually called the main thread of execution, which is the part of the program that the user interacts with most directly. By moving expensive operations off of the main thread, the responsiveness of the program increases.
Have you ever seen the mouse turn into an hourglass, seen the “This program is not responding” message on Windows? Or the spinny colourful pinwheel on macOS? In those cases, the main thread is off doing something and never came back to give you your order or refill your water or breadsticks — that’s how it generally manifests in common operating systems. The program seems “unresponsive”, “sluggish”, “frozen”. It’s “hanging”, or “stuck”. When I hear those words, my immediate assumption is that the main thread is busy doing something — either it’s taking a long time (it’s making you your massive five course meal, maybe not as efficiently as it could), or it’s stuck (maybe they fell down a well!).
In either case, the general rule of thumb to improving program responsiveness is to keep the server filling the user’s water and breadsticks by offloading complex things on the menu to other cooks in the kitchen.
Accessing the disk on the main thread
Recall that in the common case, for a personal computer, it’s best to assume that reading and writing to the disk is the slowest operation you can perform. In our restaurant example, reading or writing to the disk on the main thread is a bit like having your server hop onto their bike and ride out to the next town over to grab some groceries to help make what you ordered.
And sometimes, because of data fragmentation (not everything is all in one place), the server has to search amongst many many shelves all widely spaced apart to get everything.
And sometimes the grocery store is very busy because there are other restaurants out there that are grabbing supplies.
And sometimes there are police checks (anti-virus / anti-malware software) occurring for passengers along the road, where they all have to show their IDs before being allowed through.
It’s an incredibly slow operation. Hopefully by the time the server comes back, they don’t realize they have to go back out again to get more, but they might if they didn’t realize they were missing some more ingredients.4
Slow slow slow. And unresponsive. And a great way to lose a hungry customer.
For super small programs, where the kitchen is well stocked, or the ride to the grocery store doesn’t need to happen often, having a single-thread and having it read or write is usually okay. I’ve certainly written my fair share of utility programs or scripts that do main thread disk access.
Firefox, the program I spend most of my time working on as my job, is not a small program. It’s a very, very, very large program. Using our restaurant model, it’s many large restaurants with many many cooks on staff. The restaurants communicate with each other and ship food and supplies back and forth using messenger bikes, to provide to you, the customer, the best meals possible.
But even with this large set of restaurants, there’s still only a single waiter / waitress / server / main thread of execution as the point of contact with the user.
Part of my job is to help organize the workflows of this restaurant so that they provide those meals as quickly as possible. Sending the server to the grocery store (main thread disk access) is part of the workflow that we absolutely need to strike from the list.
Start-up main-thread disk access
Going back to our analogy, imagine starting the program like opening the restaurant. The lights go on, the chairs come off of the tables, the kitchen gets warmed up, and prep begins.
While this is occurring, it’s all hands on deck — the server might be off in the kitchen helping to do prep, off getting cutlery organized, whatever it takes to get the restaurant open and ready to serve. Before the restaurant is open, there’s no point in having the server be idle, because the customer hasn’t been able to come in yet.
So if critical groceries and supplies needed to open the restaurant need to be gotten before the restaurant is open, it’s fine to send the server to the store. Somebody has to do it.
For Firefox, there are various things that need to take place before we can display any UI. At that point, it’s usually fine to do main-thread disk access, so long as all of the things being read or written are kept to an absolute minimum. Find how much you need to do, and reduce it as much as possible.
But as soon as UI is presented to the user, the restaurant is open. At that point, the server should stay off their bike and keep chatting with the customer, even if the kitchen hasn’t finished setting up and getting all of their supplies. So to stay responsive, don’t do disk access on the main thread of execution after you’ve started to show the user some kind of UI.
Disk contention
There’s one last complication I want to capture here with our restaurant example before I wrap up. I’ve been saying that it’s important to send anyone except the server to the grocery store for supplies. That’s true — but be careful of sending too many other people at the same time.
Moving disk access off of the main thread is good for responsiveness, full stop. However, it might do nothing to actually improve the overall time that it takes to complete some amount of work. Put it another way: just because the server is refilling your glass and giving you breadsticks doesn’t mean that your five-course meal is going to show up any faster.
Also, disk operations on magnetic drives do not have a constant speed. Having the disk do many things at once within a single program or across multiple programs can slow the whole set of operations down due to the overhead of seeking and context switching, since the operating system will try to serve all disk requests at once, more or less.5
Disk contention and main thread disk access is something I think a lot about these days while my team and I work on improving Firefox start-up performance.
Some questions to ask yourself when touching disk
So it’s important to be thoughtful about disk access. Are you working on code that touches disk? Here are some things to think about:
Is UI visible, and responsiveness a goal?
If so, best to move the disk access off of the main-thread. That was the main thing I wanted to capture, and I hope I’ve convinced you of that point by now.
Does the access need to occur?
As programs age and grow and contributors come and go, sometimes it’s important to take a step back and ask, “Are the assumptions of this disk access still valid? Does this access need to happen at all?” The fastest code is the code that doesn’t run at all.
What else is happening during this disk access? Can disk access be prioritized more efficiently?
This is often trickier to answer as a program continues to run. Thankfully, tools like profilers can help capture recordings of things like disk access to gain evidence of simultaneous disk access.
Start-up is a special case though, since there’s usually a somewhat deterministic / reliably stable set of operations that occur in the same way in roughly the same order during start-up. For start-up, using a tool like a profiler, you can gain a picture of the sorts of things that tend to happen during that special window of time. If you notice a lot of disk activity occurring simultaneously across multiple threads, perhaps ponder if there’s a better way of ordering those operations so that the most important ones complete first.
Can we reduce how much we need to read or write?
There are lots of wonderful compression algorithms out there with a variety of performance characteristics that might be worth pondering. It might be worth considering compressing the data that you’re storing before writing it so that the disk has to write less and read less.
Of course, there’s compression and decompression overhead to consider here. Is it worth the CPU time to save the disk time? Is there some other CPU intensive task that is more critical that’s occurring?
Can we organize the things that we want to read ahead of time so that they’re more likely to be read contiguously (without seeking the disk)?
If you know ahead of time the sorts of things that you’re going to be reading off of the disk, it’s generally a good strategy to store them in that read order. That way, in the best case scenario (the disk is defragmented), the read head can fly along the sectors and read everything in, in exactly the right order you want them. If the user has defragmented their disk, but the things you’re asking for are all out of order on the disk, you’re adding overhead to seek around to get what you want.
Supposing that the data on the disk is fragmented, I suspect having the files in order anyways is probably better than not, but I don’t think I know enough to prove it.
Flawed but useful
One of my mentors, Greg Wilson, likes to say that “all models are flawed, but some are useful”. I don’t think he coined it, but he uses it in the right places at the right times, and to me, that’s what counts.
The information in this post is not exhaustive — I glossed over and left out a lot. It’s flawed. Still, I hope it can be useful to you.
Thanks
Thanks to the following folks who read drafts of this and gave feedback:
Mandy Cheang
Emily Derr
Gijs Kruitbosch
Doug Thayer
Florian Quèze
There are also newer forms of disks called Flash disks and SSDs. I’m not really going to cover those in this post. ↩
The other thing to keep in mind is that the disk cache can have its contents evicted at any time for reasons that are out of your control. If you time it right, you can maybe increase the probability of a file you want to read being in the cache, but don’t bet the farm on it. ↩
When writing multi-threaded programs, this is much harder than it sounds! Mozilla actually developed a whole new programming language to make that easier to do correctly. ↩
Keen readers might notice I’m leaving out a discussion on Paging. That’s because this blog post is getting quite long, and because it kinda breaks the analogy a bit — who sends groceries back to a grocery store? ↩
I’ve never worked on an operating system, but I believe most modern operating systems try to do a bunch of smart things here to schedule disk requests in efficient ways. ↩
Hello, folks. I wanted to give a quick update on what the Firefox Front-end Performance team is up to, so let’s get into it.
The name of the game continues to be start-up performance. We made some really solid in-roads last quarter, and this year we want to continue to apply pressure. Specifically, we want to focus on reducing IO (specifically, main-thread IO) during browser start-up.
Reducing main thread IO during start-up
There are lots of ways to reduce IO – in the best case, we can avoid start-up IO altogether by not doing something (or deferring it until much later). In other cases, when the browser might be servicing events on the main thread, we can move IO onto another thread. We can also re-organize, pack or compress files differently so that they’re read off of the disk more efficiently.
If you want to change something, the first step is measuring it. Thankfully, my colleague Florian has written a rather brilliant test that lets us take accounting of how much IO is going on during start-up. The test is deterministic enough that he’s been able to write a whitelist for the various ways we touch the disk on the main thread during start-up, and that whitelist means we’ve made it much more difficult for new IO to be introduced on that thread.
That whitelist has been processed by the team, and have been turned into bugs, bucketed by the start-up phasewheretheIO is occurring. The next step is to estimate the effort and potential payoff of fixing those bugs, and then try to whittle down the whitelist.
And that’s effectively where we’re at. We’re at the point now where we’ve got a big list of work in front of us, and we have the fun task of burning that list down!
Being better at loading DLLs on Windows
While investigating the warm-up service for Windows, Doug Thayer noticed that we were loading DLLs during start-up oddly. Specifically, using a tool called RAMMap, he noticed that we were loading DLLs using “read ahead” (eagerly reading the entirety of the DLL into memory) into a region of memory marked as not-executable. This means that anytime we actually wanted to call a library function within that DLL, we needed to load it again into an executable region of memory.
Doug also noticed that we were unnecessarily doing ReadAhead for the same libraries in the content process. This wasn’t necessary, because by the time the content process wanted to load these libraries, the parent process would have already done it and it’d still be “warm” in the system file cache.
We’re not sure why we were doing this ReadAhead-into-unexecutable-memory work – it’s existence in the Firefox source code goes back many many years, and the information we’ve been able to gather about the change is pretty scant at best, even with version control. Our top hypothesis is that this was a performance optimization that made more sense in the Windows XP era, but has since stopped making sense as Windows has evolved.
UPDATE: Ehsan pointed us to this bug where the change likely first landed. It’s a long and wind-y bug, but it seems as if this was indeed a performance optimization, and efforts were put in to side-step effects from Prefetch. I suspect that later changes to how Prefetch and SuperFetch work ultimately negated this optimization.
Doug hacked together a quick prototype to try loading DLLs in a more sensible way, and the he was able to capture quite an improvement in start-up time on our reference hardware:
<figcaption>This graph measures various start-up metrics. The scatter of datapoints on the left show the “control” build, and they tighten up on the right with the “test” build. Lower is better. </figcaption>
At this point, we all got pretty excited. The next step was to confirm Doug’s findings, so I took his control and test builds, and tested them independently on the reference hardware using frame recording. There was a smaller1, but still detectable improvement in the test build. At this point, we decided it was worth pursuing.
We’re also seeing the impact reflected in Telemetry. The first Nightly build with Doug Thayer’s patch went out on April 14th, and we’re starting to see a nice dip in some of our graphs here:
<figcaption>This graph measures the time at which the browser window reports that it has first painted. April 14th is the second last date on the X axis, and the Y axis is time. The top-most line is plotting the 95th percentile, and there’s a nice dip appearing around April 14th. </figcaption>
We expect this improvement to have the greatest impact on weaker hardware with slower disks, but we’ll be avoiding some unnecessary work for all Windows users, and that gets a thumbs-up in my books.
If all goes well, this fix should roll out in Firefox 68, which reaches our release audience on July 9th!
My test machine has SuperFetch disabled to help reduce noise and inconsistency with start-up tests, and we suspect SuperFetch is able to optimize start-up better in the test build ↩
With Firefox 67 only a few short weeks away, I thought it might be interesting to take a step back and talk about some of the work that the Firefox Front-end Performance team is shipping to users in that particular release.
To be clear, this is not an exhaustive list of the great performance work that’s gone into Firefox 67 – but I picked a few things that the front-end team has been focused on to talk about.
Stop loading things we don’t need right away
The fastest code is the code that doesn’t run at all. Sometimes, as the browser evolves, we realize that there are components that don’t need to be loaded right away during start-up, and can instead of deferred until sometime after start-up. Sometimes, that means we can wait until the very last moment to initialize some component – that’s called loading something lazily.
Here’s a list of things that either got deferred until sometime after start-up, or made lazy:
The Hidden Window is a mysterious chunk of code that manages the state of the global menu bar on macOS when there are no open windows. The Hidden Window is also sometimes used as a singleton DOM window where various operations can take place. On Linux and Windows, it turns out we were creating this Hidden Window far early than needs be, and now it’s quite a bit lazier.
Page style
Page Style is a menu you can find under View in the main menu bar, and it’s used to switch between alternative style sheets on a page. It’s a pretty rarely used feature from what we can tell, but we were scanning pages for their alternative stylesheets far earlier than we needed to. We were also scanning pages that we know don’t have alternative stylesheets, like the about:home / about:newtab page. Now we only scan normal web pages, and we do so only after we service the idle event queue.
Cache invalidation
The Startup Cache is an important part of Firefox startup performance. It’s primary job is to cache computations that occur during each startup so that they only have to happen every once in a while. For example, the mark-up of the browser UI often doesn’t change from startup to startup, so we can cache a version of the mark-up that’s faster to read from disk, and only invalidate that during upgrades.
We were invalidating the whole startup cache every time a WebExtension was installed or uninstalled. This used to be necessary for old-style XUL add-ons (since those could cause changes to the UI that would need to go into the cache), but with those add-ons no longer available, we were able to remove the invalidation. This means faster startups more often.
Don’t touch the disk
The disk is almost always the slowest part of the system. Reading and writing to the disk can take a long time, especially on spinning magnetic drives. The less we can read and write, the better. And if we’re going to read, best to do it off of the main thread so that the UI remains responsive.
Old XUL icons code
We were reading from the disk on the main thread to search for window-specific icons to display in the window titlebar.
We noticed that when we were checking that a directory exists on Windows (to write a file to it), we were using the CreateDirectoryW Windows API. This API checks each folder on the way down to the last one to see if they exist. That’s a lot of disk IO! We can avoid this if we assume that the parent directories exist, and only fall into the slow path if we fail to write our file. This means that we hit the faster path with less IO more often, which is good for responsiveness and start-up time.
Enjoy your Faster Fox!
Firefox 67 is slated to ship with these improvements on May 14th – just a little over a month away. Enjoy!
Firefox 66 has been released, Firefox 67 is out on the beta channel, and Firefox 68 is cooking for the folks on the Nightly channel! These trains don’t stop!
With that, let’s take a quick peek at what the Firefox Front-end Performance team has been doing these past few weeks…
Document Splitting Foundations for WebRender (In-Progress by Doug Thayer)
An impressive set of patches were recently queued to land, which should bring document splitting to WebRender, but in a disabled state. The gfx.webrender.split-render-roots pref is what controls it, but I don’t think we can reap the full benefits of document splitting until we get retained display lists enabled in the parent process for the UI. I believe, at that point, we can start enabling document splitting, which means that updating the browser UI area will not involve sending updates to the content area for WebRender.
In other WebRender news, it looks like it should be enabled by default for some of our users on the release channel in Firefox 67, due to be released in mid-May!
Warm-up Service (In-Progress by Doug Thayer)
Doug has written the bits of code that tie a Firefox preference to an HKLM registry key, which can be read by the warm-up service at start-up. The next step is to add a mode to the Firefox executable that loads its core DLLs and then exits, and then have the warm-up service call into that mode if enabled.
Once this is done, we should be in a state where we can user test this feature.
Startup Cache Telemetry (In-Progress by Doug Thayer)
Two things of note here:
With the probes having now uplifted to Beta, data will slowly trickle in these next few days that will show us how the Firefox startup cache is behaving in the wild for users that aren’t receiving two updates a day (like our Nightly users). This important, because oftentimes, those updates cause some or all of the startup cache to be invalidated. We’re eager to see how the startup caches are behaving in the wild on Beta.
One of the tests that was landed for the startup cache Telemetry appears to have caught an issue with how the QuantumBar code works with it – this is useful, because up until now, we’ve had very little automated testing to ensure that the startup cache is working as expected.
Smoother Tab Animations (Paused by Felipe Gomes)
UX, Product and Engineering have been having discussions about how the new tab animations work, and one thing has been decided upon: we want our User Research team to run some studies to see how tab animations are perceived before we fully commit to changing one of the fundamental interactions in the browser. So, at this time, Felipe is pausing his efforts here until User Research comes back with some information on guidance.
Browser Adjustment Project (Concluded by Gijs Kruitbosch)
We originally set out to see whether or not we could do something for users running weaker hardware to improve their browsing experience. Our initial hypothesis was that by lowering the frame rate of the browser on weaker hardware, we could improve the overall page load time.
This hypothesis was bolstered by measurements done in late 2018, where it appeared that by manually lowering the frame rate on a weaker reference laptop, we could improve our internal page load benchmarks by a significant degree. This measurement was reproduced by Denis Palmeiro on Vicky Chin’s team, and so Gijs started implementing a runtime detection mechanism to do that lowering of the frame rate for machines with 2 or fewer cores where each core’s clockspeed was 1.8Ghz or slower1.
However, since then, we’ve been unable to reproduce the same positive effect on page load time. Neither has Denis. We suspect that recent work on the RefreshDriver, which changes how often the RefreshDriver runs during the page load window, is effectively getting the same kind of win2.
We did one final experiment to see whether or not lowering the frame rate would improve battery life, and it appeared to, but not to a very high degree. We might revisit that route were we tasked with trying to improve power usage in Firefox.
Better about:newtab Preloading (Completed by Gijs Kruitbosch)
The patch to preload about:newtab in an idle callback has landed and stuck! This means that we don’t preload about:newtab immediately after opening a new tab (which is good for responsiveness right around the time when you’re likely to want to do something), and also means that we have the possibility of preloading the first new tab in new windows! Great job, Gijs!
Experiments with the Process Priority Manager (In-Progress by Mike Conley)
I had a meeting today with Saptarshi, one of our illustrious Data Scientists, to talk about the upcoming experiment. One of the things he led me to conclude was that this experiment is going to have a lot of confounds, and it will be difficult to conclude things from.
Part of the reason for that is because there are often times when a background tab won’t actually have its content process priority lowered. The potential reasons for this are:
The tab is running in a content process which is also hosting a tab that is running in the foreground of either the same or some other browser window.
The tab is playing audio or video.
Because of this, we can’t actually do things like measure how page load is being impacted by this feature because we don’t have a great sense of how many tabs have their content process priorities lowered. That’s just not a thing we collect with Telemetry. It’s theoretically possible, either due to how many windows or videos or tabs our Beta users have open, that very few of them will ever actually have their content process priorities lowered, and then the data we’d draw from Telemetry would be useless.
I’m working with Saptarshi now to try to find ways of either altering the process priority manager or adding new probes to reduce the number of potential confounds.
We’re only a few weeks away from Firefox 67 merging from the Nightly channel to Beta, and since my last update, a number of things have landed.
It’s the end of a long week for me, so I apologize for the brevity here. Let’s check it out!
Document Splitting Foundations for WebRender (In-Progress by Doug Thayer)
dthayer is still trucking along here – he’s ironed out a number of glitches, and kats is giving feedback on some APZ-related changes. dthayer is also working on a WebRender API endpoint for generating frames for multiple documents in a single transaction, which should help reduce the window of opportunity for nasty synchronization bugs.
Warm-up Service (In-Progress by Doug Thayer)
dthayer is pressing ahead with this experiment to warm up a number of critical files for Firefox shortly after the OS boots. He is working on a prototype that can be controlled via a pref that we’ll be able to test on users in a lab-setting (and perhaps in the wild as a SHIELD experiment).
Startup Cache Telemetry (In-Progress by Doug Thayer)
dthayer landed this Telemetry early in the week, and data has started to trickle in. After a few more days, it should be easier for us to make inferences on how the startup caches are operating out in the wild for our Nightly users.
Smoother Tab Animations (In-Progress by Felipe Gomes)
After a few rounds of landings and backouts, this appears to have stuck! The hidden window is now created after the main window has finished painting, and this has resulted in a nice ts_paint (startup paint) win on our Talos benchmark!
<figcaption>This is a graph of the ts_paint startup paint Talos benchmark. The highlighted node is the first mozilla-central build with the hidden window work. Lower is better, so this looks like a nice win!</figcaption>
Browser Adjustment Project (In-Progress by Gijs Kruitbosch)
This project appears to be reaching its conclusion, but with rather unsatisfying results. Denis Palmeiro from Vicky Chin’s team has done a bunch of testing of both the original set of patches that Gijs landed to lower the global frame rate (painting and compositing) from 60fps to 30fps for low-end machines, as well as the new patches that decrease the frequency of main-thread painting (but not compositing) to 30fps. Unfortunately, this has not yielded the page load wins that we wanted1. We’re still waiting to see if there’s a least a power-usage win here worth pursuing, but we’re almost ready the pull the plug on this one.
Better about:newtab Preloading (In-Progress by Gijs Kruitbosch)
Unfortunately, there’s a small snag with a test failure in automation, but Gijs is on the case.
Experiments with the Process Priority Manager (In-Progress by Mike Conley)
The Process Priority Manager has been enabled in Nightly for a number of weeks now, and no new bugs have been filed against it. I filed a bug earlier this week to run a pref-flip experiment on Beta after the Process Priority Manager patches are uplifted later this month. Our hope is that this has a neutral or positive impact on both page load time and user retention!
Make the PageStyleChild load lazily (Completed by Mike Conley)
There’s an infrequently used feature in Firefox that allows users to switch between different CSS stylesheets that a page might offer. I’ve made the component that scans the document for alternative stylesheets much lazier, and also made it skip non web-pages, which means (at the very least) less code running when loading about:home and about:newtab
This was unexpected – we ran an experiment late in 2018 where we noticed that lowering the frame rate manually via the layout.frame_rate pref had a positive impact on page load time… unfortunately, this effect is no longer being observed. This might be due to other refresh driver work that has occurred in the meantime. ↩
It’s been just a little over two weeks since my last update, so let’s see where we are!
A number of our projects are centered around trying to improve start-up time. Start-up can mean a lot of things, so we’re focused specifically on cold start-up on the Windows 10 2018 reference device when the machine is at rest.
If you want to improve something, the first thing to do is measure it. There are lots of ways to measure start-up time, and one of the ways we’ve been starting to measure is by doing frame recording analysis. This is when we capture display output from a testing device, and then analyze the videos.
This animated GIF shows eight videos. The four on the left are Firefox Nightly, and the four on the right are Google Chrome (71.0.3578.98). The videos are aligned so that both browsers are started at the same time.
<figcaption>The four on the left are Firefox Nightly, and the four on the right are Google Chrome (71.0.3578.98)</figcaption>
Some immediate observations:
Firefox Nightly is consistently faster to reach first paint (that’s the big white window)
Firefox Nightly is consistently faster to draw its toolbar and browser UI
Google Chrome is faster at painting its initial content
This last bullet is where the team will be focusing its efforts – we want to have the initial content painted and settled much sooner than we currently do.
Document Splitting Foundations (In-Progress by Doug Thayer)
After some pretty significant refactorings to work better with APZ, Doug posted a new stack of patches late last week which will sit upon the already large stack of patches that have already landed. There are still a number of reviews pending on the main stack, but this work appears to be getting pretty close to conclusion, as the patches are in the final review and polish stage.
After this, once retained display lists are enabled in the parent process, and an API is introduced to WebRender to generate frames for multiple documents in a single transaction, we can start thinking about enabling document splitting by default.
Warm-up Service (In-Progress by Doug Thayer)
A Heartbeat survey went out a week or so back to get some user feedback about a service that would speed up the launching of Firefox at the cost of adding some boot time to Windows. The responses we’ve gotten back have been quite varied, but can be generally bucketed into three (unsurprising) groups:
Users who say they do not want to make this trade
Users who say they would love to make this trade
Users who don’t care at all about this trade
Each group is sufficiently large to warrant further exploration. Our next step is to build a version of this service that we can turn on and off with a pref and test either in a lab and/or out in the wild with a SHIELD study.
Startup Cache Telemetry (In-Progress by Doug Thayer)
We do a number of things to try to improve real and perceived start-up time. One of those things is to cache things that we calculate at runtime during start-up to the disk, so that for subsequent start-ups, we don’t have to do those calculations again.
There are a number of mechanisms that use this technique, and Doug is currently adding some Telemetry to see how they’re behaving in the wild. We want to measure cache hits and misses, so that we know how healthy our cache system is out in the wild. If we get signals back that our start-up caches are missing more than we expect, this will highlight an important area for us to focus on.
Smoother Tab Animations (In-Progress by Felipe Gomes)
UX has gotten back to us with valuable feedback on the current implementation, and Felipe is going through it and trying to find the simplest way forward to address their concerns.
Having been available (though disabled by default) on Nightly, we’ve discovered one bug where the tab strip can become unresponsive to mouse events. Felipe is currently working on this.
Lazy Hidden Window (In-Progress by Felipe Gomes)
Under the hood, Firefox’s front-end has a notion of a “hidden window”. This mysterious hidden window was originally introduced long long ago1 for MacOS, where it’s possible to close all windows yet keep the application running.
Since then, it’s been (ab)used for Linux and Windows as well, as a safe-ish place to do various operations that require a window (since that window will always be around, and not go away until shutdown).
Browser Adjustment Project (In-Progress by Gijs Kruitbosch)
Gijs updated the patch so that the adjustment causes the main thread to skip every other VSync rather than swithing us to 30fps globally2.
We passed the patch off to Denis Palmeiro, who has a sophisticated set-up that allows him to measure a pageload benchmark using frame recording. Unfortunately, the results we got back suggested that the new approach regressed visual page load time significantly in the majority of cases.
We’re in the midst of using the same testing rig to test the original global 30fps patch to get a sense of the magnitude of any improvements we could get here. Denis is also graciously measuring the newer patch to see if it has any positive benefits towards power consumption.
Better about:newtab Preloading (In-Progress by Gijs Kruitbosch)
By default, users see about:newtab / a.k.a Activity Stream when they open new tabs. One of the perceived performance optimizations we’ve done for many years now is to preload the next about:newtab in the background so that the next time that the user opens a tab, the about:newtab is all ready to roll.
This is a perceived performance optimization where we’re moving work around rather than doing less work.
Right now, we preload a tab almost immediately after the first tab is opened in a window. That means that the first opened tab is never preloaded, but the second one is. This is for historical reasons, but we think we can do better.
Gijs is working on making it so that we choose a better time to preload the tab – namely, when we’ve found an idle pocket of time where the user doesn’t appear to be doing anything. This should also mean that the first new tab that gets opened might also be preloaded, assuming that enough idle time was made available to trigger the preload. And if there wasn’t any idle time, that’s also good news – we never got in the users way by preloading when it’s clear they were busy doing something else
Experiments with the Process Priority Manager (In-Progress by Mike Conley)
The Process Priority Manager has been enabled on Nightly for a few weeks now. Except for a (now fixed) issue where audio playing in background tabs would drop samples periodically, it’s been all quiet for regression reports.
The next step is to file a bug to run an experiment on Beta to see how this work impacts page load time.
Enable the separate Activity Stream content process by default (Stalled by Mike Conley)
This work is temporarily stalled while I work on other things, so there’s not too much to report here.
Well, here I am again – apologizing about a late update. Lots of stuff has been going on performance-wise in the Firefox code-base, and I’ll just be covering a small section of it here.
You might also notice that I changed the title of the blog series from “Firefox Performance Update” to “Firefox Front-end Performance Update”, to reflect that the things the Firefox Front-end Performance team is doing to keep Firefox speedy (though I’ll still add a grab-bag of other performance related work at the end).
So what are we waiting for? What’s been going on?
Migrate consumers to the new Places Observer system (Paused by Doug Thayer)
Doug was working on this later in 2018, and successfully ported a good chunk of our bookmarks code to use the new batched Places Observer system. There’s still a long-tail of other call sites that need to be updated to the new system, but Doug has shifted focus from this to other things in the meantime.
This has been a pretty long-haul task, but Doug has been plugging away, and landed a significant chunk of the infrastructure for this. At this time, Doug is working with kats to make Document Splitting integrate nicely with Async-Pan-Zooming (APZ).
The current plan is for Document Splitting to land disabled by default, since it’s blocked by parent-process retained display lists (which still have a few bugs to shake out).
Warm-up Service (In-Progress by Doug Thayer)
Doug is investigating the practicalities of having a service run during Windows start-up to preload various files that Firefox will need when started.
We’re still researching this at multiple levels, and haven’t yet determined if this is a thing that we’d eventually want to ship. Stay tuned.
Smoother Tab Animations (In-Progress by Felipe Gomes)
After much ado, simplification, and review back-and-forth, the initial set of new tab animations have landed in Nightly. You can enable them by setting browser.tabs.newanimations to true in about:config and then restarting the browser. These new animations run entirely on the compositor, instead of painting at each refresh driver tick, so they should be smoother than the current animations that we ship.
There are still some cases that need new animations, and Felipe is waiting on UX for those.
Overhauling about:performance (V1 Completed by Florian Quèze)
The new about:performance shipped late last year, and now shows both energy as well as memory usage of your tabs and add-ons.
The current iteration allows you to close the tabs that are hogging your resources. Current plans should allow users to pause JavaScript execution in busy background tabs as well.
Browser Adjustment Project (In-Progress by Gijs Kruitbosch)
Gijs has landed some patches in Nightly (which have recently uplifted to Beta, and are only enabled on early Betas), which lowers the default frame rate of Firefox from 60fps to 30fps on devices that are considered “low-end”1.
This has been on Nightly for a while, but as our Nightly population tends to skew to more powerful hardware, we expect not a lot of users have experienced the impact there.
While the lowered frame rate seemed to have a positive impact on page load time in the lab on our “low-end” reference hardware, we’re having a much harder time measuring any appreciable improvement in CI. We have scheduled an experiment to see if improvements are detectable via our Telemetry system on Beta.
We need to be prepared that this particular adjustment will either not have the desired page load improvement, or will result in a poorer quality of experience that is not worth any page load improvement. If that’s the case, we still have a few ideas to try, including:
Lowering the refresh driver tick, rather than the global frame rate. This would mean things like scrolling and videos would still render at 60fps, but painting the UI and web content would occur at a lower frequency.
Use the hardware vsync again (switching to 30fps turns hardware vsync off), but just paint every other time. This is to test whether or not software vsync results in worse page load times than hardware vsync.
Avoiding spurious about:blank loads in the parent process (Completed by Gijs Kruitbosch)
Gijs short-circuited a bunch of places where we were needlessly creating about:blank documents that we were just going to throw away (see this bug and dependencies). There are still a long tail of cases where we still do this in some cases, but they’re not the common cases, and we’ve decided to apply effort for other initiatives in the meantime.
Experiments with the Process Priority Manager (In-Progress by Mike Conley)
This was originally Doug Thayer’s project, but I’ve taken it on while Doug focuses on the epic mountain that is WebRender Document Splitting.
If you recall, the goal of this project is to lower the process priority for tabs that are only sitting in the background. This means that if you have tabs in the background that are attempting to use system resources (running JavaScript for example), those tabs will have less priority at the operating system level than tabs that are in the foreground. This should make it harder for background tabs to cause foreground tabs to be starved of processing resources.
After clearing a few finalblockers, we enabled the Process Priority Manager by default last week. We also filed a bug to keep background tabs at a higher priority if they’re playing audio and video, and the fix for that just landed in Nightly today.
So if you’re on Windows on Nightly, and you’re curious about this, you can observe the behaviour by opening up the Windows Task Manager, switching to the “Details” tab, and watching the “Base priority” reading on your firefox.exe processes as you switch tabs.
Cheaper tabs in titlebar (Completed by Mike Conley)
After an epic round of review (thanks, Dao!), the patches to move our tabs-in-titlebar logic out of JS and into CSS landed late last year.
Enable the separate Activity Stream content process by default (In-Progress by Mike Conley
There’s one known bug remaining that’s preventing us from letting the privileged content process from being enabled by default.
Thankfully, the cause is understood, and a fix is being worked on. Unfortunately, this is one of those bugs where the proper solution involves refactoring a bit of old crufty stuff, so it’s taking longer than I’d like.
Still, if all goes well, this bug should be closed out soon, and we can see about letting the privileged content process ride the trains.
Grab bag of notable performance work
This is an informal list of things that I’ve seen land in the tree lately that I believe will have a positive performance impact for our users. Have you seen something that you’d like to nominate for a future list? Submit the bug here!
Also, keep in mind that some of these landed months ago and already shipped to release. That’s what I get for taking so long to write a blog post.
Gabriele Svelto made it so that the minidump analyzer, which runs after a crash has occurred to produce stack dumps for crash pings, runs with much lower process priority. This means that the CPU is being hogged less after a crash, when you’re likely to want to get right back to what you were doing.
Hey folks – another Performance Update coming at you! It’s been a few weeks since I posted one of these, mostly due to travel, holidays and the Mozilla SF All-Hands. However, we certainly haven’t been idle during that time. Much work has been done Performance-wise, and there’s a lot to tell. So strap in! But first…
This Performance Update is brought to you by: promiseDocumentFlushed
promiseDocumentFlushed is a utility that’s available for browser engineers in chrome documents on the window global. The goal of promiseDocumentFlushed is to help avoid synchronous layout flushes in our JavaScript code by scheduling work to only occur after the next “natural” layout flush occurs1.
promiseDocumentFlushed takes a function and returns a Promise. The function it takes will run the next time a natural layout flush and paint has finished occurring. At this point, the DOM should not be “dirty”, and size and position queries should be very cheap to calculate. It is critically important for the callback to not modify the DOM. I’ve filed bugs to make modifying the DOM inside that callback enter some kind of failure state, but it hasn’t been resolved yet.
The return value of the callback is what promiseDocumentFlushed’s returned Promise resolves with. Once the Promise resolves, it is then safe to modify the DOM.
This mechanism means that if, for some reason, you need to gather information about the size or position of things in the DOM, you can do it without forcing a synchronous layout flush – however, a paint will occur before that information is given to you. So be on the look-out for flicker, since that’s the trade-off here.
And now, here’s a list of the projects that the team has been working on lately:
ClientStorage (In-Progress by Doug Thayer)
The ClientStorage project should allow Firefox to communicate with the GPU more efficiently on macOS, which should hopefully reduce jank on the compositor thread2. This is right on the verge of landing3, and we’re very excited to see how this impacts our macOS users!
Init WindowsJumpLists off-main-thread (Completed by Doug Thayer)
The JumpList is a Windows-only feature – essentially an application-specific context menu that opens when you right-click on the application in the task bar. Adding entries to this context menu involves talking to Windows, and unfortunately, the way we were originally doing this involved writing to the disk on the main thread. Thankfully, the API is thread-safe, so Doug was able to move the operation onto a background thread. This is good, because arewesmoothyet was reporting the Windows JumpList code as one of the primary causes of main-thread hangs caused by our front-end code.
Reduce painting while scrolling panels on macOS (Completed by Doug Thayer)
Matt Woodrow noticed that the recently added All Tabs list was performing quite poorly when scrolling it on macOS. After turning on paint-flashing for our browser UI, he noticed that we were re-painting the entire menu every time it scrolled. After some investigation, Matt realized that this was because our Graphics code was skipping some optimizations due to the rounded corners of the panels on macOS. We briefly considered removing the rounded corners on macOS, but then Doug found a more general fix, and now we only re-paint the minimum necessary to scroll the menu, and it’s much smoother!
Make the RemotePageManager lazy (In-Progress by Felipe Gomes)
Overhauling about:performance (In-Progress by Florian Quèze)
Florian is working on improving about:performance, with the hopes of making it more useful for browser engineers and users for diagnosing performance problems in Firefox. Here’s a screenshot of what he has so far:
Apparently, mining cryptocurrency takes a lot of CPU!
Thanks to the work of Tarek Ziade, we now have a reliable mechanism for getting information on which tabs are consuming CPU cycles. For example, in the above screenshot, we can see that the coinhive tab that Firefox has open is consuming a bunch of CPU in some workers (mining cryptocurrency). Florian has also been clearing out some of the older code that was supporting about:performance, including the subprocess memory table. This table was useful for our browser engineers when developing and tuning the multi-process project, but we think we can replace it now with something more actionable and relevant to our users. In the meantime, since gathering the memory data causes jank on the main thread, he’s removed the table and the supporting infrastructure. The about:performance work hasn’t landed in the tree yet, but Florian is aiming to get it reviewed and landed (preffed off) soon.
Browser Adjustment Project (In-Progress by Gijs Kruitbosch)
This is a research project to find ways that Firefox can classify the hardware it’s running on, which should make it easier for the browser to make informed decisions on how to deal with things like CPU scheduling, thread and process priority, graphics and UI optimizations, and memory reclamation strategies. This project is still in its early days, but Gijs has already identified prior art and research that we can build upon, and is looking at lightweight ways we can assign grades to a user’s CPU, disk, and graphics hardware. Then the plan is to try hooking that up to the toolkit.cosmeticAnimations pref, to test disabling those animations on weaker hardware. He’s also exploring ways in which the user can override these measurements in the event that they want to bypass the defaults that we choose for each environment.
Avoiding spurious about:blank loads in the parent process (In-Progress by Gijs Kruitbosch)
When we open new browser windows, the initial browser tab inside them runs in the parent process and loads about:blank. Soon after, we do a process flip to load a page in the content process. However, that initial about:blank still has cost, and we think we can avoid it. There’s a test failure that Gijs is grappling with, but after much thorough detective work deep in the complex ball of code that supports our window opening infrastructure, he’s figured out a path forward. We expect this project to be wrapped up soon, which should hopefully make window opening cheaper and also produce less flicker.
Load Activity Stream scripts from ScriptPreloader (Completed by Jay Lim)
Defer calculating Activity Stream state until idle (In-Progress by Jay Lim)
When Firefox starts up, one of the first things it prepares to do is show you the Activity Stream page, since that’s the default home and new tab page. Jay thinks we might be able to save the state of Activity Stream at shutdown, and load it again quickly during startup within the content process, and then defer the calculations necessary to produce a more recent state until after the parent process has become idle. We’re unsure yet what this will buy us in terms of start-up speed, but Jay is hacking together a prototype to see. I’m eager to find out!
Grab bag of Notable Performance Work
Luca Greco landed all of the infrastructure to move the WebExtension storage.local backend from a file in the profile directory to indexedDB. This should particularly help the performance of the browser when WebExtensions write small changes to large storage structures, since historically this would cause the entire JSON object for the structure to be recomputed and flushed to disk. This should also help with memory consumption. The infrastructure is disabled by default, and once this bug is fixed, it will be switched on.
Dave Townsend fixed an issue where we were requesting the favicon for new pages twice instead of once. This resulted in a 2%-3% win on our internal session restoration bench on 64-bit Linux!
As I draw this update to a close, I want to give a shout-out to my intern and colleague Jay Lim, whose internship is ending in a few short days. Jay took to performance work like a duck in water, and his energy, ideas and work were greatly appreciated! Thank you so much, Jay!
By “natural”, I mean a layout flush triggered by the refresh driver, and not by some JavaScript requesting size or position information on a dirty DOM ↩
And when it comes to smoothness and responsiveness, jank on the compositor thread is deadly ↩
it landed and bounced once due to a crash test failure, but Doug has just gotten a fix for it approved ↩
As I mentioned previously, the Mixed Reality "virus" has caught me recently and I spend a good portion of my Mozilla contribution time with presenting and writing demos for WebVR/XR nowadays.
The prime driver for writing my first such demo was that I wanted to do something meaningful with A-Frame. Previously, I had only played around with the Hello WebVR example and some small alterations around the basic elements seen in that one, which is also pretty much what I taught to others in the WebVR workshops I held in Vienna last year. Now, it was time to go beyond that, and as I had recently bought a HTC Vive, I wanted something where the controllers could be used - but still something that would fall back nicely and be usable in 2D mode on a desktop browser or even mobile screens.
While I was thinking about what I could work on in that area, another long-standing thought crossed my mind: How feasible is it to render OpenStreetMap (OSM) data in 3D using WebVR and A-Frame? I decided to try and find out.
First, I built on my knowledge from Lantea Maps and the fact that I had a tile cache server set up for that, and created a layer of a certain set of tiles on the ground to for the base. That brought me to a number of issue to think about and make decisions on: First, should I respect the curvature of the earth, possibly put the tiles and the viewer on a certain place on a virtual globe? Should I respect the terrain, especially the elevation of different points on the map? Also, as the VR scene relates to real-world sizes of objects, how large is a map tile actually in reality? After a lot of thinking, I decided that this would be a simple demo so I would assume the earth is flat - both in terms of curvature or "the globe" and terrain, and the viewer would start off at coordinates 0/0/0 with x and z coordinates being horizontal and y the vertical component, as usual in A-Frame scenes. For the tile size, I found that with OpenStreetMap using Mercator projection, the tiles always stayed squares, with different sizes based on the latitude (and zoom level, but I always use the same high zoom there). In this respect, I still had to take account of the real world being a globe.
Once I had those tiles rendering on the ground, I could think about navigation and I added teleport controls, later also movement controls to fly through the scene. With W/A/S/D keys on the desktop (and later the fly controls), it was possible to "fly" underneath the ground, which was awkward, so I wrote a very simple "position-limit" A-Frame control later on, which prohibits that and also is a very nice example for how to build a component, because it's short and easy to understand.
All this isn't using OSM data per se, but just the pre-rendered tiles, so it was time to go one step further and dig into the Overpass API, which allows to query and retrieve raw geo data from OSM. With Overpass Turbo I could try out and adjust the queries I wanted to use ad then move those into my code. I decided the first exercise would be to get something that is a point on the map, a single "node" in OSM speak, and when looking at rendered maps, I found that trees seemed to fit that requirement very well. An Overpass query for "node[natural=tree]" later and some massaging the result into a format that JavaScript can nicely work with, I was able to place three-dimensional A-Frame entities in the places where the tiles had the symbols for trees! I started with simple brown cylinders for the trunks, then placed a sphere on top of them as the crown, later got fancy by evaluating various "tags" in the data to render accurate height, crown diameter, trunk circumference and even a different base model for needle-leaved trees, using a cone for the crown.
But to make the demo really look like a map, it of course needed buildings to be rendered as well. Those are more complex, as even the simpler buildings are "ways" with a variable amount of "nodes", and the more complex ones have holes in their base shape and therefore require a compound (or "relation" in OSM speak) of multiple "ways", for the outer shape and the inner holes. And then, the 2D shape given by those properties needs to be extruded to a certain height to form an actual 3D building. After finding the right Overpass query, I realized it would be best to create my own "building" geometry in A-Frame, which would get the inner and outer paths as well as the height as parameters. In the code for that, I used the THREE.js library underlying A-Frame to create a shape (potentially with holes), extrude it to the right height and rotate it to actually stand on the ground. Then I used code similar to what I had for trees to actually create A-Frame entities that had that custom geometry. For the height, I would use the explicit tags in the OSM database, estimate from its levels/floors if given or else fall back to a default. And I would even respect the color of the building if there was a tag specifying it.
With that in place, I had a pretty nice demo that uses data directly from OpenStreetMap to render Virtual Reality scenes that could be viewed in the desktop or mobile browser, or even in a full VR headset!
It's available under the name of "VR Map" at vrmap.kairo.at, and of course the source code can also be expected, copied and forked on GitHub.
Again, this is intended as a demo, not a full-featured product, and e.g. does at this time only render an area of a defined size and does not include any code to load additional scenery as you are moving around. Also, it does not support "building parts", which are the way to specify in OSM that a different pieces of a building have e.g. different heights or colors. It could also be extended to actually render models of the buildings when they exist and are referred in the database (so e.g. the Eiffel Tower would look less weird when going to the Paris preset). There are a lot of things that still can be done to improve on this demo for sure, but as it stands, it's a pretty simple piece of code that shows the power of both A-Frame and the OpenStreetMap data, and that's what I set out to do, after all.
My plan is to take this to multiple meetups and conferences to promote both underlying projects and get people inspired to think about what they can do with those ideas. Please let me know if you know of a good event where I can present this work. The first of those presentations happened a at the ViennaJS May Meetup, see the slides and video.
I'm also in an email conversation with another OSM contributor who is using this demo as a base for some of his work, e.g. on rendering building models in 3D and VR and allowing people to correct their position data.
I hope that this demo spawns more ideas of what people can do with this toolset, and I'll also be looking into more demos that will probably move into different directions.
Ever since a close encounter with burning out (thankfully, I didn't quite get there) forced me to leave my job with Mozilla more than two years ago, I have been looking for a place and role that feels good for me in the Mozilla community. I immediately signed up to join Tech Speakers as I always loved talking about Mozilla tech topics and after all breaking down complicated content and communicating it to different groups is probably my biggest strength - but finding the topics I want to present at conferences and other events has been a somewhat harder journey.
I knew I had to keep my distance to crash stats, despite knowing the area in and out and having developed some passion for it, but staying in the same area as a volunteer than in a job that almost burned me out was just not a good idea, from multiple points of view. I thought about building up some talks about working with data but it still was a bit too close to that past and not what I presently do a lot (I work in blockchain technology mostly today), so that didn't go far (but maybe it will happen at some point).
On the other hand, I got more and more interested in some things the Open Innovation group at Mozilla was doing, and even more in what the Emerging Technologies teams bring into the Mozilla and web sphere. My talk (slides) at this year's local "Linuxwochen Wien" conference was a very quick run-through of what's going on there and it's a whole stack of awesomeness, from Mixed Reality via codecs, Rust, Voice and whatnot to IoT. I would love to dig a bit into the latter but I didn't yet find the time.
What I did find some time for is digging into WebVR (now WebXR, where "XR" means "Mixed Reality") and the A-Frame library that Mozilla has created to make it dead simple to create your own VR/XR experiences. Last year I did two workshops in Vienna on that area, another one this year and I'm planning more of them. It's great how people with just some HTML knowledge can build something easily there as well as people who are more into JS programming, who can dig even deeper. And the immersiveness of VR with a real headset blows people away again and again in any case, so a good thing to show off.
While last year I only had cardboards with some left-over Sony Z3C phones (thanks to Mozilla) to show some basic 3DoF (rotation only) VR with low resolution, this proved to be interesting already to people I presented to or made workshops with. Now, this year I decided to buy a HTC Vive, seeing its price go down somewhat before the next generation of headsets would be shipped. (As a side note, I chose the Vive over the Rift because of Linux drivers being available and because I don't want to give money to Facebook.) Along with a new laptop with a high-end GPU that can drive the VR headset, I got into fully immersive 6DoF VR and, I have to say, got somewhat addicted to the experience.
I ran a demo booth with A-Painter at "Linuxwochen Wien" in May, and people were both awed at the VR experience and that this was all running in plain Firefox! Spreading the word about new web technologies can be really fun and rewarding with experiences like that! Next to showing demos and using VR myself, I also got into building WebVR/XR demos myself (I'm more the person to do demos and prototypes and spread the word, rather than building long-lasting products) - but I'll leave that to another blog post that will be upcoming very soon!
So, for the moment, I have found a place I feel very comfortable with in the community, doing demos and presentations about WebVR or "Mixed Reality" (still need to dig into AR but I don't have fitting hardware for that yet) as well as giving people and overview of the Emerging Technologies "we" (MoCo and the Mozilla community) are bringing to the web, and trying to make people excited and use the technologies or hopefully even contribute to them. Being at the forefront of innovation for once feels really good, I hope it lasts long!
Hello, Internet! Here we are with yet another Firefox Performance Update for your consumption. Hold onto your hats – we’re going in!
But first a word from our sponsor: ScriptPreloader!
A lot of the Firefox front-end is written using JavaScript. With the possible exception of system add-ons that update outside of the normal release cycle, these scripts tend to be the same until you update.
About a year ago, Mozilla developer Kris Maglione had an idea: let’s try to optimize browser start time by noticing which scripts are being loaded during start-up, and then converting those scripts into a binary representation1 that we can cache on disk. That way, next time we start up, we can just grab the cached binaries off of the disk, skip the parsing step and start executing the JavaScript right away.
Long-time Mozillians might know that we already do some aggressive caching to improve start time for things like XUL, XBL, manifests and other things that are read at start-up. I think we actually were already caching JavaScript files too – but I don’t think we were storing them pre-parsed. And the old caching stuff was definitely not caching scripts that were loading in content processes (since content processes didn’t exist when the old caching stuff was designed).
At any rate, my understanding is that the ScriptPreloader pays attention to script loads between main process start and the point where the first browser window fires the “browser-delayed-startup-finished” observer notification (after the window paints and does post-painting script loading). At that point, the ScriptPreloader examines the list of scripts that the parent and content processes have loaded, and2 writes their pre-parsed bytecode representation to disk.
After that cache is written, the next time the main process or content processes start up, the cache is checked for the binary data. If it exists, this means that we can skip the parsing step. The ScriptPreloader goes one step further and starts to “decode”3 that binary format off of the main thread, even before those scripts are requested. Then, when the scripts are finally requested, they’re very much ready to execute right away.
I’m now working on a series of patches in this bug that will widen the window of time where we note scripts that we can cache. This will hopefully improve the speed of privileged scripts that run up until the idle point of the first browser window.
And now for some Performance Project updates!
Early first blank paint (lead by Florian Quèze)
User Research has hired a contractor to perform a study to validate our hypothesis that the early first blank paint perceived performance optimization will make Firefox seem like it’s starting faster. More data to come out of that soon!
Faster content process start-up time (lead by Felipe Gomes)
The patchesthatFelipe wrote a few weeks back have landed and have had a positive impact! The proof is in the pudding – let’s look at some graphs:
The cpstartup impact. Those two clusters are test runs “before” and “after” Felipe’s patches landed, respectively.
The above graph shows a nice drop in the cpstartup Talos test. The cpstartup test measures the time it takes to boot up the content process and have it be ready to show you web pages.
This is a screen capture of a Base Content JS improvement in the AreWeSlimYet test. This graph measures the amount of memory that content processes consume via JavaScript not long after starting up.
In the graph above, we can see that the patches also helped reduce the memory that content processes use by default, by making more scripts only load when they’re needed.
It’s always nice to see our work have an impact in our graphs. Great work, Felipe! Keep it up!
The good news is that it appears that this has had a significant and positive impact on tab switch times – tab switch times went down, and the number of Nightly instances reporting tab switch spinners went down by about 10%. Great work, Doug!
A numberofbugswere filed against the original bug due to some glitchy edge-cases that we don’t handle well just yet.
We also detected a ~8% resident memory regression in our automated testing suites. This was expected (keeping layers around isn’t free!) and gave us a sense of how much memory we might consume were we to enable this by default.
The experiment is concluded for now, and we’re going to disable the cache for a bit while we think about ways to improve the implementation.
ClientStorageTextureSource for macOS (lead by Doug Thayer)
Swapping DataURLs for Blobs in Activity Stream (lead by Jay Lim)
Jay’s patch to swap out DataURLs for Blobs for Activity Stream images has passed a first round of review from Mardak! He’s now waiting for a second review from k88hudson, and then hopefully this can land and give us a bit of a memory win. Having done some analysis, we expect this buy back quite a bit of memory that was being contained within those long DataURL strings.
Caching Activity Stream JS in the JS Bytecode Cache (lead by Jay Lim)
After examining the JavaScript Bytecode Cache that’s used for Web Content, Jay has determined that it’s really not the right mechanism for caching the Activity Steam scripts.
However, that ScriptPreloader that I was talking about earlier sounds like a much more reasonable candidate. Jay is now doing a deep dive on the ScriptPreloader to see whether or not the Activity Stream scripts are already being cached – and if not, why not.
Tab warming (lead by Mike Conley)
No news is good news here. Tab warming continues to ride and no new bugs have been filed. The work to reduce the number of paints when warming tabs has stalled a bit while I dealt with a rather strange cpstartup Talos regression. Ultimately, I think I can get rid of the second paint when warming by keeping background tabs display port suppressed4, and then only triggering the display port unsuppression after a tab switch. This will happily take advantage of a painting mechanism that Doug Thayer put in as part of the LRU cache experiment.
Firefox’s Most Wanted: Performance Wins (lead by YOU!)
Before we go into the grab-bag list of performance-related fixes – have you seen any patches landing that should positively impact Firefox’s performance? Let me know about it so I can include it in the list, and give appropriate shout-outs to all of the great work going on! That link again!
Grab-bag time
And now, without further ado, a list of performance work that took place in the tree:
Ryan Hunt made it so that paint threads (see Off Main Thread Painting) no longer block Display List and Frame Layer building. This means we can get more done before offloading instructions to the paint threads, and this gives paint threads more time to finish up their work, increasing the probability that they’ll be free by the time the transaction to the paint thread needs to take place.
This is an optimization that we do that shrinks the painted area to just the region that’s visible to the browser. We normally paint a bit outside the viewable area so that it’s ready when a user starts scrolling ↩
We’ve just released a new Thunderbird Conversations (previously know as Gmail Conversation View) with full support for Thunderbird 52. We’re sorry for the delay, but the good news is it should now work fine.
I’d like to thank Jonathan for letting me help out with the release process, and for all those who contributed to release or filed issues.
The add-on should work with the current Thunderbird Beta versions (56), but won’t currently work in Daily (57) due to some compatibility issues. We’re hoping to get those resolved in the next week or so.
If you want to help out with future releases, then find the source code here and come and help us with supporting users or fixing issues.
On Monday, I, along with several million other people, decided to view the Great American Eclipse. Since I presently live in Urbana, IL, that meant getting in my car and driving down I-57 towards Carbondale. This route is also what people from Chicago or Milwaukee would have taken, which means traffic was heavy. I ended up leaving around 5:45 AM, which puts me around the last clutch of people leaving.
Our original destination was Goreville, IL (specifically, Ferne Clyffe State Park), but some people who arrived earlier got dissatisfied with the predicted cloudy forecast, so we moved the destination out to Cerulean, KY, which meant I ended up arriving around 11:00 AM, not much time before the partial eclipse started.
Partial eclipses are neat, but they're very much a see-them-once affair. When the moon first entered the sun, you get a flurry of activity as everyone puts on the glasses, sees it, and then retreats back into the shade (it was 90°F, not at all comfortable in the sun). Then the temperature starts to drop—is that the eclipse, or this breeze that started up? As more and more gets covered, then it starts to dim: I had the impression that a cloud had just passed in front of the sun, and I wanted to turn and look at that non-existent cloud. And as the sun really gets covered, then trees start acting as pinhole cameras and the shadows take on a distinctive scalloped pattern.
A total eclipse though? Completely different. The immediate reaction of everyone in the group was to start planning to see the 2024 eclipse. For those of us who spent 10, 15, 20 hours trying to see 2-3 minutes of glory, the sentiment was not only that it was time well spent, but that it was worth doing again. If you missed the 2017 eclipse and are able to see the 2024 eclipse, I urge you to do so. Words and pictures simply do not do it justice.
What is the eclipse like? In the last seconds of partiality, everyone has their eyes, eclipse glasses on of course, staring at the sun. The thin crescent looks first like a side picture of an eyeball. As the time ticks by, the tendrils of orange slowly diminish until nothing can be seen—totality. Cries come out that it's safe to take the glasses off, but everyone is ripping them off anyways. Out come the camera phones, trying to capture that captivating image. That not-quite-perfect disk of black, floating in a sea of bright white wisps of the corona, not so much a circle as a stretched oval. For those who were quick enough, the Baily's beads can be seen. The photos, of course, are crap: the corona is still bright enough to blot out the dark disk of the moon.
Then, our attention is drawn away from the sun. It's cold. It's suddenly cold; the last moment of totality makes a huge difference. Probably something like 20°F off the normal high in that moment? Of course, it's dark. Not midnight, all-you-see-are-stars dark; it's more like a dusk dark. But unlike normal dusk, you can see the fringes of daylight in all directions. You can see some stars (or maybe that's just Venus; astronomy is not my strong suit), and of course a few planes are in the sky. One of them is just a moving, blinking light in the distance; another (chasing the eclipse?) is clearly visible with its contrail. And the silence. You don't notice the usual cacophony of sounds most of the time, but when everyone shushes for a moment, you hear the deafening silence of insects, of birds, of everything.
Naturally, we all point back to the total eclipse and stare at it for most of the short time. Everything else is just a distraction, after all. How long do we have? A minute. Still more time for staring. A running commentary on everything I've mentioned, all while that neck is craned skyward and away from the people you're talking to. When is it no longer safe to keep looking? Is it still safe—no orange in the eclipse glasses, should still be fine. How long do we need to look at the sun to damage our eyes? Have we done that already? Are the glasses themselves safe? As the moon moves off the sun, hold that stare until that last possible moment, catch the return of the Baily's beads. A bright spark of sun, the photosphere is made visible again, and then clamp the eyes shut as hard as possible while you fumble the glasses back on to confirm that orange is once again visible.
Finally, the rush out of town. There's a reason why everyone leaves after totality is over. Partial eclipses really aren't worth seeing twice, and we just saw one not five minutes ago. It's just the same thing in reverse. (And it's nice to get back in the car before the temperature gets warm again; my dark grey car was quite cool to the touch despite sitting in the sun for 2½ hours). Forget trying to beat the traffic; you've got a 5-hour drive ahead of you anyways, and the traffic is going to keep pouring onto the roads over the next several hours anyways (10 hours later, as I write this, the traffic is still bad on the eclipse exit routes). If you want to avoid it, you have to plan your route away from it instead.
I ended up using this route to get back, taking 5 hours 41 minutes and 51 seconds including a refueling stop and a bathroom break. So I don't know how bad I-57 was (I did hear there was a crash on I-57 pretty much just before I got on the road, but I didn't know that at the time), although I did see that I-69 was completely stopped when I crossed it. There were small slowdowns on the major Illinois state roads every time there was a stop sign that could have been mitigated by sitting police cars at those intersections and effectively temporarily signalizing them, but other than that, my trip home was free-flowing at speed limit the entire route.
Some things I've learned:
It's useful to have a GPS that doesn't require cellphone coverage to figure out your route.
It's useful to have paper maps to help plan a trip that's taking you well off the beaten path.
It's even more useful to have paper maps of the states you're in when doing that.
The Ohio River is much prettier near Cairo, IL than it is near Wheeling, WV.
The Tennessee River dam is also pretty.
Driving directions need to make the "I'm trying to avoid anything that smells like a freeway because it's going to be completely packed and impassable" mode easier to access.
Passing a car by crossing the broken yellow median will never not be scary.
Being passed by a car crossing the broken yellow median is still scary.
Driving on obscure Kentucky state roads while you're playing music is oddly peaceful and relaxing.
The best test for road hypnosis is seeing how you can drive a long, straight, flat, featureless road. You have not seen a long, straight, flat, featureless road until you've driven something like an obscure Illinois county road where the "long, straight" bit means "20 miles without even a hint of a curve" and the "featureless" means "you don't even see a house, shed, barn, or grain elevator to break up corn and soy fields." Interstates break up the straight bit a lot, and state highways tend to have lots of houses and small settlements on them that break up endless farm fields.
Police direction may not permit you to make your intended route work.
<figcaption class="wp-element-caption">This view of the Crab Nebula in visible light comes from the Hubble Space Telescope and spans 12 light-years. The supernova remnant, located 6,500 light-years away in the constellation Taurus, is among the best-studied objects in the sky. Credits: NASA/ESA/ASU/J. Hester</figcaption>
<figcaption class="wp-element-caption">Authenticate with your OAuth provider</figcaption>
TIP #1: Are you having trouble finding that menu? The area of Thunderbird where you activate Cards View is called the Message List Header. Some people choose to hide this section to reclaim a bit of vertical space. It’s easy to get it back: Just return to the App menu (≡), then select View 
Layout, and make sure that “Message List Header” is checked ✓
Download Episode 3 Directly
.
<figcaption>Conversations’ threaded message layout</figcaption>
<figcaption>Message Controls Menu</figcaption>
<figcaption>Attachment preview area with gallery view available for images.</figcaption>
<figcaption>There are several of those circles stacked on top of one another, and several of those arms in between them. We’re only seeing the top one in this photo.</figcaption>
<figcaption>Vinyl’s back, baby!</figcaption>
<figcaption>This graph measures various start-up metrics. The scatter of datapoints on the left show the “control” build, and they tighten up on the right with the “test” build. Lower is better.
<figcaption>This graph measures the time at which the browser window reports that it has first painted. April 14th is the second last date on the X axis, and the Y axis is time. The top-most line is plotting the 95th percentile, and there’s a nice dip appearing around April 14th.
<figcaption>This is a graph of the ts_paint startup paint Talos benchmark. The highlighted node is the first mozilla-central build with the hidden window work. Lower is better, so this looks like a nice win!</figcaption>
<figcaption>The four on the left are Firefox Nightly, and the four on the right are Google Chrome (71.0.3578.98)</figcaption>
indicates a volunteer contributor)