This is the story of the biggest struggle Dygma has ever faced.
Our first product is the Dygma Raise.
We believe the Raise the best keyboard you can get to make the jump from normal to ergonomic keyboards.
It has the perfect mix of a normal easy-to-use layout with ergonomic features such as it being split, having extra comfortable thumb keys, and allowing you to configure you keys. Plus it looks really cool, as an icing on the cake!
However, there have been some challenges along the way that have threatened the existence of the Raise.
On March 2020, we received an unusual email from a Raise user.
He reported that the Raise had become unresponsive and he was unable to type. He had to unplug the cable and plug it again to make it work. This apparently small issue, would later become our biggest nightmare. But we yet didn't know.
Back when we created the Raise with the support our crowdfunding backers, we didn't have a lot or resources. So we created our firmware based on an existing open source firmware called Kaleidoscope. This base firmware was still under development. Some of its changes were pushed into the Raise firmware.
Since the report from March was the first reported freezing issue, we assumed that there was a bug in the recent update in Kaleidoscope. Then other users started reporting it. So we talked to our developer at that time. He then released an update that would fix it. Or so we thought.
Fast forward a few months and we get new reports from users that were having freezing issues. Their Raise keyboard would suddenly become unresponsive while using it. The only solution was to unplug it. This sounded familiar. The same issue we had solved a few months earlier was coming back to bite us in the a**.
This was a deal-breaker.
For some, this was a mild inconvenience. And for others, this was a deal-breaker and they returned their keyboards and asked for refunds.
Every customer had a different experience with the freezes. For some of them, it happened while typing. It could freeze in the middle of a keystroke, having that key repeat itself indefinitely, until they unplugged the keyboard. For others, it only froze when they walked away from the computer. As soon as they came back to it, they weren't able to type. Several users reported having to unplug their keyboard dozens of times per day.
We immediately started investigating it.
We had already changed the firmware, so that couldn't be it. We tried changing the cables for those users, and this solved the issue for most of them. So the freezing was caused by faulty cables. Or was it?
There were some users who replaced all of their cables, but still experienced freezes. At this point, we already knew that there was something wrong in the communication between the keyboard sides and the Neuron. We were suspecting that these cases, where cables were not the issue, were caused by a faulty USB C socket.
The Neuron has 3 USB C sockets, so if just one of them is faulty, it would cause interruptions in the communication, and freeze the Neuron. So we replaced the Neuron for these users. Some of them found that it helped with freezing, and this was a good step.
But there were still a few users who got the issue. This meant that the faulty USB C socket could be in the keyboard side. So we replaced those keyboards completely.
And this is when something unexpected happened.
A few of the users, who had their whole Raise keyboard replaced, were still having freezes.
How could this be? Did they get two faulty Raises in a row?
We got back these keyboards and tested them on our workshop. But they had no freezing.
This was starting to be worrying. We had this mysterious issue that was affecting at least 2% of our users. Some cases were solved by changing the cables. But other cases persisted even after replacing the whole keyboard. What's more: if those keyboards were shipped back to us, the freezing disappeared.
There was only one possible conclusion: The environment was causing this. Something was present on the environment of these keyboards that was causing these freezes. This was not present on our workshop, so we couldn't reproduce the issue there.
We asked customers to disconnect other devices from their computers, and this helped some users. So we were on the right path. However, we had to get to the root cause and solve it, because we couldn't just have every Raise user remove all other devices.
At this point, there was a lot of uncertainty. We didn't really know if we would be able to fix it, and how we would solve it. We were even afraid of sending a Raise to big a YouTube Channel, in case they had they suffered freezing, and would create a terrible review for the Raise that could really hurt us.
This could not go on.
We had to solve this asap. We had an emergency meeting in Dygma. All Dygma workers stopped what they were doing, and we would now focus all of our efforts in solving the freezing issue.
There was a customer who had found a possible cause for this issue. So we arranged a video call with him.
Enter Babak, our first savior.
He explained that electrostatic discharges were causing the freezes, and a few experiments he did to trigger it. We were able to learn a lot from him, and it helped us understand the causes of the freezing. It could be triggered by a faulty cable, from an electrostatic discharge, or by a strong electromagnetic field. Any event that caused an interference in the communication between the Neuron and the keyboard sides.
We now knew the cause, but we didn't know how to solve it. We needed help. So we did what we've always done in the past. We decided to tell everyone about it. This was a risky move. We were about to expose ourselves publicly as having a faulty product, before having the solution to it. What if we couldn't find a solution? What if the solution implied replacing all Raise keyboards ever sold? This could be the end.
We explained publicly all the details about the issue, and looked for electronic engineers that could shed some light on it. We got some answers. Several of our Dygmates were experts on I2C, the communications protocol that the Raise uses. The same communications that appeared to be causing the freezes. We saw the light at the end for the tunnel. We talked to two of them: Keith, and Leo. Leo finally helped us develop a solution that, hopefully, only involved changing the firmware.
Our solutions
Our first solution involved resetting the whole Raise keyboard every time it froze. There was a watchdog in the Neuron that checked every now and then to see if the processor was frozen. If it was, it would reset it completely. This was a little overkill. Like we say in Spain, it was like killing flies with a cannonballs. But it was the best we had.
Users who used to have freezes, now had 3 second resets. Their keyboard would automatically disconnect and re-connect every time it froze. It was an improvement, but still inconvenient if you experienced it in the middle of a game, or when typing.
Alex, our developer, stopped developing Bazecor during this time. He focused all his efforts on finding a better solution. We needed this to succeed. And he finally did it.
Electrostatic discharges and electromagnetic pulses create noise in the cables. This noise makes the Neuron think that the keyboard side started talking. The Neuron waits for the keyboard side to finish the "sentence". How polite of it, right? The problem is that sentence never ends, because it never started in the first place. It was just noise.
Alex implemented a system that finishes "sentences" automatically. Every time the Neuron detects an incomplete communication on the I2C channel, it creates an empty message with an end. This unblocks the I2C channel, and it can accept new communications.
So, did the fix work?
Absolutely. We tested it with the help of dozens of Dygmates, on reddit and by email. They all reported that the freezes and resets were over. We continued testing to make sure. And only the most extreme electric discharges were able to shut off the processor, triggering a Reset. For everything else, the fix worked perfectly.
The freezing issue is officially over.
And the Raise is saved.
Thank you, Dygmates, for all your love and support!
Watch how we were able to solve the freezing issue in the video below:
