Hey Dygmates!
TL;DR. ~2% of Raise users reported experiencing keyboard freezes where no key input would register while the LEDs remained on. If you are one of those who experienced the freezing issue, download the latest Bazecor and update your firmware to the latest version to turn the freezes into ~3-second resets. Read the post to learn about the problem and how we will fix it long term.
Windows 10 • MAC OS • LINUX
☝️ Context
A couple of weeks back, some Dygmates rightfully complained about the freezing issue.
We listened to your feedback and re-evaluated how we were dealing with the freezing issue. This post (and short-term fix) is the result of it.
Usually, companies don't talk openly about the issues they face. But you know, we're not like most companies. We believe in open-development and learning from the community. We believe that any problem is temporary and everything will be solved with enough effort and work.
Our mission is to make the best possible keyboard ever. With your constant help and feedback, we will achieve this together.
❄️ History of the freezing issue
The first time we got contacted by a customer about a Raise freezing was in March, right after we released the Bazecor 0.2 update.
Thinking it was related to our latest update, we spent over two months working with a Kaleidoscope firmware developer. After improving the Raise firmware during those months, he told us that he couldn't reproduce the freeze anymore and that he believed it was fixed.
Although the number of freeze cases dropped, occasionally a keyboard would still freeze. So it wasn't fixed.
We started considering other potential causes.
Some clients that bought the first batch of Raise keyboards (up to March 2020) experienced keyboard disconnections related to cables. The cables had faulty soldering, so we changed the supplier and improved the cable quality.
Changing the faulty cables fixed many freezing cases, but again, it didn't fix 100% of them.
We picked up keyboards that experienced freezing and tested them. In most cases, the keyboards we picked up from customers wouldn't freeze.
This made us think that it could be related to their setup around the Raise.
In some cases, we were able to fix the freezes by uninstalling drivers of other peripherals. In other cases, by using a different USB port or by updating or downgrading the firmware. And for others, by removing a device such as a KVM switch.
In no case were we able to replicate it consistently ourselves.
This is the data we gathered from talking to customers:
- 2% of Dygmates contacted us because of freezing issues.
From those Dygmates:
- ~50% of the issues were fixed by swapping one or multiple cables.
- ~15% were fixed by swapping the Neuron or PCBs.
- ~15% were fixed by a combination of firmware updates/downgrades or uninstalling other drivers' peripherals.
- ~20% were caused by an external device (wireless charging pads or KVM switches).
We were able to fix most keyboards that experienced freezes except a handful where the cause was related to their environment.
Even if we are able to fix most cases, this situation is still not ideal. It's annoying for the customer and it requires a lot of work from our customer support team. Plus, we were paying the extra cost of shipping replacement components around the world.
We needed to find a solution ASAP.
🤨 So, what was going on?
After analyzing deeply what was happening, having many internal meetings and bumping our head against the wall a few times, we reached a few conclusions:
- The freezing is a symptom of a few different "illnesses". That's why there's no simple fix that works in every case.
- The reason the keyboard freezes is because the communication in the I2C* bus is blocked by electrical interference. The chip sends a message but doesn't receive an answer.
- What causes the keyboard to freeze is a combination of two things:
- An electrical interference interrupts the communications in the I2C channel and blocks the Neuron microprocessor.
- When the communication gets blocked, the microprocessor should reset itself. But there's a bug in the firmware that freezes the keyboard instead of allowing it to reset.
- We've been able to replicate the freezing by intentionally blocking the communication between the Neuron and the sides, but we haven't been able to accomplish it between the Neuron and the computer. This means that any Firmware issue is related specifically to I2C, not to USB.
- The electrical interference can be caused externally and internally. Externally would be by something like a wireless mousepad. Internally, it could be by a faulty cable that in some specific angle doesn't connect properly.
*The I2C is the communication protocol used so both sides speak to the Neuron. But I2C was designed so chips talked to each other in the same PCB. The Raise has 5 PCBs, 3 chips in 3 different PCBs and they are connected by two different cables.
The I2C protocol wasn't created for this kind of "electrical stress" environment.
Now that we know what's going on, we can focus on fixing it. Hurraaay!!🎉
🎯 Fixing it. The 4-step masterplan
Now that we know that the main issue is the I2C, we can focus all our efforts on "strengthening" it. This can be done short term via firmware and long term via hardware. We can even solve the issues provoked by external devices.
We have a 4-step masterplan to fix the freezing.
1. Turn the freezes into 3-second resets.
This is a short term band-aid, but the experience for any user is much better than having to unplug and replug the Neuron.
What we are doing here is telling the keyboard to completely reset when the communications in the I2C channel gets blocked.
It's kind of an overkill solution because we aren't just resetting the I2C channel, but the whole keyboard.
The next step addresses this.
2. Turning the 3-second resets into 0.2-second refreshes.
This sounds a bit better.
Instead of resetting the whole keyboard, we will refresh the side of the keyboard that has the I2C channel issue.
This would be close to instant and in most cases even imperceptible by the user.
3. Turn the 0.2-second refreshes into <0.002s refreshes.
Now we're talking!
This would make any I2C refresh unnoticeable by any user.
4. Reinforce the PCBs to be a better shield against electromagnetic radiation.
The Raise passed all electronic certifications (ESD, EMI, and EFT), so it already has the required electronics to be safe and stable.
But there are some scenarios that we didn't consider when designing our PCBs, thus have made us want to reinforce the design:
- Not all electronic devices we can buy have passed the certifications, and they can cause other devices to malfunction. For example, every cheap electronic device bought in your local convenience store probably didn't pass the ESD, EMI, and EFT even though they have the logos printed on a sticker. Who knows?
- Old houses have low insulation. And usually have old electronic devices wouldn't pass today's electronic radiation tests.
- The I2C protocols were designed so that chips would communicate with each other in the same PCB. Our chips communicate through cables, making the communication channel more susceptible to electromagnetic radiation.
With this in mind, we can improve the sturdiness of our electronics by:
- Tweaking the design of the cables and increasing the shielding. This would make them stiffer, so we need to find the right balance.
- Make changes in the Neuron PCB so it's better insulated against any kind of external electrical radiation.
- Add components to the PCBs of the keyboard sides, such as diodes and capacitors, to better isolate the working circuits from its environment.
📌 Where are we at right now?
We are on Step 1. You can now download the latest Bazecor version and update your keyboard to the latest firmware that turns the freezes into resets.
Windows 10 • MAC OS • LINUX
🐾 Next steps
Our developer, Alex, has already started working on the Step 2, turning the freezes into instant refreshes.
On top of that we are looking for a firmware developer to help us with this. If you know of any, please email us at contact@dygma.com.
And at the same time, our Product Manager Manel is working on the Step 4. He's looking for an electronic engineer with experience in consumer electronics or medical devices and specializes in emissions. If you know of any electronic engineer that fits the criteria, please tell them to contact us at contact@dygma.com or let us know.
We hope we can have Step 2 by February. We still don't know exactly when to expect Step 3. We will learn more about this once we've investigated more into the previous step.
Finally, we plan to do Step 4 for the next mass production batch of Raise keyboards, which will be by mid-2021. We will keep you posted on our developments!
❤️ An awesome Dygmate
A very special thanks to Babak Javadi!
He reached out to us when he experienced the freezing issue. He went out of his way to do experiments to find where the problem was coming from. We had a video call with him and he explained to us how he was able to replicate the freezing issue and offered his solutions based on his expertise.
It's so amazing to have great Dygmates like him! Thank you Babak!
Please check our Reddit thread to join the conversation about this topic and to provide feedback and suggestions! 😊
