Meta Issue: Keyboard input

[maroider]

There are a bunch of issues related to keyboard input on the issue tracker, but it's currently difficult to get an overview of currently open issues in a short amount of time.
By opening this issue, I hope to consolidate information about what's currently broken about keyboard input in Winit in one place.

Two issues which are of particular importance here are:

• A keyboard input model #753, where the API for individual keypresses is discussed.
  There is some additional discussion in (PR) New keyboard API for Windows #1788, which tries to implement the API.
• Tracking issue for IME / composition support #1497, where the IME API is discussed.

There has been a lot of progress on the implementation of the API discussed in #753, and there is now a PR moving all the backends (bar Redox) to the new API: #2662

This PR supersedes the previous backend-specific PRs:

• Windows: New keyboard API for Windows #1788 (merged)
• macOS: New keyboard API for macOS #1890
• WASM (web-sys): New keyboard API for the web #1888
• Linux: New keyboard API for Linux #1932

There are two milestones which track some issues related to keyboard input:

• Keyboard Events Overhaul
• IME Events Overhaul

These milestones don't track all issues related to keyboard input, so here's a list of these untracked issues:

May be resolved by adoption of #753

Issue	Platform	Notes
#440	Linux	Panic when using optirun and typing dead keys.
#714	macOS	ReceivedCharacter gets infinitely repeated under certain circumstances.
#734	macOS	"in certain circumstances the Paste key is sent multiple times to the winit application."
#773	Cross-platform	Ignoring keyboard input when IME is active.
#1208	macOS	Might be another instance of a VirtualKeyCode not being delivered.
#1267	macOS	Certain key combinations don't send ReceivedCharacter events.
#1272	Cross-platform	Some platforms don't give you key release events if you press a key, then unfocus the window and then release the key. This is papered over by winit through synthesized keyboard events (WindowEvent::KeyboardInput.is_synthetic).
#1396	macOS	ModifiersChanged isn't emitted in certain cases.
#1443	Linux	DeviceEvent delivers a VirtualKeyCode while WindowEvent doesn't.
#1470	macOS	No keyboard input through DeviceEvent on macOS.
#1488	macOS	Key repeat is inconsistent.
#1656	Linux	Shift+Alt produces the wrong virtual keycode on Alt "key down".

Various bugs and requests for API changes

Issue	Platform	Notes
#347	Linux	Mutex issues & segfault when using IME.
#361	Linux	DeviceEvent delivers a VirtualKeyCode while WindowEvent doesn't.
#477	Cross-platform	ReceivedCharacter & Enter/Return.
#732	Cross-platform	Proposal for an API to convert between ScanCode and VirtualKeyCode.
#756	Cross-platform	ModifiersState should differentiate between "right Alt" and AltGr.
#788	Linux	XIMPreeditPosition should be used instead of XIMPreeditNothing in a particular spot.
#1010	Cross-platform	Request for a keyboard key state query API.
#1426	Cross-platform	A request for a NumLock status query API.
#1452	Cross-platform	ReceivedCharacter is not emitted while IME is active.
#1497	Cross-platform	Tracking issue for CompositionEvent support.
#1768	Web	Web hotkeys may need to be optionally allowed.

NOTE: This comment received a substantial edit on 2020-04-07.

[ArturKovacs]

Thank you for putting all this work into this topic. Your contributions have had a great effect on getting this beast moving forward.

EDIT: I will go through the list sometime and try adding those to the milestone which we expect to resolve by resolving #753. On that note, we may need to place the IME related issues and the keypress related ones in two separate milestones.

[maroider]

Actually, do you mind pinning this issue, @ArturKovacs?

[ArturKovacs]

Good idea!

[ArturKovacs]

I categorized the listed issues.

IME / Composition

I put some of the IME related ones to the IME Events Overhaul. Optimally only the issues that will be solved by #1497 should be part of this milestone but I don't have much time right now to go into the IME aspect so that milestone might not hold the correct set of issues.

Keyboard Events

The ones that we expect to be resolved by #753 are put into Keyboard Events Overhaul.

The ones that are uncertain to be resolved by #753 are: #440, #714, #734, #773, #1208, #1267, #1272, #1396, #1443, #1470, #1488, #1656

The ones that are expeccted to persist after #753 (until addressed later) are: #732, #756, #1010, #1426, #1768

[ctrlcctrlv]

Unfortunately, MFEKglif needed to drop winit due to this bug. Cf. MFEK/glif#31.

Due to the difficulty of SDL adoption, even if it were fixed we wouldn't switch back. I was a little mad in MFEK/glif#31, but no longer really have an emotional attachment to this, other than to say that if it isn't fixed winit is certainly going to keep losing mindshare in a similar manner, and new projects probably shouldn't adopt it until this is closed.

[ArturKovacs]

I completely understand your frustration @ctrlcctrlv. We understand the importance of this issue and currently working on solving it. We are hoping that the new API will land in winit 0.26.0. I'm also thinking that it would be a good idea to cut a release as soon as the the new keyboard API is done.

[maroider]

Now that the Linux PR is up, we should have the major desktop platforms covered.
Once we've got all platforms up to speed on the new-keyboard branch, I'm going to summarize #753 and potential deviations from it introduced #1788 and other PRs, and bring up some mostly bike-shed-y things which I'd like to resolve before merging into master.

If you'd like to test any of the PRs, you can use this program I made for this exact purpose.

[lunabunn]

Any updates on this? All the related issues/PRs seem stagnant except for the IME stuff.

[maroider]

I went off on an entirely unplanned hiatus, and if no movement has occurred while I was gone, then I suppose it might be the case that others were waiting for me to get the ball rolling again. I'm currently clearing my GitHub notifications backlog, and I'll try to get back into the swing of things in the next week or so.

[maroider]

#2662 now consolidates all the various branches into one.

[TimJentzsch]

Is there a way to determine the logical key of the current keyboard layout, given a scan code?
I found the from_scancode function in this PR, but this doesn't seem to consider the keyboard layout.

This would be useful for games where you specify input in terms of the physical key position (e.g. the classic WASD controls), but need to show the logical key in the game UI.

[kchibisov]

You don't know the layout at all on hardware level. Every other layout is software, thus scancodes are reliable. You may manually craft some mappings, but the thing is that you can have arbitrary layout on the hardware, so it's all irrelevant.

[maroider]

There has been talk of adding an API to let you query the active software layout, which would (mostly) do what you want, but it's not a part of the initial API rework.

[kchibisov]

There has been talk of adding an API to let you query the active software layout, which would (mostly) do what you want, but it's not a part of the initial API rework.

But it's not hardware layout? Since if you use scancodes like they want for wasd the stuff is already software agnostic. Software layout is mostly useless for anything. Though most games I know bind by character, so you have to remap stuff on the software dvorak layout. For software dvorak it'll solve issues if you use scancodes, but not for hardware dvorak.

[minecrawler]

Scan codes are useful to know where a button is on the hardware, but key codes are what a user expects to see when they press a button. The standard example is a French person pressing "Z" on their keyboard and expecting to walk forward, even though the game developer was using a US layout and programmed that functionality for the "W" key.

Also, let's say that game has a tutorial, which teaches the user the basic game controls. Such a tutorial should not show "W" for a French person, but instead "Z", because of the player's keyboard layout.

Imho, winit should hence query the layout as configured (and watch it for changes; also: IME might be interesting) and on key press pass both, the scan code and the key code. Then the game could know that the "forward" key was pressed, but show the correct key in the bindings menu :)

Taking it a step further, it may even be possible to store the scan code and adapt the key code in case the layout switches. For example, I use an US/INT keyboard, but my laptop came with a German layout. I'm not interested in what's labeled on the key when playing a game, so it should just display "Y" for the US/INT layout and "Z" for the German layout. But this last part is game-specific and just a personal flavor 😆

[dhardy]

Looks to me like people are talking past each other...

You don't know the layout at all on hardware level. Every other layout is software, thus scancodes are reliable. You may manually craft some mappings, but the thing is that you can have arbitrary layout on the hardware, so it's all irrelevant.

Yes, there are keyboards that let you switch to Dvorak, Colemak, Azerty etc. in hardware (especially low-volume mechanical keyboards), but this stuff is pretty niche. In this case the OS doesn't know the physical position of the key pressed, hence pre-made game key-maps are useless (and I'd recommend against remapping keys in hardware specifically because of this problem). TLDR: niche issue, don't worry about it.

Though most games I know bind by character, so you have to remap stuff on the software dvorak layout.

Like, 20 years ago, this seemed to be the case. More recently, most games I've tried seem to get this right (presumably so that AZERTY users don't have to remap keys). Anyway, irrelevant.

What I have seen a lot is keys matched by scan code but report US QWERTY names for those keys. Annoying, but quite common and what @TimJentzsch's request is about.

Scan codes are useful to know where a button is on the hardware

Note also these are OS dependent. The new keyboard event is going to look vaguely like this, where code provides a device-independent mapping.

For example, I use an US/INT keyboard, but my laptop came with a German layout.

I guess the way to approach this is (1) have an API returning a handle to the current [software] keyboard layout, (2) for querying a key code's label in a given layout, and (3) allow the user to specify a different layout to use when querying labels.

Sounds nice, but I've never heard of either the OS or a game supporting part (3), and I doubt many would bother (most people's software layouts match the key labels).

---

I opened #2678 to track the request since it's off-topic here.

[Systemcluster]

Yes, there are keyboards that let you switch to Dvorak, Colemak, Azerty etc. in hardware (especially low-volume mechanical keyboards), but this stuff is pretty niche. In this case the OS doesn't know the physical position of the key pressed, hence pre-made game key-maps are useless (and I'd recommend against remapping keys in hardware specifically because of this problem). TLDR: niche issue, don't worry about it.

I use a mechanical split keyboard with a Colemak layout on the firmware (QMK) layer and EurKEY layout (basically US QWERTY) on the OS layer, and I have to unfortunately agree here. This is a known drawback when using keyboards with custom layouts like this, and I wouldn't expect winit or any other software to deal with this. Personally I use a hotkey to switch to US QWERTY in the firmware when playing games that use scan-codes for inputs, and I know of other people who remap their keys in-game or use a second keyboard when necessary.

It is theoretically possible to obtain the firmware-level mapping by communicating with e.g. the QMK firmware and getting the VIA definitions, but that is widely out-of-scope for winit.

[maroider]

Please don't discuss this any further in this issue. This issue is not intended for the discussion of the specifics of how the keyboard API will or should change, or any accompanying rationale.

[kchibisov]

I don't think this issue has any value anymore.