Compare commits

...

2 Commits

Author SHA1 Message Date
郑博文
8993a38658 fix(android): BACK 键改用自测键盘可见性——isAcceptingText 会让会话永远断不掉
真机实测(vivo S6 + 讯飞,2026-10-30):软键盘已隐藏、mInputShown=false,
但 imm.isAcceptingText 仍返回 true——只要 InputConnection 还绑着它就是 true。
原先用它判断「先关键盘还是先断会话」,后果是第一次 BACK 空转,第二次也进不来
(我们已 return true),会话再也断不掉。这不是推测,是完整链路 proof 跑到
step 5 时实测失败的。

改为自测:比较根窗口可见区域与窗口高度,被 IME 吃掉超过 15% 视为可见;
用 OnGlobalLayoutListener 被动跟踪(没有公开 API 能直接查询,API 34 上
getInputMethodWindowVisibleHeight 也不是公开接口)。阈值用比例不用固定像素。

验证:diag 依次显示「键盘可见」→「键盘隐藏」→ 会话断开回表单;
随后完整链路 proof 与中文 proof 均复跑通过。

同时修正 phone_chain_proof 的 V4 断言:uiautomator 属性顺序是 text= 在
class= 之前,原正则要求反序,导致打印空表单行仍判 PASS(假绿)。
空 EditText 的 text 会回显 hint,所以判据改为比对实际输入值。
2026-10-30 13:31:28 +08:00
郑博文
9792662181 test(android-2b): 真机中文候选词 proof——拼音组词经 commitText 落到 Windows
V6 的判据是「手机中文输入法能把汉字送进远端」,此前只能靠人手点候选词。
实测发现两条关键事实,决定了 proof 的可行形态:
1. adb keyevent 相当于物理键盘输入,中文 IME **不组词**,字母原样透传到
   Activity.onKeyDown——所以驱动候选词必须用 adb input text(走 IME)。
2. IME 自己的按键视图对 uiautomator 不可见(dump 里只有我们 App 的 15 个节点),
   中/英切换按钮读不到坐标,只能按截图标定一次点击位;且该按钮**有状态**,
   于是写成幂等重试:先试,只收到字母才翻转模式再来一次。

跑通结果:讯飞拼音 nihao + 空格 -> commitText 2u -> 靶窗收到 U+4F60 U+597D。
这补上了方案风险 R1(vivo ROM + 第三方中文 IME 行为未观测)的核心一环。
2026-10-30 13:15:27 +08:00
3 changed files with 265 additions and 6 deletions

View File

@ -94,6 +94,10 @@ class MainActivity : Activity(), RemoteSurfaceView.Listener {
root.addView(remotePanel, LinearLayout.LayoutParams(LinearLayout.LayoutParams.MATCH_PARENT, 0, 1f))
setContentView(root)
// 键盘可见性只能靠窗口布局变化被动感知(没有公开 API 能直接查询)。
// 每次根视图布局完成都重算一次,覆盖「用户点键盘按钮」「IME 自己收起」两条路径。
root.viewTreeObserver.addOnGlobalLayoutListener { updateKeyboardVisibility() }
showConnectState("未连接")
// intent 入口优先级高于表单默认值:build-android.ps1 与调试脚本依赖它
@ -275,9 +279,44 @@ class MainActivity : Activity(), RemoteSurfaceView.Listener {
override fun isReadyForInput(): Boolean = handle != 0L
/**
* 软键盘是否可见(自测,不依赖 InputMethodManager 的间接信号)。
*
* 为什么必须自测:真机实测(vivo S6 + 讯飞,2026-10-30)键盘隐藏后
* `imm.isAcceptingText` **仍返回 true**——只要 InputConnection 还绑着它就一直 true。
* 用它判断"先关键盘还是先断会话",会让第一次 BACK 空转、第二次也进不来
* (我们已 return true),会话再也断不掉。这是真机暴露的缺陷,不是推测。
*
* 做法:比较根窗口可见区域高度与窗口高度。键盘占位时可见区域显著变矮。
* 阈值用 15% 而不是固定像素,避免不同屏幕密度下误判。
*/
private var keyboardVisible = false
private val visibleFrame = android.graphics.Rect()
private fun updateKeyboardVisibility() {
val decor = window.decorView
decor.getWindowVisibleDisplayFrame(visibleFrame)
val screenH = decor.height.takeIf { it > 0 } ?: return
val visibleH = visibleFrame.height()
// 底部被 IME 吃掉的高度超过屏幕 15% → 认为键盘可见
val hiddenByIme = screenH - visibleH
val nowVisible = hiddenByIme > screenH * 0.15
if (nowVisible != keyboardVisible) {
keyboardVisible = nowVisible
lastTouchDbg = if (nowVisible) "键盘可见" else "键盘隐藏"
refreshStatus()
}
}
override fun onConfigurationChanged(newConfig: android.content.res.Configuration) {
super.onConfigurationChanged(newConfig)
updateKeyboardVisibility()
}
/**
* 按键转发(物理键盘 / adb 注入 / 无 IME 时的软键盘):
* - 返回键:先收 IME,否则断开回表单(阶段 2a 语义延续);
* - 返回键:**键盘可见时**先收键盘,否则断开回表单(阶段 2a 语义延续);
* - **可打印**的 `unicodeChar`(0x20 起、排除 0x7F)→ v3 **Text** 通道:
* 字符注入与远端键盘布局无关。注意 `getUnicodeChar()` 给的是单个
* UTF-16 code unit 而非码点,所以这条路只覆盖 BMP;增补平面字符由
@ -295,10 +334,13 @@ class MainActivity : Activity(), RemoteSurfaceView.Listener {
*/
override fun onKeyDown(keyCode: Int, event: KeyEvent?): Boolean {
if (keyCode == KeyEvent.KEYCODE_BACK) {
val imm = getSystemService(Context.INPUT_METHOD_SERVICE) as InputMethodManager
// isAcceptingText = 当前有 IME 连接且正在接收输入(API 17+)
if (imm.isAcceptingText) {
// 用自测的键盘可见性,不用 imm.isAcceptingText(见 updateKeyboardVisibility
// 注释:真机上键盘隐藏后它仍为 true,会让会话永远断不掉)
updateKeyboardVisibility()
if (keyboardVisible) {
val imm = getSystemService(Context.INPUT_METHOD_SERVICE) as InputMethodManager
imm.hideSoftInputFromWindow(surfaceView.windowToken, 0)
keyboardVisible = false
return true
}
if (handle != 0L) {

205
tools/ime_chinese_proof.ps1 Normal file
View File

@ -0,0 +1,205 @@
# 2b V6 on-device CHINESE: drive the IME through pinyin + candidate selection.
# ASCII-only source. Non-ASCII expectations are built from code points.
#
# Measured path differences (2026-10-30, vivo S6 + iFlytek IME):
# - `adb input keyevent <letter>` behaves like a HARDWARE keyboard: the IME
# does not compose, the event lands in Activity.onKeyDown and our printable
# unicodeChar branch sends it as text. So keyevents CANNOT exercise Chinese
# candidate selection.
# - `adb input text <latin>` is delivered to the IME, which composes it;
# SPACE (keyevent 62) then accepts the highlighted candidate, and the Hanzi
# arrive at RemoteInputConnection.commitText.
# This script therefore types pinyin with `input text` and commits with SPACE.
#
# Acceptance: the Windows target window receives the exact Han characters the
# IME committed - Chinese input works end to end over protocol v3.
# Not covered: swipe/handwriting input, browsing the candidate list.
param(
[string]$PhoneIp = "192.168.1.180",
[int]$Port = 47800,
[string]$Code = "t"
)
$ErrorActionPreference = "Stop"
$repo = (Resolve-Path "$PSScriptRoot\..").Path
$adb = "D:\working\android\sdk\platform-tools\adb.exe"
$tmp = "$env:TEMP\remote-cjk2"
New-Item -ItemType Directory -Force -Path $tmp | Out-Null
$pidFile = "$tmp\target.pid"
$stopFile = "$tmp\stop.flag"
$resultFile = "$tmp\received.txt"
Remove-Item $pidFile,$stopFile,$resultFile -ErrorAction SilentlyContinue
Add-Type -TypeDefinition @"
using System;
using System.Runtime.InteropServices;
using System.Text;
public static class Fg5 {
[DllImport("user32.dll")] public static extern IntPtr GetForegroundWindow();
[DllImport("user32.dll")] public static extern uint GetWindowThreadProcessId(IntPtr h, out uint pid);
[DllImport("user32.dll", CharSet=CharSet.Auto)] public static extern int GetWindowText(IntPtr h, StringBuilder s, int n);
public static string Desc() {
IntPtr h = GetForegroundWindow(); uint pid; GetWindowThreadProcessId(h, out pid);
var sb = new StringBuilder(256); GetWindowText(h, sb, 256);
return h.ToInt64() + " [" + sb.ToString() + "] pid=" + pid;
}
}
"@
function Adb-Shell {
param([Parameter(Mandatory)][string[]]$Cmd)
$old = $ErrorActionPreference
$ErrorActionPreference = "Continue"
try {
$out = & $adb shell @Cmd 2>$null
return @{ out = ($out -join "`n"); code = $LASTEXITCODE }
}
finally { $ErrorActionPreference = $old }
}
function Wait-File {
param([string]$Path, [int]$TimeoutMs = 20000)
$t = 0
while (-not (Test-Path $Path) -and $t -lt ($TimeoutMs / 150)) { Start-Sleep -Milliseconds 150; $t++ }
if (-not (Test-Path $Path)) { throw "timeout waiting for $Path" }
}
function Assert-Foreground([IntPtr]$hwnd, [string]$where) {
$deadline = (Get-Date).AddMilliseconds(2500)
while ((Get-Date) -lt $deadline) {
if ([Fg5]::GetForegroundWindow() -eq $hwnd) { return }
Start-Sleep -Milliseconds 50
}
throw "$where : foreground is $([Fg5]::Desc()), want target $($hwnd.ToInt64()) - refusing to inject"
}
function Get-Ui {
[void](Adb-Shell @("uiautomator", "dump", "/sdcard/ui.xml"))
$old = $ErrorActionPreference
$ErrorActionPreference = "Continue"
try { & $adb pull /sdcard/ui.xml "$tmp\ui.xml" 2>$null | Out-Null }
finally { $ErrorActionPreference = $old }
if (-not (Test-Path "$tmp\ui.xml")) { return "" }
return [System.IO.File]::ReadAllText("$tmp\ui.xml", [System.Text.Encoding]::UTF8)
}
function Read-Received {
if (-not (Test-Path $resultFile)) { return "" }
$b = [System.IO.File]::ReadAllBytes($resultFile)
$s = [System.Text.Encoding]::UTF8.GetString($b)
if ($s.Length -gt 0 -and $s[0] -eq [char]0xFEFF) { $s = $s.Substring(1) }
return $s
}
function Show-Cps([string]$s) {
($s.ToCharArray() | ForEach-Object { "U+{0:X4}" -f [int]$_ }) -join " "
}
Write-Host "=== 0. retire ALL earlier targets, verify none holds foreground ==="
$flags = @(Get-ChildItem -Path $env:TEMP -Recurse -Filter "stop.flag" -ErrorAction SilentlyContinue |
Where-Object { $_.FullName -like "*remote*cjk*" -or $_.FullName -like "*remote*proof*" })
foreach ($f in $flags) { (Get-Item $f.FullName).LastWriteTime = Get-Date }
if ($flags.Count -gt 0) { Start-Sleep -Seconds 3 }
$hosts = @(Get-CimInstance Win32_Process -Filter "Name='powershell.exe'" |
Where-Object { $_.CommandLine -match 'text_target\.ps1' -and $_.ProcessId -ne $PID })
foreach ($hx in $hosts) { Stop-Process -Id $hx.ProcessId -Force -ErrorAction SilentlyContinue }
if ($hosts.Count -gt 0) { Start-Sleep -Seconds 1 }
Write-Host " retired $($flags.Count) flag(s), killed $($hosts.Count) host(s); fg now: $([Fg5]::Desc())"
Write-Host "=== 1. agent freshness ==="
$agentExe = "$repo\target\release\agent.exe"
$ag = Get-Process agent -ErrorAction SilentlyContinue
if ($ag -and (Get-Item $agentExe).LastWriteTime -gt $ag.StartTime) {
Stop-Process -Name agent -Force -ErrorAction SilentlyContinue
Start-Sleep -Milliseconds 800; $ag = $null
}
if (-not $ag) {
Start-Process $agentExe -ArgumentList "--port",$Port,"--code",$Code -WindowStyle Hidden
Start-Sleep -Seconds 1
}
Write-Host "=== 2. fresh target window (empty field) ==="
Start-Process powershell.exe -ArgumentList "-NoProfile","-ExecutionPolicy","Bypass",`
"-File","$repo\tools\text_target.ps1","-PidFile",$pidFile,`
"-ResultFile",$resultFile,"-StopFile",$stopFile,"-GrabForeground"
Wait-File $pidFile
$parts = @((Get-Content $pidFile -Raw) -split "`n")
$tid = [int]$parts[0].Trim()
$hwnd = [IntPtr][long]$parts[1].Trim()
Assert-Foreground $hwnd "after launch"
$base0 = Read-Received
Write-Host " target pid=$tid hwnd=$($hwnd.ToInt64()) baseline=[$(Show-Cps $base0)]"
Write-Host "=== 3. phone session ==="
[void](Adb-Shell @("am","force-stop","com.verify.remote"))
[void](Adb-Shell @("am","start","-n","com.verify.remote/.MainActivity",
"--es","host",$PhoneIp,"--ei","port","$Port","--es","code",$Code))
$deadline = (Get-Date).AddSeconds(25)
$ui = ""
while ((Get-Date) -lt $deadline) { Start-Sleep -Milliseconds 900; $ui = Get-Ui; if ($ui -match "RTT=") { break } }
if ($ui -notmatch "RTT=") { throw "phone did not connect" }
Write-Host "=== 4. show IME (tap keyboard button) ==="
$btns = [regex]::Matches($ui, '<node[^>]*class="android\.widget\.Button"[^>]*bounds="\[(\d+),(\d+)\]\[(\d+),(\d+)\]"')
if ($btns.Count -lt 1) { throw "keyboard button not found" }
$b0 = $btns[0]
$kx = [int](([int]$b0.Groups[1].Value + [int]$b0.Groups[3].Value) / 2)
$ky = [int](([int]$b0.Groups[2].Value + [int]$b0.Groups[4].Value) / 2)
[void](Adb-Shell @("input","tap","$kx","$ky"))
Start-Sleep -Seconds 3
$ime = (Adb-Shell @("dumpsys","input_method")).out
if ($ime -notmatch "mInputShown=true") { throw "IME not shown" }
if ($ime -notmatch "mHaveConnection=true") { throw "IME not bound to our InputConnection" }
Write-Host " IME: $([regex]::Match($ime,'mCurId=(\S+)').Groups[1].Value)"
Write-Host "=== 4b/5. pinyin -> IME candidate -> commitText (mode-tolerant) ==="
# Both IMEs on this device may remember an ENGLISH mode, in which case letters
# pass through without composing - which looks identical to "our pipeline lost
# the Hanzi" unless you check. The IME's key views are NOT visible to
# uiautomator (its dump contains only our app's nodes), so the ab/zh toggle is
# reached by a calibrated tap: iFlytek's key sits at display (73,1048) in a
# 1080x2400 capture => device (146,2096) at 2x. Verified 2026-10-30.
# The toggle is STATEFUL, so this is written as an idempotent attempt loop:
# try, and only if we see letters-without-Hanzi do we flip the mode and retry.
# A stray tap here changes a keyboard setting, nothing destructive.
$modeToggle = @(146, 2096)
$expect = -join @([char]0x4F60, [char]0x597D) # NI HAO
$attempt = 0
$new = ""
while ($attempt -lt 2) {
$attempt++
$base = Read-Received
Assert-Foreground $hwnd "attempt ${attempt}: before pinyin"
[void](Adb-Shell @("input","text","nihao"))
Start-Sleep -Milliseconds 1500
$mid = Read-Received
Write-Host " attempt ${attempt} after pinyin (composing): [$(Show-Cps $mid)]"
Assert-Foreground $hwnd "attempt ${attempt}: before SPACE"
[void](Adb-Shell @("input","keyevent","62"))
Start-Sleep -Seconds 2
$got = Read-Received
$new = if ($got.Length -ge $base.Length) { $got.Substring($base.Length) } else { "" }
Write-Host " attempt ${attempt} newly received: [$(Show-Cps $new)]"
if ($new -match [regex]::Escape($expect)) { break }
if ($attempt -eq 1) {
Write-Host " no Hanzi yet -> flipping IME language mode and retrying once"
[void](Adb-Shell @("input","tap","$($modeToggle[0])","$($modeToggle[1])"))
Start-Sleep -Seconds 2
}
}
$diag = [regex]::Match((Get-Ui), 'text="([^"]*RTT=[^"]*)"').Groups[1].Value
Write-Host " phone status: $diag"
Write-Host " expected to contain: $(Show-Cps $expect)"
if ($new -notmatch [regex]::Escape($expect)) {
Write-Host " GROUND TRUTH: [$(Show-Cps $new)]"
Write-Host " U+006E.. letters only => IME stayed in English mode (re-calibrate the toggle tap)"
throw "V6 CHINESE FAIL: IME did not deliver Hanzi through commitText"
}
Write-Host " V6 PASS: pinyin -> IME candidate -> commitText -> v3 Text -> Windows Hanzi"
Write-Host "=== 6. cleanup ==="
[void](Adb-Shell @("input","keyevent","4"))
New-Item -ItemType File -Force -Path $stopFile | Out-Null
Start-Sleep -Milliseconds 500
Write-Host "=== CHINESE IME PROOF COMPLETE ==="

View File

@ -230,9 +230,21 @@ try {
for ($i = 0; $i -lt 6; $i++) { [void](Adb-Shell @("input", "keyevent", "67")) }
[void](Adb-Shell @("input", "keyevent", "48"))
$xml3 = Get-PhoneUiText
$vals = [regex]::Matches($xml3, '<node[^>]*class="android\.widget\.EditText"[^>]*text="([^"]*)"') |
ForEach-Object { $_.Groups[1].Value }
# Attribute order in uiautomator dumps is text= ... class= ... bounds=;
# matching "class=...text=" silently finds nothing (seen 2026-10-30: printed
# an empty form line). Parse per-node and pull attributes independently.
$vals = @()
foreach ($n in [regex]::Matches($xml3, '<node[^>]*>')) {
$v = $n.Value
if ($v -match 'class="android\.widget\.EditText"') {
$vals += [regex]::Match($v, 'text="([^"]*)"').Groups[1].Value
}
}
# NOTE: an EMPTY EditText reports its hint as text (verified: text="被控端 IP"),
# so "field text == typed value" is the assertion, not "field text non-empty".
Write-Host " form now: $($vals -join ' | ')"
if ($vals.Count -lt 3) { throw "expected 3 fields after typing, got $($vals.Count)" }
if ($vals[0] -ne "192.168.1.180") { throw "V4 FAIL: IP field is [$($vals[0])], typing did not land" }
$btn = [regex]::Match($xml3, '<node[^>]*class="android\.widget\.Button"[^>]*bounds="\[(\d+),(\d+)\]\[(\d+),(\d+)\]"')
if (-not $btn.Success) { throw "connect button not found" }
$bc = Center $btn