Arch Linux

The test still fail in my chroot env:


Executing Test_whichwrap()
Executing Test_win_tab_autocmd()
Executing Test_window_cmd_cmdwin_with_vsp()
Executing Test_window_cmd_ls0_with_split()
Executing Test_window_cmd_wincmd_gf()
Executing Test_with_directories()
Executing Test_with_partial_callback()
Executing Test_with_tilde()
Executing Test_wrong_arguments()
Executed 189 tests
SKIPPED Test_popup_complete2(): Bug with <c-e> and popupmenu not fixed yet


From test_netbeans.vim:
Found errors in Test_nb_basic():
function RunTheTest[9]..Test_nb_basic[2]..<SNR>5_run_server[1]..RunServer[21]..Nb_basic line 12: Expected 3 but got 49
function RunTheTest[9]..Test_nb_basic[2]..<SNR>5_run_server[1]..RunServer[21]..Nb_basic line 13: Expected 20 but got 3
function RunTheTest[9]..Test_nb_basic[2]..<SNR>5_run_server[1]..RunServer line 23: 'Caught exception: Vim(call):E684: list index out of range: 6'

Test results:


From test_netbeans.vim:
Found errors in Test_nb_basic():
function RunTheTest[9]..Test_nb_basic[2]..<SNR>5_run_server[1]..RunServer[21]..Nb_basic line 12: Expected 3 but got 49
function RunTheTest[9]..Test_nb_basic[2]..<SNR>5_run_server[1]..RunServer[21]..Nb_basic line 13: Expected 20 but got 3
function RunTheTest[9]..Test_nb_basic[2]..<SNR>5_run_server[1]..RunServer line 23: 'Caught exception: Vim(call):E684: list index out of range: 6'
TEST FAILURE
make[2]: *** [Makefile:43: report] Error 1
make[2]: Leaving directory '/build/vim/src/vim-8.0.0002/src/testdir'
make[1]: *** [Makefile:1995: scripttests] Error 2
make[1]: Leaving directory '/build/vim/src/vim-8.0.0002/src'
make: *** [Makefile:39: test] Error 2
==> ERROR: A failure occurred in check().

OK - so the failure here means tests are running correctly (i.e. not "freezing").

I'm now getting the same test failures. It appears that 7.4.2367 [1] fixes running of so-called "flaky" tests which 7.4.2364 broke [2]. During this window the tests were aborted but not classed as failed, hence why it "worked" for me last night. Test_nb_basic() is one of these "flaky" tests.

The next step is working out why this test fails (or if it's too important to just skip):

src/testdir/test_netbeans.vim, 20-47
```
func Nb_basic(port)
call delete("Xnetbeans")
exe 'nbstart :localhost:' . a:port . ':bunny'
call assert_true(has("netbeans_enabled"))

call WaitFor('len(readfile("Xnetbeans")) > 2')
split +$ README.txt

" Opening README.txt will result in a setDot command
call WaitFor('len(readfile("Xnetbeans")) > 4')
call WaitFor('getcurpos()[1] == 2')
let pos = getcurpos()
call assert_equal(3, pos[1])
call assert_equal(20, pos[2])
close
nbclose

call WaitFor('len(readfile("Xnetbeans")) > 6')
let lines = readfile("Xnetbeans")
call assert_equal('AUTH bunny', lines[0])
call assert_equal('0:version=0 "2.5"', lines[1])
call assert_equal('0:startupDone=0', lines[2])
call assert_equal('0:fileOpened=0 "README.txt" T F', substitute(lines[3], '".*/', '"', ''))

call assert_equal('0:disconnect=1', lines[6])

call delete("Xnetbeans")
endfunc
```

[1] https://github.com/vim/vim/commit/edeb846c1f04a49466992077eaea3396838bf4fd
[2] https://github.com/vim/vim/commit/e1c8c7a6742be6072290f9aa54ae358060d9c42f

It's down to the default symlink of /usr/bin/python -> /usr/bin/python3. The test seems to expect /usr/bin/python to be python2 (which is against PEP 394 [1], "python should be used in the shebang line only for scripts that are source compatible with both Python 2 and 3.").

Manually switching the symlink gets all tests to pass; I'll look into patching the failing test and reporting upstream.

[1] https://www.python.org/dev/peps/pep-0394/

Upstream bug report: https://github.com/vim/vim/issues/1070

The underlying cause might take some time to track down; upstream can replicate but don't see anything obviously wrong.

In the meantime, the simple fix is to force the tests to use python2:

```
prepare() {
cd vim-$pkgver/src

# define the place for the global (g)vimrc file (set to /etc/vimrc)
sed -i 's|^.*\(#define SYS_.*VIMRC_FILE.*"\) .*$|\1|' feature.h
sed -i 's|^.*\(#define VIMRC_FILE.*"\) .*$|\1|' feature.h

# Use Python 2 to avoid tests failing under Python 3, https://github.com/vim/vim/issues/1070
sed -i "11,14s|'python'|'python2'|" testdir/shared.vim

autoconf

cd "$srcdir"
cp -a vim-$pkgver gvim-$pkgver
}
```

The sed replacement specifies which Python interpreter under which (all) tests should be run. Although each test also has a shebang this isn't used (i.e. it is run as `python test.py`), hence this doesn't work: sed -i 's|python$|python2|' testdir/test_netbeans.py.

Upstream patch [1]:

```
prepare() {
cd vim-$pkgver/src

# define the place for the global (g)vimrc file (set to /etc/vimrc)
sed -i 's|^.*\(#define SYS_.*VIMRC_FILE.*"\) .*$|\1|' feature.h
sed -i 's|^.*\(#define VIMRC_FILE.*"\) .*$|\1|' feature.h

# https://github.com/vim/vim/issues/1070
sed -i "55s|self.request.sendall(response)|self.request.sendall(response.encode('utf-8'))|" testdir/test_netbeans.py

autoconf

cd "$srcdir"
cp -a vim-$pkgver gvim-$pkgver
}
```

[1] https://github.com/vim/vim/issues/1070#issuecomment-247141940

Fix in new upstream release:

patch 8.0.0005
Problem: Netbeans test fails with Python 3. (Jonathonf)
Solution: Encode the string before sending it. (closes #1070)

https://github.com/vim/vim/releases/tag/v8.0.0005

Yay!

Now we can go back to the original request. :)


Comment on below as this has been closed:
You know I was the one reporting and debugging it upsteam, right?

Thank you for tracking it with upstream