Arch Linux

Please read this before reporting a bug:
https://wiki.archlinux.org/title/Bug_reporting_guidelines

Do NOT report bugs when a package is just outdated, or it is in the AUR. Use the 'flag out of date' link on the package page, or the Mailing List.

REPEAT: Do NOT report bugs for outdated packages!
Tasklist

FS#50742 - [vim] Re-enable tests

Attached to Project: Arch Linux
Opened by Jonathon (jonathon) - Sunday, 11 September 2016, 19:27 GMT
Last edited by Anatol Pomozov (anatolik) - Thursday, 15 September 2016, 17:20 GMT
Task Type Feature Request
Category Packages: Extra
Status Closed
Assigned To Anatol Pomozov (anatolik)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 1
Private No

Details

Description:
Tests are currently disabled during check() (83, "disable tests because they seem to freeze"). Depending on the build host environment one or both of two extra configuration options will allow tests to complete successfully. Disabling threading is used by Debian to fix builds on their buildhosts [1], Ubuntu also requires setting TERM explicitly to allow builds on Launchpad [2].

This can be accomplished on Arch by switching lines 85 and 88 to one of:

Option 1 (Debian equivalent):
make -j1 test

Option 2 (Ubuntu equivalent):
TERM=xterm make -j1 test

Testing within a Manjaro buildpkg chroot suggests option 2 is most reliable (tests fail with option 1).

[1] https://anonscm.debian.org/cgit/pkg-vim/vim.git/commit/debian/rules?id=a97dd481da59932a8490e7dc3fd8fd8bc9d3af26
[2] http://changelogs.ubuntu.com/changelogs/pool/main/v/vim/vim_7.4.1829-1ubuntu2/changelog
This task depends upon

Closed by  Anatol Pomozov (anatolik)
Thursday, 15 September 2016, 17:20 GMT
Reason for closing:  Fixed
Additional comments about closing:  8.0.0005
Comment by Anatol Pomozov (anatolik) - Monday, 12 September 2016, 15:52 GMT
The test still fail in my chroot env:


Executing Test_whichwrap()
Executing Test_win_tab_autocmd()
Executing Test_window_cmd_cmdwin_with_vsp()
Executing Test_window_cmd_ls0_with_split()
Executing Test_window_cmd_wincmd_gf()
Executing Test_with_directories()
Executing Test_with_partial_callback()
Executing Test_with_tilde()
Executing Test_wrong_arguments()
Executed 189 tests
SKIPPED Test_popup_complete2(): Bug with <c-e> and popupmenu not fixed yet


From test_netbeans.vim:
Found errors in Test_nb_basic():
function RunTheTest[9]..Test_nb_basic[2]..<SNR>5_run_server[1]..RunServer[21]..Nb_basic line 12: Expected 3 but got 49
function RunTheTest[9]..Test_nb_basic[2]..<SNR>5_run_server[1]..RunServer[21]..Nb_basic line 13: Expected 20 but got 3
function RunTheTest[9]..Test_nb_basic[2]..<SNR>5_run_server[1]..RunServer line 23: 'Caught exception: Vim(call):E684: list index out of range: 6'

Test results:


From test_netbeans.vim:
Found errors in Test_nb_basic():
function RunTheTest[9]..Test_nb_basic[2]..<SNR>5_run_server[1]..RunServer[21]..Nb_basic line 12: Expected 3 but got 49
function RunTheTest[9]..Test_nb_basic[2]..<SNR>5_run_server[1]..RunServer[21]..Nb_basic line 13: Expected 20 but got 3
function RunTheTest[9]..Test_nb_basic[2]..<SNR>5_run_server[1]..RunServer line 23: 'Caught exception: Vim(call):E684: list index out of range: 6'
TEST FAILURE
make[2]: *** [Makefile:43: report] Error 1
make[2]: Leaving directory '/build/vim/src/vim-8.0.0002/src/testdir'
make[1]: *** [Makefile:1995: scripttests] Error 2
make[1]: Leaving directory '/build/vim/src/vim-8.0.0002/src'
make: *** [Makefile:39: test] Error 2
==> ERROR: A failure occurred in check().
Comment by Jonathon (jonathon) - Monday, 12 September 2016, 18:39 GMT
OK - so the failure here means tests are running correctly (i.e. not "freezing").

I'm now getting the same test failures. It appears that 7.4.2367 [1] fixes running of so-called "flaky" tests which 7.4.2364 broke [2]. During this window the tests were aborted but not classed as failed, hence why it "worked" for me last night. Test_nb_basic() is one of these "flaky" tests.

The next step is working out why this test fails (or if it's too important to just skip):

src/testdir/test_netbeans.vim, 20-47
```
func Nb_basic(port)
call delete("Xnetbeans")
exe 'nbstart :localhost:' . a:port . ':bunny'
call assert_true(has("netbeans_enabled"))

call WaitFor('len(readfile("Xnetbeans")) > 2')
split +$ README.txt

" Opening README.txt will result in a setDot command
call WaitFor('len(readfile("Xnetbeans")) > 4')
call WaitFor('getcurpos()[1] == 2')
let pos = getcurpos()
call assert_equal(3, pos[1])
call assert_equal(20, pos[2])
close
nbclose

call WaitFor('len(readfile("Xnetbeans")) > 6')
let lines = readfile("Xnetbeans")
call assert_equal('AUTH bunny', lines[0])
call assert_equal('0:version=0 "2.5"', lines[1])
call assert_equal('0:startupDone=0', lines[2])
call assert_equal('0:fileOpened=0 "README.txt" T F', substitute(lines[3], '".*/', '"', ''))

call assert_equal('0:disconnect=1', lines[6])

call delete("Xnetbeans")
endfunc
```

[1] https://github.com/vim/vim/commit/edeb846c1f04a49466992077eaea3396838bf4fd
[2] https://github.com/vim/vim/commit/e1c8c7a6742be6072290f9aa54ae358060d9c42f
Comment by Jonathon (jonathon) - Monday, 12 September 2016, 20:58 GMT
It's down to the default symlink of /usr/bin/python -> /usr/bin/python3. The test seems to expect /usr/bin/python to be python2 (which is against PEP 394 [1], "python should be used in the shebang line only for scripts that are source compatible with both Python 2 and 3.").

Manually switching the symlink gets all tests to pass; I'll look into patching the failing test and reporting upstream.

[1] https://www.python.org/dev/peps/pep-0394/
Comment by Jonathon (jonathon) - Tuesday, 13 September 2016, 17:40 GMT Comment by Jonathon (jonathon) - Wednesday, 14 September 2016, 18:55 GMT
The underlying cause might take some time to track down; upstream can replicate but don't see anything obviously wrong.

In the meantime, the simple fix is to force the tests to use python2:

```
prepare() {
cd vim-$pkgver/src

# define the place for the global (g)vimrc file (set to /etc/vimrc)
sed -i 's|^.*\(#define SYS_.*VIMRC_FILE.*"\) .*$|\1|' feature.h
sed -i 's|^.*\(#define VIMRC_FILE.*"\) .*$|\1|' feature.h

# Use Python 2 to avoid tests failing under Python 3, https://github.com/vim/vim/issues/1070
sed -i "11,14s|'python'|'python2'|" testdir/shared.vim

autoconf

cd "$srcdir"
cp -a vim-$pkgver gvim-$pkgver
}
```

The sed replacement specifies which Python interpreter under which (all) tests should be run. Although each test also has a shebang this isn't used (i.e. it is run as `python test.py`), hence this doesn't work: sed -i 's|python$|python2|' testdir/test_netbeans.py.
Comment by Jonathon (jonathon) - Wednesday, 14 September 2016, 20:43 GMT
Upstream patch [1]:

```
prepare() {
cd vim-$pkgver/src

# define the place for the global (g)vimrc file (set to /etc/vimrc)
sed -i 's|^.*\(#define SYS_.*VIMRC_FILE.*"\) .*$|\1|' feature.h
sed -i 's|^.*\(#define VIMRC_FILE.*"\) .*$|\1|' feature.h

# https://github.com/vim/vim/issues/1070
sed -i "55s|self.request.sendall(response)|self.request.sendall(response.encode('utf-8'))|" testdir/test_netbeans.py

autoconf

cd "$srcdir"
cp -a vim-$pkgver gvim-$pkgver
}
```

[1] https://github.com/vim/vim/issues/1070#issuecomment-247141940
Comment by Jonathon (jonathon) - Thursday, 15 September 2016, 13:53 GMT
Fix in new upstream release:

patch 8.0.0005
Problem: Netbeans test fails with Python 3. (Jonathonf)
Solution: Encode the string before sending it. (closes #1070)

https://github.com/vim/vim/releases/tag/v8.0.0005

Yay!

Now we can go back to the original request. :)


Comment on below as this has been closed:
You know I was the one reporting and debugging it upsteam, right?
Comment by Anatol Pomozov (anatolik) - Thursday, 15 September 2016, 17:20 GMT
Thank you for tracking it with upstream

Loading...