It has been awhile since I released the TopIcons-plus Gnome-Shell extension.

I had not advertised it here because it was not really ready or stable, but now I believe it is taking shape.

How is Topicons-plus useful ?

The Gnome developers want to kill system tray icons, which are displayed in what they call the legacy tray.

Such icons are familiar to everybody: messaging programs like RocketChat or Telegram, e-mail clients like Thunderbird, Dropbox, KeepassX, etc.

Gnome designers think such a design belongs to past, is flawed in many ways (status or menu?) and should be useless with modern environments with a dock and a powerful notification system.

I would not comment on that and I actually believe they are right.

However, the legacy tray they propose is horrible. It is hidden most of the time and you have to click to open it before accessing to your icons. It is very painful, and it is done on purpose, to clearly send a message that it should not be used anymore by application developers.

Well, but what about the existing applications ?

They are not going away all the sudden. As a user, I still need them.

And it is open-source, mostly developed on free time: developers are not going to re-implement everything just for the Gnome ecosystem…

That is where I think an extension like TopIcons-plus is useful. It removes the hassle of this legacy tray by bringing back the icons to the top bar, so they are always visible.

Latest release

It comes with extra features, like styling (opacity, desaturation, size) and positioning.

The latest release should be in pretty good shape. If you don’t want to use the Github code, be patient: it should get validated on the Gnome website within the next days.

Enjoy!

TopIcons-Plus v18, tray icons centered

TopIcons-Plus v18, tray icons centered

I am, like many, the unfortunate user of a laptop with Intel graphics (Thinkpad T460 to be precise). Why unfortunate? Because the graphic driver provided by Intel sucks.

i915, as it is being called, really has been sucking for years, and it is known for that (just google it, if you don’t believe me).

For the sake of completeness, here is the exact model with which I experienced some issues:

%  lspci
00:00.0 Host bridge: Intel Corporation Skylake Host Bridge/DRAM Registers (rev 08)
00:02.0 VGA compatible controller: Intel Corporation HD Graphics 520 (rev 07)
...

For performance, remove the X11 Intel driver

First, its X11 module is generally under-performing under X11, so I just removed it to have X11 using modsettings. These are the instructions for Fedora (24), but you can virtually do something similar for any distribution:

% dnf remove xorg-x11-drv-intel

Do not worry, it just remove the X11 part of the driver, not the kernel driver itself.

Login, logout, job done: you should have less lags with desktop environments like gnome-shell.

For stability, disable RC6

I experienced frequent, daily freezes of my work session. The display would totally hang or display a blank screen, forcing me to cold reboot the computer.

Here is an extract of the dmesg kernel traces leading the the crash (it is a bit lengthy, but that may help people to find this post):

oct. 06 11:00:25 localhost.localdomain kernel: [drm] RC6 on
oct. 06 11:00:44 localhost.localdomain kernel: [drm] RC6 on
oct. 06 11:01:01 localhost.localdomain kernel: [drm] RC6 on
oct. 06 11:01:25 localhost.localdomain kernel: [drm] RC6 on
oct. 06 11:01:43 localhost.localdomain kernel: [drm] RC6 on
oct. 06 11:02:01 localhost.localdomain kernel: [drm] RC6 on
oct. 06 11:02:22 localhost.localdomain kernel: [drm] RC6 on
oct. 06 11:04:08 localhost.localdomain kernel: [drm:intel_pipe_update_end [i915]] *ERROR* Atomic update failure on pipe B (start=6364 end=6365) time 287 us, min 954, max 959, scanline start 950, end 967
oct. 06 11:13:58 localhost.localdomain kernel: [drm:intel_pipe_update_end [i915]] *ERROR* Atomic update failure on pipe B (start=41777 end=41778) time 340 us, min 954, max 959, scanline start 946, end 967
oct. 06 11:20:18 localhost.localdomain kernel: [drm:intel_pipe_update_end [i915]] *ERROR* Atomic update failure on pipe B (start=64583 end=64584) time 284 us, min 954, max 959, scanline start 946, end 964
oct. 06 11:20:33 localhost.localdomain kernel: [drm:intel_pipe_update_end [i915]] *ERROR* Atomic update failure on pipe A (start=65517 end=65518) time 284 us, min 1073, max 1079, scanline start 1071, end 1091
oct. 06 11:28:27 localhost.localdomain kernel: [drm:intel_cpu_fifo_underrun_irq_handler [i915]] *ERROR* CPU pipe B FIFO underrun
oct. 06 11:31:53 localhost.localdomain kernel: [drm:intel_pipe_update_end [i915]] *ERROR* Atomic update failure on pipe A (start=106339 end=106340) time 285 us, min 1073, max 1079, scanline start 1066, end 1086
oct. 06 11:33:58 localhost.localdomain kernel: [drm:intel_pipe_update_end [i915]] *ERROR* Atomic update failure on pipe B (start=113803 end=113804) time 287 us, min 954, max 959, scanline start 948, end 966
oct. 06 11:35:13 localhost.localdomain kernel: [drm:intel_pipe_update_end [i915]] *ERROR* Atomic update failure on pipe A (start=118345 end=118346) time 285 us, min 1073, max 1079, scanline start 1062, end 1081
oct. 06 11:52:59 localhost.localdomain kernel: [drm:intel_pipe_update_end [i915]] *ERROR* Atomic update failure on pipe A (start=182278 end=182279) time 282 us, min 1073, max 1079, scanline start 1064, end 1084
oct. 06 12:01:29 localhost.localdomain kernel: [drm:intel_pipe_update_end [i915]] *ERROR* Atomic update failure on pipe A (start=212893 end=212894) time 284 us, min 1073, max 1079, scanline start 1068, end 1088
oct. 06 12:02:44 localhost.localdomain kernel: [drm:intel_pipe_update_end [i915]] *ERROR* Atomic update failure on pipe A (start=217395 end=217396) time 282 us, min 1073, max 1079, scanline start 1068, end 1088
oct. 06 12:02:49 localhost.localdomain kernel: [drm:intel_pipe_update_end [i915]] *ERROR* Atomic update failure on pipe B (start=217642 end=217643) time 247 us, min 954, max 959, scanline start 949, end 964
oct. 06 12:03:54 localhost.localdomain kernel: [drm:intel_pipe_update_end [i915]] *ERROR* Atomic update failure on pipe A (start=221597 end=221598) time 281 us, min 1073, max 1079, scanline start 1067, end 1086
oct. 06 12:05:49 localhost.localdomain kernel: [drm:intel_pipe_update_end [i915]] *ERROR* Atomic update failure on pipe B (start=228446 end=228447) time 290 us, min 954, max 959, scanline start 948, end 966
oct. 06 12:17:32 localhost.localdomain kernel: [drm] RC6 on
oct. 06 12:18:01 localhost.localdomain kernel: [drm] RC6 on
oct. 06 12:18:23 localhost.localdomain kernel: [drm] RC6 on
oct. 06 12:18:44 localhost.localdomain kernel: [drm] RC6 on
oct. 06 12:19:01 localhost.localdomain kernel: [drm] RC6 on
oct. 06 12:19:25 localhost.localdomain kernel: [drm] RC6 on
oct. 06 12:19:44 localhost.localdomain kernel: [drm] RC6 on
oct. 06 12:20:01 localhost.localdomain kernel: [drm] RC6 on
oct. 06 12:20:25 localhost.localdomain kernel: [drm] RC6 on
oct. 06 12:20:44 localhost.localdomain kernel: [drm] RC6 on
oct. 06 12:20:52 localhost.localdomain kernel: [drm] stuck on render ring
oct. 06 12:20:52 localhost.localdomain kernel: [drm] GPU HANG: ecode 9:0:0xfffffffe, in Xorg [2322], reason: Engine(s) hung, action: reset
oct. 06 12:20:52 localhost.localdomain kernel: [drm] GPU hangs can indicate a bug anywhere in the entire gfx stack, including userspace.
oct. 06 12:20:52 localhost.localdomain kernel: [drm] Please file a _new_ bug report on bugs.freedesktop.org against DRI -> DRM/Intel
oct. 06 12:20:52 localhost.localdomain kernel: [drm] drm/i915 developers can then reassign to the right component if it's not a kernel issue.
oct. 06 12:20:52 localhost.localdomain kernel: [drm] The gpu crash dump is required to analyze gpu hangs, so please always attach it.
oct. 06 12:20:52 localhost.localdomain kernel: [drm] GPU crash dump saved to /sys/class/drm/card0/error
oct. 06 12:20:52 localhost.localdomain kernel: drm/i915: Resetting chip after gpu hang
oct. 06 12:20:54 localhost.localdomain kernel: [drm] RC6 on
oct. 06 12:21:05 localhost.localdomain kernel: [drm] stuck on render ring
oct. 06 12:21:05 localhost.localdomain kernel: [drm] GPU HANG: ecode 9:0:0xfffffffe, in gnome-shell [2532], reason: Engine(s) hung, action: reset
oct. 06 12:21:05 localhost.localdomain kernel: drm/i915: Resetting chip after gpu hang
oct. 06 12:21:07 localhost.localdomain kernel: [drm] RC6 on
oct. 06 12:21:15 localhost.localdomain kernel: [drm] stuck on render ring
oct. 06 12:21:15 localhost.localdomain kernel: [drm] GPU HANG: ecode 9:0:0xfffffffe, in gnome-shell [2532], reason: Engine(s) hung, action: reset
oct. 06 12:21:15 localhost.localdomain kernel: drm/i915: Resetting chip after gpu hang
oct. 06 12:21:17 localhost.localdomain kernel: [drm] RC6 on
oct. 06 12:21:27 localhost.localdomain kernel: [drm] stuck on render ring
oct. 06 12:21:27 localhost.localdomain kernel: [drm] GPU HANG: ecode 9:0:0xfffffffe, in Xorg [2322], reason: Engine(s) hung, action: reset
oct. 06 12:21:27 localhost.localdomain kernel: drm/i915: Resetting chip after gpu hang
oct. 06 12:21:29 localhost.localdomain kernel: [drm] RC6 on
oct. 06 12:21:37 localhost.localdomain kernel: [drm] stuck on render ring
oct. 06 12:21:37 localhost.localdomain kernel: [drm] GPU HANG: ecode 9:0:0xfffffffe, in Xorg [2322], reason: Engine(s) hung, action: reset
oct. 06 12:21:37 localhost.localdomain kernel: drm/i915: Resetting chip after gpu hang
oct. 06 12:21:39 localhost.localdomain kernel: [drm] RC6 on
oct. 06 12:21:43 localhost.localdomain kernel: ------------[ cut here ]------------
oct. 06 12:21:43 localhost.localdomain kernel: WARNING: CPU: 0 PID: 1109 at drivers/gpu/drm/i915/intel_display.c:13533 intel_atomic_commit+0x13b8/0x1470 [i915]
oct. 06 12:21:43 localhost.localdomain kernel: pipe A vblank wait timed out
oct. 06 12:21:43 localhost.localdomain kernel: Modules linked in: tun nfnetlink_queue nfnetlink_log uas usb_storage xt_nat veth rfcomm ccm ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype br_netfilter ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack dm_thin_pool dm_persistent_data dm_bio_prison loop ip_set nfnetlink ebtable_nat ebtable_broute bridge ip6table_raw ip6table_mangle ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_security iptable_raw iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_security ebtable_filter ebtables ip6table_filter ip6_tables vmnet(O) ppdev parport_pc parport vboxpci(O) vboxnetadp(O) vboxnetflt(O) fuse vmw_vsock_vmci_transport vsock vmw_vmci vmmon(O) vboxdrv(O) cmac bnep cpufreq_stats vfat fat arc4 iTCO_wdt snd_soc_skl iTCO_vendor_support
oct. 06 12:21:43 localhost.localdomain kernel:  snd_soc_skl_ipc snd_hda_codec_hdmi snd_soc_sst_ipc intel_rapl snd_soc_sst_dsp x86_pkg_temp_thermal snd_hda_codec_realtek snd_hda_ext_core intel_powerclamp snd_hda_codec_generic coretemp snd_soc_sst_match kvm_intel snd_soc_core kvm snd_compress snd_pcm_dmaengine ac97_bus snd_hda_intel snd_hda_codec iwlmvm snd_hda_core mac80211 irqbypass intel_cstate intel_rapl_perf snd_hwdep snd_seq snd_seq_device btusb snd_pcm btrtl uvcvideo btbcm btintel videobuf2_vmalloc videobuf2_memops joydev bluetooth videobuf2_v4l2 iwlwifi i2c_i801 snd_timer videobuf2_core cfg80211 rtsx_pci_ms videodev memstick media mei_me mei shpchp thinkpad_acpi intel_pch_thermal snd soundcore rfkill wmi tpm_tis tpm nfsd auth_rpcgss nfs_acl lockd grace sunrpc xfs libcrc32c dm_crypt hid_logitech_hidpp hid_logitech_dj 8021q garp
oct. 06 12:21:43 localhost.localdomain kernel:  stp llc mrp i915 rtsx_pci_sdmmc mmc_core crct10dif_pclmul crc32_pclmul crc32c_intel e1000e i2c_algo_bit drm_kms_helper ghash_clmulni_intel drm serio_raw ptp pps_core rtsx_pci video fjes
oct. 06 12:21:43 localhost.localdomain kernel: CPU: 0 PID: 1109 Comm: systemd-logind Tainted: G     U  W  O    4.7.5-200.fc24.x86_64 #1
oct. 06 12:21:43 localhost.localdomain kernel: Hardware name: LENOVO 20FNCTO1WW/20FNCTO1WW, BIOS R06ET42W (1.16 ) 09/20/2016
oct. 06 12:21:43 localhost.localdomain kernel:  0000000000000286 0000000018e0c148 ffff8800d283b850 ffffffffb63daaaf
oct. 06 12:21:43 localhost.localdomain kernel:  ffff8800d283b8a0 0000000000000000 ffff8800d283b890 ffffffffb60a0b0b
oct. 06 12:21:43 localhost.localdomain kernel:  000034dd00000000 ffff88040f607000 0000000000000000 0000000000000000
oct. 06 12:21:43 localhost.localdomain kernel: Call Trace:
oct. 06 12:21:43 localhost.localdomain kernel:  [<ffffffffb63daaaf>] dump_stack+0x63/0x84
oct. 06 12:21:43 localhost.localdomain kernel:  [<ffffffffb60a0b0b>] __warn+0xcb/0xf0
oct. 06 12:21:43 localhost.localdomain kernel:  [<ffffffffb60a0b8f>] warn_slowpath_fmt+0x5f/0x80
oct. 06 12:21:43 localhost.localdomain kernel:  [<ffffffffb60e4483>] ? finish_wait+0x53/0x70
oct. 06 12:21:43 localhost.localdomain kernel:  [<ffffffffc05046a8>] intel_atomic_commit+0x13b8/0x1470 [i915]
oct. 06 12:21:43 localhost.localdomain kernel:  [<ffffffffb60e46e0>] ? prepare_to_wait_event+0xf0/0xf0
oct. 06 12:21:43 localhost.localdomain kernel:  [<ffffffffc0380ba7>] drm_atomic_commit+0x37/0x60 [drm]
oct. 06 12:21:43 localhost.localdomain kernel:  [<ffffffffc03e21e8>] restore_fbdev_mode+0x238/0x260 [drm_kms_helper]
oct. 06 12:21:43 localhost.localdomain kernel:  [<ffffffffc03e45d4>] drm_fb_helper_restore_fbdev_mode_unlocked+0x34/0x80 [drm_kms_helper]
oct. 06 12:21:43 localhost.localdomain kernel:  [<ffffffffc03e464d>] drm_fb_helper_set_par+0x2d/0x50 [drm_kms_helper]
oct. 06 12:21:43 localhost.localdomain kernel:  [<ffffffffc051ea4a>] intel_fbdev_set_par+0x1a/0x60 [i915]
oct. 06 12:21:43 localhost.localdomain kernel:  [<ffffffffb645a6b6>] fb_set_var+0x236/0x460
oct. 06 12:21:43 localhost.localdomain kernel:  [<ffffffffb60e4004>] ? __wake_up+0x44/0x50
oct. 06 12:21:43 localhost.localdomain kernel:  [<ffffffffb67ea562>] ? down_write+0x12/0x40
oct. 06 12:21:43 localhost.localdomain kernel:  [<ffffffffb64caabb>] ? tty_unthrottle+0x3b/0x60
oct. 06 12:21:43 localhost.localdomain kernel:  [<ffffffffb645074f>] fbcon_blank+0x30f/0x350
oct. 06 12:21:43 localhost.localdomain kernel:  [<ffffffffb64db0b2>] do_unblank_screen+0xd2/0x1a0
oct. 06 12:21:43 localhost.localdomain kernel:  [<ffffffffb64d0ef6>] vt_ioctl+0x4f6/0x1270
oct. 06 12:21:43 localhost.localdomain kernel:  [<ffffffffb64c537a>] tty_ioctl+0x35a/0xc50
oct. 06 12:21:43 localhost.localdomain kernel:  [<ffffffffb625f909>] ? dput+0xd9/0x260
oct. 06 12:21:43 localhost.localdomain kernel:  [<ffffffffb625b4b2>] do_vfs_ioctl+0xa2/0x5d0
oct. 06 12:21:43 localhost.localdomain kernel:  [<ffffffffb60be9b8>] ? task_work_run+0x88/0xb0
oct. 06 12:21:43 localhost.localdomain kernel:  [<ffffffffb625ba59>] SyS_ioctl+0x79/0x90
oct. 06 12:21:43 localhost.localdomain kernel:  [<ffffffffb67ec572>] entry_SYSCALL_64_fastpath+0x1a/0xa4
oct. 06 12:21:43 localhost.localdomain kernel: ---[ end trace 9f62268cfd97b6cb ]---

As seen above, it would always happen after a while and when the graphic chip goes to the RC6 power saving mode.

After searching on different forum and wikis, I applied the proposed solution of completly disabling the RC6 mode. Add the part in red to the kernel options in your grub configuration file:

%  cat /etc/default/grub 
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.lvm.lv=fedora/root rd.luks.uuid=luks-a9d14a0e-6c22-4976-919a-d216bd69d563 rd.lvm.lv=fedora/swap resume=/dev/dm-2 quiet splash i915.enable_rc6=0"
GRUB_DISABLE_RECOVERY="true"

Then, just rebuild grub and reboot. If you are on a UEFI system (as root):

grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg

Or, for legacy BIOS:

grub2-mkconfig -o /boot/grub2/grub.cfg

Finally reboot and you are done.

There is a caveat however, as it will probably cause some battery drain. With Powertop, I measured a consumption increase of around 6 W (8 to 13W), which caused my battery life to drop from approximately 10h to 5h30.

Still enough and a acceptable price to pay to work reliably without risking a complete system hang.

But, if I had to buy a computer personally, I would make sure that it has an nvidia card. Yeah, I know that there proprietary blob has its caveats too, but from what I heard it is probably more stable.

Graphic drivers have always been a problem for “Linux on the desktop”.

References

• https://wiki.archlinux.org/index.php/intel_graphics
• https://wiki.gentoo.org/wiki/Intel

I have kept playing with Docker recently, just for fun and to learn.

It is very powerful, but still young. It quickly shows some limit when it comes to security or persistence. There are some workarounds, yet more or less complex, more or less hacky.

Indeed, I had some issues with Etherpad, which is a Nodejs application, and its integration into Docker.

Initially, I made something quite simple, so my Dockerfile ended like that:

USER etherpad
CMD ["node","/opt/etherpad-lite/node_modules/ep_etherpad-lite/node/server.js"]

Thus, I simply start the app with a low privileges user.

It worked, but I had two issues:

1. Docker was not able to stop it nicely. Instead, it timed out after 10 sec and finally killed the app and the container altogether.
2. No persistence of any kind, of course.

I decided to tackle these two issues to understand what was going on behind.

The PID 1 issue

I could not understand immediately the first issue: why was Docker unable to terminate the container properly?

After wandering a few hours on wrong paths (trying to get through with Nodejs nodemon or supervisor), I finally found some good articles, explaining that Docker misses an init system to catch signals, wich causes some issues with applications started with a PID = 1, which cannot be killed, or with Bash (the shell doesn’t handle transmitted signals.

I am not going to repeat poorly what has already been explained very well, so I encourage you to read this two excellent posts:

• The PID 1 zombie reaping problem
• Trapping signals in Docker containers

You will also find a lot of bug reports in the Docker github about this issue, and a lot of hacky or overkilling solutions.

In my opinion, the most elegant solution among them is to use a launcher program, very simple and dedicated to catch and handle signal.

I chose to use Dumb-init, as it is well packaged (there are plenty of options) and seems to be well maintained.

So, after installing Dump-init in the Dockerfile, the CMD line should now look like this:

USER etherpad
CMD ["dumb-init","node","/opt/etherpad-lite/node_modules/ep_etherpad-lite/node/server.js"]

And indeed, as expected, docker stop now works flawlessly.

Volume permissions

This is where I had the toughest issue, although it is supposed to be straightforward with volumes.

Volumes enable to share files or folders between host and containers, or between containers solely. There are plenty of possibilities, nicely illustrated on this blog:

• Docker: storage patterns for persistence

And it works very well…. as long as you application runs as root.

In my case, for instance, Etherpad runs with a low privileged user, which is highly recommended. At startup, it creates a sqlite database, etherpad.db,  in its ./var folder.

Mounting a volume, of any kind, over the ./var folder, would result in a folder with root only permissions. Subsequently, of course, the launch of Etherpad from the CMD command would fail miserably.

Simple solutions like chown in the Dockerfile don’t work, because they apply before the mount. The mount occurs at runtime and works like a standard Linux mount: it is created by the docker daemon, with root permissions, over possibly existing data.

My solution was to completely change the way Etherpad is started. I now use an external script which is started at runtime:

1. First, it applies the appropriate permissions to the mounted volume with chown,
2. Then, it starts Etherpad with a low privileged user thanks to a su hack.

So now the Dockerfile ends with:

VOLUME /opt/etherpad-lite/var
ADD run-docker.sh ./bin/
CMD ["./bin/run-docker.sh"]

And here is the script:

#!/bin/bash

chown -R etherpad:etherpad /opt/etherpad-lite/var
su etherpad -s /bin/bash -c  "dumb-init node /opt/etherpad-lite/node_modules/ep_etherpad-lite/no
de/server.js"

I use a data volume for persistency, so the run command looks like this:

docker run -d --name etherpad -p 80:9001 -v etherpad:/opt/etherpad-lite/var -t debian-etherpad

Far from being ideal, but it works. I really hope some features are coming to bring more options in this area, especially in the Dockerfile.

Some final thoughts

Globally, we can still hope a lot of improvements in security, because when I look at many Dockerfiles around, I see two behaviors:

• A lot of people don’t care and everything is happily running as root, from unauthenticated third-party images or binaries…
• Some people do care but end up with dirty hacks, because there is no other way to do so.

It is scary and so far from the Linux philosophy. Let’s wait for the enhancements to come.

You can find the complete updated Dockerfile on this github page.

While we are on this topic, have a look to this nice post with some nice tips and tricks for Docker.

Here they are:

• Debian-based Etherpad
• Debian-based Phishing Frenzy
• Debian-based Tor Browser

I used to use VirtualBox guests, but maintaining them was a hassle (updates, snapshots, disk defragmation and shrinking, etc.).

It makes perfect sense to use Docker just for that, and on top of that it consumes much fewer resources. Starting with the disk usage : all these containers along with their image stands below 1 GB!

The fact that I am using Btrfs as the underlying storage driver is not for nothing: compression is extremely efficient on images!

Note that my Dockerfiles have nothing special, you can actually find others on the Internet (and I was inspired by some).

There are a few differences, however:

• I care much about security, so at least I try to make Web services not running as root, even if it is inside a container (the root user is still the same as on the host, so let’s make a compromise as unlikely as possible).
• I like simple things, so I tried to keep everything straightforward and simplified some stuff.
• I don’t like to waste disk space. So when I some Dockerfiles based on Ubuntu, Debian Wheezy, Debian Jessie, Fedora, etc., I try to unify all of them under Debian “stable” (so as of today, Jessie). Why bother with useless images? I chose a versatile and common server distribution and I am trying to stick with it.

While I was playing, I had two things bothering me:

• No quota support: for a Samba sharing guest that I have, I would have liked to implement quotas from within the container. There is no support for that at the moment, and the global limitation by container is not nice (and once you choose a big size, you can’t go backward for existing containers…). I have a dedicated partition for Docker, so, while not perfect, it is okay for now.
• The devicemapper storage driver totally sucks at this time: free space is never reclaimed after you delete images or containers! So the more you use Docker, the more your partition gets full.

I have recently tested Btrfs as the file system for my /home partition (which was previously on ext4).

I have been impressed by what this file system enables to do, but also came to the conclusion that it is not for me.

As a quick reminder, the goal of this file system is to bring to Linux a fully featured file system similar to zfs. Some of these features promise a lot of awesomeness: snapshots, native RAID, automatic defragmentation and repairs, etc.

Wouldn’t it be cool to have such a file system for your data? Among them, snapshotting really is a killer feature. See it as a global git for all your data. You can track any file history, make a diff comparison on them and revert back to a chosen version, anytime and on-line.

Btrfs has been under development for a while and it is still undergoing. However, the first stable version has finally been released last year.

Many people warn that it is not production ready yet. It seems obvious for critical production systems, under heavy load or using the most advanced features (e.g. RAID). But what about a simple /home, mainly using snapshots (which have been around for a while)?

You will see that there are still some issues with virtualization.

Disclaimer 1: this is in no way a review or a benchmark of Btrfs. Consider it simply as some feedback for my specific use case.

Getting ready

This chapter is a summary of procedures found in various resources, along with my feedback.

Disclaimer 2: First of all, make several backup of your entire /home. And make sure that it is operational and complete. Anyway, beware that there is obviously some inherent risk for your data in manipulating your home partition. So, do not come back to insult me if you lose any data.

First, note that there is a conversion utility btrfs-convert, to convert an existing ext4 partition to btrfs. While this sounds cool, it did not work well with my partition, leading to many corrupted inodes.

So my advice is to just make a good backup of your home:

% rsync -av /home /your/backup/

Then, log out and format the partition as root:

# mount | grep home
/dev/mapper/system-home on /home type ext4 (rw,noatime,data=ordered)
# umount /home
# mkfs.btrfs /dev/mapper/system-home

Change the file system and its options in /etc/fstab. For example:

/dev/system/home     /home     ext4     defaults,noatime     1 1

should become (also note the change on the last digit):

/dev/system/home   /home    btrfs  defaults,noatime,ssd,space_cache,compress=lzo    1 0

Re-mount /home and you are done!

Snapper

The main purpose for me to test Btrfs was the snapshot feature, in the hope to keep a version history of each file and avoid accidental deletions and changes.

Of course, one could use the Btrfs commands and implement snapshots manually. But why reinventing the wheel?

The guys behind snapper  already made a service especially for that. It is basically a wrapper over Btrfs that will make automatic snapshots in the background, based on your frequency settings, and ease their handling.

Once installed, it can be enabled with the following command:

# snapper -c home create-config /home

It has the effect of creating a configuration file, where you can adjust the number of snapshots you want to keep per day, week, month, etc. Of course, don’t keep too much data as it will waste free space, especially if you happen to move large amounts of data. Hourly and daily snapshots are OK, as they would be cleaned up quickly. But monthly or yearly snapshots would consume a lot of space and would be pretty useless for a /home.

Here is what I used, without consuming much more than 10 GB:

# subvolume to snapshot
SUBVOLUME="/home"

# filesystem type
FSTYPE="btrfs"

# users and groups allowed to work with config
ALLOW_USERS=""
ALLOW_GROUPS="

# sync users and groups from ALLOW_USERS and ALLOW_GROUPS to .snapshots
# directory
SYNC_ACL="no"

# start comparing pre- and post-snapshot in background after creating
# post-snapshot
BACKGROUND_COMPARISON="yes"

# run daily number cleanup
NUMBER_CLEANUP="yes"

# limit for number cleanup
NUMBER_MIN_AGE="1800"
NUMBER_LIMIT="10"
NUMBER_LIMIT_IMPORTANT="5"

# create hourly snapshots
TIMELINE_CREATE="yes"

# cleanup hourly snapshots after some time
TIMELINE_CLEANUP="yes"

# limits for timeline cleanup
TIMELINE_MIN_AGE="1800"
TIMELINE_LIMIT_HOURLY="10"
TIMELINE_LIMIT_DAILY="7"
TIMELINE_LIMIT_WEEKLY="2"
TIMELINE_LIMIT_MONTHLY="0"
TIMELINE_LIMIT_YEARLY="0"

# cleanup empty pre-post-pairs
EMPTY_PRE_POST_CLEANUP="yes"

# limits for empty pre-post-pair cleanup
EMPTY_PRE_POST_MIN_AGE="1800"

Now, let’s play a little. In the following sequence, we create a file containing “Hello World!”, we then create a manual snapshot, change the file and display the differences:

# vim test.txt
# snapper -c home create --description "before test"
# vim test.txt
# sudo snapper -c home list
Type   | # | Pre # | Date                     | User | Cleanup  | Description  | Userdata
-------+---+-------+--------------------------+------+----------+--------------+---------
single | 0 |       |                          | root |          | current      | 
single | 1 |       | Sun Mar 13 19:44:21 2016 | root |          | before test  | 
single | 2 |       | Sun Mar 13 19:45:12 2016 | root |          | created test | 
single | 3 |       | Sun Mar 13 19:52:39 2016 | root |          | update test  | 
single | 4 |       | Sun Mar 13 20:00:01 2016 | root | timeline | timeline     | 
single | 5 |       | Sun Mar 13 21:00:01 2016 | root | timeline | timeline     | 
single | 6 |       | Sun Mar 13 22:00:01 2016 | root | timeline | timeline     | 
# snapper -c home status 1..0
--- "/home/.snapshots/2/snapshot/phocean/test.txt" 2016-03-13 19:44:53.370641373 +0100
+++ "/home/phocean/test.txt" 2016-03-13 19:45:27.226586459 +0100
@@ -1 +1,2 @@
Hell World!
+Good bye.
@@ -0,0 +1,2 @@
+Hell World!
+Good bye

Neat, isn’t it? Now, what if we decide to restore the file to this snapshot:

snapper -c home undochange 1..0 /home/phocean/test.txt

That’s it!

Note that all these operations can be done against the entire partition (no argument needed), a folder or a file.

Pros

Regarding regular files, I had no issue at all. After a week of intensive use, I already the occasion to enjoy the benefits of having snapshots and being able to restore a file.

On the performance side, even though I haven’t done any benchmark, it is a least as fast as ext4. It is said that under some conditions, compression can be a big read rate boost.

On the compression side, on my partition of 400 GB, it allowed me to reclaim around 20 GB of space. Of course, the gain you can expect is totally related to the sorts of files you have (you won’t gain much on files that are already compressed or encrypted).

Cons

As warned on the official wiki itself, you should not use Btrfs as-is with database or virtualization solutions.

Dixit the official wiki:

Files with a lot of random writes can become heavily fragmented (10000+ extents) causing trashing on HDDs and excessive multi-second spikes of CPU load on systems with an SSD or large amount a RAM.

Indeed, I quickly experienced some issues with Virtualbox. Under heavy I/O operations, and having several machines running at a time, I had the guest file systems corrupted more than once. And so badly that the guest machine was unrecoverable (even with snapshots). Sometimes I got plenty of ext4 errors, or sometimes it just froze, while copying a bunch of file or doing an apt-get upgrade...

The workarounds did not make it for me:

1. I even did not test disabling CoW for the whole partition. It kills one of the main advantages of using Btrfs.
2. I tried disabling CoW for all the VM folder. While the corruption frequency decreased, it still occurred after a while.

So, I would simply adivse of not putting any virtual machine on the Btrfs partitions, until this thing definitely get sorted. I use virtual machines intensively at work and need them to be reliable.

Conclusion

Btrfs is awesome and pretty stable at this time, unless you need to host virtual machines. You could still have a dedicate ext4 partition for you VMs, and enjoy Btrfs for the rest of your home.

To be honest, I did not bother (not wanting to manage several partitions), and switched back to ext4 for all, in the expectation of better days. I am not sure if this should be addressed on the Btrfs, or the Virtualbox side (or both).

References

• Snapper FAQ
• Snapper tutorial
• Arch Linux Btrfs wiki
• Gentoo Btrfs wiki
• The joys of btrfs and opensuse or no space left on device
• CoW workarounds
• Btrfs wiki: gotchas (virtual machines and databases)

The password filter architecture is useful to check that a password is compliant with the system security policy. It will typically check that when a user changes his password, it follows the required complexity.

Microsoft opened the API so that users can extend the functionality with their own filters.

Mubix diverted this API by developing a password logger: the DLL just logs the password both on the disk and a remote server,  and does nothing else.

A perfect way to maintain a persistent access… I tested it:

Evilpassfilter exploitation process

1. Evilpassfilter.dll is loaded into lsass.exe
2. A user updates his password
3. The password goes through the Evilpassfilter password filter, which notifies the attacker through HTTP and also logs it locally.

Here is what I did to get it work (Windows 7 x64):

• Make sure the local password security policy is enabled on the target
• Create a new Win32 project in Visual Studio (2012)
• Eventually delete unnecessary files, to start with an empty project (stadfx.h and cie)
• Import the source code
• Create a Evilpassfilter.def file, which defines the exports:

  LIBRARY Evilpassfilter
  EXPORTS
     InitializeChangeNotify
     PasswordFilter
     PasswordChangeNotify

• In the project properties, make sure to select the appropriate architecture, matching with the one of your target.
  

  Selecting the compilation target architecture (win32/x64)

• In the input settings of the link editor, add wininet.lib as additional dependancy.
• Also add Evilpassfilter.def as module definition file.
  

  Evilpassfilter Visual Studio settings

• In the source code, fix line 72: return; –> return 1;
• Now you should be able to compile the library. You may want to make sure that the DLL is valid and integrated the exports (open it with IDA or a PE tool):
  

  Evilpassfilter.dll exports seen in IDA

• Copy the resulting DLL to the system32 folder.
• Open regedit HKLM\System\CurrentControlSet\Control\Lsa
  and add Evilpassfilter to the Notification Packages

Reboot and… now you should know what to do next :-)

I just watched the presentation of Phil Young at Shmoocon 2013: “Mainframed: the secrets inside that black box“. I truly loved it. I thought mainframes where disappearing, but I was surprised to see that it still alive. I was even more surprised to find out that they have some Unix interface, and that there is a emulator for x86. Where it was less of a surprise is that their security is pretty low :-)

Anyway, don’t miss watching the video. Phil’s blog, “Soldier of Fortran”, is also a gold mine, he wrote many tips, tutos and tools.

It made me very curious and just in case I find some IBM Z/OS during a pentest, I though it would be nice to run it.

Installing

Disclaimer:

Although some Z/OS files are available for download on the Internet, you must own a legal license of Z/OS. This tutorial is exclusively for education-purpose, use it only for testing, never in production nor for illegal activities.

Also, I am a noob in the area. So if some of you are skilled and find mistakes or improvements, please let me know in the comments. I give a great importance to your feedback and it encourages me to continue.

I glued the pieces in the following steps (Mac OS oriented and tested only with it, the same should work for Linux with minor adjustments and see the reference otherwise):

1. Download and install tn3270 (Mac) or x3270 (Windows, Linux, Mac): this will be the client terminal used to connect to the mainframe.
2. Download the emulator, Hercules. Install it, following the README instructions relevant to your system. Note that the instructions for Mac OS are outdated and won’t work. I followed Phil’s instructions:

git clone git://github.com/s390guy/hercules-390.git
cd hercules-390
sh autogen.sh
./configure
make
make install

1. Take some IBM Z/OS release, and install it:

mv IBM\ ZOS\ 1.10/Z110SA/images/Z110\ -\ Copy /YOUR/PATH/HERE/Z110
cd /YOUR/PATH/HERE/Z110
mkdir PRTR
cd CONF
cp ADCD_LINUX.CONF ADCD_MAC.CONF
sed -i '' 's/\/home\/ehrocha\/hercules\/images/\/YOUR\/PATH\/HERE/g' ADCD_MAC.CONF
sed -i '' 's/CNSLPORT \{2\}23/CNSLPORT  3270/g' ADCD_MAC.CONF
sed -i '' 's/0E20.2   LCS  10.0.1.20/0E20.2 3088 CTCI \/dev\/tun0 1500 10.10.10.11 10.10.10.12 255.255.255.255/g' ADCD_MAC.CONF

1. Getting the network to work on Mac OS require some extra steps (skip it if your are using Linux).

Download tuntaposx, uncompress and install the package. No reboot it necessary, you should now have plenty of tun* (and tap*) interfaces:

$ ls /dev/tun*
/dev/tun0 /dev/tun10 /dev/tun12 /dev/tun14 /dev/tun2 /dev/tun4 /dev/tun6 /dev/tun8
/dev/tun1 /dev/tun11 /dev/tun13 /dev/tun15 /dev/tun3 /dev/tun5 /dev/tun7 /dev/tun9

1. Okay, now we can start the emulator (we need to sudo to access to the tun0 interface, among other reasons):

sudo hercules -f ADCD_MAC.CONF

First of all, checks that the network is fine:

# From Mac OS:
$ ifconfig tun0
tun0: flags=8851<UP,POINTOPOINT,RUNNING,SIMPLEX,MULTICAST> mtu 1500
 inet 10.10.10.12 --> 10.10.10.11 netmask 0xff000000 
 open (pid 98687)

# From Hercules:
herc =====> devlist
[...]
HHC02279I 0:0E20 3088 CTCI 10.10.10.11/10.10.10.12 (tun0) IO[0] open
HHC02279I 0:0E21 3088 CTCI 10.10.10.11/10.10.10.12 (tun0) IO[0] open

Open tn3270 and connect with default settings on localhost:

And then in the hercules terminal, enter ipl a80

Hercules390 console: booting Z/OS

It is very long to boot, don’t worry. You will actually have to use 2 terminals, so open the second one, which will show the logon screen (see screenshot below) after booting is done. It will be used for “userland” aka TSO commands.

The first terminal shall be kept open as the master console, which receive system logs and can be used for “system-level”* commands (e.g root level).

Z/OS “Duza” logon screen

1. At the prompt, enter TSO, then IBMUSER as the login, and SYS1 as the password. It will automatically launch the ISPF menu:

ISPF menu

1. Now, you are good to go ahead with Z/OS commands…

This video demonstrates the boot process:

Z/OS emulation with Hercules390 from phocean on Vimeo.

1. Now, let’s get the network up.

Prepare Mac OS:

• Make sure that the Mac OS firewall is deactivated or/and that you configured pf to allow the tun0 interface (another article coming soon on this topic).
• Add a route to tun0

sudo route add -net 10.10.10.0/24 -interface tun0

• You may want to activate ip forwarding, to have the Z/OS reach other interfaces through the kernel:

sudo sysctl -w net.inet.ip.forwarding=1

Now every thing is in place to allow the mainframe to reach the outside. Further routing considerations are outside the scope of this article.

Prepare Z/OS:

• In TSO menu, choose 3 (utilities), 4 (Dslist)
• Click on the line besides Dsname Level and type-in ADCD and then press [Enter]. ADCD is what is called a dataset.
• In the Command column, on the left of ADCD.Z110S.PROCLIB, type in e (stands for edit, reproduce the same pattern when I say “edit” in the following steps)
• Edit the TCPIP member, and make sure that the //PROFILE line looks like this:

//PROFILE DD DISP=SHR,DSN=ADCD.Z110S.TCPPARMS(DUZA)

You could change the DUZA string, but you would have to make sure that the corresponding profile exists in ADCD.Z110S.TCPPARMS (see TODO section).

• Go back to Dslist page using end or exit as a command. This time, type DUZA as dataset.
• Edit the TCPARMS member, then PROFILE. Once in the file, edit carefuly the following lines (at the bottom, around line 90):

000090 DEVICE CTCA1 CTC e20
000091 LINK CTC1 CTC 1 CTCA1
000092
000093 HOME
000094    10.10.10.11  CTC1
000095
000096 GATEWAY
000097    10.10.10.12  = CTC1 1492 HOST
000098
000099 DEFAULTNET 10.10.10.12 CTC1 1492 0
[...]
000109 START CTCA1

• In the console window, restart the network stack:

stop tcpip
# wait for termination message
start tcpip

• If every is going well, the tunnel should get up and you should be able to ping both side (use the ping command in Z/OS from the command menu).

This video illustrates some of this networking stuff:

Hercules390 and Z/OS, getting the network up from phocean on Vimeo.

Useful commands

• Ifconfig

netstat home

• Shutdown

# in "system" terminal:
S SHUTSYS
Z EOD

# then, once finished, in Hercules:
exit

Tips

• I was stuck at an early moment during the boot process with:

IXC208I THE RESPONSE TO MESSAGE IXC420D IS INCORRECT: IS NOT A VALID 
ACTION 
 IXC420D REPLY I TO INITIALIZE SYSPLEX ADCDPL, OR R TO REINITIALIZE 
XCF.     
  REPLYING I WILL IMPACT OTHER ACTIVE SYSTEMS.

You can go over it by entering this in your terminal session (tn3270):

R 00, I

• After the long process, I actually had to open a second connection with the terminal to get the logon screen. So, just check from time to time instead of waiting for nothing in front of the first window.
• To logoff, type X from the ISPF main menu. The first time, you have to configure the printer. Choose LOCAL as print mode, and give it any name as Local printer ID. Then press [Enter], and if you are asked for a sysout class, choose "J". You should be back in TSO, where you can execute logoff. Next time, it will default to these values, so you should get straight from ISPF to TSO.
• Don’t forget that TSO is a CLI where you can type Z/OS and Unix commands. You actually don’t need or have to use ISPF, so don’t hesitate to use it!

Of course, a good source of information is the hercules390 forum may also be of help.

Voilà, happy hacking! WTF, it seems I got mainframed too! Did you?

Big thanks again to Phil Young for catching our attention on this stuff.

TODO

• Understand and get rid off the DUZO profile: you probably noticed that we are using the DUZO  profile to load the network stack (which is after the name of the torrent, and does probably more stuff behind). For example, there is no DUZO profile in ADCD.Z110S.TCPPARMS, so I still have no idea how it actually gets loaded. It has been only 2 days that I work on Z/OS, so I still have to read the doc (and any help is welcome).
• Change the logon screen (see references).

References

• Hercules 3.08 on Mac OS X Lion
• Instructions to installing z/OS in Hercules
• Installin’ that sweet sweet big iron on your Linux laptop or server (local mirror)
• Z/OS files
• Hercules and Z/OS TCP/IP networking for ADCD versions
• Changing the logon screen on the mainframe
• tuntaposx
• TSO tutorial
• Mainframe – using TSO and ISPF
• IBM online documentation

Getting Metasploit

First, let’s fetch Metasploit. Adjust the last two lines by replacing .zshrc (I am using Zsh) with .bash_profile if you are using Bash, for instance.

This will download, create symlinks and set the database settings path (we will come back on it later) in your environment:

cd /usr/local/share/
git clone https://github.com/rapid7/metasploit-framework.git
cd metasploit-framework
for MSF in $(ls msf*); do ln -s /usr/local/share/metasploit-framework/$MSF /usr/local/bin/$MSF;done
ln -s /usr/local/share/metasploit-framework/armitage /usr/local/bin/armitage
echo export MSF_DATABASE_CONFIG=/usr/local/share/metasploit-framework/config/database.yml >> ~/.zshrc
source ~/.zshrc

Metasploit is almost ready, but don’t run anything yet. There a still quite a few steps…

Getting Postgres

We use Homebrew:

brew install postgresql --without-ossp-build

Initialization stuff:

initdb /usr/local/var/postgres

To have launchd start postgresql at login:

ln -sfv /usr/local/opt/postgresql/*.plist ~/Library/LaunchAgents

But I prefer to keep my startup clean, so I added two aliases in my .zshrc

alias pg_start='pg_ctl -D /usr/local/var/postgres -l /usr/local/var/postgres/server.log start'
alias pg_stop='pg_ctl stop'

So you now have two commands, pg_start and pg_stop, to use for Metasploit.
Finally, we create the msf user that will connect to the database from within Metasploit:

createuser msf -P -h localhost  
createdb -O msf msf -h localhost 

While we are at the database stuff, let’s configure Metasploit to use it. Create a database.yml file in  /usr/local/share/metasploit-framework/config/ and put these lines:

production:
    adapter: postgresql
    database: msf
    username: msf
    password: <password>
    host: 127.0.0.1
    port: 5432
    pool: 75
    timeout: 5

The database is ready!

Getting Ruby

The last big step is to install Ruby. The one provided by Mac Os is a little too old, and you don’t want to mess with system libraries, so let’s leave it untouched. You could install Ruby with Homebrew, but it happens that the latest version (2.0.0-p0) is not working with Metasploit (OpenSSL libraries conflicts). So we need to use something like the 1.9.3 version of Ruby.

Anyway, a good practice is to have some flexibility on the version you are going to use, so you would be able to switch between 1.9.3, 2.0.0 or whatever and that whenever you need.

Here comes rbenv. For the next steps, I will assume that you have a working homebrew setting.

Let’s go:

brew install rbenv ruby-build

Add this line to your .zshrc or bash_profile:

eval "$(rbenv init -)"

Now you should be able to list all installable versions of Ruby:

rbenv install --list

Let’s pick up 1.9.3:

rbenv install 1.9.3-p392

It takes a while, but after it is completed, you can set it as your default:

rbenv rehash
rbenv global 1.9.3-p392

Note that you could use the local command instead of global to set it for the current terminal only.

Let’s check that everything is correctly set. This is where the Ruby versions are stored:

$ ls ~/.rbenv/versions/
1.9.3-p392 2.0.0-p0

ruby and gem MUST point to the 1.9.3 version:

$ rbenv which ruby
$HOME/.rbenv/versions/1.9.3-p392/bin/ruby
$ rbenv which gem
$HOME/.rbenv/versions/1.9.3-p392/bin/gem

Looks good, let’s go ahead.

We are now able to install up the required gems for Metasploit. They made it easy by packaging these in a Gemfile that can be read by the “bundle” utility:

gem install bundle
cd /usr/local/share/metasploit-framework
rbenv rehash
bundle install

Final steps

Create an vncviewer wrapper to facilitate use from within Metasploit:

echo '#!/usr/bin/env bash'  >> /usr/local/bin/vncviewer   
echo open vnc://\$1 >> /usr/local/bin/vncviewer  
chmod +x /usr/local/bin/vncviewer

Get and compile the pcaprub library (optional):

cd /usr/local/share/metasploit-framework/external
git clone http://github.com/shadowbq/pcaprub.git
cd ./ext/pcaprub
ruby extconf.rb && make && make install

Have fun!

If you haven’t, don’t forget to start Postgres, and you are ready to play:

sudo -E msfconsole

It should deploy the database structure and then start to work without warning. Hurrah! That was not hard, but a bit long, wasn’t it?

In case it still fails for you, it means that something went wrong with the setup. Check the steps again, and then leave a comment as it may be the time for an update or a correction of this article.

Credits

As stated in the introduction, this article is mostly taken from darkoperator.com with minor adjustments (it actually did not work out of the box for me), so the use of rbenv. I hope it will be helpful to other people in the same case as me.

UPDATE 09/07/2013:

• change in pcaprub directory (./pcaprub –> ./ext/pcaprub)

UPDATE 07/23/2013:

• add missing rbenv rehash command (thanks @amukofes)
• add missing commands to retrieve pcaprub (thanks @Ton)
• fix indentation in postgres config file

Meanwhile, I had received a laconic answer from Apple to my bug report saying that they “are aware of this issue”.

Note that Safari 6.0 on Lion did not (at least on my computer, if someone could confirm)… so same browser version, different OS, the system SSL library must have been – silently – updated.

Anyway, good move finally.

I decided to analyze one the crash I experienced during registry analysis.
I could reproduce all the time a BSOD with Regshot. I thought it would be nice to see what I could get with WinDBG.

I had my environment set up with the suspicious VM configured to debug activated on the serial port, which is a simple pipe on Mac OS X.
Another VM is configured with a serial port as the other end of this pipe, and WinDBG attached to it.
Another method would be to just configure Windows to create a crashdump file with kernel symbols, that you can later load into WinDBG. Of course, the first method offers more opportunities to check and play with the live system.

Then, I just boot the target and trigger the crash, simply by starting a scan with Regshot:

Windows then crashes, WinDBG catches the exception and stops.

So what do we have ?

First, the error type, PAGE_FAULT_IN_NONPAGED_AREA (50), means that an instruction pointed to an invalid memory address. Let’s check this.

With !analyse -v, you get the full error dump.

Crashing Instruction

It shows the function (nt!CmpGetValueKeyFromCache, offset 0x89) and the memory address where the crash was triggered.

The instruction at this address is:

80637807 f3a5 rep movs dword ptr es:[edi],dword ptr [esi]

This instruction is trying to copy 8 bytes at the address pointed by EDI.
EDI has the value of 0xe1285050 at execution time.

And what do we have at this memory location ?

EDI pointing to invalid memory section

Nothing indeed. Note that this corruption persists at every boot.

So what can we conclude?
We can certainly exclude hardware failure, because it is a virtual machine and because the corruption always occur at the same memory region, even after a reboot.
At least, I can now be sure that something in the kernel is definitely corrupted.

Could it be a rootkit trick? Still the question remains, but to me it now looks very, very suspicious. Some rootkit code, poorly written, could have sat in this non-paged memory area and been paged out, causing the BSOD. I have not much knowledge about it at this time but I am going to search on this. At least, I now have good starting point to look at.

That’s all for today, folks. I wrote it while I am still working on it, so sorry if it looks rough and incomplete. It is sort of live, thoughts are still in process.

Again, I am looking forward to reading your comments and suggestions. (Hopefully) there will be a part IV!