I have been working on an implementation of RSync specifically for Google Drive. I now have it synchronizing between client and server using many of the basic rsync options. I am limited on the time I have with which to develop it and test it, so am looking to find volunteers willing to use the software and identify issues. The project is hosted on git hub here:
GSync on Github
Update: I have published the GSync application to pypi.python.org.
https://pypi.python.org/pypi/gsync/0.1.0
To install using pypi, you just need to execute the following command:
$ sudo pypi-install gsync
It will autoresolve dependencies and install require packages. To install the pypi-install utility on debian, execute the following command:
$ sudo apt-get install python-stdeb
Thanks.
Tuesday, 3 September 2013
Friday, 30 August 2013
Linux Mint - Upgrading from Maya(13) to Olivia(15)
I thought I'd give this unsupported upgrade a go to see if it would actually work. Wow, just as I thought, it totally nose dived! I love you Debian, for all your flaws. Anyway, all was not lost. The failure occurred as a result of glibc6 being uninstalled without being replaced by the upgraded glibc6. Something went horribly wrong after uninstall, when something couldn't be restarted as a result of not having glibc6 available; yet another argument that suggests package managers should queue post install scripts until the end of installing all packages rather than trying to do a half arsed install and messing it up.
Anyway, since aptititude kindly caches all the downloaded packages in /var/cache/apt/archive, it was easy to reinstate a working glibc6, providing I didn't reboot!
No matter how bad you think the situation is, you can make it worse by hitting the reset button!
Anyway, since aptititude kindly caches all the downloaded packages in /var/cache/apt/archive, it was easy to reinstate a working glibc6, providing I didn't reboot!
$ dpkg -i $(ls -t1 /var/cache/apt/archive/*glibc6*.deb | head -1)After that, just running the dist-upgrade again managed to get the upgrade merrily on its way. So my tip of the day:
No matter how bad you think the situation is, you can make it worse by hitting the reset button!
Thursday, 25 July 2013
Fanotify NFS root_squash woes
This just plainly doesn't work for root_squash file systems. Here's my bug report:
https://bugzilla.kernel.org/show_bug.cgi?id=60563
https://bugzilla.kernel.org/show_bug.cgi?id=60563
Tuesday, 4 December 2012
MK802 Project - NAS Assembly
Using some parts I had lying around, namely an external USB SATA HDD caddy, the HDD from my existing NAS and a few USB cables, I was able to assemble the MK802 NAS - Phase One:
This all runs using the single DC adapter for the external HDD caddy. I am powering the MK802 using the first USB port on the HDD. The second port has a second WiFi adapter plugged in, to allow me to bridge the WiFi interface that will run in AP mode on the MK802, with the existing WiFi router. This will allow me to set up the device with access to the local network and internet (required for updating / downloading packages). All of this is hooked up to the second USB port on the MK802, via the USB input on the HDD caddy, thus allowing the HDD caddy to act as a four port hub too.
The HDD caddy was purchased a few years back on Amazon for around £15 including delivery. It has served me well and makes for a good hot-pluggable HDD caddy.
I have Debian running on the MK802 with a sunxi 3.0.36 kernel. I will provide details on how to set up the MK802 with Debian using an SD card.
Related: MK802 Project - Heat sinks
This all runs using the single DC adapter for the external HDD caddy. I am powering the MK802 using the first USB port on the HDD. The second port has a second WiFi adapter plugged in, to allow me to bridge the WiFi interface that will run in AP mode on the MK802, with the existing WiFi router. This will allow me to set up the device with access to the local network and internet (required for updating / downloading packages). All of this is hooked up to the second USB port on the MK802, via the USB input on the HDD caddy, thus allowing the HDD caddy to act as a four port hub too.
The HDD caddy was purchased a few years back on Amazon for around £15 including delivery. It has served me well and makes for a good hot-pluggable HDD caddy.
I have Debian running on the MK802 with a sunxi 3.0.36 kernel. I will provide details on how to set up the MK802 with Debian using an SD card.
Related: MK802 Project - Heat sinks
Saturday, 1 December 2012
MK802 Project - Adding heat sinks
I have just bought an MK802 Mini Android PC. My intention is to use this powerful but small piece of kit to create a replacement for my dated NAS. It will serve as a router, wireless AP, firewall, NAS, NIS and NFS server. It is some 6 times more powerful than it's predecessor, with 10 times more memory. It cost just over £30 including delivery and runs at 3 watts; some 8 watts less than the current NAS. It also has USB 2.0, replacing the previously poor USB 1.1 from the predecessor.
Phase one will involve using my existing 3.5" HDD, running in an external USB caddy. This will server as the NAS storage, while the OS is run directly from the 8GB UHS Micro SD card. Using u-boot, it is possible to boot a custom linux kernel from the SD card and boot an entirely different OS. I will be sticking with Debian ARM, since it has served me so well. I will be restricted to the 3.0.36 kernel for a while, downgrading from 3.4, since the Allwinner A10 SOC drivers have not yet been ported to 3.4. However, 3.4 is currently in WIP status, so I should be able to return to this version very soon.
Phase two will see the introduction of the 2.5" external USB HDD, powered by USB and thus losing the need to power the drive separately. I'm not sure how the MK802 will cope with providing this power throughput, though you have to wonder how affordable a 500GB SSD will be by that point...
Before doing anything though, I have read a few reviews that suggest the MK802 gets hot when operational. This is understandable given it's running a 1.6 GHz CPU and a dual core 500 MHz GPU and 2 x 512 MB DDR3 memory modules. The solution is quite brutal, but simple: add heat sinks! Since the whole design is based on it being minimalistic, some surgery is required. If you care to follow it, I have prepared step by step instructions:
Step 1: Dismantling the MK802
You will notice a thin layer of plastic attached to the thicker base layer. This thin layer is just clipped in place and can be removed by carefully inserting a Stanley blade into the end with the HDMI connector, and carefully twisting until the two elements separate.
Before getting dirty and hacking a whole in your casing, you should try to work out an area that will provide the maximum coverage, while not shorting anything; remember, the heat sinks are metal, so have the potential to short something if carelessly placed. Once you know where that will go, line up your four heat sinks
in single file on the inside of the lid part of the casing, precisely where they will protrude. Now carefully mark an outline using your Stanley knife. Don't press hard, you are only making a guide for when you will cut, so just a light scratch in the surface will suffice.
Step 3: Modification of the casing
With the template engraved in the lid, you can remove the heat sinks and begin cutting. Be patient and gentle with this, since trying to press too hard and cut all the way through will result in cracking the plastic or slicing a finger. Just gently score the surface, one edge at a time, repeating the same pattern until you notice the knife begin to make it's way through. This part took me about 20 minutes to complete, but it's worth taking your time. I have seen some people use electric hand tools and make a complete mess of the casing, so it pays to be patient here.
If you want to improve the finish even further, then you can also use a nail file to get rid of any burrs.
Step 4: Assembly and adhesion of the heat sinks
It is better to assemble the casing first, before adhering the heat sinks, since it will be easier to get them in the right place. You can use the hole in the case as a guide for alignment.
After adhering all the heat sinks, you should have something that looks a little like this:
I have seen some other articles where holes are drilled in the surface to allow more ventilation. This really isn't at all necessary, since the heat sinks are fairly efficient at radiating heat on their own, thus drawing cooler air through other cavities such as the USB ports, SD slot or HDMI slot.
Remember, when operating, keep the heat sinks on the top most surface, since they will not function correctly if they cannot radiate heat upwards. It is okay if the heat sinks are on the side, providing the assembly is kept horizontal. Vertical positioning will result in radiated heat preventing either the GPU or DDR memory from getting sufficient cooling.
To test that they are working adequately, turn on your device and wait a minute or two. If working correctly, the heat sinks will be hot to the touch.
Phase one will involve using my existing 3.5" HDD, running in an external USB caddy. This will server as the NAS storage, while the OS is run directly from the 8GB UHS Micro SD card. Using u-boot, it is possible to boot a custom linux kernel from the SD card and boot an entirely different OS. I will be sticking with Debian ARM, since it has served me so well. I will be restricted to the 3.0.36 kernel for a while, downgrading from 3.4, since the Allwinner A10 SOC drivers have not yet been ported to 3.4. However, 3.4 is currently in WIP status, so I should be able to return to this version very soon.
Phase two will see the introduction of the 2.5" external USB HDD, powered by USB and thus losing the need to power the drive separately. I'm not sure how the MK802 will cope with providing this power throughput, though you have to wonder how affordable a 500GB SSD will be by that point...
Before doing anything though, I have read a few reviews that suggest the MK802 gets hot when operational. This is understandable given it's running a 1.6 GHz CPU and a dual core 500 MHz GPU and 2 x 512 MB DDR3 memory modules. The solution is quite brutal, but simple: add heat sinks! Since the whole design is based on it being minimalistic, some surgery is required. If you care to follow it, I have prepared step by step instructions:
Step 1: Dismantling the MK802
You will notice a thin layer of plastic attached to the thicker base layer. This thin layer is just clipped in place and can be removed by carefully inserting a Stanley blade into the end with the HDMI connector, and carefully twisting until the two elements separate.
Before getting dirty and hacking a whole in your casing, you should try to work out an area that will provide the maximum coverage, while not shorting anything; remember, the heat sinks are metal, so have the potential to short something if carelessly placed. Once you know where that will go, line up your four heat sinks
in single file on the inside of the lid part of the casing, precisely where they will protrude. Now carefully mark an outline using your Stanley knife. Don't press hard, you are only making a guide for when you will cut, so just a light scratch in the surface will suffice.
Step 3: Modification of the casing
With the template engraved in the lid, you can remove the heat sinks and begin cutting. Be patient and gentle with this, since trying to press too hard and cut all the way through will result in cracking the plastic or slicing a finger. Just gently score the surface, one edge at a time, repeating the same pattern until you notice the knife begin to make it's way through. This part took me about 20 minutes to complete, but it's worth taking your time. I have seen some people use electric hand tools and make a complete mess of the casing, so it pays to be patient here.
If you want to improve the finish even further, then you can also use a nail file to get rid of any burrs.
Step 4: Assembly and adhesion of the heat sinks
It is better to assemble the casing first, before adhering the heat sinks, since it will be easier to get them in the right place. You can use the hole in the case as a guide for alignment.
After adhering all the heat sinks, you should have something that looks a little like this:
I have seen some other articles where holes are drilled in the surface to allow more ventilation. This really isn't at all necessary, since the heat sinks are fairly efficient at radiating heat on their own, thus drawing cooler air through other cavities such as the USB ports, SD slot or HDMI slot.
Remember, when operating, keep the heat sinks on the top most surface, since they will not function correctly if they cannot radiate heat upwards. It is okay if the heat sinks are on the side, providing the assembly is kept horizontal. Vertical positioning will result in radiated heat preventing either the GPU or DDR memory from getting sufficient cooling.
To test that they are working adequately, turn on your device and wait a minute or two. If working correctly, the heat sinks will be hot to the touch.
Saturday, 13 October 2012
PPPoE and obscure packet dropping (some websites not working)...
I have been having a mare with packets just disappearing or packets being marked as invalid and dropped, then resulting in established and related packets being dropped in response to mangled requests. I spent ages trying to debug this, logging anything and everything with iptables log rules. I couldn't determine what on earth was going on, until I started reading about MSS exceeding the MTU of 1500 set by the ISP. This is only apparent with large amounts of data and even more apparent on devices connecting from behind the firewall, with mangling going on. Essentially, it boils down to packets being mangled, increasing the segment size, which eventually leads to a packet that exceeds the MTU when the ISP reroutes it. At the firwall level, the packet is fine and fits just within the 1500 limit. Proposed solutions seem to be centred around adjusting the PPPoE interface MTU to something that would not exceed the limit of 1500 when mangled. This is a good idea to have set. For me, I modified my PPP configuration and changed the MTU:
$ sed -i.bak 's/^mtu [0-9]\+$/mtu 1454/' /etc/ppp/peers/dsl-provider
This is not always guaranteed to work alone though, as I found out. This is because you don't know what additional mangling is going on at the ISP's end. In the end, I started reading the iptables man page for the 3.4 kernel I am running. I found something very interesting...
TCPMSS
This target allows to alter the MSS value of TCP SYN packets, to control the maximum size for that connection (usually limiting it to your outgoing inter-
face's MTU minus 40 for IPv4 or 60 for IPv6, respectively). Of course, it can only be used in conjunction with -p tcp.
This target is used to overcome criminally braindead ISPs or servers which block "ICMP Fragmentation Needed" or "ICMPv6 Packet Too Big" packets. The symp-
toms of this problem are that everything works fine from your Linux firewall/router, but machines behind it can never exchange large packets:
1) Web browsers connect, then hang with no data received.
2) Small mail works fine, but large emails hang.
3) ssh works fine, but scp hangs after initial handshaking.
Workaround: activate this option and add a rule to your firewall configuration like:
iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN
-j TCPMSS --clamp-mss-to-pmtu
--set-mss value
Explicitly sets MSS option to specified value. If the MSS of the packet is already lower than value, it will not be increased (from Linux 2.6.25
onwards) to avoid more problems with hosts relying on a proper MSS.
--clamp-mss-to-pmtu
Automatically clamp MSS value to (path_MTU - 40 for IPv4; -60 for IPv6). This may not function as desired where asymmetric routes with differing
path MTU exist -- the kernel uses the path MTU which it would use to send packets from itself to the source and destination IP addresses. Prior to
Linux 2.6.25, only the path MTU to the destination IP address was considered by this option; subsequent kernels also consider the path MTU to the
source IP address.
These options are mutually exclusive.
$ sed -i.bak 's/^mtu [0-9]\+$/mtu 1454/' /etc/ppp/peers/dsl-provider
This is not always guaranteed to work alone though, as I found out. This is because you don't know what additional mangling is going on at the ISP's end. In the end, I started reading the iptables man page for the 3.4 kernel I am running. I found something very interesting...
TCPMSS
This target allows to alter the MSS value of TCP SYN packets, to control the maximum size for that connection (usually limiting it to your outgoing inter-
face's MTU minus 40 for IPv4 or 60 for IPv6, respectively). Of course, it can only be used in conjunction with -p tcp.
This target is used to overcome criminally braindead ISPs or servers which block "ICMP Fragmentation Needed" or "ICMPv6 Packet Too Big" packets. The symp-
toms of this problem are that everything works fine from your Linux firewall/router, but machines behind it can never exchange large packets:
1) Web browsers connect, then hang with no data received.
2) Small mail works fine, but large emails hang.
3) ssh works fine, but scp hangs after initial handshaking.
Workaround: activate this option and add a rule to your firewall configuration like:
iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN
-j TCPMSS --clamp-mss-to-pmtu
--set-mss value
Explicitly sets MSS option to specified value. If the MSS of the packet is already lower than value, it will not be increased (from Linux 2.6.25
onwards) to avoid more problems with hosts relying on a proper MSS.
--clamp-mss-to-pmtu
Automatically clamp MSS value to (path_MTU - 40 for IPv4; -60 for IPv6). This may not function as desired where asymmetric routes with differing
path MTU exist -- the kernel uses the path MTU which it would use to send packets from itself to the source and destination IP addresses. Prior to
Linux 2.6.25, only the path MTU to the destination IP address was considered by this option; subsequent kernels also consider the path MTU to the
source IP address.
These options are mutually exclusive.
So I wasn't going mad after all and it does appear to be a widely known issue. So I gave it a try...
-A FORWARD -p tcp --tcp-flags SYN,RST SYN -j LOG --log-prefix "CLAMP-MSS-TO-PMTU" --log-tcp-options --log-ip-options --log-level 7
-A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
I am only specifying the LOG action here to ensure that it works. Indeed it does. After enabling it, I instantly started seeing entries appear in the logs and all the websites that weren't previously accessible suddenly started working. I don't need to explain why this works, since it's explained enough in the above manual excerpt. But if you share my blight, then hopefully this helps you too!
Friday, 12 October 2012
Broadcast ping for host discovery
I saw a question posted on an exchange website today that took me back a few years. Often, people assume that broadcast ping can be used for host discovery on a network. But these days, broadcast ICMP requests are dropped silently to avoid stimulating springboard style DoS attacks. This is essentially where you ping a number of hosts, as many as possible across a multitude of subnets, but have all the responses returned to the machine you want to attack. This overloads the machine's network interface with ICMP traffic, obstructing real network traffic. It also poses as a threat to ADSL connections, when you consider that most people have caps on their broadband. It would be quite easy to render someone's internet connection useless through this attack mechanism. For this reason, the default response for any machine, is to not respond to broadcast ICMP requests. Most routers also filter these out, to prevent ICMP broadcast packets spreading across different subnets.
Anyway, host discovery can actually be obtained through a much simpler mechanism, given that you can ping specific hosts for a reply; again, providing that those hosts reply to ICMP requests. The solution is to ping each possible permutation of addresses on a given subnet, looking for a response from any of them. This sounds like a really heavyweight process, but it's actually very easy and very fast. The following command demonstrates this, which can be incorporated into a script if you wish (brackets are important).
$ time ( s=192.168.0 ; for i in $(seq 1 254) ; do ( ping -n -c 1 -w 1 $s.$i 1>/dev/null 2>&1 && printf "%-16s %s\n" $s.$i responded ) & done ; wait ; echo )
192.168.0.5 responded
192.168.0.11 responded
192.168.0.2 responded
192.168.0.254 responded
192.168.0.4 responded
real 0m1.317s
user 0m0.004s
sys 0m0.084s
Anyway, host discovery can actually be obtained through a much simpler mechanism, given that you can ping specific hosts for a reply; again, providing that those hosts reply to ICMP requests. The solution is to ping each possible permutation of addresses on a given subnet, looking for a response from any of them. This sounds like a really heavyweight process, but it's actually very easy and very fast. The following command demonstrates this, which can be incorporated into a script if you wish (brackets are important).
$ time ( s=192.168.0 ; for i in $(seq 1 254) ; do ( ping -n -c 1 -w 1 $s.$i 1>/dev/null 2>&1 && printf "%-16s %s\n" $s.$i responded ) & done ; wait ; echo )
192.168.0.5 responded
192.168.0.11 responded
192.168.0.2 responded
192.168.0.254 responded
192.168.0.4 responded
real 0m1.317s
user 0m0.004s
sys 0m0.084s
Subscribe to:
Posts (Atom)