script News - CNX Software - Embedded Systems News

October 3, 2016October 3, 2016 by Jean-Luc Aufranc (CNXSoft) - 1 Comment

How to check HTTP Header and Connection Stats from the Command Line

A few days ago, I discussed with somebody whether a file was cached by Cloudflare or not, and this involved getting the HTTP header, and checking for CF-RAY field to see if data is going through one of Cloudflare data centers. This can be done with curl:

curl -svo /dev/null http://www.cnx-software.com
* Rebuilt URL to: http://www.cnx-software.com/
*   Trying 104.28.19.95...
* Connected to www.cnx-software.com (104.28.19.95) port 80 (#0)
> GET / HTTP/1.1
> Host: www.cnx-software.com
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Mon, 03 Oct 2016 09:57:17 GMT
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Set-Cookie: __cfduid=d90ff49c11865e8fda1331c2977559f521475488637; expires=Tue, 03-Oct-17 09:57:17 GMT; path=/; domain=.cnx-software.com; HttpOnly
< X-Powered-By: PHP/5.5.9-1ubuntu4.19
< Expires: Wed, 11 Jan 1984 05:00:00 GMT
< Cache-Control: no-cache, must-revalidate, max-age=0
< Pragma: no-cache
< X-UA-Compatible: IE=edge
< Link: <http://www.cnx-software.com/wp-json/>; rel="https://api.w.org/"
< Server: cloudflare-nginx
< CF-RAY: 2ebf876da273114d-SIN
< 
{ [2307 bytes data]
* Connection #0 to host www.cnx-software.com left intact

curl -svo /dev/null http://www.cnx-software.com

* Rebuilt URL to: http://www.cnx-software.com/

* Trying 104.28.19.95...

* Connected to www.cnx-software.com (104.28.19.95) port 80 (#0)

> GET / HTTP/1.1

> Host: www.cnx-software.com

> User-Agent: curl/7.47.0

> Accept: */*

< HTTP/1.1 200 OK

< Date: Mon, 03 Oct 2016 09:57:17 GMT

< Content-Type: text/html; charset=UTF-8

< Transfer-Encoding: chunked

< Connection: keep-alive

< Set-Cookie: __cfduid=d90ff49c11865e8fda1331c2977559f521475488637; expires=Tue, 03-Oct-17 09:57:17 GMT; path=/; domain=.cnx-software.com; HttpOnly

< X-Powered-By: PHP/5.5.9-1ubuntu4.19

< Expires: Wed, 11 Jan 1984 05:00:00 GMT

< Cache-Control: no-cache, must-revalidate, max-age=0

< Pragma: no-cache

< X-UA-Compatible: IE=edge

< Link: <http://www.cnx-software.com/wp-json/>; rel="https://api.w.org/"

< Server: cloudflare-nginx

< CF-RAY: 2ebf876da273114d-SIN

{ [2307 bytes data]

* Connection #0 to host www.cnx-software.com left intact

In the command above, -s stands for silent so that curl does not show the progress meter, -v stands for verbose to show the header, and -o /dev/null is used to discard the packet load. You can also use -I option (fetch the HTTP-header only) with curl, which – if all you need is the HTTP header – provides a cleaner output:

curl -I http://www.cnx-software.com
HTTP/1.1 200 OK
Date: Mon, 03 Oct 2016 10:06:51 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Set-Cookie: __cfduid=d4dda8a9ec8370cf0950d26e5faf37cc21475489211; expires=Tue, 03-Oct-17 10:06:51 GMT; path=/; domain=.cnx-software.com; HttpOnly
X-Powered-By: PHP/5.5.9-1ubuntu4.19
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Cache-Control: no-cache, must-revalidate, max-age=0
Pragma: no-cache
Set-Cookie: bb2_screener_=1475489211+1.1.174.2+1.1.174.2; path=/
X-UA-Compatible: IE=edge
Link: &lt;http://www.cnx-software.com/wp-json/&gt;; rel="https://api.w.org/"
Server: cloudflare-nginx
CF-RAY: 2ebf9574c129081d-SIN

curl -I http://www.cnx-software.com

HTTP/1.1 200 OK

Date: Mon, 03 Oct 2016 10:06:51 GMT

Content-Type: text/html; charset=UTF-8

Connection: keep-alive

Set-Cookie: __cfduid=d4dda8a9ec8370cf0950d26e5faf37cc21475489211; expires=Tue, 03-Oct-17 10:06:51 GMT; path=/; domain=.cnx-software.com; HttpOnly

X-Powered-By: PHP/5.5.9-1ubuntu4.19

Expires: Wed, 11 Jan 1984 05:00:00 GMT

Cache-Control: no-cache, must-revalidate, max-age=0

Pragma: no-cache

Set-Cookie: bb2_screener_=1475489211+1.1.174.2+1.1.174.2; path=/

X-UA-Compatible: IE=edge

Link: <http://www.cnx-software.com/wp-json/>; rel="https://api.w.org/"

Server: cloudflare-nginx

CF-RAY: 2ebf9574c129081d-SIN

I also came across httpstat Python script recently via n0where, doing much of the same thing, except it also adds transfer statistics. It can be installed by downloading httpstat.py, or better using pip:

sudo pip install httpstat

1	sudo pip install httpstat

Let’s try it with this very […]

September 13, 2016September 13, 2016 by Jean-Luc Aufranc (CNXSoft) - 4 Comments

Borg Lets You Search for Bash Commands within the Terminal

There are so many commands available in bash that it’s impossible to know them all, and some commands are rarely run. Usually, you have to switch to your web browser to search for a solution or spending time reading through the manpage. In case you’d like to search for a specific command line you can now do so with borg script. If you want to give it a try, you can install it in Linux as follows:

sudo wget https://github.com/crufter/borg/blob/master/builds/borg_linux_amd64\?raw\=true -O /usr/local/bin/borg
sudo chmod 755 /usr/local/bin/borg

1 2	sudo wget https://github.com/crufter/borg/blob/master/builds/borg_linux_amd64\?raw\=true -O /usr/local/bin/borg sudo chmod 755 /usr/local/bin/borg

Then run the script with your search query in quotes. Some explanations for the output: () denotes hits for your query [] denotes possible solutions … under a [] means more lines to display a “–” in a solution means separate code examples extracted from the same source Some queries are returning useful results such as “list all files in dir” as show in the screenshot above, but others not so, as […]

November 30, 2014 by Jean-Luc Aufranc (CNXSoft) - 9 Comments

Create-android-sdcard Script Creates a Bootable SD Card for any Rockchip RK3188 Devices

Bootable Linux SD card images for some Rockchip RK3188 devices such as MINIX NEO X7, MK809III, Tronsmart T428 have already been released this summer, and it provided any easy way to try Linux on Rockchip RK3188 devices without messing with your Android installation. But now you can create Android bootable images with any Rockchip RK3188 devices thanks to create-android-sdcard script written by Ian Morrison, as long as you have the firmware for your Android mini PC, and I don’t see why it would not even work for tablets. It allows you to try Android firmwwre before replacing your existing installation (great for developers), and it can also be a way to handle multiple users on a single device. There could be one SD card for dad, one for mum, and one for the kid, and all you have to do is to take out and replace the SD card with […]

February 9, 2014February 9, 2014 by Jean-Luc Aufranc (CNXSoft) - 9 Comments

How to Upload YouTube Videos with the Command Line in Linux

Like many people, I access Internet via an ADSL connection at home. ADSL stands for “Asymmetric Digital Subscriber Line”, with Asymmetric being the key word here, as it just mean your download speed will be (much) higher than your upload speed. My ISP promises a theoretical 10 Mbps download speed, and 512 Kbps upload speed, and this is pretty close in reality: Data Rate: 10240 (downstream), 509 (upstream) kbps. Assuming a 265MB video, in the very best case (63 KB/s), it would take 1 hour and 12 minutes to upload a video to YouTube, but in practice it’s often closer to 2 or 3 hours. If it’s a video you’ve shot yourself, and copied inside your computer, there’s very little you can do, except processing the video with tools such as HandBrake to make it smaller before upload. But if the video files are located somewhere in the Internet, and […]

April 22, 2013 by Jean-Luc Aufranc (CNXSoft) - 22 Comments

You-Get – Video Download Script for YouTube, YouKu, DailyMotion, and More

YouKu is the equivalent of YouTube in China, and earlier today I wanted to download a video from the service, so I’ve looked for an application or script that can do the job in Linux. I’ve finally come across you-get, a python 3 script that claims to be able to download videos from an impressive number of websites, namely: YouTube Vimeo Coursera Blip Dailymotion Facebook Google+ Google Drive Tumblr Vine SoundCloud Mixcloud Freesound JPopsuki VID48 Niconico (ニコニコ動画) Youku (优酷) Tudou (土豆) YinYueTai (音悦台) AcFun bilibili CNTV (中国网络电视台) Douban (豆瓣) ifeng (凤凰视频) iQIYI (爱奇艺) Joy.cn (激动网) Ku6 (酷6网) MioMio h NetEase (网易视频) (v.163.com) PPTV.com QQ (腾讯视频) (v.qq.com) Sina (新浪视频) (video.sina.com.cn) Sohu (搜狐视频) (tv.sohu.com) 56 (56网) Xiami (虾米) Baidu (百度音乐) (music.baidu.com) SongTaste I won’t try all, but just test it with YouKu and YouTube. First things first, let’s install it:

git clone git://github.com/soimort/you-get.git
cd you-get
sudo apt-get install python3 python3-setuptools
./you-get -V
make install

git clone git://github.com/soimort/you-get.git

cd you-get

sudo apt-get install python3 python3-setuptools

./you-get -V

make install

Time to go to youku.com and download a video:

you-get http://v.youku.com/v_show/id_XNTQ1OTk3NTAw.html
Video Site: Youku.com
Title:      1 second boot QT app in linux-sunxi with  A10
Type:       Flash video (video/x-flv)
Size:       5.45 MB (5711001 Bytes)

Downloading 1 second boot QT app in linux-sunxi with  A10.flv ...
100.0% (  5.4/5.4  MB) [========================================] 1/1

you-get http://v.youku.com/v_show/id_XNTQ1OTk3NTAw.html

Video Site: Youku.com

Title: 1 second boot QT app in linux-sunxi with A10

Type: Flash video (video/x-flv)

Size: 5.45 MB (5711001 Bytes)

Downloading 1 second boot QT app in linux-sunxi with A10.flv ...

100.0% ( 5.4/5.4 MB) [========================================] 1/1

Success! […]

February 12, 2013February 12, 2013 by Jean-Luc Aufranc (CNXSoft) - 6 Comments

How to Display dd Progress and dd.sh Script

Upgrading software on Linux / Android boards or devices often involves copying images to an SD card or microSD. In Linux, you’d usually do that with dd, a utility that provides binary copy of data to files or block devices. A typical command would be:

sudo dd if=new_firmware.bin of=/dev/sdc

1	sudo dd if=new_firmware.bin of=/dev/sdc

However, during the copy, dd does not show a progress bar by default. But dd actually supports progress report, as indicated in the manpage: you can run dd, and send USR1 signal to display the current progress once, and resume copying. Linux commando explains how to continuously return the progress. First run the dd command:

sudo dd if=/dev/random of=/dev/null bs=1K count=100

1	sudo dd if=/dev/random of=/dev/null bs=1K count=100

Open another terminal window to find out the process id:

pgrep -l '^dd$'
10152 dd

1 2	pgrep -l '^dd$' 10152 dd

And use the watch command to send USR1 at regular interval.

watch -n 10 sudo kill -USR1 10152

1	watch -n 10 sudo kill -USR1 10152

You should see dd progress in the first window every 10 seconds. It works, but the output is not very nice because dd will just […]

May 12, 2012May 29, 2017 by Jean-Luc Aufranc (CNXSoft) - 3 Comments

makeSD Script to Write Image to SD Card for Mele A1000 / AllWinner A10 Devices

Most images released for Mele A1000 (Ubuntu, Puppy Linux..) won’t fit in my SD card, and until now I had to manually partition the SD card, extract the data and copy it to the SD card. I’ve also noticed the size of the SD card slowly creeps lower overtime. I bought an SD card last week and fdisk reported 3901685760 bytes and this morning the same command reported 3898782720 bytes. That’s probably due to new bad sectors which I believe is actually normal for this type of device. The problem is that a backup of “last week” SD card done with dd might not be restored properly with dd since the SD card is now smaller. So I decided to write a shell script “makeSD.sh” that will do the following: Umount the SD card if needed Partition the SD card Copy uboot to the SD card Mount the image file […]