Showing posts with label python. Show all posts
Showing posts with label python. Show all posts

Saturday, July 28, 2012

Working With Binary Data in Python

This puzzled me for quite a time, so here is a post explaining some of my findings.
First we define a bytes array:
 s = bytearray(b"Hello World")
 for i in s: 
     print i
 ...
 72
 101
 108
 108
 111
 32
 87
 111
 114
 108
 100

now, let's write this data to a file:
 f = open('helloword.bin','wb')
 for i in s:                                                                                                                                       
    f.write(struct.pack("I",i))
 f.close()

Let's inspect the file created:
 
 $ du -h points.bin
 4.0K    points.bin
 $ file points.bin
 points.bin: data
 $ less points.bin
 "points.bin" may be a binary file.  See it anyway?

Binary file sizes

let's us write "hello world" into a text file in a text form:

 f = open('helloword.txt")
 f.write("hello world")
 f.close()

Once again we can inspect the file:

$ du -h   helloworld.txt
 4.0K    helloworld.txt


Now, what happens if we make a longer binary array?


 s.split()
 s.append(33)
 bytearray(b'Hello World!')
 for i in range(10000): s.append(33)
 f = open('longhelloword.bin', 'wb')
 import struct      
 for i in s: f.write(struct.pack("I",i))
 f.close()
 f = open('longhelloword.txt', 'w')
 hello = "Hello World!"
 for i in range(10000): hello = hello+"!"
In a shell, examaining the file sizes:
 
 $ du -h longhelloword.bin
 40K     longhelloword.bin
 $ du -h longhelloword.txt
 12K     longhelloword.txt

Wait a minute ! Why is the binary file almost 4 times bigger?

The answer is: it depends on the format specifier in struct.pack.
Namely, we used an unsigned int, for each character we then reserved 4 bytes!.
When we saved the text, every character was assigned to the file exactly as a char which takes one byte only.

If we repeat the above with struct.pack("b",i) the sizes of the file won't differ:

 f = open('longhellowordwithchar.bin', 'w')
 for i in s: f.write(struct.pack('b',i))
 f.close()

and in the shell:

 $ du  longhellowordwithchar.bin
 12      longhellowordwithchar.bin
 $ du  longhelloword.txt       
 12      longhelloword.txt

Credits:
http://dabeaz.blogspot.de/2010/01/few-useful-bytearray-tricks.html
http://docs.python.org/library/struct.html#module-struct

Tuesday, March 20, 2012

Good Bye Latex ? Hello WeasyPrint ...

Latex, is my main way of producing reports and scientific works. Usually these are formed as PDF before being actually printed. Latex is AWESOME. I love writing documents with latex. That said, although I am not a beginner with latex, when I want to deviate from normal good old latex patterns things become a bit rough.
Despite using the Latex language for almost 5 years, I still don't fully understand the syntax of writing packages, although mostly a long enough search will yield some solution which does not require me to hack on my own. Never the less this is a bit tiresome sometimes.
Enter WeasyPrint. This project seems really cool because it allows anyone with basic knowledge of HTML and CSS to produce good looking PDFs. Further more, I can really think of a simple work flow once a template is made:
Write your documents  with Markdown, use something like Jekyll to convert your easy to understand Markdown to HTML with some Jinja or CSS and finally convert your pages to good looking PDF's.
I think this is more or less what Sphinx does (except that Sphinx uses RST instead of Markdown). I never really mastered CSS or HTML, but this seems like a good opportunity to start mastering both

Besides, consider this:

MarkDown + Convert to HTML +  WeasyPrint  = Sphinx from Scratch.

Besides being a very cool educational project, a project like this can allow people with CSS + HTML knowledge to produce some great looking PDF. It also allowed a lot of flexibility if you are good at both. 

Never the less, Latex has still one big advantage: Latex is well designed to produce good looking documents even though you are not an expert on typography and page layout. Additionally it is thousands of ready made packages and kind community that can help solving many existing problems. So maybe after all, Latex is not going away so fast!

Friday, February 24, 2012

VIM Quicky: set tabs and spaces behaviour for different files

Never really bothered me before, because I only used Python and C, so I always replaces tabs with spaces.
However, with more C programming lately, I needed to start working with Makefiles. These require tabs instead of spaces.
So here is how to. In the end of my .vimrc file I put:

" python python pyhton
" convert tabs to spaces before writing python files or C files
autocmd! bufwritepre *.py,*.c,*.h set expandtab | retab! 4

" convert spaces to tabs when reading python file or C files
autocmd! bufreadpost *.py,*.c,*.h set noexpandtab | retab! 4


Et viola, not so complicated!

Wednesday, October 19, 2011

When friends spit FUD

Warning: This is not a technical post, rather just a boring rant and a call for advice. A friend of mine asked me what programming language he should learn. He was thinking of dwelling into modeling after doing some programming with R and Matlab. Then, I suggested he should give Python a spin. I think python has much nicer syntax than R or Matlab. Regarding speed, I have seen many many times that Python is simply faster than Matlab and R when using modules written in C or Fortran (which is the case for Numpy and most of scipy). And finally, both R and Python are FREE, so their amount of packages is extensive so every user has a great choice of ready made recipes. So, it seemed my friend was kind of convinced and thought of trying Python. But a couple of days later, when we chatted, he told me the following sentence:  

"2. anything that's open source is inherently less user friendly than commercial products."

Now, I am a dickhead, so I get mad when friends of mine, whom I consider to be smart, say stuff like that. Especially, when they sound so convinced. If you read my blog, you are likely to label this sentence as FUD. And you are likely to disagree like me. Personally, I don't know how to change a man's opinion who is so convinced. I also don't know if I should try. But it sure does piss me off and sets me on angry mode. It would be nice to know what other people do in such cases. I just don't want to be an evangelist anymore.

So, please give me and advice how to fight FUD when it comes from friends you really love.

Finally, apologies to my friend who got quoted without reference, and had to experience me getting upset.

Sunday, June 19, 2011

One more python learning resource

This one is pretty cool. It is a learning environment, which allows you also to create more learning environment. I wish I had more time to play with it!
Python Crucny: http://code.google.com/p/crunchy/

Have fun!


Thursday, May 26, 2011

2 Cool Python Learning resources

A couple of nights ago I was browsing code.google.com in random search for interesting Python projects.
Here are two nice catches from my random fishing, which are worth mentioning. The first one is a Free python book, which attempts to bring Python and the joy of creating software for kiddies, here is what they write as an intro:

"Snake Wrangling for Kids" is a printable electronic book, for children 8 years and older, who would like to learn computer programming. It covers the very basics of programming, and uses the Python 3 programming language to teach the concepts.

The second one, is a Python interactive learning program. Although I have not played a lot with it, it seems fun. However, the project seems a little bit sleepy (no updates for almost a year):

RUR-PLE is an environment designed to help you learn computer programming using the language Python. Within an artificial world in which a robot can be programmed to perform various tasks, you will learn what it means to write a computer program, using Python's syntax

If you are a self learner type like me, you would appreciate these resources.



Friday, February 25, 2011

oh, Yesterday I was a bad programmer ...

Forgetting my first lesson in programming, I looked for a solution to convert IP addresses to country. I found out tables, and SQL files, and went on and on trying to figure how to use them, and do SQL in python...
Until, I remembered that a good programmer should not re-invent what has been invented.
And then I looked around and found: pygeoip.
I honestly felt sorry for the time I wasted on that problem ...