After reading the article at unitstep.net, it does seem interesting.
The challenge is a cryptographic nonce, which means is only sent once to the client and after successful login, it is invalidated, so if someone were sniffing the connection and they received the challenge and the response, it wouldn't work since next time around it will be different. As far as the login credentials are concerned, someone sniffing the connection will not feasibly be able to hack your login information by using the data sent during login.
However, when not using an encrypted connection, there are other downfalls. A few I can think of are:
- If a hacker got the challenge and your response, they could perform a dictionary/brute force attack on it to retrieve your password. If poor (fast) cryptography is used and a simple password is used, this may be done relatively quickly, so make sure to use a slow algorithm or use many rounds if it is a fast one to achieve a slowing effect.
- This method does not provide authentication, which could lead to a man-in-the-middle attack.
- Since it is done client-side, you have no control over the password strength. For all you know, they could be using an empty string.
- A hacker can sniff what the user is doing while logged in
- Your site would be far more vulnerable to session hijacking
- Even though your login credentials are safe in themselves (since they are never transmitted), the next challenge is always stored ahead of time in the database in plain text.
Valid SSL certificates are not expensive. Doing a quick search for "cheap ssl certificates" found a few matches under $10/year (about the cost of your domain name). This is definitely the way to go if you can.
Now, to answer your actual question:
Do you know any class / lib that provides challenge-response feature?
No, sorry.
Data must be hashed at client side before sending to server - isn't
that unsafe by definition?
I don't see how this would be unsafe. I have seen a few websites implement browser-side certificate logins (such as http://www.startssl.com and Webmin also has the same feature). A certificate pair is calculated by the browser and the public key sent to the server for authentication.
Edit 2016:
If you're looking for completely free SSL certificates I highly recommend https://letsencrypt.org/. I currently use them for about 10 domains without problem and have the certificates automatically renew using a simple cron job, so now I basically never have to worry about certificates again. They also accept donations which I would encourage anyone using them to do, especially since they are saving you ~$10/year per domain.