Feb 16, 2010

bash and konsole

So I've had lots of problems using a multi-tab xterminal, for the casual user I'm sure they'd not even notice them, or think I'm silly, but when you spend all day in a shell they become less so.

History


one of the big problems I've had is with history. see when you close a shell then it writes the history. well if you have 5 tabs open each with with a super long history they'll write those histories out in the order they close in. basically what happens is all the history saves at once so one of your open shells histories will be before another's and if each has like 50 entries it can be quite a way back. In fact I'm sure I've had it happen where parts of my history was lost. So what do you do? you have to separate them per tab. One way to do this is with gnu screen. I don't like screen's history because it records the output of the terminal as well as the command, so this wasn't a good solution for me. The solution is that each tab must have its own history file. Well how do we do that? before I get to it I have another problem.

PS1, umask, and default group

I also have a need to have different settings depending on what I'm working on. If I'm working on files that need to be read and written to by my web server they need to have a different group and umask. first I solved this by su - to a different user, then I solved it by having different bashrc's but neither fully solved the history problem.

howto do it

first you have to have one konsole profile for each of the different settings you need. so I have one for web development called webdev. Then you have to add an environment variable to it I'm going to call it KONSOLE_PROFILE and set that to whatever your profile is called. Neither name really matters so long as they don't conflict with others. Then you need to code your bashrc (or other shell to work with it). I have a shortened sample bashrc that you can use in a gist

Feb 14, 2010

Can't Find Google's OpenID

I think that Google's (not Blogger's) OpenID is a failure. It's not memorable and you can't find it. I've been trying to figure out where google has my OpenID listed for 30 minutes now. This is not the first time I've had to do this. I usually use Blogger's because it uses my blog's url, but it's currently error-ing with a typepad blog. Why can't google use something like google.com/username ? something we could remember. Why isn't it listed in an obvious place in my account details? Why is it when I search "google openid" it talks about the announcements all over the place, but never how to find out what your url is.

UPDATE:
Found it on lenni.info

UPDATE:
it's this https://www.google.com/accounts/o8/id

Feb 11, 2010

KDE 4 is finally ready for users

I just installed KDE SC 4.4 last night and I can say that I feel that it's finally general user ready. The last bug that I felt would prevent average use is now gone. What was that bug? when you pressed print screen nothing would happen. There was a workaround to make print screen work but it didn't work out of the box. I feel that there are some things that should just work and that's one of them.

I'm not sure I'm yet happy with the state of the device notifier, and automounting as the UI doesn't seem perfect yet, but it can now do all the things that are necessary.

There will always be improvements to be made, but I think that if you've been not sure about using KDE 4 go give it a try now. (I'm about to find out if I can replace vim with kate's vi bindings).

Jan 20, 2010

empty() function for postgresql in sql

In PostgreSQL the ASCII NULL or empty string \0 is seen as NOT NULL. This is because postgres developers consider any characters data. I know there are better discussions on it but I can't find them right now. Unfortunately the programming language you are using probably doesn't see it the same way. There's a good chance that initialized variables are set to \0 and so when you try to insert from your language to a NOT NULL field with variables that are seen as undefined in your language, postgres accepts it, and now you have fields that you probably consider to have no data but are NOT NULL.

My function possibly would be better called empty_or_whitespace but in my mind any text field that contains only whitespace is empty, and I'd rather have 1 function and 1 regex deal with both than have 2 functions have to be called on every insert/update.

Here is the gist

It returns true if an empty string is found. It really requires a knowledge of SQL, PostgreSQL's CREATE FUNCTION, and Regular Expressions, to understand. Thanks to the wonderfully helpful PostgreSQL community with perfecting it.

In order to use it to keep ASCII NULLS and fields that someone has just entered whitespace into you need to add it to your table as a constraint.

Here's an example create table gist using it.

remember it returns true on null's and whitespace so you have to say NOT emtpy( field ).

Jan 19, 2010

PostgreSQL initial setup (authentication) Part 2

I ran into so problems and lack of information with my last post on this topic. Firstly my syntax for local all all to local all all ident devel
doesn't seem to work in my current setup. It's possible that it has something to do with the configuration of the Debian/Ubuntu server I was basing that against, and now my targets are Arch Linux and Slackware.

So our goal here will be to provide an alternate user that can log in as postgres via ident. why would you want to do this? maybe you're tired of su - to the postgres unix user after logging in as your administrative user (root?).

First we need to open our pg_ident.conf and add the following line.

admin root postgres

So as I said in the last post on this: "admin" is an arbitrary identifier, root is the system user, and postgres is the database user. Now let's go ahead and make sure the postgres system user can still login.

admin postgres postgres

ok that's all we need to do in the pg_ident.conf. Go ahead and open pg_hba.conf and add this line above the default 3 lines in the file.

local all postgres ident map=admin

as a breakdown: local is for socket connections, all means all databases, ident means it's checking for local users, and map=admin says look at the users in the admin identifier in pg_ident.conf.

Now you should be able to run psql dname postgres as root. You will not be able to use the db postgres user as any system user not in the admin map.

Nov 16, 2009

Bypassing disabled accounts with KDM

So the most common way of disabling an account in a unix system is changing the users shell in /etc/passwd to /bin/false or /sbin/nologin. However, I've discovered on Arch Linux that if I do this only shell login's are disabled, I was still able to log the user in with gui via kdm. I also tried using usermod --expiredate 1. this was not effective either however. the only way I found to lock the account from kdm login was to do a passwd -l accountname, which only locks password authentication. This means key and token authentication should still work. My real concern is that if the user was set to login without a password that it would still be bypass-able. My personal opinion is that I shouldn't have to do more than 1 thing to disable a user account in a 100% effective manner. Currently Arch Linux (and maybe more) fails at this. I'll post a fix later as I investigate further.

Bugs I filed are at kde and Arch Linux

EDIT: expire date worked... just not immediately, very odd.

EDIT: Here's a proper /etc/pam.d/kde

#%PAM-1.0
auth        required    pam_nologin.so
auth        required    pam_unix.so nullok
auth        required    pam_shells.so
auth required pam_tally.so onerr=succeedfile=/var/log/faillog
account     required    pam_access.so
account     required    pam_time.so
account     required    pam_unix.so
password    required    pam_unix.so
session     required    pam_unix.so
session     required    pam_env.so
session     required    pam_limits.so

Nov 10, 2009

Difference Between a Database and a Database Instance

I was prowling StackOverflow today and the question was asked when you would use 2 databases. I'm amazed at the number of answers of replication and sharding (some under the guise of some other description). The only answer I can think of to be correct is Different Applications/Services (shared hosting probably falls under those 2 in a weird way). I'm going to explain why a replicated database and/or a sharded database are not multiple databases.

Firstly, we must understand what a database is and is not. A database is not a computer program that runs and stores data, this is DBMS (Database Management System). A database simply is a collection of data. A database is an organized collection of data. Many Definitions of a Database include computers, however not all do, in fact Joe Celko argues that first database were precursors to written language in the Middle East (see Data and Databases Chapter 1). It may be true that database are organized for quick retrieval of information in a certain way, however they are not necessarily designed for quick retrieval of all information, in any way.

So what are some example's of a Non Computer Database? A phone book is a good example of a database. It's really fast to look someone up if you know there last name, but not if all you have is their phone number and you want their address. You'll Also note that it's got various ways of searching if you're looking for businesses. It is not a good example of a normalized database, but I'm not sure that's possible with a paper database that allows you to find the same information in multiple ways. A phone book is simply a collection of Names, Addresses, and Phone Numbers, and Categories (tags).

So wouldn't two phone books be two databases? well if they are published by different vendors, or at different times, then yes. However, if you have two copies of the same phone book, then you have two instances of that database, that have been 'replicated' from a 'master' copy. You can't update one of these 'slave' copy's and expect them to replicate, you must update the 'master' and publish the updates. Unlike computer Phone books are extremely slow to push updates. Let's say you want to call a friend, Steven, over for pizza, and your roommate, John, wants to try a new pizza place. John can look at one copy of the phone book to look up the addresses and phone numbers of to the new pizza places you haven't tried, while you look up Steven's number in your copy of the book. This is exactly what happens with replication, different requesters doing lookups from different databases.

Now let's talk sharding, phone books aren't sharded right? wrong! they are phone books are sharded by location. The entire phone book database for even 1 state would be too big for them to distribute, so they split it up into smaller regions. Only the phone book company has the entire list. So that makes it kind of a bad example. Let's talk about encyclopedia's instead. They are a much better example since you don't generally buy one encyclopedia, you buy the whole set (which is also replicated and you can think of it's replication in the same way as phone books). Each encyclopedia volume tells on the side which letters it contains, so when you open it you know you're in the right vicinity. This is how a shard works, your database got too big an unmanageable as a whole so you split it up into 'shards' or 'partitions' to make it faster to search and easier to handle and store.

Now to clarify my answer to that original question. You wouldn't put a phone number (phone entry not included) in an encyclopedia even though I'm sure Bill Clinton has one, you might list it in the phone book though. So now you see when you would use a different database. Very clearly you would use one when the type of data is different enough, when it's for different organizations, or when it's for kind of shared service.

I hope these analogies to computer databases have helped explain the concepts.

Oct 3, 2009

Port forwarding in OpenWRT

So you need the outside world to have access to some box's on your internal network. first you should use Static DHCP to tell static the IP's of the computer you're routing to. Then you of course need to know the inbound from the outside world and the port on the computer you are forwarding too. For this example we'll forward WAN (Wide Area Network (or (probably) the Internet)) port 8080 to a local dev box running Apache on httpd (we assume you know how to set apache up and make sure it's working on the LAN(Local Area Network)).

edit /etc/config/firewall. note: /etc/firewall.user is for manual iptables commands and is basically a shell script. I only recommend this if /etc/config/firewall doesn't do what you want (like multiport).
config redirect
        option src              wan
        option src_dport        8080
        option dest             lan
        option dest_ip          192.168.1.3
        option dest_port        80
        option proto            tcp


So options src and dest merely define the too and from interfaces (to be honest I'm not sure they actually do anything. as I'm not seeing any changes to what's in iptables). src_dport is what external port you want to listen on. dest_port is what you want it to go to. dest_ip is what ip you want it to go to and proto is the protocol it listens on (yes you have to specify separately for udp if what you're forwarding uses both tcp and udp).

After you've edited and saved the file to fit your case run /etc/init.d/firewall restart and it should work

IMPORTANT: there's a bug in 7.x and pre 8.09.2 with 2.4-bcrm kernels and netfilter code... it 'causes port drift. if the bug is affecting you. (it did me) reboot the system. you will be able to see it with a -j LOG iptables rule. the output was thus for me
IN=eth0.1 OUT=br-lan SRC=66.98.131.131 DST=192.168.1.3 LEN=44 TOS=0x00 PREC=0x20 TTL=47 ID=63450 DF PROTO=TCP SPT=54402 DPT=82 WINDOW=5840 RES=0x00 SYN URGP=0
as you can see the destination port(DPT) is 82 instead of 80 like it's supposed to be. 8.09.2 should be out in a few weeks it's currently at rc2.

Sep 22, 2009

Quick sha1sum with Crypto++

I've been working on Korama. I was originally planning on hashing each music file and using the sha1 for the track primary key. However, after benchmarking an actual implementation. I've decided that it's too slow, 20 minutes for 3k+ tracks, also my program was about 5 minutes slower than find ... -exec sha1sum not sure on the reason why. However, since I had problems figuring out how to do it I figure I'll post a basic sha1sum program that I made with the help of people on the Crypto++ Mailing List.

Include's are mangled due to blogger filtering

#include < cryptopp/sha.h >
#include < cryptopp/hex.h >
#include < cryptopp/files.h >
#include < string >
#include < iostream >

using namespace std;

int main(int argc, char *argv[])
{
char *file = argv[1];
string result;
CryptoPP::SHA1 hash;

CryptoPP::FileSource( ( file ),true,
new CryptoPP::HashFilter(
hash, new CryptoPP::HexEncoder(
new CryptoPP::StringSink(result), false)
)
);
cout << result << endl;
return 0;
}


it can be built with.

g++ sha1.cpp -lcryptopp -o sha1sum

and tested with

./sha1sum filename
It's not meant to be a exactly compatible implementation of sha1sum. it doesn't output the filename, it isn't capable of handling more than one file argument, and if no argument is provided it crashes. I know I haven't actually explained how the crypo++ code works, but I hope just posting this here will help someone in the future.

Sep 5, 2009

FSDaily security fail (see image)




Hint, Hint! FSDaily don't access your db with the root account. and definitely make sure that, that error page doesn't show that to users.