Sysadmin Stories: All about file permissions…

by Stephen on October 19, 2009 · 0 comments

in Sysadmin Stories

1

From: jdell@maggie.mit.edu (John Ellithorpe)
Organization: Massachusetts Institute of Technology

Here’s a pretty bad story. I wanted to have root use tcsh instead of the
Bourne shell. So I decided to copy tcsh to /usr/local/bin. I created the
file, /etc/shells, and put in /usr/local/bin/tcsh, along with /bin/sh and
/bin/csh.

All seems fine, so I used the chsh command and changed root’s shell to
/usr/local/bin/tcsh. So I logged out and tried to log back in. Only to find
out that I couldn’t get back in. Every time I tried to log in, I only got
the statement: /usr/local/bin/tcsh: permission denied!

I instantly realized what I had done. I forgot to check that tcsh has
execute privileges and I couldn’t get in as root!

After about 30 minutes of getting mad at myself, I finally figured out to just
bring the system down to single-user mode, which ONLY uses the /bin/sh,
thankfully, and edited the password file back to /bin/sh.

2

From: djd@csg.cs.reading.ac.uk (David J Dawkins)
Organization: University of Reading

About a year back, I was looking through /etc and found that a few
system files had world write permission. Gasping with horror, I went
to put it right with something like

dipshit# chmod -r 664 /etc/*

(I know, I know, goddamnit!.. now)

Everything was OK for about two to three weeks, then the machine went
down for some reason (other than the obvious). Well, I expect that you
can imagine the result. The booting procedure was unable to run fsck,
so barfed and mounted the file systems read-only, and bunged me into
single-user mode. Dumb expression..gradual realisation..cold sweat. Of
course, now I can’t do a frigging chmod +x on anything because it’s all
read-only. In fact I can’t run anything that isn’t part of sh.
Wedgerama. Hysteria time. Consider reformatting disks. All sorts of
crap ideas. Headless chicken scene. Confession.

“You did WHAT??!!”

Much forehead slapping, solemn oaths and floor pacing.

Luckily, we have a local MegaUnixGenius who, having sat puzzled for an hour
or more, decided to boot from a cdrom and take things from there. He fixed
it.

My boss, totally amazed at the fix I’d got the system into, luckily
saw the funny side of it. I didn’t. Even though at that stage, I didn’t
know much about unix/suns/booting/admin, I did actually know enough to NOT
use a command like the one above. Don’t ask. Must be the drugs.

BTW, if my future employer _is_ reading this (like they say he/she might),
then I have certainly learned tonnes of stuff in the last year, especially
having had to set up a complete Sun system, fix local problems, etc 🙂

Anyone else got a tale of SGS (Spontaneous Gross Stupidity) ?

3

From: mfraioli@grebyn.com (Marc Fraioli)
Organization: Grebyn Timesharing

I was happily churning along developing something on a Sun workstation,
and was getting a number of annoying permission denieds from trying to
write into a directory heirarchy that I didn’t own. Getting tired of
that, I decided to set the permissions on that subtree to 777 while I
was working, so I wouldn’t have to worry about it. Someone had recently
told me that rather than using plain “su”, it was good to use “su -“,
but the implications had not yet sunk in. (You can probably see where
this is going already, but I’ll go to the bitter end.) Anyway, I cd’d
to where I wanted to be, the top of my subtree, and did su -. Then I
did chmod -R 777. I then started to wonder why it was taking so damn
long when there were only about 45 files in 20 directories under where I
(thought) I was. Well, needless to say, su – simulates a real login,
and had put me into root’s home directory, /, so I was proceeding to set
file permissions for the whole system to wide open. I aborted it before
it finished, realizing that something was wrong, but this took quite a
while to straighten out.

4

From: jerry@incc.com (Jerry Rocteur)
Organization: InCC.com Perwez Belgium

I sent one of my support guys to do an Oracle update in Madrid.

As instructed he created a new user called esf and changed the files
in /u/appl to owner esf, however in doing so he *must* have cocked up
his find command, the command was:

find /u/appl -user appl -exec chown esf {} \;

He rang me up to tell me there was a problem, I logged in via x25 and
about 75% of files on system belonged to owner esf.

VERY little worked on system.

What a mess, it took me a while and I came up with a brain wave to
fix it but it really screwed up the system.

Moral: be *very* careful of find execs, get the syntax right!!!

5

From: weave@bach.udel.edu (Ken Weaverling)
Organization: University of Delaware

A friend of mine called me up saying he no longer could log into his
system. I asked him what he had done recently, and found out that he
thought that all executable programs in /bin /usr/bin /etc and so on
should be owned by bin, since they were all binaries! So he had
chown’ed them all.

6

From: rob@wzv.win.tue.nl (Rob J. Nauta)
Organization: None

At my previous employer, the sysadmin would create new user accounts by
hand by editing the passwd file, create a home dir, put some files in
it, and chown ‘*’ and ‘.*’ to that new user. Thus, /home/machine
was also chowned (‘.*’ also matches ‘..’). It was quite handy to see
who was added last, but after a while I slipped him the hint to
chown ‘.[a-z]*’ which works much better of course.

But the stories told now are more folklore than real horror. Having read
2 Stephen Kings this weekend I beg everyone to tell more interesting
stories, about demons, the system clock running backwards, old files
reappearing etc !

7

From: alan@spuddy.uucp (Alan Saunders)
Organization: Spuddy’s Public Usenet Domain

About inexperienced sysadmins .. One such had been on a Sun syasadmin
course, and learned all about security. One of the topics was on file
and group access. On his return, he decided to put what he had learned
into practice, and changed the ownership of all files in /bin, /usr/bin
to bin.bin! I was called in when no one could log in to the system
(of course /bin/login needs to be setuid root!)

8

From: pete@tecc.co.uk (Pete Bentley)
Organization: T.E.C.C. Ltd, London, England

The guys next door had just got a Sun 3/360 (or some such) to host a
VME-bus image processing system – none of them knew much (or cared
much) about Un*x and so early on a student on loan to them got a
space in the wrong place and did
pillock# chmod -r -x ~ /*
with the same results (system in single user, refusing to run any commands
or go multi-user).

As it happened
a) This was a government establishment, and so the order for the QIC tapes
for backups had not yet been approved, hence no backups…
b) The install script for the kernel drivers for the image processing stuff
had not worked ‘out of the box’, and so the company had sent an
engineer down to install it. I hadn’t been around when he came and
built their drivers, and they hadn’t a clue what he had done. So,
there was no way to rebuild the drivers without another engineer call
and because of (a) there were no backups of the driver…Anyway, a complete
reload was therefore out of the question.

These were the days before SunOS on CD-ROM. In the end I managed to get
the thing up by booting from tape, installing the miniroot into the swap
partition and booting from that. This gave me a working tar and a working
mount, but no chmod. Also no mt command. Also at this time very little
of my Un*x experience was on Suns, so I had no idea of the layout of the
distribution tape. Various experiments with dd and the non-rewinding tape
device eventually found the file on the tape with a chmod I could extract.
chmod +x /etc/* /bin/* /usr/bin/* on the system’s existing disk was enough
to make it bootable. After that I sat the student down with a SunOS manual
and let him figure out the mess and correct the permissions that had been
todged all over the system…

9

From: dvsc-a@minster.york.ac.uk
Organization: Department of Computer Science, University of York, England

I was changing the UIDs of a few users on one of our major servers, due to
a clash with some machines newly connected to the net. Fine, edit
/etc/passwd then chown all their files to the new UID. So, rather than just
assume that all files owned by “fred” live in /home/machine/fred I did this:

machine# find / -user old_uid -exec chown username {} \;

This was fine… except it was late at night and I was tired, and in a hurry
to get home. I had six of these commands to type, and as they would take a
long time I’d just let them run in the background over night…..

So, you come in the next morning and a user compains… I can’t login to the
4/490 – it says “/bin/login: setgid: not owner”.

Okay…. naive user problem no?

rlogin machine -l root
/bin/login: setgid: not owner

machine console
login: root
/bin/login: setgid: not owner

Okay – I REALLY can’t get in… lets reboot single user and see whats on…
this worked. /bin/login is owned (and setuid to) one of the users whos UID
I changed the previous day… infact ALL FILES in the ENTIRE filesystem are
owned by this user..problem!

We `only’ lost about 200 man hours through my little typing mistake. The
moral: Beware anything recursive when logged in as root!

10

From: joslin_paul@ae.ge.com
Organization: GE Aircraft Engines

True confession time: Cron is a great way to hide your flubs. I installed
the COPS security package on a system, then set up cron to recheck the
system once a month. No problem, right? Except that I had configured COPS
to put the reports in /. As a security measure, COPS chmods its directory
to u-rwx,w-rwx so that only the COPS owner can read the reports.

The chronology was

1) Run cops. Add cops entry to root’s crontab. Later that day, notice
that / was 600; change it back.

2) 30 days later: get calls from users – can’t log in, “No shell” error
messages. Find / is 600; change it. Vaguely remember that this
happened once before. The machine was a sandbox, so almost anything
could have changed /.

3) 30 days later: get calls from users – can’t log in, “No shell” error
messages. Find / is 600; change it. Vaguely remember that this
happened once before. Happen to think “cron”; notice that the only cron
activity for root last night was COPS. Read COPS source and discover
problem.

Moral: RTFM. Keep logs, so that you can notice patterns in your data.
Don’t do anything as root that you can do as a mortal.

11

From: johnd@cortex.physiol.su.oz.au (John Dodson)
Organization: Department of Physiology, University of Sydney, NSW, Australia

Some years ago when we went from Version 7 Unix on a PDP11 to a flavour of BSD
on a Vax, I was working on the Vax in my home directory & came across a file
that I had no permission on (I’d created it as root) so the following ensued…

$ /bin/su –
Password:
# chown -R me *

mmmmm this seems to be taking a long time !
kill.
# ls -l

the result was that I was in / after the su !
(good old V7 su used to leave you in the current directory 😉

It took me quite a while to restore all the right ownerships to /bin /etc &
/dev (especially the suid/sgid files)
I’d managed to kill it before it got off the root filesystem.

12

From: adb@geac.com (Anthony DeBoer)
Organization: Geac Computer Corporation

I was once called in to save a system where most things worked, but the
main application package being used on it hung the moment you entered it
(leaving the system more than a little useless for getting things done).
I poked around for awhile, verified that the application’s files were all
present, undamaged, and had the right permissions. The folks who
normally used the machine had also discovered that all was well if root
tried to run it. But nothing was visibly wrong anywhere. So, being a
bit hungry by then, I took a break for supper, and about halfway through,
the little voice at the back of my head that sometimes helps me said,
“/dev/tty”. Sure enough, somebody had chmod’ded it to 0644, and the
application directed (or tried to direct, in this case) all its I/O
through it rather than just using stdin/stdout like a sane normal process.

13

From: mike@sojurn.lns.pa.us (Mike Sangrey)

To set the stage:
We used the csh.
We were fairly new to Unix.
We were developing a fairly eloborate system in “C”.

We made some fairly harmless (most of the time) mistakes:
We had “.” (dot) in root’s PATH. (Yeah, I know, so sue me.)

We had the forsight to set up a pseudo-user for our package. Certain
of these programs were to run setuid as the pseudo-user others weren’t
setuid and were to be only run as that psuedo-user. You know the
scenario. The problem was that sometimes during development, one of
us didn’t have the permission to execute a program. We frequently
fell into executing things as root. One particularly frustrating day
we did something even more stupid:

chmod 777 *.

Then, just to make sure (of how stupid we can be) we flipped to a
virtual terminal that was su’ed to root. The next command, which used
the csh’s history mechanism, executed a “C” program — NOT the
executable, mind you, the source. Believe it or not, the end effect
was the same as

cd /
rm -fr *

Sort of reminds me of the story of a hurricane, a junk yard and the
creation of a 747. Who’d a thunk it?!!

Take some inexperienced people and a powerful system; add profuse
doses of frustration and wha-la! — You have a Stephen King shell
script.

14

From: mba@controls.ccd.harris.com (Belinda Asbell)
Organization: Harris Controls

In article , JRowe@cen.ex.ac.uk (J.Rowe) writes:
>> Am I the only one to have mangled a root shell?

Probably not. I learned the hard way to be careful if messing with
/etc/passwd. One day, for some reason, I couldn’t login as root (pretty
scary, since I knew the root passwd and hadn’t changed it).

Turned out that somehow I’d blitzed the first letter of /etc/passwd somehow
(vi does bizarre things sometimes). So I logged in as ‘oot’ and fixed it.

NEVER do a “chmod -R u-s .”, especially not in /usr….

I think that “mount -o” or something similar will mount a filesystem read-write
if it’s come up in singleuser mode and is mounted read-only…..

{ 0 comments… add one now }

Leave a Comment

Previous post:

Next post: